I have been playing with Numpy and matplotlib in the last few days. I am having problems trying to make matplotlib plot a function without blocking execution. I know there are already many threads here on SO asking similar questions, and I ‘ve googled quite a lot but haven’t managed to make this work.
I have tried using show(block=False) as some people suggest, but all I get is a frozen window. If I simply call show(), the result is plotted properly but execution is blocked until the window is closed. From other threads I ‘ve read, I suspect that whether show(block=False) works or not depends on the backend. Is this correct? My back end is Qt4Agg. Could you have a look at my code and tell me if you see something wrong? Here is my code. Thanks for any help.
from math import *
from matplotlib import pyplot as plt
print plt.get_backend()
def main():
x = range(-50, 51, 1)
for pow in range(1,5): # plot x^1, x^2, ..., x^4
y = [Xi**pow for Xi in x]
print y
plt.plot(x, y)
plt.draw()
#plt.show() #this plots correctly, but blocks execution.
plt.show(block=False) #this creates an empty frozen window.
_ = raw_input("Press [enter] to continue.")
if __name__ == '__main__':
main()
PS. I forgot to say that I would like to update the existing window every time I plot something, instead of creating a new one.
I spent a long time looking for solutions, and found this answer.
It looks like, in order to get what you (and I) want, you need the combination of plt.ion(), plt.show() (not with block=False) and, most importantly, plt.pause(.001) (or whatever time you want). The pause is needed because the GUI events happen while the main code is sleeping, including drawing. It’s possible that this is implemented by picking up time from a sleeping thread, so maybe IDEs mess with that—I don’t know.
Here’s an implementation that works for me on python 3.5:
import numpy as np
from matplotlib import pyplot as plt
def main():
plt.axis([-50,50,0,10000])
plt.ion()
plt.show()
x = np.arange(-50, 51)
for pow in range(1,5): # plot x^1, x^2, ..., x^4
y = [Xi**pow for Xi in x]
plt.plot(x, y)
plt.draw()
plt.pause(0.001)
input("Press [enter] to continue.")
if __name__ == '__main__':
main()
回答 1
一个对我有用的简单技巧如下:
在show内使用block = False参数:plt.show(block = False)
在.py脚本的末尾使用另一个plt.show() 。
范例:
import matplotlib.pyplot as plt
plt.imshow(add_something)
plt.xlabel("x")
plt.ylabel("y")
plt.show(block=False)#more code here (e.g. do calculations and use print to see them on the screen
plt.show()
A simple trick that works for me is the following:
Use the block = False argument inside show: plt.show(block = False)
Use another plt.show()at the end of the .py script.
Example:
import matplotlib.pyplot as plt
plt.imshow(add_something)
plt.xlabel("x")
plt.ylabel("y")
plt.show(block=False)
#more code here (e.g. do calculations and use print to see them on the screen
plt.show()
import pyformulas as pf
import matplotlib.pyplot as plt
import numpy as np
import time
fig = plt.figure()
canvas = np.zeros((480,640))
screen = pf.screen(canvas,'Sinusoid')
start = time.time()whileTrue:
now = time.time()- start
x = np.linspace(now-2, now,100)
y = np.sin(2*np.pi*x)+ np.sin(3*np.pi*x)
plt.xlim(now-2,now+1)
plt.ylim(-3,3)
plt.plot(x, y, c='black')# If we haven't already shown or saved the plot, then we need to draw the figure first...
fig.canvas.draw()
image = np.fromstring(fig.canvas.tostring_rgb(), dtype=np.uint8, sep='')
image = image.reshape(fig.canvas.get_width_height()[::-1]+(3,))
screen.update(image)#screen.close()
You can avoid blocking execution by writing the plot to an array, then displaying the array in a different thread. Here is an example of generating and displaying plots simultaneously using pf.screen from pyformulas 0.2.8:
import pyformulas as pf
import matplotlib.pyplot as plt
import numpy as np
import time
fig = plt.figure()
canvas = np.zeros((480,640))
screen = pf.screen(canvas, 'Sinusoid')
start = time.time()
while True:
now = time.time() - start
x = np.linspace(now-2, now, 100)
y = np.sin(2*np.pi*x) + np.sin(3*np.pi*x)
plt.xlim(now-2,now+1)
plt.ylim(-3,3)
plt.plot(x, y, c='black')
# If we haven't already shown or saved the plot, then we need to draw the figure first...
fig.canvas.draw()
image = np.fromstring(fig.canvas.tostring_rgb(), dtype=np.uint8, sep='')
image = image.reshape(fig.canvas.get_width_height()[::-1] + (3,))
screen.update(image)
#screen.close()
If you’re using an actual figure.
I used @krs013, and @Default Picture’s answers to figure this out
Hopefully this saves someone from having launch every single figure on a separate thread, or from having to read these novels just to figure this out
回答 4
实时绘图
import numpy as np
import matplotlib.pyplot as plt
x = np.linspace(0,2* np.pi,100)# plt.axis([x[0], x[-1], -1, 1]) # disable autoscalingfor point in x:
plt.plot(point, np.sin(2* point),'.', color='b')
plt.draw()
plt.pause(0.01)# plt.clf() # clear the current figure
import numpy as np
import matplotlib.pyplot as plt
x = np.linspace(0, 2 * np.pi, 100)
# plt.axis([x[0], x[-1], -1, 1]) # disable autoscaling
for point in x:
plt.plot(point, np.sin(2 * point), '.', color='b')
plt.draw()
plt.pause(0.01)
# plt.clf() # clear the current figure
if the amount of data is too much you can lower the update rate with a simple counter
cnt += 1
if (cnt == 10): # update plot each 10 points
plt.draw()
plt.pause(0.01)
cnt = 0
Holding Plot after Program Exit
This was my actual problem that couldn’t find satisfactory answer for, I wanted plotting that didn’t close after the script was finished (like MATLAB),
If you think about it, after the script is finished, the program is terminated and there is no logical way to hold the plot this way, so there are two options
block the script from exiting (that’s plt.show() and not what I want)
run the plot on a separate thread (too complicated)
this wasn’t satisfactory for me so I found another solution outside of the box
SaveToFile and View in external viewer
For this the saving and viewing should be both fast and the viewer shouldn’t lock the file and should update the content automatically
Selecting Format for Saving
vector based formats are both small and fast
SVG is good but coudn’t find good viewer for it except the web browser which by default needs manual refresh
PDF can support vector formats and there are lightweight viewers which support live updating
Fast Lightweight Viewer with Live Update
For PDF there are several good options
On Windows I use SumatraPDF which is free, fast and light (only uses 1.8MB RAM for my case)
On Linux there are several options such as Evince (GNOME) and Ocular (KDE)
Sample Code & Results
Sample code for outputing plot to a file
import numpy as np
import matplotlib.pyplot as plt
x = np.linspace(0, 2 * np.pi, 100)
y = np.sin(2 * x)
plt.plot(x, y)
plt.savefig("fig.pdf")
after first run, open the output file in one of the viewers mentioned above and enjoy.
Here is a screenshot of VSCode alongside SumatraPDF, also the process is fast enough to get semi-live update rate (I can get near 10Hz on my setup just use time.sleep() between intervals)
def plt_show():'''Text-blocking version of plt.show()
Use this instead of plt.show()'''
plt.draw()
plt.pause(0.001)
input("Press enter to continue...")
plt.close()
Iggy’s answer was the easiest for me to follow, but I got the following error when doing a subsequent subplot command that was not there when I was just doing show:
MatplotlibDeprecationWarning: Adding an axes using the same arguments
as a previous axes currently reuses the earlier instance. In a future
version, a new instance will always be created and returned.
Meanwhile, this warning can be suppressed, and the future behavior
ensured, by passing a unique label to each axes instance.
In order to avoid this error, it helps to close (or clear) the plot after the user hits enter.
Here’s the code that worked for me:
def plt_show():
'''Text-blocking version of plt.show()
Use this instead of plt.show()'''
plt.draw()
plt.pause(0.001)
input("Press enter to continue...")
plt.close()
The Python package drawnow allows to update a plot in real time in a non blocking way.
It also works with a webcam and OpenCV for example to plot measures for each frame.
See the original post.
lxml is the only package failing to install and this leads to everything failing (expected results as pointed out by larsks in the comments). However, after lxml fails pip still runs through and downloads the rest of the packages.
From what I understand the pip install -r requirements.txt command will fail if any of the packages listed in the requirements.txt fail to install.
Is there any argument I can pass when running pip install -r requirements.txt to tell it to install what it can and skip the packages that it cannot, or to exit as soon as it sees something fail?
import sys
from pip._internal import main as pip_main
def install(package):
pip_main(['install', package])if __name__ =='__main__':with open(sys.argv[1])as f:for line in f:
install(line)
点版本<18
import sys
import pip
def install(package):
pip.main(['install', package])if __name__ =='__main__':with open(sys.argv[1])as f:for line in f:
install(line)
import sys
from pip._internal import main as pip_main
def install(package):
pip_main(['install', package])
if __name__ == '__main__':
with open(sys.argv[1]) as f:
for line in f:
install(line)
pip version <18
import sys
import pip
def install(package):
pip.main(['install', package])
if __name__ == '__main__':
with open(sys.argv[1]) as f:
for line in f:
install(line)
The xargs solution works but can have portability issues (BSD/GNU) and/or be cumbersome if you have comments or blank lines in your requirements file.
As for the usecase where such a behavior would be required, I use for instance two separate requirement files, one which is only listing core dependencies that need to be always installed and another file with non-core dependencies that are in 90% of the cases not needed for most usecases. That would be an equivalent of the Recommends section of a debian package.
I use the following shell script (requires sed) to install optional dependencies:
#!/bin/sh
while read dependency; do
dependency_stripped="$(echo "${dependency}" | sed -e 's/^[[:space:]]*//' -e 's/[[:space:]]*$//')"
# Skip comments
if [[ $dependency_stripped == \#* ]]; then
continue
# Skip blank lines
elif [ -z "$dependency_stripped" ]; then
continue
else
if pip install "$dependency_stripped"; then
echo "$dependency_stripped is installed"
else
echo "Could not install $dependency_stripped, skipping"
fi
fi
done < recommends.txt
回答 4
谢谢Etienne Prothon提供的Windows保护套。
但是,升级到pip 18后,pip包不会将main公开。因此,您可能需要像这样更改代码。
# This code install line by line a list of pip package import sys
from pip._internal import main as pip_main
def install(package):
pip_main(['install', package])if __name__ =='__main__':with open(sys.argv[1])as f:for line in f:
install(line)
But, after upgrading to pip 18, pip package don’t expose main to public. So you may need to change code like this.
# This code install line by line a list of pip package
import sys
from pip._internal import main as pip_main
def install(package):
pip_main(['install', package])
if __name__ == '__main__':
with open(sys.argv[1]) as f:
for line in f:
install(line)
回答 5
对于Windows:
import os
from pip.__main__ import _main as main
error_log = open('error_log.txt','w')def install(package):try:
main(['install']+[str(package)])exceptExceptionas e:
error_log.write(str(e))if __name__ =='__main__':
f = open('requirements1.txt','r')for line in f:
install(line)
f.close()
error_log.close()
import os
from pip.__main__ import _main as main
error_log = open('error_log.txt', 'w')
def install(package):
try:
main(['install'] + [str(package)])
except Exception as e:
error_log.write(str(e))
if __name__ == '__main__':
f = open('requirements1.txt', 'r')
for line in f:
install(line)
f.close()
error_log.close()
Create a local directory, and put your requirements.txt file in it.
Copy the code above and save it as a python file in the same directory. Remember to use .py extension, for instance, install_packages.py
Run this file using a cmd: python install_packages.py
All the packages mentioned will be installed in one go without stopping at all. :)
You can add other parameters in install function. Like:
main(['install'] + [str(package)] + ['--update'])
You’ve already got it: A if test else B is a valid Python expression. The only problem with your dict comprehension as shown is that the place for an expression in a dict comprehension must have two expressions, separated by a colon:
{ (some_key if condition else default_key):(something_if_true if condition
else something_if_false) for key, value in dict_.items() }
The final if clause acts as a filter, which is different from having the conditional expression.
# here we assume that the values in d2 are unique# Python 2
dout2 ={d2.keys()[d2.values().index(v1)]if v1 in d2.values()else k1: v1 for k1, v1 in d1.items()}# Python 3
dout2 ={list(d2.keys())[list(d2.values()).index(v1)]if v1 in d2.values()else k1: v1 for k1, v1 in d1.items()}
and you want to replace the keys in d1 by the keys of d2 if there respective values are identical, you could do
# here we assume that the values in d2 are unique
# Python 2
dout2 = {d2.keys()[d2.values().index(v1)] if v1 in d2.values() else k1: v1 for k1, v1 in d1.items()}
# Python 3
dout2 = {list(d2.keys())[list(d2.values()).index(v1)] if v1 in d2.values() else k1: v1 for k1, v1 in d1.items()}
Assume you want to suffix only keys with a in its value and you want the value replaced with the length of the set in such a case. Otherwise, the key-value pair should stay unchanged.
dict((f"{k}_a", len(v)) if "a" in v else (k, v) for k, v in d.items())
# {'key1_a': 3, 'key2': {'bar', 'foo'}, 'key3': {'sad', 'so'}}
entries ={'name':'Material Name','maxt':'Max Working Temperature','ther':{100:1.1,200:1.2}}
a_dic, b_dic ={},{}for field, value in entries.items():if field =='ther':for k,v in value.items():
b_dic[k]= v
a_dic[field]= b_dicelse:
a_dic[field]= valueprint(a_dic)“{'name':'Material Name','maxt':'Max Working Temperature','ther':{100:1.1,200:1.2}}”
第二轮我尝试使用字典理解,但是循环仍然存在(6行):
entries ={'name':'Material Name','maxt':'Max Working Temperature','ther':{100:1.1,200:1.2}}for field, value in entries.items():if field =='ther':
b_dic ={k:v for k,v in value.items()}
a_dic[field]= b_dicelse:
a_dic[field]= valueprint(a_dic)“{'name':'Material Name','maxt':'Max Working Temperature','ther':{100:1.1,200:1.2}}”
最后,使用单行字典理解语句(1行):
entries ={'name':'Material Name','maxt':'Max Working Temperature','ther':{100:1.1,200:1.2}}
a_dic ={field:{k:v for k,v in value.items()}if field =='ther'else value for field, value in entries.items()}print(a_dic)“{'name':'Material Name','maxt':'Max Working Temperature','ther':{100:1.1,200:1.2}}”
Another example in using if/else in dictionary comprehension
I am working on data-entry desktop application for my own office work, and it is common for such data-entry application to get all entries from input widget and dump it into a dictionary for further processing like validation, or editing which we must return selected data from file back to entry widgets, etc.
The first round using traditional coding (8 lines):
entries = {'name': 'Material Name', 'maxt': 'Max Working Temperature', 'ther': {100: 1.1, 200: 1.2}}
a_dic, b_dic = {}, {}
for field, value in entries.items():
if field == 'ther':
for k,v in value.items():
b_dic[k] = v
a_dic[field] = b_dic
else:
a_dic[field] = value
print(a_dic)
“ {'name': 'Material Name', 'maxt': 'Max Working Temperature', 'ther': {100: 1.1, 200: 1.2}}”
Second round I tried to use dictionary comprehension but the loop still there (6 lines):
entries = {'name': 'Material Name', 'maxt': 'Max Working Temperature', 'ther': {100: 1.1, 200: 1.2}}
for field, value in entries.items():
if field == 'ther':
b_dic = {k:v for k,v in value.items()}
a_dic[field] = b_dic
else:
a_dic[field] = value
print(a_dic)
“ {'name': 'Material Name', 'maxt': 'Max Working Temperature', 'ther': {100: 1.1, 200: 1.2}}”
Finally, with a one-line dictionary comprehension statement (1 line):
entries = {'name': 'Material Name', 'maxt': 'Max Working Temperature', 'ther': {100: 1.1, 200: 1.2}}
a_dic = {field:{k:v for k,v in value.items()} if field == 'ther'
else value for field, value in entries.items()}
print(a_dic)
“ {'name': 'Material Name', 'maxt': 'Max Working Temperature', 'ther': {100: 1.1, 200: 1.2}}”
Traceback(most recent call last):File"C:\Python34\lib\site-packages\graphviz\files.py", line 220,in render
proc = subprocess.Popen(cmd, startupinfo=STARTUPINFO)File"C:\Python34\lib\subprocess.py", line 859,in __init__
restore_signals, start_new_session)File"C:\Python34\lib\subprocess.py", line 1112,in _execute_child
startupinfo)FileNotFoundError:[WinError2]The system cannot find the file specified
During handling of the above exception, another exception occurred:Traceback(most recent call last):File"C:\Users\Documents\Kissmetrics\curves and lines\eventNodes.py", line 56,in<module>
filename=dot.render(filename='test')File"C:\Python34\lib\site-packages\graphviz\files.py", line 225,in render
'are on your systems\' path'% cmd)RuntimeError: failed to execute ['dot','-Tpdf','-O','test'], make sure the Graphviz executables are on your systems' path
I downloaded Graphviz 2.38 MSI version and installed under folder C:\Python34, then I run pip install Graphviz, everything went well. In system’s path I added C:\Python34\bin. When I tried to run a test script, in line filename=dot.render(filename='test'), I got a message
RuntimeError: failed to execute ['dot', '-Tpdf', '-O', 'test'], make sure the Graphviz executables are on your systems' path
I tried to put "C:\Python34\bin\dot.exe" in system’s path, but it didn’t work, and I even created a new environment variable "GRAPHVIZ_DOT" with value "C:\Python34\bin\dot.exe", still not working. I tried to uninstall Graphviz and pip uninstall graphviz, then reinstall it and pip install again, but nothing works.
The whole traceback message is:
Traceback (most recent call last):
File "C:\Python34\lib\site-packages\graphviz\files.py", line 220, in render
proc = subprocess.Popen(cmd, startupinfo=STARTUPINFO)
File "C:\Python34\lib\subprocess.py", line 859, in __init__
restore_signals, start_new_session)
File "C:\Python34\lib\subprocess.py", line 1112, in _execute_child
startupinfo)
FileNotFoundError: [WinError 2] The system cannot find the file specified
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\Documents\Kissmetrics\curves and lines\eventNodes.py", line 56, in <module>
filename=dot.render(filename='test')
File "C:\Python34\lib\site-packages\graphviz\files.py", line 225, in render
'are on your systems\' path' % cmd)
RuntimeError: failed to execute ['dot', '-Tpdf', '-O', 'test'], make sure the Graphviz executables are on your systems' path
Does anybody have any experience with it?
回答 0
import os
os.environ["PATH"]+= os.pathsep +'D:/Program Files (x86)/Graphviz2.38/bin/'
在Windows中,只需在开头添加这两行,其中“ D:/ Program Files(x86)/Graphviz2.38/bin/”将替换为bin文件所在的地址。
import os
os.environ["PATH"] += os.pathsep + 'D:/Program Files (x86)/Graphviz2.38/bin/'
In windows just add these 2 lines in the beginning, where ‘D:/Program Files (x86)/Graphviz2.38/bin/’ is replaced by the address of where your bin file is.
Using conda install graphviz and conda install python-graphviz to install GraphViz on Windows10 the path needed was C:/ProgramData/Anaconda3/Library/bin/graphviz/ for me. I.e. adding
import os
os.environ["PATH"] += os.pathsep + 'C:/ProgramData/Anaconda3/Library/bin/graphviz/'
When solving this issue for myself, I used this GitHub tutorial, which analysed the cause of this issue. If we read in between the lines, it says it needs system as well as python graph viz. In addition to conda install, we would need to run:
1) Graphviz – download unzip in a particular place in the system (pip does not work in windows ) and include the bin folder in the path (‘set environment variables in windows’ OR) set manually in each program
import os
os.environ["PATH"] += os.pathsep + 'C:/GraphViz/bin'
First, you should use pip install, and then download another package in http://www.graphviz.org/Download_windows.php
and add the install location into the environmental path, then it works.
I had the same error message on Mac OS (El Capitan), using the PyCharm IDE.
I had installed Graphviz using brew, as recommended in RZK’s answer, and installed the graphviz python package using PyCharm (I could check Graphviz was installed correctly by trying dot -V in a terminal and getting: dot - graphviz version 2.40.1 (20161225.0304)).
Yet I was still getting the error message when trying to call Graphviz from PyCharm.
I had to add the path /usr/local/bin in PyCharm options, as recommended in the answer to this question to resolve the problem.
我在macOS Catalina 10.15.3上,并且遇到了类似的错误: ExecutableNotFound: failed to execute ['dot', '-Tsvg'], make sure the Graphviz executables are on your systems' PATH
I’m on macOS Catalina 10.15.3, and I had a similar error: ExecutableNotFound: failed to execute ['dot', '-Tsvg'], make sure the Graphviz executables are on your systems' PATH
Fixed it with:
pip3 install graphviz AND brew install graphviz
Note the pip3 install will only return the success message Successfully installed graphviz-0.13.2 so we still need to run brew install to get graphviz 2.42.3 (as of 10 Mar 2020, 6PM).
now press Ctrl+O and then Ctrl+X to save and exit.
Problem should be solved by now.
Pycharm users, please note: Pycharm does not always see the PATH variable the same as your terminal. This solution does not work for Pycharm, and maybe other IDEs. But you can fix this by adding this line of code:
Edit: If you don’t want to use conda, you can still install graphviz from here without any root permissions and add the bin folder to your PATH variable. I didn’t test this.
Using pip install graphviz had good feedback in terminal, but lead to this error when I tried to make a graph in a Jupyter notebook. I then ran brew install graphviz, which gave an error in terminal. Then I ran conda install graphviz and the graph worked.
From @Leighton’s comment: pip only gets path problem same as yours and conda only gets import error.
回答 24
import os
os.environ["PATH"]+= os.pathsep +"/Macintosh HD/anaconda3/lib/python3.7/site-packages/sphinx/templates/graphviz"
If you are not using Conda but vanilla Python, ‘brew install graphviz’ works.
回答 26
#Write this on anaconda prompt in admin mode
conda install -c anaconda graphviz
conda install -c conda-forge python-graphviz
conda install -c conda-forge/label/broken python-graphviz
conda install -c conda-forge/label/cf201901 python-graphviz
conda install -c conda-forge/label/cf202003 python-graphviz
#check dot -v in window's cmd prompt
C:\WINDOWS\system32>dot -V
dot - graphviz version 2.38.0(20140413.2041)(this means graphviz installed successfully)#Add path to sys and user eve variables
PATH
C:\Anaconda3\pkgs\graphviz-2.38-hfd603c8_2\Library\bin(search bin folder of graphviz and then copy n paste path in env variables)#Re-run all cmds in jyupter notebook#if error occurs (less chances)#then #Restart anaconda and again run all cmds in jyupter notebook
eg.import graphviz as gp
with open("tree.dot")as f:
dot_read=f.read()
display(gp.Source(dot_read))
#Write this on anaconda prompt in admin mode
conda install -c anaconda graphviz
conda install -c conda-forge python-graphviz
conda install -c conda-forge/label/broken python-graphviz
conda install -c conda-forge/label/cf201901 python-graphviz
conda install -c conda-forge/label/cf202003 python-graphviz
#check dot -v in window's cmd prompt
C:\WINDOWS\system32>dot -V
dot - graphviz version 2.38.0 (20140413.2041)
(this means graphviz installed successfully)
#Add path to sys and user eve variables
PATH
C:\Anaconda3\pkgs\graphviz-2.38-hfd603c8_2\Library\bin
(search bin folder of graphviz and then copy n paste path in env variables)
#Re-run all cmds in jyupter notebook
#if error occurs (less chances)
#then
#Restart anaconda and again run all cmds in jyupter notebook
eg.
import graphviz as gp
with open("tree.dot") as f:
dot_read=f.read()
display(gp.Source(dot_read))
I’m building an app with Flask, but I don’t know much about WSGI and it’s HTTP base, Werkzeug. When I start serving a Flask application with gunicorn and 4 worker processes, does this mean that I can handle 4 concurrent requests?
I do mean concurrent requests, and not requests per second or anything else.
When running the development server – which is what you get by running app.run(), you get a single synchronous process, which means at most 1 request is being processed at a time.
By sticking Gunicorn in front of it in its default configuration and simply increasing the number of --workers, what you get is essentially a number of processes (managed by Gunicorn) that each behave like the app.run() development server. 4 workers == 4 concurrent requests. This is because Gunicorn uses its included sync worker type by default.
It is important to note that Gunicorn also includes asynchronous workers, namely eventlet and gevent (and also tornado, but that’s best used with the Tornado framework, it seems). By specifying one of these async workers with the --worker-class flag, what you get is Gunicorn managing a number of async processes, each of which managing its own concurrency. These processes don’t use threads, but instead coroutines. Basically, within each process, still only 1 thing can be happening at a time (1 thread), but objects can be ‘paused’ when they are waiting on external processes to finish (think database queries or waiting on network I/O).
This means, if you’re using one of Gunicorn’s async workers, each worker can handle many more than a single request at a time. Just how many workers is best depends on the nature of your app, its environment, the hardware it runs on, etc. More details can be found on Gunicorn’s design page and notes on how gevent works on its intro page.
Currently there is a far simpler solution than the ones already provided. When running your application you just have to pass along the threaded=True parameter to the app.run() call, like:
Another option as per what we can see in the werkzeug docs, is to use the processes parameter, which receives a number > 1 indicating the maximum number of concurrent processes to handle:
threaded – should the process handle each request in a separate thread?
processes – if greater than 1 then handle each request in a new process up to this maximum number of concurrent processes.
Something like:
app.run(host="your.host", port=4321, processes=3) #up to 3 processes
More info on the run() method here, and the blog post that led me to find the solution and api references.
Note: on the Flask docs on the run() methods it’s indicated that using it in a Production Environment is discouraged because (quote): “While lightweight and easy to use, Flask’s built-in server is not suitable for production as it doesn’t scale well.”
However, they do point to their Deployment Options page for the recommended ways to do this when going for production.
Its important to remember that deep deep down, assuming you are running a single core machine, the CPU really only runs one instruction* at a time.
Namely, the CPU can only execute a very limited set of instructions, and it can’t execute more than one instruction per clock tick (many instructions even take more than 1 tick).
Therefore, most concurrency we talk about in computer science is software concurrency.
In other words, there are layers of software implementation that abstract the bottom level CPU from us and make us think we are running code concurrently.
These “things” can be processes, which are units of code that get run concurrently in the sense that each process thinks its running in its own world with its own, non-shared memory.
Another example is threads, which are units of code inside processes that allow concurrency as well.
The reason your 4 worker processes will be able to handle more than 4 requests is that they will fire off threads to handle more and more requests.
The actual request limit depends on HTTP server chosen, I/O, OS, hardware, network connection etc.
Good luck!
*instructions are the very basic commands the CPU can run. examples – add two numbers, jump from one instruction to another
My problem (and so solution) was yet different from those described above.
I wasn’t using models.py file, but created a models directory and created the my_model.py file there, where I put my model. Django couldn’t find my model so it wrote that there are no migrations to apply.
My solution was: in the my_app/models/__init__.py file I added this line:
from .my_model import MyModel
There are multiple possible reasons for django not detecting what to migrate during the makemigrations command.
migration folder You need a migrations package in your app.
INSTALLED_APPS You need your app to be specified in the INSTALLED_APPS .dict
Verbosity start by running makemigrations -v 3 for verbosity. This might shed some light on the problem.
Full path In INSTALLED_APPS it is recommended to specify the full module app config path ‘apply.apps.MyAppConfig’
–settings you might want to make sure the correct settings file is set: manage.py makemigrations --settings mysite.settings
specify app name explicitly put the app name in manage.py makemigrations myapp – that narrows down the migrations for the app alone and helps you isolate the problem.
model meta check you have the right app_label in your model meta
Debug django debug django core script. makemigrations command is pretty much straight forward. Here’s how to do it in pycharm. change your script definition accordingly (ex: makemigrations --traceback myapp)
Multiple databases:
Db Router when working with django db router, the router class (your custom router class) needs to implement the allow_syncdb method.
makemigrations always creates migrations for model changes, but if
allow_migrate() returns False,
I’ve read many answers to this question often stating to simply run makemigrations in some other ways. But to me, the problem was in the Meta subclass of models.
I have an app config that says label = <app name> (in the apps.py file, beside models.py, views.py etc). If by any chance your meta class doesn’t have the same label as the app label (for instance because you are splitting one too big app into multiple ones), no changes are detected (and no helpful error message whatsoever). So in my model class I have now:
class ModelClassName(models.Model):
class Meta:
app_label = '<app name>' # <-- this label was wrong before.
field_name = models.FloatField()
...
classMyModel(models.Model):
name = models.CharField(max_length=64, null=True)# works
language_code = models.CharField(max_length=2, default='en')# works
is_dumb = models.BooleanField(default=False),# doesn't work
I had another problem not described here, which drove me nuts.
class MyModel(models.Model):
name = models.CharField(max_length=64, null=True) # works
language_code = models.CharField(max_length=2, default='en') # works
is_dumb = models.BooleanField(default=False), # doesn't work
I had a trailing ‘,’ in one line perhaps from copy&paste. The line with is_dumb doesn’t created a model migration with ‘./manage.py makemigrations’ but also didn’t throw an error. After removing the ‘,’ it worked as expected.
There are sometimes when ./manage.py makemigrations is superior to ./manage.py makemigrations <myapp> because it can handle certain conflicts between apps.
Those occasions occur silently and it takes several hours of swearing to understand the real meaning of the dreaded No changes detected message.
Therefore, it is a far better choice to make use of the following command:
Erase the “db.sqlite3” file. The issue here is that your current data base will be erased, so you will have to remake it again.
Inside the migrations folder of your edited app, erase the last updated file. Remember that the first created file is: “0001_initial.py”. For example: I made a new class and register it by the “makemigrations” and “migrate” procedure, now a new file called “0002_auto_etc.py” was created; erase it.
Go to the “pycache” folder (inside the migrations folder) and erase the file “0002_auto_etc.pyc”.
Finally, go to the console and use “python manage.py makemigrations” and “python manage.py migrate”.
回答 10
我忘了提出正确的论点:
classLineInOffice(models.Model):# here
addressOfOffice = models.CharField("Корхоная жош",max_length=200)#and here...
My problem was much simpler than the above answers and probably a far more common reason as long as your project is already set up and working. In one of my applications that had been working for a long time, migrations seemed wonky, so in a hurry, I did the following:
I had mistakenly also removed all the __init__.py files :( – Everything was working again after I went in and:
touch ads1/migrations/__init__.py
For each of my applications then the makemigrations worked again.
It turns out that I had manually created a new application by copying another and forgot to put the __init__.py in the migrations folder and that confinved me that everything was wonky – leading my making it worse with an rm -r as described above.
Hope this helps someone from swearing at the “No changes detected” error for a few hours.
A very dumb issue you can have as well is to define two class Meta in your model. In that case, any change to the first one won’t be applied when running makemigrations.
class Product(models.Model):
somefield = models.CharField(max_length=255)
someotherfield = models.CharField(max_length=255)
class Meta:
indexes = [models.Index(fields=["somefield"], name="somefield_idx")]
def somefunc(self):
pass
# Many lines...
class Meta:
indexes = [models.Index(fields=["someotherfield"], name="someotherfield_idx")]
And since all the other models up until the one I had a problem with were being imported somewhere else that ended up importing from main_app which was registered in the INSTALLED_APPS, I just got lucky that they all worked.
But since I only added each app to INSTALLED_APPS and not the app_sub* when I finally added a new models file that wasn’t imported ANYWHERE else, Django totally ignored it.
My fix was adding a models.py file to the base directory of each app like this…
I had a similar issue with django 3.0, according migrations section in the official documentation, running this was enough to update my table structure:
But the output was always the same: ‘no change detected’ about my models after I executed ‘makemigrations’ script.
I had a syntax error on models.py at the model I wanted to update on db:
In my case, I first added a field to the model, and Django said there’re no changes.
Than I decided to change the “table name” of the model, makemigrations worked.
Than I changed table name back to default, and the new field was also there.
There is a “bug” in django migration system, sometimes it doesn’t see the new field. Might be related with date field.
The possible reason could be deletion of the existing db file and migrations folder
you can use python manage.py makemigrations <app_name> this should work. I once faced a similar problem.
I added a boolean field, and at the same time added an @property referencing it, with the same name (doh). Commented the property and migration sees and adds the new field. Renamed the property and all is good.
If you have the managed = True in yout model Meta, you need to remove it and do a migration. Then run the migrations again, it will detect the new updates.
When adding new models to the django api application and running the python manage.py makemigrations the tool did not detect any new models.
The strange thing was that the old models did got picked by makemigrations, but this was because they were referenced in the urlpatterns chain and the tool somehow detected them. So keep an eye on that behavior.
The problem was because the directory structure corresponding to the models package had subpackages and all the __init__.py files were empty. They must explicitly import all the required classes in each subfolder and in the models__init__.py for Django to pick them up with the makemigrations tool.
Try registering your model in admin.py, here’s an example:-
admin.site.register(YourModelHere)
You can do the following things:-
1. admin.site.register(YourModelHere) # In admin.py
2. Reload the page and try again
3. Hit CTRL-S and save
4. There might be an error, specially check models.py and admin.py
5. Or, at the end of it all just restart the server
The Best Thing You can do is, Delete the existing database. In my case, I were using phpMyAdmin SQL database, so I manually delete the created database overthere.
After Deleting:
I create database in PhpMyAdmin, and doesn,t add any tables.
Again run the following Commands:
python manage.py makemigrations
python manage.py migrate
After These Commands: You can see django has automatically created other necessary tables in Database(Approx there are 10 tables).
python manage.py makemigrations <app_name>
python manage.py migrate
And Lastly: After above commands all the model(table) you have created are directly imported to the database.
I had a different issue while creating a new app called deals. I wanted to separate the models inside that app so I had 2 model files named deals.py and dealers.py.
When running python manage.py makemigrations I got: No changes detected.
I went ahead and inside the __init__.py which lives on the same directory where my model files lived (deals and dealer) I did
from .deals import *
from .dealers import *
And then the makemigrations command worked.
Turns out that if you are not importing the models anywhere OR your models file name isn’t models.py then the models wont be detected.
Another issue that happened to me is the way I wrote the app in settings.py:
I had:
apps.deals
It should’ve been including the root project folder:
classPerson:def __init__(self,name,age):
self.name = name
self.age = age
def __getitem__(self,key):print("Inside `__getitem__` method!")return getattr(self,key)
p =Person("Subhayan",32)print(p["age"])
I have gone through most of the documentation of __getitem__ in the Python docs, but I am still unable to grasp the meaning of it.
So all I can understand is that __getitem__ is used to implement calls like self[key]. But what is the use of it?
Lets say I have a python class defined in this way:
class Person:
def __init__(self,name,age):
self.name = name
self.age = age
def __getitem__(self,key):
print ("Inside `__getitem__` method!")
return getattr(self,key)
p = Person("Subhayan",32)
print (p["age"])
This returns the results as expected. But why use __getitem__ in the first place? I have also heard that Python calls __getitem__ internally. But why does it do it?
Cong Ma does a good job of explaining what __getitem__ is used for – but I want to give you an example which might be useful.
Imagine a class which models a building. Within the data for the building it includes a number of attributes, including descriptions of the companies that occupy each floor :
Without using __getitem__ we would have a class like this :
class Building(object):
def __init__(self, floors):
self._floors = [None]*floors
def occupy(self, floor_number, data):
self._floors[floor_number] = data
def get_floor_data(self, floor_number):
return self._floors[floor_number]
building1 = Building(4) # Construct a building with 4 floors
building1.occupy(0, 'Reception')
building1.occupy(1, 'ABC Corp')
building1.occupy(2, 'DEF Inc')
print( building1.get_floor_data(2) )
We could however use __getitem__ (and its counterpart __setitem__) to make the usage of the Building class ‘nicer’.
class Building(object):
def __init__(self, floors):
self._floors = [None]*floors
def __setitem__(self, floor_number, data):
self._floors[floor_number] = data
def __getitem__(self, floor_number):
return self._floors[floor_number]
building1 = Building(4) # Construct a building with 4 floors
building1[0] = 'Reception'
building1[1] = 'ABC Corp'
building1[2] = 'DEF Inc'
print( building1[2] )
Whether you use __setitem__ like this really depends on how you plan to abstract your data – in this case we have decided to treat a building as a container of floors (and you could also implement an iterator for the Building, and maybe even the ability to slice – i.e. get more than one floor’s data at a time – it depends on what you need.
The [] syntax for getting item by key or index is just syntax sugar.
When you evaluate a[i] Python calls a.__getitem__(i) (or type(a).__getitem__(a, i), but this distinction is about inheritance models and is not important here). Even if the class of a may not explicitly define this method, it is usually inherited from an ancestor class.
import copy
# Constants that can be used to index date of birth's Date-Month-Year
D =0; M =1; Y =-1classPerson(object):def __init__(self, name, age, dob):
self.name = name
self.age = age
self.dob = dob
def __getitem__(self, indx):print("Calling __getitem__")
p = copy.copy(self)
p.name = p.name.split(" ")[indx]
p.dob = p.dob[indx]# or, p.dob = p.dob.__getitem__(indx)return p
假设一个用户输入如下:
p =Person(name ='Jonab Gutu', age =20, dob=(1,1,1999))
借助__getitem__方法,用户可以访问索引属性。例如,
print p[0].name # print first (or last) nameprint p[Y].dob # print (Date or Month or ) Year of the 'date of birth'
The magic method __getitem__ is basically used for accessing list items, dictionary entries, array elements etc. It is very useful for a quick lookup of instance attributes.
Here I am showing this with an example class Person that can be instantiated by ‘name’, ‘age’, and ‘dob’ (date of birth). The __getitem__ method is written in a way that one can access the indexed instance attributes, such as first or last name, day, month or year of the dob, etc.
import copy
# Constants that can be used to index date of birth's Date-Month-Year
D = 0; M = 1; Y = -1
class Person(object):
def __init__(self, name, age, dob):
self.name = name
self.age = age
self.dob = dob
def __getitem__(self, indx):
print ("Calling __getitem__")
p = copy.copy(self)
p.name = p.name.split(" ")[indx]
p.dob = p.dob[indx] # or, p.dob = p.dob.__getitem__(indx)
return p
Suppose one user input is as follows:
p = Person(name = 'Jonab Gutu', age = 20, dob=(1, 1, 1999))
With the help of __getitem__ method, the user can access the indexed attributes. e.g.,
print p[0].name # print first (or last) name
print p[Y].dob # print (Date or Month or ) Year of the 'date of birth'
# use tensorflow.keras... to benchmark tf.keras; used GPU for all above benchmarksfrom keras.layers importInput,Dense, LSTM,Bidirectional,Conv1Dfrom keras.layers importFlatten,Dropoutfrom keras.models importModelfrom keras.optimizers importAdamimport keras.backend as K
import numpy as np
from time import time
batch_shape =(32,400,16)
X, y = make_data(batch_shape)
model_small = make_small_model(batch_shape)
model_small.train_on_batch(X, y)# skip first iteration which builds graph
timeit(model_small.train_on_batch,200, X, y)
K.clear_session()# in my testing, kernel was restarted instead
model_medium = make_medium_model(batch_shape)
model_medium.train_on_batch(X, y)# skip first iteration which builds graph
timeit(model_medium.train_on_batch,10, X, y)
使用的功能:
def timeit(func, iterations,*args):
t0 = time()for _ in range(iterations):
func(*args)print("Time/iter: %.4f sec"%((time()- t0)/ iterations))def make_small_model(batch_shape):
ipt =Input(batch_shape=batch_shape)
x =Conv1D(128,400, strides=4, padding='same')(ipt)
x =Flatten()(x)
x =Dropout(0.5)(x)
x =Dense(64, activation='relu')(x)
out =Dense(1, activation='sigmoid')(x)
model =Model(ipt, out)
model.compile(Adam(lr=1e-4),'binary_crossentropy')return model
def make_medium_model(batch_shape):
ipt =Input(batch_shape=batch_shape)
x =Bidirectional(LSTM(512, activation='relu', return_sequences=True))(ipt)
x = LSTM(512, activation='relu', return_sequences=True)(x)
x =Conv1D(128,400, strides=4, padding='same')(x)
x =Flatten()(x)
x =Dense(256, activation='relu')(x)
x =Dropout(0.5)(x)
x =Dense(128, activation='relu')(x)
x =Dense(64, activation='relu')(x)
out =Dense(1, activation='sigmoid')(x)
model =Model(ipt, out)
model.compile(Adam(lr=1e-4),'binary_crossentropy')return model
def make_data(batch_shape):return np.random.randn(*batch_shape), np.random.randint(0,2,(batch_shape[0],1))
It’s been cited by many users as the reason for switching to Pytorch, but I’ve yet to find a justification / explanation for sacrificing the most important practical quality, speed, for eager execution.
Below is code benchmarking performance, TF1 vs. TF2 – with TF1 running anywhere from 47% to 276% faster.
My question is: what is it, at the graph or hardware level, that yields such a significant slowdown?
Looking for a detailed answer – am already familiar with broad concepts. Relevant Git
Specs: CUDA 10.0.130, cuDNN 7.4.2, Python 3.7.4, Windows 10, GTX 1070
Benchmark results:
UPDATE: Disabling Eager Execution per below code does not help. The behavior, however, is inconsistent: sometimes running in graph mode helps considerably, other times it runs slower relative to Eager.
As TF devs don’t appear around anywhere, I’ll be investigating this matter myself – can follow progress in the linked Github issue.
UPDATE 2: tons of experimental results to share, along explanations; should be done today.
Benchmark code:
# use tensorflow.keras... to benchmark tf.keras; used GPU for all above benchmarks
from keras.layers import Input, Dense, LSTM, Bidirectional, Conv1D
from keras.layers import Flatten, Dropout
from keras.models import Model
from keras.optimizers import Adam
import keras.backend as K
import numpy as np
from time import time
batch_shape = (32, 400, 16)
X, y = make_data(batch_shape)
model_small = make_small_model(batch_shape)
model_small.train_on_batch(X, y) # skip first iteration which builds graph
timeit(model_small.train_on_batch, 200, X, y)
K.clear_session() # in my testing, kernel was restarted instead
model_medium = make_medium_model(batch_shape)
model_medium.train_on_batch(X, y) # skip first iteration which builds graph
timeit(model_medium.train_on_batch, 10, X, y)
Functions used:
def timeit(func, iterations, *args):
t0 = time()
for _ in range(iterations):
func(*args)
print("Time/iter: %.4f sec" % ((time() - t0) / iterations))
def make_small_model(batch_shape):
ipt = Input(batch_shape=batch_shape)
x = Conv1D(128, 400, strides=4, padding='same')(ipt)
x = Flatten()(x)
x = Dropout(0.5)(x)
x = Dense(64, activation='relu')(x)
out = Dense(1, activation='sigmoid')(x)
model = Model(ipt, out)
model.compile(Adam(lr=1e-4), 'binary_crossentropy')
return model
def make_medium_model(batch_shape):
ipt = Input(batch_shape=batch_shape)
x = Bidirectional(LSTM(512, activation='relu', return_sequences=True))(ipt)
x = LSTM(512, activation='relu', return_sequences=True)(x)
x = Conv1D(128, 400, strides=4, padding='same')(x)
x = Flatten()(x)
x = Dense(256, activation='relu')(x)
x = Dropout(0.5)(x)
x = Dense(128, activation='relu')(x)
x = Dense(64, activation='relu')(x)
out = Dense(1, activation='sigmoid')(x)
model = Model(ipt, out)
model.compile(Adam(lr=1e-4), 'binary_crossentropy')
return model
def make_data(batch_shape):
return np.random.randn(*batch_shape), np.random.randint(0, 2, (batch_shape[0], 1))
UPDATE 2/18/2020: I’ve benched 2.1 and 2.1-nightly; the results are mixed. All but one configs (model & data size) are as fast as or much faster than the best of TF2 & TF1. The one that’s slower, and slower dramatically, is Large-Large – esp. in Graph execution (1.6x to 2.5x slower).
Furthermore, there are extreme reproducibility differences between Graph and Eager for a large model I tested – one not explainable via randomness/compute-parallelism. I can’t currently present reproducible code for these claims per time constraints, so instead I strongly recommend testing this for your own models.
Haven’t opened a Git issue on these yet, but I did comment on the original – no response yet. I’ll update the answer(s) once progress is made.
VERDICT: it isn’t, IF you know what you’re doing. But if you don’t, it could cost you, lots – by a few GPU upgrades on average, and by multiple GPUs worst-case.
THIS ANSWER: aims to provide a high-level description of the issue, as well as guidelines for how to decide on the training configuration specific to your needs. For a detailed, low-level description, which includes all benchmarking results + code used, see my other answer.
I’ll be updating my answer(s) w/ more info if I learn any – can bookmark / “star” this question for reference.
ISSUE SUMMARY: as confirmed by a TensorFlow developer, Q. Scott Zhu, TF2 focused development on Eager execution & tight integration w/ Keras, which involved sweeping changes in TF source – including at graph-level. Benefits: greatly expanded processing, distribution, debug, and deployment capabilities. The cost of some of these, however, is speed.
The matter, however, is fairly more complex. It isn’t just TF1 vs. TF2 – factors yielding significant differences in train speed include:
TF2 vs. TF1
Eager vs. Graph mode
keras vs. tf.keras
numpy vs. tf.data.Dataset vs. …
train_on_batch() vs. fit()
GPU vs. CPU
model(x) vs. model.predict(x) vs. …
Unfortunately, almost none of the above are independent of the other, and each can at least double execution time relative to another. Fortunately, you can determine what’ll work best systematically, and with a few shortcuts – as I’ll be showing.
WHAT SHOULD I DO? Currently, the only way is – experiment for your specific model, data, and hardware. No single configuration will always work best – but there are do’s and don’t’s to simplify your search:
fit() + numpy + tf.keras + TF1/TF2 + Graph + large model & data
>> DON’T:
fit() + numpy + keras for small & medium models and data
fit() + numpy + tf.keras + TF1/TF2 + Eager
train_on_batch() + numpy + keras + TF1 + Eager
[Major]tf.python.keras; it can run 10-100x slower, and w/ plenty of bugs; more info
This includes layers, models, optimizers, & related “out-of-box” usage imports; ops, utils, & related ‘private’ imports are fine – but to be sure, check for alts, & whether they’re used in tf.keras
Refer to code at bottom of my other answer for an example benchmarking setup. The list above is based mainly on the “BENCHMARKS” tables in the other answer.
LIMITATIONS of the above DO’s & DON’T’s:
This question’s titled “Why is TF2 much slower than TF1?”, and while its body concerns training explicitly, the matter isn’t limited to it; inference, too, is subject to major speed differences, even within the same TF version, import, data format, etc. – see this answer.
RNNs are likely to notably change the data grid in the other answer, as they’ve been improved in TF2
Models primarily used Conv1D and Dense – no RNNs, sparse data/targets, 4/5D inputs, & other configs
Input data limited to numpy and tf.data.Dataset, while many other formats exist; see other answer
GPU was used; results will differ on a CPU. In fact, when I asked the question, my CUDA wasn’t properly configured, and some of the results were CPU-based.
Why did TF2 sacrifice the most practical quality, speed, for eager execution? It hasn’t, clearly – graph is still available. But if the question is “why eager at all”:
Superior debugging: you’ve likely come across multitudes of questions asking “how do I get intermediate layer outputs” or “how do I inspect weights”; with eager, it’s (almost) as simple as .__dict__. Graph, in contrast, requires familiarity with special backend functions – greatly complicating the entire process of debugging & introspection.
Faster prototyping: per ideas similar to above; faster understanding = more time left for actual DL.
HOW TO ENABLE/DISABLE EAGER?
tf.enable_eager_execution() # TF1; must be done before any model/tensor creation
tf.compat.v1.disable_eager_execution() # TF2; above holds
ADDITIONAL INFO:
Careful with _on_batch() methods in TF2; according to the TF dev, they still use a slower implementation, but not intentionally – i.e. it’s to be fixed. See other answer for details.
REQUESTS TO TENSORFLOW DEVS:
Please fix train_on_batch(), and the performance aspect of calling fit() iteratively; custom train loops are important to many, especially to me.
Add documentation / docstring mention of these performance differences for users’ knowledge.
Improve general execution speed to keep peeps from hopping to Pytorch.
11/14/19 – found a model (in my real application) that that runs slower on TF2 for all* configurations w/ Numpy input data. Differences ranged 13-19%, averaging 17%. Differences between keras and tf.keras, however, were more dramatic: 18-40%, avg. 32% (both TF1 & 2). (* – except Eager, for which TF2 OOM’d)
11/17/19 – devs updated on_batch() methods in a recent commit, stating to have improved speed – to be released in TF 2.1, or available now as tf-nightly. As I’m unable to get latter running, will delay benching until 2.1.
2/20/20 – prediction performance is also worth benching; in TF2, for example, CPU prediction times can involve periodic spikes
training_v2.Loop()
training_distributed.DistributionMultiWorkerTrainingLoop(
training_v2.Loop())# multi-worker mode# Case 1: distribution strategy
training_distributed.DistributionMultiWorkerTrainingLoop(
training_distributed.DistributionSingleWorkerTrainingLoop())# Case 2: generator-like. Input is Python generator, or Sequence object,# or a non-distributed Dataset or iterator in eager execution.
training_generator.GeneratorOrSequenceTrainingLoop()
training_generator.EagerDatasetOrIteratorTrainingLoop()# Case 3: Symbolic tensors or Numpy array-like. This includes Datasets and iterators # in graph mode (since they generate symbolic tensors).
training_generator.GeneratorLikeTrainingLoop()# Eager
training_arrays.ArrayLikeTrainingLoop()# Graph
THIS ANSWER: aims to provide a detailed, graph/hardware-level description of the issue – including TF2 vs. TF1 train loops, input data processors, and Eager vs. Graph mode executions. For an issue summary & resolution guidelines, see my other answer.
PERFORMANCE VERDICT: sometimes one is faster, sometimes the other, depending on configuration. As far as TF2 vs TF1 goes, they’re about on par on average, but significant config-based differences do exist, and TF1 trumps TF2 more often than vice versa. See “BENCHMARKING” below.
EAGER VS. GRAPH: the meat of this entire answer for some: TF2’s eager is slower than TF1’s, according to my testing. Details further down.
The fundamental difference between the two is: Graph sets up a computational network proactively, and executes when ‘told to’ – whereas Eager executes everything upon creation. But the story only begins here:
Eager is NOT devoid of Graph, and may in fact be mostly Graph, contrary to expectation. What it largely is, is executed Graph – this includes model & optimizer weights, comprising a great portion of the graph.
Eager rebuilds part of own graph at execution; direct consequence of Graph not being fully built — see profiler results. This has a computational overhead.
Eager is slower w/ Numpy inputs; per this Git comment & code, Numpy inputs in Eager include the overhead cost of copying tensors from CPU to GPU. Stepping through source code, data handling differences are clear; Eager directly passes Numpy, while Graph passes tensors which then evaluate to Numpy; uncertain of the exact process, but latter should involve GPU-level optimizations
TF2 Eager is slower than TF1 Eager – this is… unexpected. See benchmarking results below. Differences span from negligible to significant, but are consistent. Unsure why it’s the case – if a TF dev clarifies, will update answer.
TF2 vs. TF1: quoting relevant portions of a TF dev’s, Q. Scott Zhu’s, response – w/ bit of my emphasis & rewording:
In eager, the runtime needs to execute the ops and return the numerical value for every line of python code. The nature of single step execution causes it to be slow.
In TF2, Keras leverages tf.function to build its graph for training, eval and prediction. We call them “execution function” for the model. In TF1, the “execution function” was a FuncGraph, which shared some common component as TF function, but has a different implementation.
During the process, we somehow left an incorrect implementation for train_on_batch(), test_on_batch() and predict_on_batch(). They are still numerically correct, but the execution function for x_on_batch is a pure python function, rather than a tf.function wrapped python function. This will cause slowness
In TF2, we convert all input data into a tf.data.Dataset, by which we can unify our execution function to handle the single type of the inputs. There might be some overhead in the dataset conversion, and I think this is a one-time only overhead, rather than a per-batch cost
With the last sentence of last paragraph above, and last clause of below paragraph:
To overcome the slowness in eager mode, we have @tf.function, which will turn a python function into a graph. When feed numerical value like np array, the body of the tf.function is converted into static graph, being optimized, and return the final value, which is fast and should have similar performance as TF1 graph mode.
I disagree – per my profiling results, which show Eager’s input data processing to be substantially slower than Graph’s. Also, unsure about tf.data.Dataset in particular, but Eager does repeatedly call multiple of the same data conversion methods – see profiler.
Train Loops: depending on (1) Eager vs. Graph; (2) input data format, training in will proceed with a distinct train loop – in TF2, _select_training_loop(), training.py, one of:
training_v2.Loop()
training_distributed.DistributionMultiWorkerTrainingLoop(
training_v2.Loop()) # multi-worker mode
# Case 1: distribution strategy
training_distributed.DistributionMultiWorkerTrainingLoop(
training_distributed.DistributionSingleWorkerTrainingLoop())
# Case 2: generator-like. Input is Python generator, or Sequence object,
# or a non-distributed Dataset or iterator in eager execution.
training_generator.GeneratorOrSequenceTrainingLoop()
training_generator.EagerDatasetOrIteratorTrainingLoop()
# Case 3: Symbolic tensors or Numpy array-like. This includes Datasets and iterators
# in graph mode (since they generate symbolic tensors).
training_generator.GeneratorLikeTrainingLoop() # Eager
training_arrays.ArrayLikeTrainingLoop() # Graph
Each handles resource allocation differently, and bears consequences on performance & capability.
Train Loops: fit vs train_on_batch, keras vs. tf.keras: each of the four uses different train loops, though perhaps not in every possible combination. keras‘ fit, for example, uses a form of fit_loop, e.g. training_arrays.fit_loop(), and its train_on_batch may use K.function(). tf.keras has a more sophisticated hierarchy described in part in previous section.
Train Loops: documentation — relevant source docstring on some of the different execution methods:
Unlike other TensorFlow operations, we don’t convert python
numerical inputs to tensors. Moreover, a new graph is generated for each
distinct python numerical value
functioninstantiates a separate graph for every unique set of input
shapes and datatypes.
A single tf.function object might need to map to multiple computation graphs
under the hood. This should be visible only as performance (tracing graphs has
a nonzero computational and memory cost)
Input data processors: similar to above, the processor is selected case-by-case, depending on internal flags set according to runtime configurations (execution mode, data format, distribution strategy). The simplest case’s with Eager, which works directly w/ Numpy arrays. For some specific examples, see this answer.
MODEL SIZE, DATA SIZE:
Is decisive; no single configuration crowned itself atop all model & data sizes.
Data size relative to model size is important; for small data & model, data transfer (e.g. CPU to GPU) overhead can dominate. Likewise, small overhead processors can run slower on large data per data conversion time dominating (see convert_to_tensor in “PROFILER”)
Speed differs per train loops’ and input data processors’ differing means of handling resources.
% computed as (1 - longer_time / shorter_time)*100; rationale: we’re interested by what factor one is faster than the other; shorter / longer is actually a non-linear relation, not useful for direct comparison
% sign determination:
TF2 vs TF1: + if TF2 is faster
GvE (Graph vs. Eager): + if Graph is faster
TF2 = TensorFlow 2.0.0 + Keras 2.3.1; TF1 = TensorFlow 1.14.0 + Keras 2.2.5
PROFILER:
PROFILER – Explanation: Spyder 3.3.6 IDE profiler.
Some functions are repeated in nests of others; hence, it’s hard to track down the exact separation between “data processing” and “training” functions, so there will be some overlap – as pronounced in the very last result.
% figures computed w.r.t. runtime minus build time
Build time computed by summing all (unique) runtimes which were called 1 or 2 times
Train time computed by summing all (unique) runtimes which were called the same # of times as the # of iterations, and some of their nests’ runtimes
Functions are profiled according to their original names, unfortunately (i.e. _func = func will profile as func), which mixes in build time – hence the need to exclude it
TESTING ENVIRONMENT:
Executed code at bottom w/ minimal background tasks running
GPU was “warmed up” w/ a few iterations before timing iterations, as suggested in this post
CUDA 10.0.130, cuDNN 7.6.0, TensorFlow 1.14.0, & TensorFlow 2.0.0 built from source, plus Anaconda
Python 3.7.4, Spyder 3.3.6 IDE
GTX 1070, Windows 10, 24GB DDR4 2.4-MHz RAM, i7-7700HQ 2.8-GHz CPU
METHODOLOGY:
Benchmark ‘small’, ‘medium’, & ‘large’ model & data sizes
Fix # of parameters for each model size, independent of input data size
“Larger” model has more parameters and layers
“Larger” data has a longer sequence, but same batch_size and num_channels
Models only use Conv1D, Dense ‘learnable’ layers; RNNs avoided per TF-version implem. differences
Always ran one train fit outside of benchmarking loop, to omit model & optimizer graph building
Not using sparse data (e.g. layers.Embedding()) or sparse targets (e.g. SparseCategoricalCrossEntropy()
LIMITATIONS: a “complete” answer would explain every possible train loop & iterator, but that’s surely beyond my time ability, nonexistent paycheck, or general necessity. The results are only as good as the methodology – interpret with an open mind.
CODE:
import numpy as np
import tensorflow as tf
import random
from termcolor import cprint
from time import time
from tensorflow.keras.layers import Input, Dense, Conv1D
from tensorflow.keras.layers import Dropout, GlobalAveragePooling1D
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import Adam
import tensorflow.keras.backend as K
#from keras.layers import Input, Dense, Conv1D
#from keras.layers import Dropout, GlobalAveragePooling1D
#from keras.models import Model
#from keras.optimizers import Adam
#import keras.backend as K
#tf.compat.v1.disable_eager_execution()
#tf.enable_eager_execution()
def reset_seeds(reset_graph_with_backend=None, verbose=1):
if reset_graph_with_backend is not None:
K = reset_graph_with_backend
K.clear_session()
tf.compat.v1.reset_default_graph()
if verbose:
print("KERAS AND TENSORFLOW GRAPHS RESET")
np.random.seed(1)
random.seed(2)
if tf.__version__[0] == '2':
tf.random.set_seed(3)
else:
tf.set_random_seed(3)
if verbose:
print("RANDOM SEEDS RESET")
print("TF version: {}".format(tf.__version__))
reset_seeds()
def timeit(func, iterations, *args, _verbose=0, **kwargs):
t0 = time()
for _ in range(iterations):
func(*args, **kwargs)
print(end='.'*int(_verbose))
print("Time/iter: %.4f sec" % ((time() - t0) / iterations))
def make_model_small(batch_shape):
ipt = Input(batch_shape=batch_shape)
x = Conv1D(128, 40, strides=4, padding='same')(ipt)
x = GlobalAveragePooling1D()(x)
x = Dropout(0.5)(x)
x = Dense(64, activation='relu')(x)
out = Dense(1, activation='sigmoid')(x)
model = Model(ipt, out)
model.compile(Adam(lr=1e-4), 'binary_crossentropy')
return model
def make_model_medium(batch_shape):
ipt = Input(batch_shape=batch_shape)
x = ipt
for filters in [64, 128, 256, 256, 128, 64]:
x = Conv1D(filters, 20, strides=1, padding='valid')(x)
x = GlobalAveragePooling1D()(x)
x = Dense(256, activation='relu')(x)
x = Dropout(0.5)(x)
x = Dense(128, activation='relu')(x)
x = Dense(64, activation='relu')(x)
out = Dense(1, activation='sigmoid')(x)
model = Model(ipt, out)
model.compile(Adam(lr=1e-4), 'binary_crossentropy')
return model
def make_model_large(batch_shape):
ipt = Input(batch_shape=batch_shape)
x = Conv1D(64, 400, strides=4, padding='valid')(ipt)
x = Conv1D(128, 200, strides=1, padding='valid')(x)
for _ in range(40):
x = Conv1D(256, 12, strides=1, padding='same')(x)
x = Conv1D(512, 20, strides=2, padding='valid')(x)
x = Conv1D(1028, 10, strides=2, padding='valid')(x)
x = Conv1D(256, 1, strides=1, padding='valid')(x)
x = GlobalAveragePooling1D()(x)
x = Dense(256, activation='relu')(x)
x = Dropout(0.5)(x)
x = Dense(128, activation='relu')(x)
x = Dense(64, activation='relu')(x)
out = Dense(1, activation='sigmoid')(x)
model = Model(ipt, out)
model.compile(Adam(lr=1e-4), 'binary_crossentropy')
return model
def make_data(batch_shape):
return np.random.randn(*batch_shape), \
np.random.randint(0, 2, (batch_shape[0], 1))
def make_data_tf(batch_shape, n_batches, iters):
data = np.random.randn(n_batches, *batch_shape),
trgt = np.random.randint(0, 2, (n_batches, batch_shape[0], 1))
return tf.data.Dataset.from_tensor_slices((data, trgt))#.repeat(iters)
batch_shape_small = (32, 140, 30)
batch_shape_medium = (32, 1400, 30)
batch_shape_large = (32, 14000, 30)
batch_shapes = batch_shape_small, batch_shape_medium, batch_shape_large
make_model_fns = make_model_small, make_model_medium, make_model_large
iterations = [200, 100, 50]
shape_names = ["Small data", "Medium data", "Large data"]
model_names = ["Small model", "Medium model", "Large model"]
def test_all(fit=False, tf_dataset=False):
for model_fn, model_name, iters in zip(make_model_fns, model_names, iterations):
for batch_shape, shape_name in zip(batch_shapes, shape_names):
if (model_fn is make_model_large) and (batch_shape is batch_shape_small):
continue
reset_seeds(reset_graph_with_backend=K)
if tf_dataset:
data = make_data_tf(batch_shape, iters, iters)
else:
data = make_data(batch_shape)
model = model_fn(batch_shape)
if fit:
if tf_dataset:
model.train_on_batch(data.take(1))
t0 = time()
model.fit(data, steps_per_epoch=iters)
print("Time/iter: %.4f sec" % ((time() - t0) / iters))
else:
model.train_on_batch(*data)
timeit(model.fit, iters, *data, _verbose=1, verbose=0)
else:
model.train_on_batch(*data)
timeit(model.train_on_batch, iters, *data, _verbose=1)
cprint(">> {}, {} done <<\n".format(model_name, shape_name), 'blue')
del model
test_all(fit=True, tf_dataset=False)
I don’t like how it looks like (not too readable) but I don’t have any better idea to limit lines to 79 characters in this situation. Is there a better way of breaking it (preferably without backslashes)?
This is a case where a line continuation character is preferred to open parentheses. The need for this style becomes more obvious as method names get longer and as methods start taking arguments:
PEP 8 is intend to be interpreted with a measure of common-sense and an eye for both the practical and the beautiful. Happily violate any PEP 8 guideline that results in ugly or hard to read code.
That being said, if you frequently find yourself at odds with PEP 8, it may be a sign that there are readability issues that transcend your choice of whitespace :-)
According to Python Language Reference
You can use a backslash.
Or simply break it. If a bracket is not paired, python will not treat that as a line. And under such circumstance, the indentation of following lines doesn’t matter.
回答 5
它与其他人提供的解决方案有些不同,但我的最爱,因为它有时会导致漂亮的元编程。
base =[Subkeyword.subkeyword_id,Subkeyword_word]
search ={'subkeyword_company_id':self.e_company_id,'subkeyword_word':subkeyword_word,'subkeyword_active':True,}
subkeyword =Session.query(*base).filter_by(**search).one()
This is a nice technique for building searches. Go through a list of conditionals to mine from your complex query form (or string-based deductions about what the user is looking for), then just explode the dictionary into the filter.
I had an interview with a hedge fund company in New York a few months ago and unfortunately, I did not get the internship offer as a data/software engineer. (They also asked the solution to be in Python.)
I pretty much screwed up on the first interview problem…
Question: Given a string of a million numbers (Pi for example), write
a function/program that returns all repeating 3 digit numbers and number of
repetition greater than 1
For example: if the string was: 123412345123456 then the function/program would return:
123 - 3 times
234 - 3 times
345 - 2 times
They did not give me the solution after I failed the interview, but they did tell me that the time complexity for the solution was constant of 1000 since all the possible outcomes are between:
000 –> 999
Now that I’m thinking about it, I don’t think it’s possible to come up with a constant time algorithm. Is it?
#include<stdio.h>#include<string.h>int main(void){staticchar inpStr[100000000+1];staticint freq[1000];// Set up test data.
memset(inpStr,'1',sizeof(inpStr));
inpStr[sizeof(inpStr)-1]='\0';// Need at least three digits to do anything useful.if(strlen(inpStr)<=2)return0;// Get initial feed from first two digits, process others.int val =(inpStr[0]-'0')*10+ inpStr[1]-'0';char*inpPtr =&(inpStr[2]);while(*inpPtr !='\0'){// Remove hundreds, add next digit as units, adjust table.
val =(val %100)*10+*inpPtr++-'0';
freq[val]++;}// Output (relevant part of) table.for(int i =0; i <1000;++i)if(freq[i]>1)
printf("%3d -> %d\n", i, freq[i]);return0;}
You got off lightly, you probably don’t want to be working for a hedge fund where the quants don’t understand basic algorithms :-)
There is no way to process an arbitrarily-sized data structure in O(1) if, as in this case, you need to visit every element at least once. The best you can hope for is O(n) in this case, where n is the length of the string.
Although, as an aside, a nominal O(n) algorithm will be O(1) for a fixed input size so, technically, they may have been correct here. However, that’s not usually how people use complexity analysis.
It appears to me you could have impressed them in a number of ways.
First, by informing them that it’s not possible to do it in O(1), unless you use the “suspect” reasoning given above.
Second, by showing your elite skills by providing Pythonic code such as:
inpStr = '123412345123456'
# O(1) array creation.
freq = [0] * 1000
# O(n) string processing.
for val in [int(inpStr[pos:pos+3]) for pos in range(len(inpStr) - 2)]:
freq[val] += 1
# O(1) output of relevant array values.
print ([(num, freq[num]) for num in range(1000) if freq[num] > 1])
This outputs:
[(123, 3), (234, 3), (345, 2)]
though you could, of course, modify the output format to anything you desire.
And, finally, by telling them there’s almost certainly no problem with an O(n) solution, since the code above delivers results for a one-million-digit string in well under half a second. It seems to scale quite linearly as well, since a 10,000,000-character string takes 3.5 seconds and a 100,000,000-character one takes 36 seconds.
And, if they need better than that, there are ways to parallelise this sort of stuff that can greatly speed it up.
Not within a single Python interpreter of course, due to the GIL, but you could split the string into something like (overlap indicated by vv is required to allow proper processing of the boundary areas):
vv
123412 vv
123451
5123456
You can farm these out to separate workers and combine the results afterwards.
The splitting of input and combining of output are likely to swamp any saving with small strings (and possibly even million-digit strings) but, for much larger data sets, it may well make a difference. My usual mantra of “measure, don’t guess” applies here, of course.
This mantra also applies to other possibilities, such as bypassing Python altogether and using a different language which may be faster.
For example, the following C code, running on the same hardware as the earlier Python code, handles a hundred million digits in 0.6 seconds, roughly the same amount of time as the Python code processed one million. In other words, much faster:
#include <stdio.h>
#include <string.h>
int main(void) {
static char inpStr[100000000+1];
static int freq[1000];
// Set up test data.
memset(inpStr, '1', sizeof(inpStr));
inpStr[sizeof(inpStr)-1] = '\0';
// Need at least three digits to do anything useful.
if (strlen(inpStr) <= 2) return 0;
// Get initial feed from first two digits, process others.
int val = (inpStr[0] - '0') * 10 + inpStr[1] - '0';
char *inpPtr = &(inpStr[2]);
while (*inpPtr != '\0') {
// Remove hundreds, add next digit as units, adjust table.
val = (val % 100) * 10 + *inpPtr++ - '0';
freq[val]++;
}
// Output (relevant part of) table.
for (int i = 0; i < 1000; ++i)
if (freq[i] > 1)
printf("%3d -> %d\n", i, freq[i]);
return 0;
}
Constant time isn’t possible. All 1 million digits need to be looked at at least once, so that is a time complexity of O(n), where n = 1 million in this case.
For a simple O(n) solution, create an array of size 1000 that represents the number of occurrences of each possible 3 digit number. Advance 1 digit at a time, first index == 0, last index == 999997, and increment array[3 digit number] to create a histogram (count of occurrences for each possible 3 digit number). Then output the content of the array with counts > 1.
from collections importCounterdef triple_counter(s):
c =Counter(s[n-3: n]for n in range(3, len(s)))for tri, n in c.most_common():if n >1:print('%s - %i times.'%(tri, n))else:breakif __name__ =='__main__':import random
s =''.join(random.choice('0123456789')for _ in range(1_000_000))
triple_counter(s)
A million is small for the answer I give below. Expecting only that you have to be able to run the solution in the interview, without a pause, then The following works in less than two seconds and gives the required result:
from collections import Counter
def triple_counter(s):
c = Counter(s[n-3: n] for n in range(3, len(s)))
for tri, n in c.most_common():
if n > 1:
print('%s - %i times.' % (tri, n))
else:
break
if __name__ == '__main__':
import random
s = ''.join(random.choice('0123456789') for _ in range(1_000_000))
triple_counter(s)
Hopefully the interviewer would be looking for use of the standard libraries collections.Counter class.
Parallel execution version
I wrote a blog post on this with more explanation.
回答 3
简单的O(n)解决方案是对每个3位数字进行计数:
for nr in range(1000):
cnt = text.count('%03d'% nr)if cnt >1:print'%03d is found %d times'%(nr, cnt)
这将搜索全部100万个数字1000次。
仅遍历数字一次:
counts =[0]*1000for idx in range(len(text)-2):
counts[int(text[idx:idx+3])]+=1for nr, cnt in enumerate(counts):if cnt >1:print'%03d is found %d times'%(nr, cnt)
The simple O(n) solution would be to count each 3-digit number:
for nr in range(1000):
cnt = text.count('%03d' % nr)
if cnt > 1:
print '%03d is found %d times' % (nr, cnt)
This would search through all 1 million digits 1000 times.
Traversing the digits only once:
counts = [0] * 1000
for idx in range(len(text)-2):
counts[int(text[idx:idx+3])] += 1
for nr, cnt in enumerate(counts):
if cnt > 1:
print '%03d is found %d times' % (nr, cnt)
Timing shows that iterating only once over the index is twice as fast as using count.
def setup_data(n):import random
digits ="0123456789"return dict(text =''.join(random.choice(digits)for i in range(n)))def f_np(text):# Get the data into NumPyimport numpy as np
a = np.frombuffer(bytes(text,'utf8'), dtype=np.uint8)- ord('0')# Rolling triplets
a3 = np.lib.stride_tricks.as_strided(a,(3, a.size-2),2*a.strides)
bins = np.zeros((10,10,10), dtype=int)# Next line performs O(n) binning
np.add.at(bins, tuple(a3),1)# Filtering is left as an exercisereturn bins.ravel()def f_py(text):
counts =[0]*1000for idx in range(len(text)-2):
counts[int(text[idx:idx+3])]+=1return counts
import numpy as np
import types
from timeit import timeit
for n in(10,1000,1000000):
data = setup_data(n)
ref = f_np(**data)print(f'n = {n}')for name, func in list(globals().items()):ifnot name.startswith('f_')ornot isinstance(func, types.FunctionType):continuetry:assert np.all(ref == func(**data))print("{:16s}{:16.8f} ms".format(name[2:], timeit('f(**data)', globals={'f':func,'data':data}, number=10)*100))except:print("{:16s} apparently crashed".format(name[2:]))
毫不奇怪,在大型数据集上,NumPy比@Daniel的纯Python解决方案要快一点。样本输出:
# n = 10# np 0.03481400 ms# py 0.00669330 ms# n = 1000# np 0.11215360 ms# py 0.34836530 ms# n = 1000000# np 82.46765980 ms# py 360.51235450 ms
Here is a NumPy implementation of the “consensus” O(n) algorithm: walk through all triplets and bin as you go. The binning is done by upon encountering say “385”, adding one to bin[3, 8, 5] which is an O(1) operation. Bins are arranged in a 10x10x10 cube. As the binning is fully vectorized there is no loop in the code.
def setup_data(n):
import random
digits = "0123456789"
return dict(text = ''.join(random.choice(digits) for i in range(n)))
def f_np(text):
# Get the data into NumPy
import numpy as np
a = np.frombuffer(bytes(text, 'utf8'), dtype=np.uint8) - ord('0')
# Rolling triplets
a3 = np.lib.stride_tricks.as_strided(a, (3, a.size-2), 2*a.strides)
bins = np.zeros((10, 10, 10), dtype=int)
# Next line performs O(n) binning
np.add.at(bins, tuple(a3), 1)
# Filtering is left as an exercise
return bins.ravel()
def f_py(text):
counts = [0] * 1000
for idx in range(len(text)-2):
counts[int(text[idx:idx+3])] += 1
return counts
import numpy as np
import types
from timeit import timeit
for n in (10, 1000, 1000000):
data = setup_data(n)
ref = f_np(**data)
print(f'n = {n}')
for name, func in list(globals().items()):
if not name.startswith('f_') or not isinstance(func, types.FunctionType):
continue
try:
assert np.all(ref == func(**data))
print("{:16s}{:16.8f} ms".format(name[2:], timeit(
'f(**data)', globals={'f':func, 'data':data}, number=10)*100))
except:
print("{:16s} apparently crashed".format(name[2:]))
Unsurprisingly, NumPy is a bit faster than @Daniel’s pure Python solution on large data sets. Sample output:
# n = 10
# np 0.03481400 ms
# py 0.00669330 ms
# n = 1000
# np 0.11215360 ms
# py 0.34836530 ms
# n = 1000000
# np 82.46765980 ms
# py 360.51235450 ms
回答 5
我将解决以下问题:
def find_numbers(str_num):
final_dict ={}
buffer ={}for idx in range(len(str_num)-3):
num = int(str_num[idx:idx +3])if num notin buffer:
buffer[num]=0
buffer[num]+=1if buffer[num]>1:
final_dict[num]= buffer[num]return final_dict
def find_numbers(str_num):
final_dict = {}
buffer = {}
for idx in range(len(str_num) - 3):
num = int(str_num[idx:idx + 3])
if num not in buffer:
buffer[num] = 0
buffer[num] += 1
if buffer[num] > 1:
final_dict[num] = buffer[num]
return final_dict
As per my understanding, you cannot have the solution in a constant time. It will take at least one pass over the million digit number (assuming its a string). You can have a 3-digit rolling iteration over the digits of the million length number and increase the value of hash key by 1 if it already exists or create a new hash key (initialized by value 1) if it doesn’t exists already in the dictionary.
The code will look something like this:
def calc_repeating_digits(number):
hash = {}
for i in range(len(str(number))-2):
current_three_digits = number[i:i+3]
if current_three_digits in hash.keys():
hash[current_three_digits] += 1
else:
hash[current_three_digits] = 1
return hash
You can filter down to the keys which have item value greater than 1.
As mentioned in another answer, you cannot do this algorithm in constant time, because you must look at at least n digits. Linear time is the fastest you can get.
However, the algorithm can be done in O(1) space. You only need to store the counts of each 3 digit number, so you need an array of 1000 entries. You can then stream the number in.
My guess is that either the interviewer misspoke when they gave you the solution, or you misheard “constant time” when they said “constant space.”
回答 8
这是我的答案:
from timeit import timeit
from collections importCounterimport types
import random
def setup_data(n):
digits ="0123456789"return dict(text =''.join(random.choice(digits)for i in range(n)))def f_counter(text):
c =Counter()for i in range(len(text)-2):
ss = text[i:i+3]
c.update([ss])return(i for i in c.items()if i[1]>1)def f_dict(text):
d ={}for i in range(len(text)-2):
ss = text[i:i+3]if ss notin d:
d[ss]=0
d[ss]+=1return((i, d[i])for i in d if d[i]>1)def f_array(text):
a =[[[0for _ in range(10)]for _ in range(10)]for _ in range(10)]for n in range(len(text)-2):
i, j, k =(int(ss)for ss in text[n:n+3])
a[i][j][k]+=1for i, b in enumerate(a):for j, c in enumerate(b):for k, d in enumerate(c):if d >1:yield(f'{i}{j}{k}', d)for n in(1E1,1E3,1E6):
n = int(n)
data = setup_data(n)print(f'n = {n}')
results ={}for name, func in list(globals().items()):ifnot name.startswith('f_')ornot isinstance(func, types.FunctionType):continueprint("{:16s}{:16.8f} ms".format(name[2:], timeit('results[name] = f(**data)', globals={'f':func,'data':data,'results':results,'name':name}, number=10)*100))for r in results:print('{:10}: {}'.format(r, sorted(list(results[r]))[:5]))
n =10
counter 0.10595780 ms
dict 0.01070654 ms
array 0.00135370 ms
f_counter :[]
f_dict :[]
f_array :[]
n =1000
counter 2.89462101 ms
dict 0.40434612 ms
array 0.00073838 ms
f_counter :[('008',2),('009',3),('010',2),('016',2),('017',2)]
f_dict :[('008',2),('009',3),('010',2),('016',2),('017',2)]
f_array :[('008',2),('009',3),('010',2),('016',2),('017',2)]
n =1000000
counter 2849.00500992 ms
dict 438.44007806 ms
array 0.00135370 ms
f_counter :[('000',1058),('001',943),('002',1030),('003',982),('004',1042)]
f_dict :[('000',1058),('001',943),('002',1030),('003',982),('004',1042)]
f_array :[('000',1058),('001',943),('002',1030),('003',982),('004',1042)]
from timeit import timeit
from collections import Counter
import types
import random
def setup_data(n):
digits = "0123456789"
return dict(text = ''.join(random.choice(digits) for i in range(n)))
def f_counter(text):
c = Counter()
for i in range(len(text)-2):
ss = text[i:i+3]
c.update([ss])
return (i for i in c.items() if i[1] > 1)
def f_dict(text):
d = {}
for i in range(len(text)-2):
ss = text[i:i+3]
if ss not in d:
d[ss] = 0
d[ss] += 1
return ((i, d[i]) for i in d if d[i] > 1)
def f_array(text):
a = [[[0 for _ in range(10)] for _ in range(10)] for _ in range(10)]
for n in range(len(text)-2):
i, j, k = (int(ss) for ss in text[n:n+3])
a[i][j][k] += 1
for i, b in enumerate(a):
for j, c in enumerate(b):
for k, d in enumerate(c):
if d > 1: yield (f'{i}{j}{k}', d)
for n in (1E1, 1E3, 1E6):
n = int(n)
data = setup_data(n)
print(f'n = {n}')
results = {}
for name, func in list(globals().items()):
if not name.startswith('f_') or not isinstance(func, types.FunctionType):
continue
print("{:16s}{:16.8f} ms".format(name[2:], timeit(
'results[name] = f(**data)', globals={'f':func, 'data':data, 'results':results, 'name':name}, number=10)*100))
for r in results:
print('{:10}: {}'.format(r, sorted(list(results[r]))[:5]))
The array lookup method is very fast (even faster than @paul-panzer’s numpy method!). Of course, it cheats since it isn’t technicailly finished after it completes, because it’s returning a generator. It also doesn’t have to check every iteration if the value already exists, which is likely to help a lot.
n = 10
counter 0.10595780 ms
dict 0.01070654 ms
array 0.00135370 ms
f_counter : []
f_dict : []
f_array : []
n = 1000
counter 2.89462101 ms
dict 0.40434612 ms
array 0.00073838 ms
f_counter : [('008', 2), ('009', 3), ('010', 2), ('016', 2), ('017', 2)]
f_dict : [('008', 2), ('009', 3), ('010', 2), ('016', 2), ('017', 2)]
f_array : [('008', 2), ('009', 3), ('010', 2), ('016', 2), ('017', 2)]
n = 1000000
counter 2849.00500992 ms
dict 438.44007806 ms
array 0.00135370 ms
f_counter : [('000', 1058), ('001', 943), ('002', 1030), ('003', 982), ('004', 1042)]
f_dict : [('000', 1058), ('001', 943), ('002', 1030), ('003', 982), ('004', 1042)]
f_array : [('000', 1058), ('001', 943), ('002', 1030), ('003', 982), ('004', 1042)]
from collections import defaultdict
string ="103264685134845354863"
d = defaultdict(int)for elt in range(len(string)-2):
d[string[elt:elt+3]]+=1
d ={key: d[key]for key in d.keys()if d[key]>1}
from collections import defaultdict
string = "103264685134845354863"
d = defaultdict(int)
for elt in range(len(string)-2):
d[string[elt:elt+3]] += 1
d = {key: d[key] for key in d.keys() if d[key] > 1}
With a bit of creativity in for loop(and additional lookup list with True/False/None for example) you should be able to get rid of last line, as you only want to create keys in dict that we visited once up to that point.
Hope it helps :)
-Telling from the perspective of C.
-You can have an int 3-d array results[10][10][10];
-Go from 0th location to n-4th location, where n being the size of the string array.
-On each location, check the current, next and next’s next.
-Increment the cntr as resutls[current][next][next’s next]++;
-Print the values of
-It is O(n) time, there is no comparisons involved.
-You can run some parallel stuff here by partitioning the array and calculating the matches around the partitions.
回答 12
inputStr ='123456123138276237284287434628736482376487234682734682736487263482736487236482634'
count ={}for i in range(len(inputStr)-2):
subNum = int(inputStr[i:i+3])if subNum notin count:
count[subNum]=1else:
count[subNum]+=1print count
inputStr = '123456123138276237284287434628736482376487234682734682736487263482736487236482634'
count = {}
for i in range(len(inputStr) - 2):
subNum = int(inputStr[i:i+3])
if subNum not in count:
count[subNum] = 1
else:
count[subNum] += 1
print count