问题:在ipython Notebook中测量单元执行时间的简单方法

除了单元的原始输出,我想花时间在单元执行上。

为此,我尝试了%%timeit -r1 -n1但它没有公开定义在单元格内的变量。

%%time 适用于仅包含1条语句的单元格。

In[1]: %%time
       1
CPU times: user 4 µs, sys: 0 ns, total: 4 µs
Wall time: 5.96 µs
Out[1]: 1

In[2]: %%time
       # Notice there is no out result in this case.
       x = 1
       x
CPU times: user 3 µs, sys: 0 ns, total: 3 µs
Wall time: 5.96 µs

最好的方法是什么?

更新资料

我已经在Nbextension中使用Execute Time了一段时间了。这太棒了。

I would like to get the time spent on the cell execution in addition to the original output from cell.

To this end, I tried %%timeit -r1 -n1 but it doesn’t expose the variable defined within cell.

%%time works for cell which only contains 1 statement.

In[1]: %%time
       1
CPU times: user 4 µs, sys: 0 ns, total: 4 µs
Wall time: 5.96 µs
Out[1]: 1

In[2]: %%time
       # Notice there is no out result in this case.
       x = 1
       x
CPU times: user 3 µs, sys: 0 ns, total: 3 µs
Wall time: 5.96 µs

What’s the best way to do it?

Update

I have been using Execute Time in Nbextension for quite some time now. It is great.


回答 0

使用单元魔术和Phillip Cloud在github上的此项目:

通过将其放在笔记本顶部或如果您始终希望默认情况下将其放在配置文件中来进行加载:

%install_ext https://raw.github.com/cpcloud/ipython-autotime/master/autotime.py
%load_ext autotime

如果加载,则后续单元执行的每个输出将包括执行时间(以分钟和秒为单位)。

Use cell magic and this project on github by Phillip Cloud:

Load it by putting this at the top of your notebook or put it in your config file if you always want to load it by default:

%install_ext https://raw.github.com/cpcloud/ipython-autotime/master/autotime.py
%load_ext autotime

If loaded, every output of subsequent cell execution will include the time in min and sec it took to execute it.


回答 1

我发现克服此问题的唯一方法是执行带有print的最后一条语句。

不要忘了单元魔术始于,%%行魔术始于%

%%time
clf = tree.DecisionTreeRegressor().fit(X_train, y_train)
res = clf.predict(X_test)
print(res)

请注意,在下一个单元格中将不考虑在单元格内执行的任何更改,这在存在管道时是很直观的: 一个例子

The only way I found to overcome this problem is by executing the last statement with print.

Do not forget that cell magic starts with %% and line magic starts with %.

%%time
clf = tree.DecisionTreeRegressor().fit(X_train, y_train)
res = clf.predict(X_test)
print(res)

Notice that any changes performed inside the cell are not taken into consideration in the next cells, something that is counter intuitive when there is a pipeline: an example


回答 2

%time%timeit现在来IPython中的一部分内置的魔法命令

%time and %timeit now come part of ipython’s built-in magic commands


回答 3

一种更简单的方法是在jupyter_contrib_nbextensions软件包中使用ExecuteTime插件。

pip install jupyter_contrib_nbextensions
jupyter contrib nbextension install --user
jupyter nbextension enable execute_time/ExecuteTime

An easier way is to use ExecuteTime plugin in jupyter_contrib_nbextensions package.

pip install jupyter_contrib_nbextensions
jupyter contrib nbextension install --user
jupyter nbextension enable execute_time/ExecuteTime

回答 4

我只是%%time在单元格的开头添加了时间。您可以在Jupyter Spark群集/虚拟环境上使用相同的名称。只需%%time在单元格的顶部添加,您将获得输出。在使用Jupyter的Spark集群上,我将其添加到单元格的顶部,并得到如下输出:-

[1]  %%time
     import pandas as pd
     from pyspark.ml import Pipeline
     from pyspark.ml.classification import LogisticRegression
     import numpy as np
     .... code ....

Output :-

CPU times: user 59.8 s, sys: 4.97 s, total: 1min 4s
Wall time: 1min 18s

I simply added %%time at the beginning of the cell and got the time. You may use the same on Jupyter Spark cluster/ Virtual environment using the same. Just add %%time at the top of the cell and you will get the output. On spark cluster using Jupyter, I added to the top of the cell and I got output like below:-

[1]  %%time
     import pandas as pd
     from pyspark.ml import Pipeline
     from pyspark.ml.classification import LogisticRegression
     import numpy as np
     .... code ....

Output :-

CPU times: user 59.8 s, sys: 4.97 s, total: 1min 4s
Wall time: 1min 18s

回答 5

import time
start = time.time()
"the code you want to test stays here"
end = time.time()
print(end - start)
import time
start = time.time()
"the code you want to test stays here"
end = time.time()
print(end - start)

回答 6

您可以使用timeit魔术功能。

%timeit CODE_LINE

或在细胞上

%%timeit 

SOME_CELL_CODE

https://nbviewer.jupyter.org/github/ipython/ipython/blob/1.x/examples/notebooks/Cell%20Magics.ipynb检查更多IPython魔术函数

You can use timeit magic function for that.

%timeit CODE_LINE

Or on the cell

%%timeit 

SOME_CELL_CODE

Check more IPython magic functions at https://nbviewer.jupyter.org/github/ipython/ipython/blob/1.x/examples/notebooks/Cell%20Magics.ipynb


回答 7

这不是很漂亮,但没有额外的软件

class timeit():
    from datetime import datetime
    def __enter__(self):
        self.tic = self.datetime.now()
    def __exit__(self, *args, **kwargs):
        print('runtime: {}'.format(self.datetime.now() - self.tic))

然后,您可以像这样运行它:

with timeit():
    # your code, e.g., 
    print(sum(range(int(1e7))))

% 49999995000000
% runtime: 0:00:00.338492

This is not exactly beautiful but without extra software

class timeit():
    from datetime import datetime
    def __enter__(self):
        self.tic = self.datetime.now()
    def __exit__(self, *args, **kwargs):
        print('runtime: {}'.format(self.datetime.now() - self.tic))

Then you can run it like:

with timeit():
    # your code, e.g., 
    print(sum(range(int(1e7))))

% 49999995000000
% runtime: 0:00:00.338492

回答 8

有时,使用时单元格中的格式会有所不同print(res),但是jupyter / ipython带有display。请参阅下面有关使用熊猫的格式差异的示例。

%%time
import pandas as pd 
from IPython.display import display

df = pd.DataFrame({"col0":{"a":0,"b":0}
              ,"col1":{"a":1,"b":1}
              ,"col2":{"a":2,"b":2}
             })

#compare the following
print(df)
display(df)

display语句可以保留格式。 屏幕截图

Sometimes the formatting is different in a cell when using print(res), but jupyter/ipython comes with a display. See an example of the formatting difference using pandas below.

%%time
import pandas as pd 
from IPython.display import display

df = pd.DataFrame({"col0":{"a":0,"b":0}
              ,"col1":{"a":1,"b":1}
              ,"col2":{"a":2,"b":2}
             })

#compare the following
print(df)
display(df)

The display statement can preserve the formatting. screenshot


回答 9

您可能还需要查看python的分析魔术命令%prun,该命令给出类似以下内容的信息-

def sum_of_lists(N):
    total = 0
    for i in range(5):
        L = [j ^ (j >> i) for j in range(N)]
        total += sum(L)
    return total

然后

%prun sum_of_lists(1000000)

将返回

14 function calls in 0.714 seconds  

Ordered by: internal time      

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    5    0.599    0.120    0.599    0.120 <ipython-input-19>:4(<listcomp>)
    5    0.064    0.013    0.064    0.013 {built-in method sum}
    1    0.036    0.036    0.699    0.699 <ipython-input-19>:1(sum_of_lists)
    1    0.014    0.014    0.714    0.714 <string>:1(<module>)
    1    0.000    0.000    0.714    0.714 {built-in method exec}

当处理大量代码时,我发现它很有用。

you may also want to look in to python’s profiling magic command %prunwhich gives something like –

def sum_of_lists(N):
    total = 0
    for i in range(5):
        L = [j ^ (j >> i) for j in range(N)]
        total += sum(L)
    return total

then

%prun sum_of_lists(1000000)

will return

14 function calls in 0.714 seconds  

Ordered by: internal time      

ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    5    0.599    0.120    0.599    0.120 <ipython-input-19>:4(<listcomp>)
    5    0.064    0.013    0.064    0.013 {built-in method sum}
    1    0.036    0.036    0.699    0.699 <ipython-input-19>:1(sum_of_lists)
    1    0.014    0.014    0.714    0.714 <string>:1(<module>)
    1    0.000    0.000    0.714    0.714 {built-in method exec}

I find it useful when working with large chunks of code.


回答 10

遇到麻烦时,意味着什么:

?%timeit 要么 ??timeit

要获取详细信息:

Usage, in line mode:
  %timeit [-n<N> -r<R> [-t|-c] -q -p<P> -o] statement
or in cell mode:
  %%timeit [-n<N> -r<R> [-t|-c] -q -p<P> -o] setup_code
  code
  code...

Time execution of a Python statement or expression using the timeit
module.  This function can be used both as a line and cell magic:

- In line mode you can time a single-line statement (though multiple
  ones can be chained with using semicolons).

- In cell mode, the statement in the first line is used as setup code
  (executed but not timed) and the body of the cell is timed.  The cell
  body has access to any variables created in the setup code.

When in trouble what means what:

?%timeit or ??timeit

To get the details:

Usage, in line mode:
  %timeit [-n<N> -r<R> [-t|-c] -q -p<P> -o] statement
or in cell mode:
  %%timeit [-n<N> -r<R> [-t|-c] -q -p<P> -o] setup_code
  code
  code...

Time execution of a Python statement or expression using the timeit
module.  This function can be used both as a line and cell magic:

- In line mode you can time a single-line statement (though multiple
  ones can be chained with using semicolons).

- In cell mode, the statement in the first line is used as setup code
  (executed but not timed) and the body of the cell is timed.  The cell
  body has access to any variables created in the setup code.

回答 11

如果要打印壁单元执行时间,这是一个技巧,请使用

%%time
<--code goes here-->

但是请确保%% time是一个魔术函数,因此请将其放在代码的第一行

如果您将其放在代码的某些行之后,则会出现用法错误,并且无法正常工作。

If you want to print wall cell execution time here is a trick, use

%%time
<--code goes here-->

but here make sure that, the %%time is a magic function, so put it at first line in your code.

if you put it after some line of your code it’s going to give you usage error and not gonna work.


声明:本站所有文章,如无特殊说明或标注,均为本站原创发布。任何个人或组织,在未征得本站同意时,禁止复制、盗用、采集、发布本站内容到任何网站、书籍等各类媒体平台。如若本站内容侵犯了原著者的合法权益,可联系我们进行处理。