分类目录归档：知识问答

如何在Python中访问命令行参数？

2021年7月27日 Python实用宝典

问题：如何在Python中访问命令行参数？

我使用python创建我的项目设置设置，但我需要获取命令行参数的帮助。

我在终端上尝试了这个：

$python myfile.py var1 var2 var3

在我的Python文件中，我想使用所有输入的变量。

I use python to create my project settings setup, but I need help getting the command line arguments.

I tried this on the terminal:

$python myfile.py var1 var2 var3

In my Python file, I want to use all variables that are input.

回答 0

Python教程对此进行了解释：

import sys

print(sys.argv)

更具体地说，如果您运行python example.py one two three：

>>> import sys
>>> print(sys.argv)
['example.py', 'one', 'two', 'three']

Python tutorial explains it:

import sys

print(sys.argv)

More specifically, if you run python example.py one two three:

>>> import sys
>>> print(sys.argv)
['example.py', 'one', 'two', 'three']

回答 1

import sys

sys.argv[1:]

将为您提供参数列表（不包括python文件的名称）

import sys

sys.argv[1:]

will give you a list of arguments (not including the name of the python file)

回答 2

我强烈建议argparsePython 2.7及更高版本随附。

该argparse模块减少了样板代码，并使您的代码更健壮，因为该模块可以处理所有标准用例（包括子命令），为您生成帮助和用法，检查并清理用户输入-您在使用时需要担心的所有事情正在使用sys.argv方法。它是免费的（内置）。

这里有个小例子：

import argparse

parser = argparse.ArgumentParser("simple_example")
parser.add_argument("counter", help="An integer will be increased by 1 and printed.", type=int)
args = parser.parse_args()
print(args.counter + 1)

和输出为 python prog.py -h

usage: simple_example [-h] counter

positional arguments:
  counter     counter will be increased by 1 and printed.

optional arguments:
  -h, --help  show this help message and exit

和python prog.py 1您所期望的：

I highly recommend argparse which comes with Python 2.7 and later.

The argparse module reduces boiler plate code and makes your code more robust, because the module handles all standard use cases (including subcommands), generates the help and usage for you, checks and sanitize the user input – all stuff you have to worry about when you are using sys.argv approach. And it is for free (built-in).

Here a small example:

import argparse

parser = argparse.ArgumentParser("simple_example")
parser.add_argument("counter", help="An integer will be increased by 1 and printed.", type=int)
args = parser.parse_args()
print(args.counter + 1)

and the output for python prog.py -h

usage: simple_example [-h] counter

positional arguments:
  counter     counter will be increased by 1 and printed.

optional arguments:
  -h, --help  show this help message and exit

and for python prog.py 1 as you would expect:

回答 3

Python代码：

import sys

# main
param_1= sys.argv[1] 
param_2= sys.argv[2] 
param_3= sys.argv[3]  
print 'Params=', param_1, param_2, param_3

调用：

$python myfile.py var1 var2 var3

输出：

Params= var1 var2 var3

Python code:

import sys

# main
param_1= sys.argv[1] 
param_2= sys.argv[2] 
param_3= sys.argv[3]  
print 'Params=', param_1, param_2, param_3

Invocation:

$python myfile.py var1 var2 var3

Output:

Params= var1 var2 var3

回答 4

您可以使用 sys.argv用来获取参数列表。

如果您需要访问各个元素，则可以使用

sys.argv[i]

i索引在哪里，0将为您提供正在执行的python文件名。之后的任何索引都是传递的参数。

You can use sys.argv to get the arguments as a list.

If you need to access individual elements, you can use

sys.argv[i]

where i is index, 0 will give you the python filename being executed. Any index after that are the arguments passed.

回答 5

如果您这样称呼它： $ python myfile.py var1 var2 var3

import sys

var1 = sys.argv[1]
var2 = sys.argv[2]
var3 = sys.argv[3]

与数组类似，您也拥有sys.argv[0]始终是当前工作目录的数组。

If you call it like this: $ python myfile.py var1 var2 var3

import sys

var1 = sys.argv[1]
var2 = sys.argv[2]
var3 = sys.argv[3]

Similar to arrays you also have sys.argv[0] which is always the current working directory.

回答 6

我可以想到的一些其他事项。

正如@allsyed所说，sys.argv提供了一个组件列表（包括程序名称），因此，如果您想知道通过命令行传递的元素数量，可以使用len（）来确定它。基于此，如果用户未传递特定数量的参数，则可以设计异常/错误消息。

另外，如果您正在寻找一种更好的方式来处理命令行参数，我建议您查看https://docs.python.org/2/howto/argparse.html

Some additional things that I can think of.

As @allsyed said sys.argv gives a list of components (including program name), so if you want to know the number of elements passed through command line you can use len() to determine it. Based on this, you can design exception/error messages if user didn’t pass specific number of parameters.

Also if you looking for a better way to handle command line arguments, I would suggest you look at https://docs.python.org/2/howto/argparse.html

知识问答

熊猫合并101

2021年7月27日 Python实用宝典

问题：熊猫合并101

如何执行与熊猫（LEFT| RIGHT| FULL）（INNER| OUTER）的联接？
合并后如何为缺失的行添加NaN？
合并后如何去除NaN？
我可以合并索引吗？
与大熊猫交叉交往？
如何合并多个DataFrame？
merge？join？concat？update？WHO？什么？为什么？！

… 和更多。我已经看到这些重复出现的问题，询问有关熊猫合并功能的各个方面。如今，有关合并及其各种用例的大多数信息都分散在数十个措辞不好，无法搜索的帖子中。这里的目的是整理后代的一些更重要的观点。

本QnA是下一章有关大熊猫习语的有用的用户指南中的下一部分（请参阅有关透视的这篇文章和有关串联的这篇文章，我将在以后进行探讨）。

请注意，本文并非旨在代替文档，因此也请阅读！一些示例是从那里获取的。

How to perform a (LEFT|RIGHT|FULL) (INNER|OUTER) join with pandas?
How do I add NaNs for missing rows after merge?
How do I get rid of NaNs after merging?
Can I merge on the index?
Cross join with pandas?
How do I merge multiple DataFrames?
merge? join? concat? update? Who? What? Why?!

… and more. I’ve seen these recurring questions asking about various facets of the pandas merge functionality. Most of the information regarding merge and its various use cases today is fragmented across dozens of badly worded, unsearchable posts. The aim here is to collate some of the more important points for posterity.

This QnA is meant to be the next installment in a series of helpful user-guides on common pandas idioms (see this post on pivoting, and this post on concatenation, which I will be touching on, later).

Please note that this post is not meant to be a replacement for the documentation, so please read that as well! Some of the examples are taken from there.

回答 0

这篇文章旨在为读者提供有关SQL风格的与熊猫的合并，使用方法以及何时不使用它的入门。

特别是，这是这篇文章的内容：

基础-联接类型（左，右，外，内）
- 与不同的列名合并
- 避免在输出中出现重复的合并键列
在不同条件下与索引合并
- 有效地使用您的命名索引
- 合并键作为一个索引，另一个索引
多路合并列和索引（唯一和非唯一）
值得注意的替代品merge和join

这篇文章不会讲的内容：

与性能相关的讨论和时间安排（目前）。在适当的地方，最引人注目的是提到更好的替代方案。
处理后缀，删除多余的列，重命名输出以及其他特定用例。还有其他（阅读：更好）的帖子可以解决这个问题，所以请弄清楚！

注意
除非另有说明，否则大多数示例在演示各种功能时会默认使用INNER JOIN操作。

此外，可以复制和复制此处的所有DataFrame，以便您可以使用它们。另外，请参阅这篇文章，了解如何从剪贴板读取DataFrames。

最后，已使用Google绘图手工绘制了JOIN操作的所有可视表示形式。从这里得到启示。

聊够了，只是告诉我如何使用`merge`！

设定

np.random.seed(0)
left = pd.DataFrame({'key': ['A', 'B', 'C', 'D'], 'value': np.random.randn(4)})    
right = pd.DataFrame({'key': ['B', 'D', 'E', 'F'], 'value': np.random.randn(4)})

left

  key     value
0   A  1.764052
1   B  0.400157
2   C  0.978738
3   D  2.240893

right

  key     value
0   B  1.867558
1   D -0.977278
2   E  0.950088
3   F -0.151357

为了简单起见，键列具有相同的名称（目前）。

一个内连接由下式表示

注意：
此规则以及即将发布的附图均遵循以下约定：

蓝色表示合并结果中存在的行

红色表示从结果中排除（即删除）的行

绿色表示缺少的值将在结果中替换为NaN

要执行INNER JOIN，请调用merge左侧的DataFrame，并指定右侧的DataFrame和联接键（至少）作为参数。

left.merge(right, on='key')
# Or, if you want to be explicit
# left.merge(right, on='key', how='inner')

  key   value_x   value_y
0   B  0.400157  1.867558
1   D  2.240893 -0.977278

这仅返回来自left并right共享一个公共密钥的行（在本示例中为“ B”和“ D”）。

甲LEFT OUTER JOIN，或LEFT JOIN由下式表示

可以通过指定执行how='left'。

left.merge(right, on='key', how='left')

  key   value_x   value_y
0   A  1.764052       NaN
1   B  0.400157  1.867558
2   C  0.978738       NaN
3   D  2.240893 -0.977278

请仔细注意NaN的位置。如果指定how='left'，则仅left使用from 的键，而rightNaN替换缺少的数据。

同样，对于RIGHT OUTER JOIN或RIGHT JOIN来说…

…指定how='right'：

left.merge(right, on='key', how='right')

  key   value_x   value_y
0   B  0.400157  1.867558
1   D  2.240893 -0.977278
2   E       NaN  0.950088
3   F       NaN -0.151357

在这里，right使用了from 的键，而缺失的数据left被NaN代替。

最后，对于FULL OUTER JOIN，由

指定how='outer'。

left.merge(right, on='key', how='outer')

  key   value_x   value_y
0   A  1.764052       NaN
1   B  0.400157  1.867558
2   C  0.978738       NaN
3   D  2.240893 -0.977278
4   E       NaN  0.950088
5   F       NaN -0.151357

这将使用两个框架中的关键点，并且会为两个框架中缺少的行插入NaN。

该文档很好地总结了这些各种合并：

其他联接-左排除，右排除和全排除/ ANTI连接

如果您需要分两个步骤进行LEFT-Exclusive JOIN和RIGHT-Exclusive JOIN。

对于不包括JOIN的LEFT，表示为

首先执行LEFT OUTER JOIN，然后过滤（不包括！）left仅来自于行的行，

(left.merge(right, on='key', how='left', indicator=True)
     .query('_merge == "left_only"')
     .drop('_merge', 1))

  key   value_x  value_y
0   A  1.764052      NaN
2   C  0.978738      NaN

哪里，

left.merge(right, on='key', how='left', indicator=True)

  key   value_x   value_y     _merge
0   A  1.764052       NaN  left_only
1   B  0.400157  1.867558       both
2   C  0.978738       NaN  left_only
3   D  2.240893 -0.977278       both

同样，对于除权利加入之外，

(left.merge(right, on='key', how='right', indicator=True)
     .query('_merge == "right_only"')
     .drop('_merge', 1))

  key  value_x   value_y
2   E      NaN  0.950088
3   F      NaN -0.151357

最后，如果您需要执行合并操作，而该合并操作只保留左侧或右侧的键，而不同时保留两者（IOW，执行ANTI-JOIN），

您可以通过类似的方式进行操作-

(left.merge(right, on='key', how='outer', indicator=True)
     .query('_merge != "both"')
     .drop('_merge', 1))

  key   value_x   value_y
0   A  1.764052       NaN
2   C  0.978738       NaN
4   E       NaN  0.950088
5   F       NaN -0.151357

键列的不同名称

如果键列的名称不同（例如，lefthas keyLeft和righthas keyRight代替），key则必须指定left_on和right_on作为参数，而不是on：

left2 = left.rename({'key':'keyLeft'}, axis=1)
right2 = right.rename({'key':'keyRight'}, axis=1)

left2

  keyLeft     value
0       A  1.764052
1       B  0.400157
2       C  0.978738
3       D  2.240893

right2

  keyRight     value
0        B  1.867558
1        D -0.977278
2        E  0.950088
3        F -0.151357

left2.merge(right2, left_on='keyLeft', right_on='keyRight', how='inner')

  keyLeft   value_x keyRight   value_y
0       B  0.400157        B  1.867558
1       D  2.240893        D -0.977278

避免在输出中重复键列

在keyLeftfrom left和keyRightfrom 上进行合并时right，如果只希望在输出中使用keyLeft或keyRight（但不同时使用）中的任何一个，则可以从将索引设置为初步步骤开始。

left3 = left2.set_index('keyLeft')
left3.merge(right2, left_index=True, right_on='keyRight')

    value_x keyRight   value_y
0  0.400157        B  1.867558
1  2.240893        D -0.977278

将此与命令输出（恰恰是的输出left2.merge(right2, left_on='keyLeft', right_on='keyRight', how='inner')）进行对比（您会发现keyLeft它丢失了）。您可以根据将哪个帧的索引设置为关键字来找出要保留的列。例如，当执行某些OUTER JOIN操作时，这可能很重要。

仅合并其中一个的单个列 `DataFrames`

例如，考虑

right3 = right.assign(newcol=np.arange(len(right)))
right3
  key     value  newcol
0   B  1.867558       0
1   D -0.977278       1
2   E  0.950088       2
3   F -0.151357       3

如果只需要合并“ new_val”（不包含任何其他列），通常可以在合并之前仅对列进行子集化：

left.merge(right3[['key', 'newcol']], on='key')

  key     value  newcol
0   B  0.400157       0
1   D  2.240893       1

如果您要进行左外部联接，则性能更高的解决方案将涉及map：

# left['newcol'] = left['key'].map(right3.set_index('key')['newcol']))
left.assign(newcol=left['key'].map(right3.set_index('key')['newcol']))

  key     value  newcol
0   A  1.764052     NaN
1   B  0.400157     0.0
2   C  0.978738     NaN
3   D  2.240893     1.0

如前所述，这类似于但比

left.merge(right3[['key', 'newcol']], on='key', how='left')

  key     value  newcol
0   A  1.764052     NaN
1   B  0.400157     0.0
2   C  0.978738     NaN
3   D  2.240893     1.0

合并多列

要加入对多列，指定列表on（或left_on和right_on，如适用）。

left.merge(right, on=['key1', 'key2'] ...)

或者，如果名称不同，

left.merge(right, left_on=['lkey1', 'lkey2'], right_on=['rkey1', 'rkey2'])

**其他有用的`merge*`操作和功能**

合并具有Series on index的DataFrame：请参阅此答案。
此外merge，DataFrame.update和DataFrame.combine_first也用在某些情况下，更新一个数据帧与另一个。
pd.merge_ordered 是有序JOIN的有用函数。
pd.merge_asof（阅读：merge_asOf）对于近似联接很有用。

本节仅介绍最基本的内容，目的只是为了激发您的胃口。更多的例子和案例，看到的文档merge，join以及concat还有链接的功能规格。

基于索引的* -JOIN（+ index-column `merge`s）

设定

np.random.seed([3, 14])
left = pd.DataFrame({'value': np.random.randn(4)}, index=['A', 'B', 'C', 'D'])    
right = pd.DataFrame({'value': np.random.randn(4)}, index=['B', 'D', 'E', 'F'])
left.index.name = right.index.name = 'idxkey'

left
           value
idxkey          
A      -0.602923
B      -0.402655
C       0.302329
D      -0.524349

right

           value
idxkey          
B       0.543843
D       0.013135
E      -0.326498
F       1.385076

通常，索引合并看起来像这样：

left.merge(right, left_index=True, right_index=True)


         value_x   value_y
idxkey                    
B      -0.402655  0.543843
D      -0.524349  0.013135

支持索引名称

如果索引被命名，然后v0.23用户还可以指定的级别名称on（或left_on和right_on必要的）。

left.merge(right, on='idxkey')

         value_x   value_y
idxkey                    
B      -0.402655  0.543843
D      -0.524349  0.013135

合并一个索引，另一个列

可以（并且非常简单）使用一个索引和另一个列进行合并。例如，

left.merge(right, left_on='key1', right_index=True)

反之亦然（right_on=...和left_index=True）。

right2 = right.reset_index().rename({'idxkey' : 'colkey'}, axis=1)
right2

  colkey     value
0      B  0.543843
1      D  0.013135
2      E -0.326498
3      F  1.385076

left.merge(right2, left_index=True, right_on='colkey')

    value_x colkey   value_y
0 -0.402655      B  0.543843
1 -0.524349      D  0.013135

在这种特殊情况下，left为命名了索引，因此您也可以将索引名称与一起使用left_on，如下所示：

left.merge(right2, left_on='idxkey', right_on='colkey')

    value_x colkey   value_y
0 -0.402655      B  0.543843
1 -0.524349      D  0.013135

DataFrame.join
除了这些，还有另一个简洁的选择。您可以使用DataFrame.join默认值来联接索引。DataFrame.join默认情况下不做LEFT OUTER JOIN，所以how='inner'在这里是必要的。

left.join(right, how='inner', lsuffix='_x', rsuffix='_y')

         value_x   value_y
idxkey                    
B      -0.402655  0.543843
D      -0.524349  0.013135

请注意，我需要指定lsuffix和rsuffix参数join，否则会出错：

left.join(right)
ValueError: columns overlap but no suffix specified: Index(['value'], dtype='object')

由于列名相同。如果它们的名称不同，这将不是问题。

left.rename(columns={'value':'leftvalue'}).join(right, how='inner')

        leftvalue     value
idxkey                     
B       -0.402655  0.543843
D       -0.524349  0.013135

pd.concat
最后，作为基于索引的联接的替代方法，可以使用pd.concat：

pd.concat([left, right], axis=1, sort=False, join='inner')

           value     value
idxkey                    
B      -0.402655  0.543843
D      -0.524349  0.013135

省略join='inner'是否需要FULL OUTER JOIN（默认）：

pd.concat([left, right], axis=1, sort=False)

      value     value
A -0.602923       NaN
B -0.402655  0.543843
C  0.302329       NaN
D -0.524349  0.013135
E       NaN -0.326498
F       NaN  1.385076

有关更多信息，请参见@piRSquared 上pd.concat的此规范帖子。

通用化：`merge`处理多个DataFrame

通常，将多个DataFrame合并在一起时会出现这种情况。天真的，这可以通过链接merge调用来完成：

df1.merge(df2, ...).merge(df3, ...)

但是，对于许多DataFrame，这很快就变得一发不可收拾。此外，可能有必要归纳为未知数量的DataFrame。

在这里，我将介绍pd.concat针对唯一键DataFrame.join的多方联接，以及针对非唯一键的多方联接。首先，设置。

# Setup.
np.random.seed(0)
A = pd.DataFrame({'key': ['A', 'B', 'C', 'D'], 'valueA': np.random.randn(4)})    
B = pd.DataFrame({'key': ['B', 'D', 'E', 'F'], 'valueB': np.random.randn(4)})
C = pd.DataFrame({'key': ['D', 'E', 'J', 'C'], 'valueC': np.ones(4)})
dfs = [A, B, C] 

# Note, the "key" column values are unique, so the index is unique.
A2 = A.set_index('key')
B2 = B.set_index('key')
C2 = C.set_index('key')

dfs2 = [A2, B2, C2]

多路合并唯一键（或索引）

如果您的键（此处的键可以是列或索引）是唯一的，则可以使用pd.concat。请注意，pd.concat将DataFrames连接到索引。

# merge on `key` column, you'll need to set the index before concatenating
pd.concat([
    df.set_index('key') for df in dfs], axis=1, join='inner'
).reset_index()

  key    valueA    valueB  valueC
0   D  2.240893 -0.977278     1.0

# merge on `key` index
pd.concat(dfs2, axis=1, sort=False, join='inner')

       valueA    valueB  valueC
key                            
D    2.240893 -0.977278     1.0

省略join='inner'完整的外部联接。请注意，您不能指定LEFT或RIGHT OUTER连接（如果需要这些连接，请使用join，如下所述）。

多路合并重复项

concat速度很快，但也有缺点。它不能处理重复项。

A3 = pd.DataFrame({'key': ['A', 'B', 'C', 'D', 'D'], 'valueA': np.random.randn(5)})

pd.concat([df.set_index('key') for df in [A3, B, C]], axis=1, join='inner')
ValueError: Shape of passed values is (3, 4), indices imply (3, 2)

在这种情况下，我们可以使用join它，因为它可以处理非唯一键（请注意，join将DataFrames连接到它们的索引上；merge除非另有说明，否则它在后台进行调用并执行LEFT OUTER JOIN）。

# join on `key` column, set as the index first
# For inner join. For left join, omit the "how" argument.
A.set_index('key').join(
    [df.set_index('key') for df in (B, C)], how='inner').reset_index()

  key    valueA    valueB  valueC
0   D  2.240893 -0.977278     1.0

# join on `key` index
A3.set_index('key').join([B2, C2], how='inner')

       valueA    valueB  valueC
key                            
D    1.454274 -0.977278     1.0
D    0.761038 -0.977278     1.0

This post aims to give readers a primer on SQL-flavoured merging with pandas, how to use it, and when not to use it.

In particular, here’s what this post will go through:

The basics – types of joins (LEFT, RIGHT, OUTER, INNER)
- merging with different column names
- avoiding duplicate merge key column in output
Merging with index under different conditions
- effectively using your named index
- merge key as the index of one and column of another
Multiway merges on columns and indexes (unique and non-unique)
Notable alternatives to merge and join

What this post will not go through:

Performance-related discussions and timings (for now). Mostly notable mentions of better alternatives, wherever appropriate.
Handling suffixes, removing extra columns, renaming outputs, and other specific use cases. There are other (read: better) posts that deal with that, so figure it out!

Note
Most examples default to INNER JOIN operations while demonstrating various features, unless otherwise specified.

Furthermore, all the DataFrames here can be copied and replicated so you can play with them. Also, see this post on how to read DataFrames from your clipboard.

Lastly, all visual representation of JOIN operations have been hand-drawn using Google Drawings. Inspiration from here.

Enough Talk, just show me how to use `merge`!

Setup

np.random.seed(0)
left = pd.DataFrame({'key': ['A', 'B', 'C', 'D'], 'value': np.random.randn(4)})    
right = pd.DataFrame({'key': ['B', 'D', 'E', 'F'], 'value': np.random.randn(4)})

left

  key     value
0   A  1.764052
1   B  0.400157
2   C  0.978738
3   D  2.240893

right

  key     value
0   B  1.867558
1   D -0.977278
2   E  0.950088
3   F -0.151357

For the sake of simplicity, the key column has the same name (for now).

An INNER JOIN is represented by

Note
This, along with the forthcoming figures all follow this convention:

blue indicates rows that are present in the merge result

red indicates rows that are excluded from the result (i.e., removed)

green indicates missing values that are replaced with NaNs in the result

To perform an INNER JOIN, call merge on the left DataFrame, specifying the right DataFrame and the join key (at the very least) as arguments.

left.merge(right, on='key')
# Or, if you want to be explicit
# left.merge(right, on='key', how='inner')

  key   value_x   value_y
0   B  0.400157  1.867558
1   D  2.240893 -0.977278

This returns only rows from left and right which share a common key (in this example, “B” and “D).

A LEFT OUTER JOIN, or LEFT JOIN is represented by

This can be performed by specifying how='left'.

left.merge(right, on='key', how='left')

  key   value_x   value_y
0   A  1.764052       NaN
1   B  0.400157  1.867558
2   C  0.978738       NaN
3   D  2.240893 -0.977278

Carefully note the placement of NaNs here. If you specify how='left', then only keys from left are used, and missing data from right is replaced by NaN.

And similarly, for a RIGHT OUTER JOIN, or RIGHT JOIN which is…

…specify how='right':

left.merge(right, on='key', how='right')

  key   value_x   value_y
0   B  0.400157  1.867558
1   D  2.240893 -0.977278
2   E       NaN  0.950088
3   F       NaN -0.151357

Here, keys from right are used, and missing data from left is replaced by NaN.

Finally, for the FULL OUTER JOIN, given by

specify how='outer'.

left.merge(right, on='key', how='outer')

  key   value_x   value_y
0   A  1.764052       NaN
1   B  0.400157  1.867558
2   C  0.978738       NaN
3   D  2.240893 -0.977278
4   E       NaN  0.950088
5   F       NaN -0.151357

This uses the keys from both frames, and NaNs are inserted for missing rows in both.

The documentation summarises these various merges nicely:

Other JOINs – LEFT-Excluding, RIGHT-Excluding, and FULL-Excluding/ANTI JOINs

If you need LEFT-Excluding JOINs and RIGHT-Excluding JOINs in two steps.

For LEFT-Excluding JOIN, represented as

Start by performing a LEFT OUTER JOIN and then filtering (excluding!) rows coming from left only,

(left.merge(right, on='key', how='left', indicator=True)
     .query('_merge == "left_only"')
     .drop('_merge', 1))

  key   value_x  value_y
0   A  1.764052      NaN
2   C  0.978738      NaN

Where,

left.merge(right, on='key', how='left', indicator=True)

  key   value_x   value_y     _merge
0   A  1.764052       NaN  left_only
1   B  0.400157  1.867558       both
2   C  0.978738       NaN  left_only
3   D  2.240893 -0.977278       both

And similarly, for a RIGHT-Excluding JOIN,

(left.merge(right, on='key', how='right', indicator=True)
     .query('_merge == "right_only"')
     .drop('_merge', 1))

  key  value_x   value_y
2   E      NaN  0.950088
3   F      NaN -0.151357

Lastly, if you are required to do a merge that only retains keys from the left or right, but not both (IOW, performing an ANTI-JOIN),

You can do this in similar fashion—

(left.merge(right, on='key', how='outer', indicator=True)
     .query('_merge != "both"')
     .drop('_merge', 1))

  key   value_x   value_y
0   A  1.764052       NaN
2   C  0.978738       NaN
4   E       NaN  0.950088
5   F       NaN -0.151357

Different names for key columns

If the key columns are named differently—for example, left has keyLeft, and right has keyRight instead of key—then you will have to specify left_on and right_on as arguments instead of on:

left2 = left.rename({'key':'keyLeft'}, axis=1)
right2 = right.rename({'key':'keyRight'}, axis=1)

left2

  keyLeft     value
0       A  1.764052
1       B  0.400157
2       C  0.978738
3       D  2.240893

right2

  keyRight     value
0        B  1.867558
1        D -0.977278
2        E  0.950088
3        F -0.151357

left2.merge(right2, left_on='keyLeft', right_on='keyRight', how='inner')

  keyLeft   value_x keyRight   value_y
0       B  0.400157        B  1.867558
1       D  2.240893        D -0.977278

Avoiding duplicate key column in output

When merging on keyLeft from left and keyRight from right, if you only want either of the keyLeft or keyRight (but not both) in the output, you can start by setting the index as a preliminary step.

left3 = left2.set_index('keyLeft')
left3.merge(right2, left_index=True, right_on='keyRight')

    value_x keyRight   value_y
0  0.400157        B  1.867558
1  2.240893        D -0.977278

Contrast this with the output of the command just before (thst is, the output of left2.merge(right2, left_on='keyLeft', right_on='keyRight', how='inner')), you’ll notice keyLeft is missing. You can figure out what column to keep based on which frame’s index is set as the key. This may matter when, say, performing some OUTER JOIN operation.

Merging only a single column from one of the `DataFrames`

For example, consider

right3 = right.assign(newcol=np.arange(len(right)))
right3
  key     value  newcol
0   B  1.867558       0
1   D -0.977278       1
2   E  0.950088       2
3   F -0.151357       3

If you are required to merge only “new_val” (without any of the other columns), you can usually just subset columns before merging:

left.merge(right3[['key', 'newcol']], on='key')

  key     value  newcol
0   B  0.400157       0
1   D  2.240893       1

If you’re doing a LEFT OUTER JOIN, a more performant solution would involve map:

# left['newcol'] = left['key'].map(right3.set_index('key')['newcol']))
left.assign(newcol=left['key'].map(right3.set_index('key')['newcol']))

  key     value  newcol
0   A  1.764052     NaN
1   B  0.400157     0.0
2   C  0.978738     NaN
3   D  2.240893     1.0

As mentioned, this is similar to, but faster than

left.merge(right3[['key', 'newcol']], on='key', how='left')

  key     value  newcol
0   A  1.764052     NaN
1   B  0.400157     0.0
2   C  0.978738     NaN
3   D  2.240893     1.0

Merging on multiple columns

To join on more than one column, specify a list for on (or left_on and right_on, as appropriate).

left.merge(right, on=['key1', 'key2'] ...)

Or, in the event the names are different,

left.merge(right, left_on=['lkey1', 'lkey2'], right_on=['rkey1', 'rkey2'])

**Other useful `merge*` operations and functions**

Merging a DataFrame with Series on index: See this answer.
Besides merge, DataFrame.update and DataFrame.combine_first are also used in certain cases to update one DataFrame with another.
pd.merge_ordered is a useful function for ordered JOINs.
pd.merge_asof (read: merge_asOf) is useful for approximate joins.

This section only covers the very basics, and is designed to only whet your appetite. For more examples and cases, see the documentation on merge, join, and concat as well as the links to the function specs.

Index-based *-JOIN (+ index-column `merge`s)

Setup

np.random.seed([3, 14])
left = pd.DataFrame({'value': np.random.randn(4)}, index=['A', 'B', 'C', 'D'])    
right = pd.DataFrame({'value': np.random.randn(4)}, index=['B', 'D', 'E', 'F'])
left.index.name = right.index.name = 'idxkey'

left
           value
idxkey          
A      -0.602923
B      -0.402655
C       0.302329
D      -0.524349

right

           value
idxkey          
B       0.543843
D       0.013135
E      -0.326498
F       1.385076

Typically, a merge on index would look like this:

left.merge(right, left_index=True, right_index=True)


         value_x   value_y
idxkey                    
B      -0.402655  0.543843
D      -0.524349  0.013135

Support for index names

If your index is named, then v0.23 users can also specify the level name to on (or left_on and right_on as necessary).

left.merge(right, on='idxkey')

         value_x   value_y
idxkey                    
B      -0.402655  0.543843
D      -0.524349  0.013135

Merging on index of one, column(s) of another

It is possible (and quite simple) to use the index of one, and the column of another, to perform a merge. For example,

left.merge(right, left_on='key1', right_index=True)

Or vice versa (right_on=... and left_index=True).

right2 = right.reset_index().rename({'idxkey' : 'colkey'}, axis=1)
right2

  colkey     value
0      B  0.543843
1      D  0.013135
2      E -0.326498
3      F  1.385076

left.merge(right2, left_index=True, right_on='colkey')

    value_x colkey   value_y
0 -0.402655      B  0.543843
1 -0.524349      D  0.013135

In this special case, the index for left is named, so you can also use the index name with left_on, like this:

left.merge(right2, left_on='idxkey', right_on='colkey')

    value_x colkey   value_y
0 -0.402655      B  0.543843
1 -0.524349      D  0.013135

DataFrame.join
Besides these, there is another succinct option. You can use DataFrame.join which defaults to joins on the index. DataFrame.join does a LEFT OUTER JOIN by default, so how='inner' is necessary here.

left.join(right, how='inner', lsuffix='_x', rsuffix='_y')

         value_x   value_y
idxkey                    
B      -0.402655  0.543843
D      -0.524349  0.013135

Note that I needed to specify the lsuffix and rsuffix arguments since join would otherwise error out:

left.join(right)
ValueError: columns overlap but no suffix specified: Index(['value'], dtype='object')

Since the column names are the same. This would not be a problem if they were differently named.

left.rename(columns={'value':'leftvalue'}).join(right, how='inner')

        leftvalue     value
idxkey                     
B       -0.402655  0.543843
D       -0.524349  0.013135

pd.concat
Lastly, as an alternative for index-based joins, you can use pd.concat:

pd.concat([left, right], axis=1, sort=False, join='inner')

           value     value
idxkey                    
B      -0.402655  0.543843
D      -0.524349  0.013135

Omit join='inner' if you need a FULL OUTER JOIN (the default):

pd.concat([left, right], axis=1, sort=False)

      value     value
A -0.602923       NaN
B -0.402655  0.543843
C  0.302329       NaN
D -0.524349  0.013135
E       NaN -0.326498
F       NaN  1.385076

For more information, see this canonical post on pd.concat by @piRSquared.

Generalizing: `merge`ing multiple DataFrames

Oftentimes, the situation arises when multiple DataFrames are to be merged together. Naively, this can be done by chaining merge calls:

df1.merge(df2, ...).merge(df3, ...)

However, this quickly gets out of hand for many DataFrames. Furthermore, it may be necessary to generalise for an unknown number of DataFrames.

Here I introduce pd.concat for multi-way joins on unique keys, and DataFrame.join for multi-way joins on non-unique keys. First, the setup.

# Setup.
np.random.seed(0)
A = pd.DataFrame({'key': ['A', 'B', 'C', 'D'], 'valueA': np.random.randn(4)})    
B = pd.DataFrame({'key': ['B', 'D', 'E', 'F'], 'valueB': np.random.randn(4)})
C = pd.DataFrame({'key': ['D', 'E', 'J', 'C'], 'valueC': np.ones(4)})
dfs = [A, B, C] 

# Note, the "key" column values are unique, so the index is unique.
A2 = A.set_index('key')
B2 = B.set_index('key')
C2 = C.set_index('key')

dfs2 = [A2, B2, C2]

Multiway merge on unique keys (or index)

If your keys (here, the key could either be a column or an index) are unique, then you can use pd.concat. Note that pd.concat joins DataFrames on the index.

# merge on `key` column, you'll need to set the index before concatenating
pd.concat([
    df.set_index('key') for df in dfs], axis=1, join='inner'
).reset_index()

  key    valueA    valueB  valueC
0   D  2.240893 -0.977278     1.0

# merge on `key` index
pd.concat(dfs2, axis=1, sort=False, join='inner')

       valueA    valueB  valueC
key                            
D    2.240893 -0.977278     1.0

Omit join='inner' for a FULL OUTER JOIN. Note that you cannot specify LEFT or RIGHT OUTER joins (if you need these, use join, described below).

Multiway merge on keys with duplicates

concat is fast, but has its shortcomings. It cannot handle duplicates.

A3 = pd.DataFrame({'key': ['A', 'B', 'C', 'D', 'D'], 'valueA': np.random.randn(5)})

pd.concat([df.set_index('key') for df in [A3, B, C]], axis=1, join='inner')
ValueError: Shape of passed values is (3, 4), indices imply (3, 2)

In this situation, we can use join since it can handle non-unique keys (note that join joins DataFrames on their index; it calls merge under the hood and does a LEFT OUTER JOIN unless otherwise specified).

# join on `key` column, set as the index first
# For inner join. For left join, omit the "how" argument.
A.set_index('key').join(
    [df.set_index('key') for df in (B, C)], how='inner').reset_index()

  key    valueA    valueB  valueC
0   D  2.240893 -0.977278     1.0

# join on `key` index
A3.set_index('key').join([B2, C2], how='inner')

       valueA    valueB  valueC
key                            
D    1.454274 -0.977278     1.0
D    0.761038 -0.977278     1.0

回答 1

一个补充的视觉观pd.concat([df0, df1], kwargs)。请注意，kwarg axis=0或axis=1的含义不如df.mean()或直观df.apply(func)

A supplemental visual view of pd.concat([df0, df1], kwargs). Notice that, kwarg axis=0 or axis=1 ‘s meaning is not as intuitive as df.mean() or df.apply(func)

知识问答

解释Python的’enter’和’exit’

2021年7月27日 Python实用宝典

问题：解释Python的’enter’和’exit’

我在某人的代码中看到了这一点。这是什么意思？

    def __enter__(self):
        return self

    def __exit__(self, type, value, tb):
        self.stream.close()

from __future__ import with_statement#for python2.5 

class a(object):
    def __enter__(self):
        print 'sss'
        return 'sss111'
    def __exit__(self ,type, value, traceback):
        print 'ok'
        return False

with a() as s:
    print s


print s

I saw this in someone’s code. What does it mean?

    def __enter__(self):
        return self

    def __exit__(self, type, value, tb):
        self.stream.close()

from __future__ import with_statement#for python2.5 

class a(object):
    def __enter__(self):
        print 'sss'
        return 'sss111'
    def __exit__(self ,type, value, traceback):
        print 'ok'
        return False

with a() as s:
    print s


print s

回答 0

使用这些魔术方法（__enter__，__exit__）使您可以实现可轻松用于该with语句的对象。

这个想法是，它使得构建需要执行一些“清除”代码的代码（将其视为一个try-finally块）变得容易。这里有更多的解释。

一个有用的例子是数据库连接对象（一旦对应的“ with”语句超出范围，它就会自动关闭连接）：

class DatabaseConnection(object):

    def __enter__(self):
        # make a database connection and return it
        ...
        return self.dbconn

    def __exit__(self, exc_type, exc_val, exc_tb):
        # make sure the dbconnection gets closed
        self.dbconn.close()
        ...

如上所述，将此对象与with语句一起使用（from __future__ import with_statement如果您使用的是Python 2.5，则可能需要在文件顶部执行此操作）。

with DatabaseConnection() as mydbconn:
    # do stuff

PEP343-‘with’语句’也有不错的写法。

Using these magic methods (__enter__, __exit__) allows you to implement objects which can be used easily with the with statement.

The idea is that it makes it easy to build code which needs some ‘cleandown’ code executed (think of it as a try-finally block). Some more explanation here.

A useful example could be a database connection object (which then automagically closes the connection once the corresponding ‘with’-statement goes out of scope):

class DatabaseConnection(object):

    def __enter__(self):
        # make a database connection and return it
        ...
        return self.dbconn

    def __exit__(self, exc_type, exc_val, exc_tb):
        # make sure the dbconnection gets closed
        self.dbconn.close()
        ...

As explained above, use this object with the with statement (you may need to do from __future__ import with_statement at the top of the file if you’re on Python 2.5).

with DatabaseConnection() as mydbconn:
    # do stuff

PEP343 — The ‘with’ statement’ has a nice writeup as well.

回答 1

如果您知道上下文管理器是什么，那么您就不需要了解__enter__和使用__exit__魔术方法。让我们看一个非常简单的例子。

在此示例中，我将借助open函数打开myfile.txt。在尝试/终于块确保即使发生意外的异常myfile.txt的将被关闭。

fp=open(r"C:\Users\SharpEl\Desktop\myfile.txt")
try:
    for line in fp:
        print(line)
finally:
    fp.close()

现在我用with语句打开相同的文件：

with open(r"C:\Users\SharpEl\Desktop\myfile.txt") as fp:
    for line in fp:
        print(line)

如果您查看代码，则我没有关闭文件，也没有try / finally块。因为with语句会自动关闭 myfile.txt。您甚至可以通过调用print(fp.closed)attribute 来检查它-返回True。

这是因为open函数返回的文件对象（在我的示例中为fp）具有两个内置方法__enter__和__exit__。它也称为上下文管理器。__enter__方法在with块的开头__exit__ 被调用，方法在结尾被调用。注意：with语句仅适用于支持上下文管理协议的对象，即它们具有__enter__和__exit__方法。实现这两种方法的类称为上下文管理器类。

现在让我们定义自己的上下文管理器类。

 class Log:
    def __init__(self,filename):
        self.filename=filename
        self.fp=None    
    def logging(self,text):
        self.fp.write(text+'\n')
    def __enter__(self):
        print("__enter__")
        self.fp=open(self.filename,"a+")
        return self    
    def __exit__(self, exc_type, exc_val, exc_tb):
        print("__exit__")
        self.fp.close()

with Log(r"C:\Users\SharpEl\Desktop\myfile.txt") as logfile:
    print("Main")
    logfile.logging("Test1")
    logfile.logging("Test2")

我希望您现在对魔术方法__enter__和两者都有基本的了解__exit__。

If you know what context managers are then you need nothing more to understand __enter__ and __exit__ magic methods. Lets see a very simple example.

In this example I am opening myfile.txt with help of open function. The try/finally block ensures that even if an unexpected exception occurs myfile.txt will be closed.

fp=open(r"C:\Users\SharpEl\Desktop\myfile.txt")
try:
    for line in fp:
        print(line)
finally:
    fp.close()

Now I am opening same file with with statement:

with open(r"C:\Users\SharpEl\Desktop\myfile.txt") as fp:
    for line in fp:
        print(line)

If you look at the code, I didn’t close the file & there is no try/finally block. Because with statement automatically closes myfile.txt . You can even check it by calling print(fp.closed) attribute — which returns True.

This is because the file objects (fp in my example) returned by open function has two built-in methods __enter__ and __exit__. It is also known as context manager. __enter__ method is called at the start of with block and __exit__ method is called at the end. Note: with statement only works with objects that support the context mamangement protocol i.e. they have __enter__ and __exit__ methods. A class which implement both methods is known as context manager class.

Now lets define our own context manager class.

 class Log:
    def __init__(self,filename):
        self.filename=filename
        self.fp=None    
    def logging(self,text):
        self.fp.write(text+'\n')
    def __enter__(self):
        print("__enter__")
        self.fp=open(self.filename,"a+")
        return self    
    def __exit__(self, exc_type, exc_val, exc_tb):
        print("__exit__")
        self.fp.close()

with Log(r"C:\Users\SharpEl\Desktop\myfile.txt") as logfile:
    print("Main")
    logfile.logging("Test1")
    logfile.logging("Test2")

I hope now you have basic understanding of both __enter__ and __exit__ magic methods.

回答 2

我发现通过Googling 定位python文档__enter__和__exit__方法非常困难，因此，这里的链接可以帮助其他人：

https://docs.python.org/2/reference/datamodel.html#with-statement-context-managers
https://docs.python.org/3/reference/datamodel.html#with-statement-context-managers
（两个版本的细节相同）

object.__enter__(self)
输入与此对象相关的运行时上下文。该with语句会将此方法的返回值绑定到该语句的as子句中指定的目标（如果有）。

object.__exit__(self, exc_type, exc_value, traceback)
退出与此对象相关的运行时上下文。参数描述导致上下文退出的异常。如果上下文无exceptions地退出，则所有三个参数均为None。

如果提供了异常，并且该方法希望抑制该异常（即，防止其传播），则它应返回一个真值。否则，将在退出此方法后正常处理异常。

注意，__exit__()方法不应该引发传入的异常。这是调用方法的责任。

我希望对__exit__方法参数进行清楚的描述。这是缺少的，但是我们可以推断出它们。

大概exc_type是异常的类。

它说您不应该重新引发传入的异常。这向我们表明，其中一个参数可能是实际的Exception实例……或者您应该自己从类型和值中实例化它？

我们可以通过看这篇文章来回答：http :
//effbot.org/zone/python-with-statement.htm

例如，以下__exit__方法可吞噬所有TypeError，但允许所有其他异常通过：

def __exit__(self, type, value, traceback):
    return isinstance(value, TypeError)

…显然value是Exception实例。

大概traceback是一个Python 回溯对象。

I found it strangely difficult to locate the python docs for __enter__ and __exit__ methods by Googling, so to help others here is the link:

https://docs.python.org/2/reference/datamodel.html#with-statement-context-managers
https://docs.python.org/3/reference/datamodel.html#with-statement-context-managers
(detail is the same for both versions)

object.__enter__(self)
Enter the runtime context related to this object. The with statement will bind this method’s return value to the target(s) specified in the as clause of the statement, if any.

object.__exit__(self, exc_type, exc_value, traceback)
Exit the runtime context related to this object. The parameters describe the exception that caused the context to be exited. If the context was exited without an exception, all three arguments will be None.

If an exception is supplied, and the method wishes to suppress the exception (i.e., prevent it from being propagated), it should return a true value. Otherwise, the exception will be processed normally upon exit from this method.

Note that __exit__() methods should not reraise the passed-in exception; this is the caller’s responsibility.

I was hoping for a clear description of the __exit__ method arguments. This is lacking but we can deduce them…

Presumably exc_type is the class of the exception.

It says you should not re-raise the passed-in exception. This suggests to us that one of the arguments might be an actual Exception instance …or maybe you’re supposed to instantiate it yourself from the type and value?

We can answer by looking at this article:
http://effbot.org/zone/python-with-statement.htm

For example, the following __exit__ method swallows any TypeError, but lets all other exceptions through:

def __exit__(self, type, value, traceback):
    return isinstance(value, TypeError)

…so clearly value is an Exception instance.

And presumably traceback is a Python traceback object.

回答 3

除了上述答案以例证调用顺序外，一个简单的运行示例

class myclass:
    def __init__(self):
        print("__init__")

    def __enter__(self): 
        print("__enter__")

    def __exit__(self, type, value, traceback):
        print("__exit__")

    def __del__(self):
        print("__del__")

with myclass(): 
    print("body")

产生输出：

__init__
__enter__
body
__exit__
__del__

提醒：使用语法时，在上述情况下with myclass() as mc，变量mc获取由返回的值！对于此类用途，需要定义返回值，例如：__enter__()None

def __enter__(self): 
    print('__enter__')
    return self

In addition to the above answers to exemplify invocation order, a simple run example

class myclass:
    def __init__(self):
        print("__init__")

    def __enter__(self): 
        print("__enter__")

    def __exit__(self, type, value, traceback):
        print("__exit__")

    def __del__(self):
        print("__del__")

with myclass(): 
    print("body")

Produces the output:

__init__
__enter__
body
__exit__
__del__

A reminder: when using the syntax with myclass() as mc, variable mc gets the value returned by __enter__(), in the above case None! For such use, need to define return value, such as:

def __enter__(self): 
    print('__enter__')
    return self

回答 4

尝试添加我的答案（我的学习思想）：

__enter__[__exit__]两者都是在“ with语句 ” 主体（PEP 343）的入口处和出口处被调用的方法，并且两者的实现都称为上下文管理器。

with语句旨在隐藏try finally子句的流控制，并使代码难以理解。

with语句的语法为：

with EXPR as VAR:
    BLOCK

转换为（如PEP 343中所述）：

mgr = (EXPR)
exit = type(mgr).__exit__  # Not calling it yet
value = type(mgr).__enter__(mgr)
exc = True
try:
    try:
        VAR = value  # Only if "as VAR" is present
        BLOCK
    except:
        # The exceptional case is handled here
        exc = False
        if not exit(mgr, *sys.exc_info()):
            raise
        # The exception is swallowed if exit() returns true
finally:
    # The normal and non-local-goto cases are handled here
    if exc:
        exit(mgr, None, None, None)

尝试一些代码：

>>> import logging
>>> import socket
>>> import sys

#server socket on another terminal / python interpreter
>>> s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> s.listen(5)
>>> s.bind((socket.gethostname(), 999))
>>> while True:
>>>    (clientsocket, addr) = s.accept()
>>>    print('get connection from %r' % addr[0])
>>>    msg = clientsocket.recv(1024)
>>>    print('received %r' % msg)
>>>    clientsocket.send(b'connected')
>>>    continue

#the client side
>>> class MyConnectionManager:
>>>     def __init__(self, sock, addrs):
>>>         logging.basicConfig(level=logging.DEBUG, format='%(asctime)s \
>>>         : %(levelname)s --> %(message)s')
>>>         logging.info('Initiating My connection')
>>>         self.sock = sock
>>>         self.addrs = addrs
>>>     def __enter__(self):
>>>         try:
>>>             self.sock.connect(addrs)
>>>             logging.info('connection success')
>>>             return self.sock
>>>         except:
>>>             logging.warning('Connection refused')
>>>             raise
>>>     def __exit__(self, type, value, tb):
>>>             logging.info('CM suppress exception')
>>>             return False
>>> addrs = (socket.gethostname())
>>> s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> with MyConnectionManager(s, addrs) as CM:
>>>     try:
>>>         CM.send(b'establishing connection')
>>>         msg = CM.recv(1024)
>>>         print(msg)
>>>     except:
>>>         raise
#will result (client side) :
2018-12-18 14:44:05,863         : INFO --> Initiating My connection
2018-12-18 14:44:05,863         : INFO --> connection success
b'connected'
2018-12-18 14:44:05,864         : INFO --> CM suppress exception

#result of server side
get connection from '127.0.0.1'
received b'establishing connection'

然后手动尝试（遵循翻译语法）：

>>> s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) #make new socket object
>>> mgr = MyConnection(s, addrs)
2018-12-18 14:53:19,331         : INFO --> Initiating My connection
>>> ext = mgr.__exit__
>>> value = mgr.__enter__()
2018-12-18 14:55:55,491         : INFO --> connection success
>>> exc = True
>>> try:
>>>     try:
>>>         VAR = value
>>>         VAR.send(b'establishing connection')
>>>         msg = VAR.recv(1024)
>>>         print(msg)
>>>     except:
>>>         exc = False
>>>         if not ext(*sys.exc_info()):
>>>             raise
>>> finally:
>>>     if exc:
>>>         ext(None, None, None)
#the result:
b'connected'
2018-12-18 15:01:54,208         : INFO --> CM suppress exception

服务器端的结果与以前相同

对不起，我的英语不好，解释不清楚，谢谢。

try adding my answers (my thought of learning) :

__enter__ and [__exit__] both are methods that are invoked on entry to and exit from the body of “the with statement” (PEP 343) and implementation of both is called context manager.

the with statement is intend to hiding flow control of try finally clause and make the code inscrutable.

the syntax of the with statement is :

with EXPR as VAR:
    BLOCK

which translate to (as mention in PEP 343) :

mgr = (EXPR)
exit = type(mgr).__exit__  # Not calling it yet
value = type(mgr).__enter__(mgr)
exc = True
try:
    try:
        VAR = value  # Only if "as VAR" is present
        BLOCK
    except:
        # The exceptional case is handled here
        exc = False
        if not exit(mgr, *sys.exc_info()):
            raise
        # The exception is swallowed if exit() returns true
finally:
    # The normal and non-local-goto cases are handled here
    if exc:
        exit(mgr, None, None, None)

try some code:

>>> import logging
>>> import socket
>>> import sys

#server socket on another terminal / python interpreter
>>> s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> s.listen(5)
>>> s.bind((socket.gethostname(), 999))
>>> while True:
>>>    (clientsocket, addr) = s.accept()
>>>    print('get connection from %r' % addr[0])
>>>    msg = clientsocket.recv(1024)
>>>    print('received %r' % msg)
>>>    clientsocket.send(b'connected')
>>>    continue

#the client side
>>> class MyConnectionManager:
>>>     def __init__(self, sock, addrs):
>>>         logging.basicConfig(level=logging.DEBUG, format='%(asctime)s \
>>>         : %(levelname)s --> %(message)s')
>>>         logging.info('Initiating My connection')
>>>         self.sock = sock
>>>         self.addrs = addrs
>>>     def __enter__(self):
>>>         try:
>>>             self.sock.connect(addrs)
>>>             logging.info('connection success')
>>>             return self.sock
>>>         except:
>>>             logging.warning('Connection refused')
>>>             raise
>>>     def __exit__(self, type, value, tb):
>>>             logging.info('CM suppress exception')
>>>             return False
>>> addrs = (socket.gethostname())
>>> s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
>>> with MyConnectionManager(s, addrs) as CM:
>>>     try:
>>>         CM.send(b'establishing connection')
>>>         msg = CM.recv(1024)
>>>         print(msg)
>>>     except:
>>>         raise
#will result (client side) :
2018-12-18 14:44:05,863         : INFO --> Initiating My connection
2018-12-18 14:44:05,863         : INFO --> connection success
b'connected'
2018-12-18 14:44:05,864         : INFO --> CM suppress exception

#result of server side
get connection from '127.0.0.1'
received b'establishing connection'

and now try manually (following translate syntax):

>>> s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) #make new socket object
>>> mgr = MyConnection(s, addrs)
2018-12-18 14:53:19,331         : INFO --> Initiating My connection
>>> ext = mgr.__exit__
>>> value = mgr.__enter__()
2018-12-18 14:55:55,491         : INFO --> connection success
>>> exc = True
>>> try:
>>>     try:
>>>         VAR = value
>>>         VAR.send(b'establishing connection')
>>>         msg = VAR.recv(1024)
>>>         print(msg)
>>>     except:
>>>         exc = False
>>>         if not ext(*sys.exc_info()):
>>>             raise
>>> finally:
>>>     if exc:
>>>         ext(None, None, None)
#the result:
b'connected'
2018-12-18 15:01:54,208         : INFO --> CM suppress exception

the result of the server side same as before

sorry for my bad english and my unclear explanations, thank you….

回答 5

这称为上下文管理器，我只想补充一下，其他编程语言也存在类似的方法。比较它们可能有助于理解python中的上下文管理器。基本上，当我们处理一些需要初始化的资源（文件，网络，数据库），并且在某些时候需要拆除（配置）时，使用上下文管理器。在Java 7及更高版本中，我们具有以下形式的自动资源管理：

//Java code
try (Session session = new Session())
{
  // do stuff
}

请注意，会话需要实现AutoClosable其（许多）子接口之一。

在C＃中，我们使用using语句来管理资源，其形式为：

//C# code
using(Session session = new Session())
{
  ... do stuff.
}

Session应该在其中实施IDisposable。

在python中，我们使用的类应实现__enter__和__exit__。因此它采取以下形式：

#Python code
with Session() as session:
    #do stuff

正如其他人指出的那样，您始终可以在所有语言中使用try / finally语句来实现相同的机制。这只是语法糖。

This is called context manager and I just want to add that similar approaches exist for other programming languages. Comparing them could be helpful in understanding the context manager in python. Basically, a context manager is used when we are dealing with some resources (file, network, database) that need to be initialized and at some point, tear downed (disposed). In Java 7 and above we have automatic resource management that takes the form of:

//Java code
try (Session session = new Session())
{
  // do stuff
}

Note that Session needs to implement AutoClosable or one of its (many) sub-interfaces.

In C#, we have using statements for managing resources that takes the form of:

//C# code
using(Session session = new Session())
{
  ... do stuff.
}

In which Session should implement IDisposable.

In python, the class that we use should implement __enter__ and __exit__. So it takes the form of:

#Python code
with Session() as session:
    #do stuff

And as others pointed out, you can always use try/finally statement in all the languages to implement the same mechanism. This is just syntactic sugar.

知识问答

有没有办法在python的lambda中执行“如果”

2021年7月27日 Python实用宝典

问题：有没有办法在python的lambda中执行“如果”

在python 2.6中，我想这样做：

f = lambda x: if x==2 print x else raise Exception()
f(2) #should print "2"
f(3) #should throw an exception

这显然不是语法。是否可以执行ifin lambda，如果可以，该怎么做？

谢谢

In python 2.6, I want to do:

f = lambda x: if x==2 print x else raise Exception()
f(2) #should print "2"
f(3) #should throw an exception

This clearly isn’t the syntax. Is it possible to perform an if in lambda and if so how to do it?

thanks

回答 0

您要寻找的语法：

lambda x: True if x % 2 == 0 else False

但是您不能使用print或raise使用lambda。

The syntax you’re looking for:

lambda x: True if x % 2 == 0 else False

But you can’t use print or raise in a lambda.

回答 1

为什么不只是定义一个函数？

def f(x):
    if x == 2:
        print(x)
    else:
        raise ValueError

在这种情况下，确实没有理由使用lambda。

why don’t you just define a function?

def f(x):
    if x == 2:
        print(x)
    else:
        raise ValueError

there really is no justification to use lambda in this case.

回答 2

到目前为止，我可能写的最糟糕的python行：

f = lambda x: sys.stdout.write(["2\n",][2*(x==2)-2])

如果您打印x == 2，

如果x！= 2，则加注。

Probably the worst python line I’ve written so far:

f = lambda x: sys.stdout.write(["2\n",][2*(x==2)-2])

If x == 2 you print,

if x != 2 you raise.

回答 3

如果您确实要这样做，则可以在lambda中轻松引发异常。

def Raise(exception):
    raise exception
x = lambda y: 1 if y < 2 else Raise(ValueError("invalid value"))

这是一个好主意吗？我的本能通常是将错误报告放在lambda之外。使其值为None并在调用方中引发错误。不过，我不认为这从本质上讲是邪恶的-我认为“ y if x else z”语法本身更糟-只需确保您没有试图将过多内容放入lambda主体即可。

You can easily raise an exception in a lambda, if that’s what you really want to do.

def Raise(exception):
    raise exception
x = lambda y: 1 if y < 2 else Raise(ValueError("invalid value"))

Is this a good idea? My instinct in general is to leave the error reporting out of lambdas; let it have a value of None and raise the error in the caller. I don’t think this is inherently evil, though–I consider the “y if x else z” syntax itself worse–just make sure you’re not trying to stuff too much into a lambda body.

回答 4

就允许使用的内容而言，Python中的Lambda相当严格。特别是，你不能有任何关键字（除了运营商喜欢and，not，or，等）在他们的身上。

因此，您无法在示例中使用lambda（因为您无法使用raise），但是如果您愿意就此作答，则可以使用：

f = lambda x: x == 2 and x or None

Lambdas in Python are fairly restrictive with regard to what you’re allowed to use. Specifically, you can’t have any keywords (except for operators like and, not, or, etc) in their body.

So, there’s no way you could use a lambda for your example (because you can’t use raise), but if you’re willing to concede on that… You could use:

f = lambda x: x == 2 and x or None

回答 5

请注意，您可以在lambda定义中使用其他else … if语句：

f = lambda x: 1 if x>0 else 0 if x ==0 else -1

note you can use several else…if statements in your lambda definition:

f = lambda x: 1 if x>0 else 0 if x ==0 else -1

回答 6

如果仍要打印，则可以导入以后的模块

from __future__ import print_function

f = lambda x: print(x) if x%2 == 0 else False

If you still want to print you can import future module

from __future__ import print_function

f = lambda x: print(x) if x%2 == 0 else False

回答 7

您还可以使用逻辑运算符来进行类似条件运算的操作

func = lambda element: (expression and DoSomething) or DoSomethingIfExpressionIsFalse

您可以在此处查看有关逻辑运算符的更多信息

You can also use Logical Operators to have something like a Conditional

func = lambda element: (expression and DoSomething) or DoSomethingIfExpressionIsFalse

You can see more about Logical Operators here

回答 8

您真正需要的是

def fun():
    raise Exception()
f = lambda x:print x if x==2 else fun()

现在以您需要的方式调用该函数

f(2)
f(3)

what you need exactly is

def fun():
    raise Exception()
f = lambda x:print x if x==2 else fun()

now call the function the way you need

f(2)
f(3)

回答 9

此代码段应帮助您：

x = lambda age: 'Older' if age > 30 else 'Younger'

print(x(40))

This snippet should help you:

x = lambda age: 'Older' if age > 30 else 'Younger'

print(x(40))

回答 10

以下示例代码对我有用。不知道它是否与这个问题直接相关，但希望在其他情况下有帮助。

a = ''.join(map(lambda x: str(x*2) if x%2==0 else "", range(10)))

Following sample code works for me. Not sure if it directly relates to this question, but hope it helps in some other cases.

a = ''.join(map(lambda x: str(x*2) if x%2==0 else "", range(10)))

回答 11

尝试一下：

is_even = lambda x: True if x % 2 == 0 else False
print(is_even(10))
print(is_even(11))

出：

True
False

Try it:

is_even = lambda x: True if x % 2 == 0 else False
print(is_even(10))
print(is_even(11))

Out:

True
False

回答 12

在lambda中执行if的一种简单方法是使用列表理解。

您不能在lambda中引发异常，但这是Python 3.x中一种类似于您的示例的方式：

f = lambda x: print(x) if x==2 else print("exception")

另一个例子：

如果M则返回1，否则返回0

f = lambda x: 1 if x=="M" else 0

An easy way to perform an if in lambda is by using list comprehension.

You can’t raise an exception in lambda, but this is a way in Python 3.x to do something close to your example:

f = lambda x: print(x) if x==2 else print("exception")

Another example:

return 1 if M otherwise 0

f = lambda x: 1 if x=="M" else 0

知识问答

在Python中连接字符串的首选方法是什么？

2021年7月27日 Python实用宝典

问题：在Python中连接字符串的首选方法是什么？

由于string无法更改Python ，因此我想知道如何更有效地连接字符串？

我可以这样写：

s += stringfromelsewhere

或像这样：

s = []
s.append(somestring)

later

s = ''.join(s)

在写这个问题时，我找到了一篇很好的文章，谈论这个话题。

http://www.skymind.com/~ocrow/python_string/

但是它在Python 2.x中，所以问题是在Python 3中会有所改变吗？

Since Python’s string can’t be changed, I was wondering how to concatenate a string more efficiently?

I can write like it:

s += stringfromelsewhere

or like this:

s = []
s.append(somestring)

later

s = ''.join(s)

While writing this question, I found a good article talking about the topic.

http://www.skymind.com/~ocrow/python_string/

But it’s in Python 2.x., so the question would be did something change in Python 3?

回答 0

将字符串附加到字符串变量的最佳方法是使用+或+=。这是因为它可读且快速。它们的速度也一样快，您选择的是一个品味问题，后者是最常见的。以下是该timeit模块的时间安排：

a = a + b:
0.11338996887207031
a += b:
0.11040496826171875

但是，那些建议拥有列表并附加到列表然后再连接这些列表的人之所以这么做，是因为将字符串附加到列表与扩展字符串相比可能非常快。在某些情况下，这可能是正确的。例如，这里是一字符字符串的一百万个追加，首先是字符串，然后是列表：

a += b:
0.10780501365661621
a.append(b):
0.1123361587524414

好的，事实证明，即使结果字符串的长度为一百万个字符，追加操作仍然更快。

现在让我们尝试将十千个字符长的字符串追加十万次：

a += b:
0.41823482513427734
a.append(b):
0.010656118392944336

因此，最终字符串的长度约为100MB。那太慢了，追加到列表上要快得多。那个时机不包括决赛a.join()。那要花多长时间？

a.join(a):
0.43739795684814453

哎呀即使在这种情况下，append / join也较慢。

那么，该建议来自何处？Python 2？

a += b:
0.165287017822
a.append(b):
0.0132720470428
a.join(a):
0.114929914474

好吧，如果您使用的是非常长的字符串（通常不是，那么内存中100MB的字符串会是什么），append / join的速度会稍微快一些。

但是真正的关键是Python 2.3。我什至不告诉您时间安排，因为它是如此之慢以至于还没有完成。这些测试突然耗时数分钟。除了append / join之外，它和以后的Python一样快。

对。在石器时代，字符串连接在Python中非常缓慢。但是在2.4上已经不存在了（或者至少是Python 2.4.7），因此在2008年Python 2.3停止更新时，使用append / join的建议已过时，您应该停止使用它。:-)

（更新：当我更仔细地进行测试时发现，使用+和+=在Python 2.3上使用两个字符串的速度也更快。关于使用的建议''.join()一定是一种误解）

但是，这是CPython。其他实现可能还有其他问题。这是过早优化是万恶之源的又一个原因。除非先进行测量，否则不要使用被认为“更快”的技术。

因此，进行字符串连接的“最佳”版本是使用+或+ =。如果事实证明这对您来说很慢，那是不太可能的，那么请执行其他操作。

那么，为什么在我的代码中使用大量的添加/联接？因为有时它实际上更清晰。尤其是当您应将其串联在一起时，应以空格，逗号或换行符分隔。

The best way of appending a string to a string variable is to use + or +=. This is because it’s readable and fast. They are also just as fast, which one you choose is a matter of taste, the latter one is the most common. Here are timings with the timeit module:

a = a + b:
0.11338996887207031
a += b:
0.11040496826171875

However, those who recommend having lists and appending to them and then joining those lists, do so because appending a string to a list is presumably very fast compared to extending a string. And this can be true, in some cases. Here, for example, is one million appends of a one-character string, first to a string, then to a list:

a += b:
0.10780501365661621
a.append(b):
0.1123361587524414

OK, turns out that even when the resulting string is a million characters long, appending was still faster.

Now let’s try with appending a thousand character long string a hundred thousand times:

a += b:
0.41823482513427734
a.append(b):
0.010656118392944336

The end string, therefore, ends up being about 100MB long. That was pretty slow, appending to a list was much faster. That that timing doesn’t include the final a.join(). So how long would that take?

a.join(a):
0.43739795684814453

Oups. Turns out even in this case, append/join is slower.

So where does this recommendation come from? Python 2?

a += b:
0.165287017822
a.append(b):
0.0132720470428
a.join(a):
0.114929914474

Well, append/join is marginally faster there if you are using extremely long strings (which you usually aren’t, what would you have a string that’s 100MB in memory?)

But the real clincher is Python 2.3. Where I won’t even show you the timings, because it’s so slow that it hasn’t finished yet. These tests suddenly take minutes. Except for the append/join, which is just as fast as under later Pythons.

Yup. String concatenation was very slow in Python back in the stone age. But on 2.4 it isn’t anymore (or at least Python 2.4.7), so the recommendation to use append/join became outdated in 2008, when Python 2.3 stopped being updated, and you should have stopped using it. :-)

(Update: Turns out when I did the testing more carefully that using + and += is faster for two strings on Python 2.3 as well. The recommendation to use ''.join() must be a misunderstanding)

However, this is CPython. Other implementations may have other concerns. And this is just yet another reason why premature optimization is the root of all evil. Don’t use a technique that’s supposed “faster” unless you first measure it.

Therefore the “best” version to do string concatenation is to use + or +=. And if that turns out to be slow for you, which is pretty unlikely, then do something else.

So why do I use a lot of append/join in my code? Because sometimes it’s actually clearer. Especially when whatever you should concatenate together should be separated by spaces or commas or newlines.

回答 1

如果要串联很多值，那么两者都不是。追加列表很昂贵。您可以为此使用StringIO。特别是如果您要通过大量操作来构建它。

from cStringIO import StringIO
# python3:  from io import StringIO

buf = StringIO()

buf.write('foo')
buf.write('foo')
buf.write('foo')

buf.getvalue()
# 'foofoofoo'

如果您已经有其他操作返回的完整列表，则只需使用 ''.join(aList)

从python常见问题解答：将许多字符串连接在一起的最有效方法是什么？

str和bytes对象是不可变的，因此将多个字符串连接在一起效率不高，因为每个串联都会创建一个新对象。在一般情况下，总运行时成本在总字符串长度中是二次方的。

要累积许多str对象，建议的惯用法是将它们放入列表中，并在最后调用str.join（）：
chunks = []
for s in my_strings:
    chunks.append(s)
result = ''.join(chunks)
（另一个合理有效的习惯用法是使用io.StringIO）

要累积许多字节对象，建议的惯用法是使用就地串联（+ =运算符）扩展一个bytearray对象：
result = bytearray()
for b in my_bytes_objects:
    result += b

编辑：我很愚蠢，并且将结果向后粘贴，使其看起来比cStringIO更快。我还添加了针对bytearray / str concat的测试，以及使用较大列表和较大字符串的第二轮测试。（python 2.7.3）

大量字符串的ipython测试示例

try:
    from cStringIO import StringIO
except:
    from io import StringIO

source = ['foo']*1000

%%timeit buf = StringIO()
for i in source:
    buf.write(i)
final = buf.getvalue()
# 1000 loops, best of 3: 1.27 ms per loop

%%timeit out = []
for i in source:
    out.append(i)
final = ''.join(out)
# 1000 loops, best of 3: 9.89 ms per loop

%%timeit out = bytearray()
for i in source:
    out += i
# 10000 loops, best of 3: 98.5 µs per loop

%%timeit out = ""
for i in source:
    out += i
# 10000 loops, best of 3: 161 µs per loop

## Repeat the tests with a larger list, containing
## strings that are bigger than the small string caching 
## done by the Python
source = ['foo']*1000

# cStringIO
# 10 loops, best of 3: 19.2 ms per loop

# list append and join
# 100 loops, best of 3: 144 ms per loop

# bytearray() +=
# 100 loops, best of 3: 3.8 ms per loop

# str() +=
# 100 loops, best of 3: 5.11 ms per loop

If you are concatenating a lot of values, then neither. Appending a list is expensive. You can use StringIO for that. Especially if you are building it up over a lot of operations.

from cStringIO import StringIO
# python3:  from io import StringIO

buf = StringIO()

buf.write('foo')
buf.write('foo')
buf.write('foo')

buf.getvalue()
# 'foofoofoo'

If you already have a complete list returned to you from some other operation, then just use the ''.join(aList)

From the python FAQ: What is the most efficient way to concatenate many strings together?

str and bytes objects are immutable, therefore concatenating many strings together is inefficient as each concatenation creates a new object. In the general case, the total runtime cost is quadratic in the total string length.

To accumulate many str objects, the recommended idiom is to place them into a list and call str.join() at the end:
chunks = []
for s in my_strings:
    chunks.append(s)
result = ''.join(chunks)
(another reasonably efficient idiom is to use io.StringIO)

To accumulate many bytes objects, the recommended idiom is to extend a bytearray object using in-place concatenation (the += operator):
result = bytearray()
for b in my_bytes_objects:
    result += b

Edit: I was silly and had the results pasted backwards, making it look like appending to a list was faster than cStringIO. I have also added tests for bytearray/str concat, as well as a second round of tests using a larger list with larger strings. (python 2.7.3)

ipython test example for large lists of strings

try:
    from cStringIO import StringIO
except:
    from io import StringIO

source = ['foo']*1000

%%timeit buf = StringIO()
for i in source:
    buf.write(i)
final = buf.getvalue()
# 1000 loops, best of 3: 1.27 ms per loop

%%timeit out = []
for i in source:
    out.append(i)
final = ''.join(out)
# 1000 loops, best of 3: 9.89 ms per loop

%%timeit out = bytearray()
for i in source:
    out += i
# 10000 loops, best of 3: 98.5 µs per loop

%%timeit out = ""
for i in source:
    out += i
# 10000 loops, best of 3: 161 µs per loop

## Repeat the tests with a larger list, containing
## strings that are bigger than the small string caching 
## done by the Python
source = ['foo']*1000

# cStringIO
# 10 loops, best of 3: 19.2 ms per loop

# list append and join
# 100 loops, best of 3: 144 ms per loop

# bytearray() +=
# 100 loops, best of 3: 3.8 ms per loop

# str() +=
# 100 loops, best of 3: 5.11 ms per loop

回答 2

在Python> = 3.6中，新的f字符串是连接字符串的有效方法。

>>> name = 'some_name'
>>> number = 123
>>>
>>> f'Name is {name} and the number is {number}.'
'Name is some_name and the number is 123.'

In Python >= 3.6, the new f-string is an efficient way to concatenate a string.

>>> name = 'some_name'
>>> number = 123
>>>
>>> f'Name is {name} and the number is {number}.'
'Name is some_name and the number is 123.'

回答 3

推荐的方法仍然是使用附加和联接。

The recommended method is still to use append and join.

回答 4

如果要串联的字符串是文字，请使用字符串文字串联

re.compile(
        "[A-Za-z_]"       # letter or underscore
        "[A-Za-z0-9_]*"   # letter, digit or underscore
    )

如果要对字符串的一部分进行注释（如上）或要使用原始字符串，这将很有用文本的一部分（但不是全部）或三引号，。

由于这是在语法层发生的，因此它使用零个串联运算符。

If the strings you are concatenating are literals, use String literal concatenation

re.compile(
        "[A-Za-z_]"       # letter or underscore
        "[A-Za-z0-9_]*"   # letter, digit or underscore
    )

This is useful if you want to comment on part of a string (as above) or if you want to use raw strings or triple quotes for part of a literal but not all.

Since this happens at the syntax layer it uses zero concatenation operators.

回答 5

你写这个函数

def str_join(*args):
    return ''.join(map(str, args))

然后，您可以随时随地调用

str_join('Pine')  # Returns : Pine
str_join('Pine', 'apple')  # Returns : Pineapple
str_join('Pine', 'apple', 3)  # Returns : Pineapple3

You write this function

def str_join(*args):
    return ''.join(map(str, args))

Then you can call simply wherever you want

str_join('Pine')  # Returns : Pine
str_join('Pine', 'apple')  # Returns : Pineapple
str_join('Pine', 'apple', 3)  # Returns : Pineapple3

回答 6

虽然有些过时，代码像Pythonista：地道的Python建议join()在+ 本节。就像PythonSpeedPerformanceTips在其有关字符串连接的部分中一样，具有以下免责声明：

对于更高版本的Python，本节的准确性存在争议。在CPython 2.5中，字符串连接相当快，尽管这可能不适用于其他Python实现。有关讨论，请参见ConcatenationTestCode。

While somewhat dated, Code Like a Pythonista: Idiomatic Python recommends join() over + in this section. As does PythonSpeedPerformanceTips in its section on string concatenation, with the following disclaimer:

The accuracy of this section is disputed with respect to later versions of Python. In CPython 2.5, string concatenation is fairly fast, although this may not apply likewise to other Python implementations. See ConcatenationTestCode for a discussion.

回答 7

就稳定性和交叉实现而言，最糟糕的串联方法是使用’+’进行就地字符串串联，因为它不支持所有值。PEP8标准不鼓励这样做，并鼓励长期使用format（），join（）和append（）。

如链接的“编程建议”部分所述：

例如，对于形式为+ = b或a = a + b的语句，请不要依赖CPython有效地实现就地字符串连接。即使在CPython中，这种优化也是脆弱的（仅适用于某些类型），并且在不使用引用计数的实现中根本没有这种优化。在库的性能敏感部分，应改用”.join（）形式。这将确保在各种实现中串联发生在线性时间内。

Using in place string concatenation by ‘+’ is THE WORST method of concatenation in terms of stability and cross implementation as it does not support all values. PEP8 standard discourages this and encourages the use of format(), join() and append() for long term use.

As quoted from the linked “Programming Recommendations” section:

For example, do not rely on CPython’s efficient implementation of in-place string concatenation for statements in the form a += b or a = a + b. This optimization is fragile even in CPython (it only works for some types) and isn’t present at all in implementations that don’t use refcounting. In performance sensitive parts of the library, the ”.join() form should be used instead. This will ensure that concatenation occurs in linear time across various implementations.

回答 8

如@jdi所述，Python文档建议使用str.join或io.StringIO进行字符串连接。并说开发人员应该从+=循环中期待二次时间，即使自Python 2.4开始进行了优化。正如这个答案所说：

如果Python检测到left参数没有其他引用，它将调用realloc来尝试通过调整字符串的大小来避免复制。这不是您应该依靠的东西，因为它是一个实现细节，并且因为如果realloc最终需要频繁移动字符串，那么性能会下降到O（n ^ 2）。

我将展示一个天真地依赖于+=这种优化的真实代码示例，但它并不适用。下面的代码将可迭代的短字符串转换为更大的块，以用于批量API。

def test_concat_chunk(seq, split_by):
    result = ['']
    for item in seq:
        if len(result[-1]) + len(item) > split_by: 
            result.append('')
        result[-1] += item
    return result

由于二次时间的复杂性，该代码可能在文学上运行数小时。以下是具有建议的数据结构的替代方案：

import io

def test_stringio_chunk(seq, split_by):
    def chunk():
        buf = io.StringIO()
        size = 0
        for item in seq:
            if size + len(item) <= split_by:
                size += buf.write(item)
            else:
                yield buf.getvalue()
                buf = io.StringIO()
                size = buf.write(item)
        if size:
            yield buf.getvalue()

    return list(chunk())

def test_join_chunk(seq, split_by):
    def chunk():
        buf = []
        size = 0
        for item in seq:
            if size + len(item) <= split_by:
                buf.append(item)
                size += len(item)
            else:
                yield ''.join(buf)                
                buf.clear()
                buf.append(item)
                size = len(item)
        if size:
            yield ''.join(buf)

    return list(chunk())

还有一个微基准测试：

import timeit
import random
import string
import matplotlib.pyplot as plt

line = ''.join(random.choices(
    string.ascii_uppercase + string.digits, k=512)) + '\n'
x = []
y_concat = []
y_stringio = []
y_join = []
n = 5
for i in range(1, 11):
    x.append(i)
    seq = [line] * (20 * 2 ** 20 // len(line))
    chunk_size = i * 2 ** 20
    y_concat.append(
        timeit.timeit(lambda: test_concat_chunk(seq, chunk_size), number=n) / n)
    y_stringio.append(
        timeit.timeit(lambda: test_stringio_chunk(seq, chunk_size), number=n) / n)
    y_join.append(
        timeit.timeit(lambda: test_join_chunk(seq, chunk_size), number=n) / n)
plt.plot(x, y_concat)
plt.plot(x, y_stringio)
plt.plot(x, y_join)
plt.legend(['concat', 'stringio', 'join'], loc='upper left')
plt.show()

As @jdi mentions Python documentation suggests to use str.join or io.StringIO for string concatenation. And says that a developer should expect quadratic time from += in a loop, even though there’s an optimisation since Python 2.4. As this answer says:

If Python detects that the left argument has no other references, it calls realloc to attempt to avoid a copy by resizing the string in place. This is not something you should ever rely on, because it’s an implementation detail and because if realloc ends up needing to move the string frequently, performance degrades to O(n^2) anyway.

I will show an example of real-world code that naively relied on += this optimisation, but it didn’t apply. The code below converts an iterable of short strings into bigger chunks to be used in a bulk API.

def test_concat_chunk(seq, split_by):
    result = ['']
    for item in seq:
        if len(result[-1]) + len(item) > split_by: 
            result.append('')
        result[-1] += item
    return result

This code can literary run for hours because of quadratic time complexity. Below are alternatives with suggested data structures:

import io

def test_stringio_chunk(seq, split_by):
    def chunk():
        buf = io.StringIO()
        size = 0
        for item in seq:
            if size + len(item) <= split_by:
                size += buf.write(item)
            else:
                yield buf.getvalue()
                buf = io.StringIO()
                size = buf.write(item)
        if size:
            yield buf.getvalue()

    return list(chunk())

def test_join_chunk(seq, split_by):
    def chunk():
        buf = []
        size = 0
        for item in seq:
            if size + len(item) <= split_by:
                buf.append(item)
                size += len(item)
            else:
                yield ''.join(buf)                
                buf.clear()
                buf.append(item)
                size = len(item)
        if size:
            yield ''.join(buf)

    return list(chunk())

And a micro-benchmark:

import timeit
import random
import string
import matplotlib.pyplot as plt

line = ''.join(random.choices(
    string.ascii_uppercase + string.digits, k=512)) + '\n'
x = []
y_concat = []
y_stringio = []
y_join = []
n = 5
for i in range(1, 11):
    x.append(i)
    seq = [line] * (20 * 2 ** 20 // len(line))
    chunk_size = i * 2 ** 20
    y_concat.append(
        timeit.timeit(lambda: test_concat_chunk(seq, chunk_size), number=n) / n)
    y_stringio.append(
        timeit.timeit(lambda: test_stringio_chunk(seq, chunk_size), number=n) / n)
    y_join.append(
        timeit.timeit(lambda: test_join_chunk(seq, chunk_size), number=n) / n)
plt.plot(x, y_concat)
plt.plot(x, y_stringio)
plt.plot(x, y_join)
plt.legend(['concat', 'stringio', 'join'], loc='upper left')
plt.show()

回答 9

您可以采用不同的方式。

str1 = "Hello"
str2 = "World"
str_list = ['Hello', 'World']
str_dict = {'str1': 'Hello', 'str2': 'World'}

# Concatenating With the + Operator
print(str1 + ' ' + str2)  # Hello World

# String Formatting with the % Operator
print("%s %s" % (str1, str2))  # Hello World

# String Formatting with the { } Operators with str.format()
print("{}{}".format(str1, str2))  # Hello World
print("{0}{1}".format(str1, str2))  # Hello World
print("{str1} {str2}".format(str1=str_dict['str1'], str2=str_dict['str2']))  # Hello World
print("{str1} {str2}".format(**str_dict))  # Hello World

# Going From a List to a String in Python With .join()
print(' '.join(str_list))  # Hello World

# Python f'strings --> 3.6 onwards
print(f"{str1} {str2}")  # Hello World

我通过以下文章创建了这个小总结。

You can do in different ways.

str1 = "Hello"
str2 = "World"
str_list = ['Hello', 'World']
str_dict = {'str1': 'Hello', 'str2': 'World'}

# Concatenating With the + Operator
print(str1 + ' ' + str2)  # Hello World

# String Formatting with the % Operator
print("%s %s" % (str1, str2))  # Hello World

# String Formatting with the { } Operators with str.format()
print("{}{}".format(str1, str2))  # Hello World
print("{0}{1}".format(str1, str2))  # Hello World
print("{str1} {str2}".format(str1=str_dict['str1'], str2=str_dict['str2']))  # Hello World
print("{str1} {str2}".format(**str_dict))  # Hello World

# Going From a List to a String in Python With .join()
print(' '.join(str_list))  # Hello World

# Python f'strings --> 3.6 onwards
print(f"{str1} {str2}")  # Hello World

I created this little summary through following articles.

Python 3’s f-Strings: An Improved String Formatting Syntax (Guide) (Speed test also included)
Formatted string literals
String Concatenation and Formatting
Splitting, Concatenating, and Joining Strings in Python

回答 10

我的用例略有不同。我不得不构造一个查询，其中有20多个字段是动态的。我遵循了这种使用格式化方法的方法

query = "insert into {0}({1},{2},{3}) values({4}, {5}, {6})"
query.format('users','name','age','dna','suzan',1010,'nda')

这对我来说相对来说比较简单，而不是使用+或其他方式

my use case was slight different. I had to construct a query where more then 20 fields were dynamic. I followed this approach of using format method

query = "insert into {0}({1},{2},{3}) values({4}, {5}, {6})"
query.format('users','name','age','dna','suzan',1010,'nda')

this was comparatively simpler for me instead of using + or other ways

回答 11

您也可以使用此（更有效）。（/software/304445/why-is-s-better-than-for-concatenation）

s += "%s" %(stringfromelsewhere)

You can use this(more efficient) too. (https://softwareengineering.stackexchange.com/questions/304445/why-is-s-better-than-for-concatenation)

s += "%s" %(stringfromelsewhere)

知识问答

如何使用Python重命名文件

2021年7月27日 Python实用宝典

问题：如何使用Python重命名文件

我想更改a.txt为b.kml。

I want to change a.txt to b.kml.

回答 0

用途os.rename：

import os

os.rename('a.txt', 'b.kml')

Use os.rename:

import os

os.rename('a.txt', 'b.kml')

回答 1

文件可能位于目录中，在这种情况下，请指定路径：

import os
old_file = os.path.join("directory", "a.txt")
new_file = os.path.join("directory", "b.kml")
os.rename(old_file, new_file)

File may be inside a directory, in that case specify the path:

import os
old_file = os.path.join("directory", "a.txt")
new_file = os.path.join("directory", "b.kml")
os.rename(old_file, new_file)

回答 2

import shutil

shutil.move('a.txt', 'b.kml')

这将可以重命名或移动文件。

import shutil

shutil.move('a.txt', 'b.kml')

This will work to rename or move a file.

回答 3

从Python 3.4开始，可以使用pathlib模块解决此问题。

如果您使用的是旧版本，则可以使用此处找到的反向移植版本

假设您不在要重命名的根路径（只是给它增加了一点难度），而不必提供完整路径，我们可以看一下：

some_path = 'a/b/c/the_file.extension'

因此，您可以按照自己的路径创建一个Path对象：

from pathlib import Path
p = Path(some_path)

只是为了提供有关我们现在拥有的该对象的一些信息，我们可以从中提取内容。例如，如果出于某种原因，我们希望通过修改从文件名的文件重命名the_file到the_file_1，那么我们可以得到的文件名部分：

name_without_extension = p.stem

并仍将扩展名放在手边：

ext = p.suffix

我们可以通过简单的字符串操作来执行修改：

Python 3.6及更高版本使用f字符串！

new_file_name = f"{name_without_extension}_1"

除此以外：

new_file_name = "{}_{}".format(name_without_extension, 1)

现在，我们可以通过rename在创建的路径对象上调用方法并附加ext来完成所需的正确的重命名结构，从而执行重命名：

p.rename(Path(p.parent, new_file_name + ext))

更简短地展示其简单性：

Python 3.6及更高版本：

from pathlib import Path
p = Path(some_path)
p.rename(Path(p.parent, f"{p.stem}_1_{p.suffix}"))

低于Python 3.6的版本改用字符串格式方法：

from pathlib import Path
p = Path(some_path)
p.rename(Path(p.parent, "{}_{}_{}".format(p.stem, 1, p.suffix))

As of Python 3.4 one can use the pathlib module to solve this.

If you happen to be on an older version, you can use the backported version found here

Let’s assume you are not in the root path (just to add a bit of difficulty to it) you want to rename, and have to provide a full path, we can look at this:

some_path = 'a/b/c/the_file.extension'

So, you can take your path and create a Path object out of it:

from pathlib import Path
p = Path(some_path)

Just to provide some information around this object we have now, we can extract things out of it. For example, if for whatever reason we want to rename the file by modifying the filename from the_file to the_file_1, then we can get the filename part:

name_without_extension = p.stem

And still hold the extension in hand as well:

ext = p.suffix

We can perform our modification with a simple string manipulation:

Python 3.6 and greater make use of f-strings!

new_file_name = f"{name_without_extension}_1"

Otherwise:

new_file_name = "{}_{}".format(name_without_extension, 1)

And now we can perform our rename by calling the rename method on the path object we created and appending the ext to complete the proper rename structure we want:

p.rename(Path(p.parent, new_file_name + ext))

More shortly to showcase its simplicity:

Python 3.6+:

from pathlib import Path
p = Path(some_path)
p.rename(Path(p.parent, f"{p.stem}_1_{p.suffix}"))

Versions less than Python 3.6 use the string format method instead:

from pathlib import Path
p = Path(some_path)
p.rename(Path(p.parent, "{}_{}_{}".format(p.stem, 1, p.suffix))

回答 4

os.rename(old, new)

可以在Python文档中找到：http : //docs.python.org/library/os.html

os.rename(old, new)

This is found in the Python docs: http://docs.python.org/library/os.html

回答 5

使用os.rename。但是您必须将两个文件的完整路径传递给函数。如果a.txt我的桌面上有文件，则可以这样做，而且我也必须提供完整的重命名文件。

 os.rename('C:\\Users\\Desktop\\a.txt', 'C:\\Users\\Desktop\\b.kml')

Use os.rename. But you have to pass full path of both files to the function. If I have a file a.txt on my desktop so I will do and also I have to give full of renamed file too.

 os.rename('C:\\Users\\Desktop\\a.txt', 'C:\\Users\\Desktop\\b.kml')

回答 6

import os

# Set the path
path = 'a\\b\\c'  
# save current working directory
saved_cwd = os.getcwd()
# change your cwd to the directory which contains files
os.chdir(path)
os.rename('a.txt', 'b.klm')
# moving back to the directory you were in 
os.chdir(saved_cwd)

import os

# Set the path
path = 'a\\b\\c'  
# save current working directory
saved_cwd = os.getcwd()
# change your cwd to the directory which contains files
os.chdir(path)
os.rename('a.txt', 'b.klm')
# moving back to the directory you were in 
os.chdir(saved_cwd)

回答 7

您可以使用os.system调用终端来完成任务：

os.system('mv oldfile newfile')

You can use os.system to invoke terminal to accomplish the task:

os.system('mv oldfile newfile')

回答 8

import os
import re
from pathlib import Path

for f in os.listdir(training_data_dir2):
  for file in os.listdir( training_data_dir2 + '/' + f):
    oldfile= Path(training_data_dir2 + '/' + f + '/' + file)
    newfile = Path(training_data_dir2 + '/' + f + '/' + file[49:])
    p=oldfile
    p.rename(newfile)

import os
import re
from pathlib import Path

for f in os.listdir(training_data_dir2):
  for file in os.listdir( training_data_dir2 + '/' + f):
    oldfile= Path(training_data_dir2 + '/' + f + '/' + file)
    newfile = Path(training_data_dir2 + '/' + f + '/' + file[49:])
    p=oldfile
    p.rename(newfile)

回答 9

import shutil
import os

files = os.listdir("./pics/") 

for key in range(0, len(files)):
   print files[key]
   shutil.move("./pics/" + files[key],"./pics/img" + str(key) + ".jpeg")

这应该做。python 3+

import shutil
import os

files = os.listdir("./pics/") 

for key in range(0, len(files)):
   print files[key]
   shutil.move("./pics/" + files[key],"./pics/img" + str(key) + ".jpeg")

This should do it. python 3+

知识问答

如何旋转数据框

2021年7月27日 Python实用宝典

问题：如何旋转数据框

什么是支点？
我如何枢纽？
这是支点吗？
长格式到宽格式？

我已经看到很多有关数据透视表的问题。即使他们不知道他们在询问数据透视表，通常也是如此。几乎不可能写出涵盖枢纽各个方面的规范问答。

…但是我要去尝试一下。

现有问题和答案的问题在于，问题通常集中在OP难以推广的细微差别上，以便使用许多现有的良好答案。但是，没有一个答案试图给出全面的解释（因为这是一项艰巨的任务）

从我的Google搜索中查找一些示例

如何在Pandas中透视数据框？
- 好问题和答案。但是答案只回答了很少的具体问题。
熊猫数据透视表到数据框
- 在此问题中，OP与枢轴的输出有关。即列的外观。OP希望它看起来像R。这对熊猫用户不是很有帮助。
旋转数据框的熊猫，重复的行
- 另一个不错的问题，但答案集中在一种方法上，即 pd.DataFrame.pivot

因此，每当有人搜索时，pivot他们都会得到零星的结果，这些结果可能不会回答他们的特定问题。

设定

您可能会注意到，我显眼地命名了我的列和相关的列值，以与我将在下面的答案中介绍的方式相对应。

import numpy as np
import pandas as pd
from numpy.core.defchararray import add

np.random.seed([3,1415])
n = 20

cols = np.array(['key', 'row', 'item', 'col'])
arr1 = (np.random.randint(5, size=(n, 4)) // [2, 1, 2, 1]).astype(str)

df = pd.DataFrame(
    add(cols, arr1), columns=cols
).join(
    pd.DataFrame(np.random.rand(n, 2).round(2)).add_prefix('val')
)
print(df)

     key   row   item   col  val0  val1
0   key0  row3  item1  col3  0.81  0.04
1   key1  row2  item1  col2  0.44  0.07
2   key1  row0  item1  col0  0.77  0.01
3   key0  row4  item0  col2  0.15  0.59
4   key1  row0  item2  col1  0.81  0.64
5   key1  row2  item2  col4  0.13  0.88
6   key2  row4  item1  col3  0.88  0.39
7   key1  row4  item1  col1  0.10  0.07
8   key1  row0  item2  col4  0.65  0.02
9   key1  row2  item0  col2  0.35  0.61
10  key2  row0  item2  col1  0.40  0.85
11  key2  row4  item1  col2  0.64  0.25
12  key0  row2  item2  col3  0.50  0.44
13  key0  row4  item1  col4  0.24  0.46
14  key1  row3  item2  col3  0.28  0.11
15  key0  row3  item1  col1  0.31  0.23
16  key0  row0  item2  col3  0.86  0.01
17  key0  row4  item0  col3  0.64  0.21
18  key2  row2  item2  col0  0.13  0.45
19  key0  row2  item0  col4  0.37  0.70

问题

我为什么得到 ValueError: Index contains duplicate entries, cannot reshape

如何旋转df以使col值成为列，row值成为索引，值的均值val0？

col   col0   col1   col2   col3  col4
row                                  
row0  0.77  0.605    NaN  0.860  0.65
row2  0.13    NaN  0.395  0.500  0.25
row3   NaN  0.310    NaN  0.545   NaN
row4   NaN  0.100  0.395  0.760  0.24

如何旋转df以使col值是列，row值是索引，值的均值val0是和缺少值是0？

col   col0   col1   col2   col3  col4
row                                  
row0  0.77  0.605  0.000  0.860  0.65
row2  0.13  0.000  0.395  0.500  0.25
row3  0.00  0.310  0.000  0.545  0.00
row4  0.00  0.100  0.395  0.760  0.24

我可以得到比其他的东西mean，如可能sum？

col   col0  col1  col2  col3  col4
row                               
row0  0.77  1.21  0.00  0.86  0.65
row2  0.13  0.00  0.79  0.50  0.50
row3  0.00  0.31  0.00  1.09  0.00
row4  0.00  0.10  0.79  1.52  0.24

我可以一次做多个聚合吗？

       sum                          mean                           
col   col0  col1  col2  col3  col4  col0   col1   col2   col3  col4
row                                                                
row0  0.77  1.21  0.00  0.86  0.65  0.77  0.605  0.000  0.860  0.65
row2  0.13  0.00  0.79  0.50  0.50  0.13  0.000  0.395  0.500  0.25
row3  0.00  0.31  0.00  1.09  0.00  0.00  0.310  0.000  0.545  0.00
row4  0.00  0.10  0.79  1.52  0.24  0.00  0.100  0.395  0.760  0.24

我可以汇总多个值列吗？

      val0                             val1                          
col   col0   col1   col2   col3  col4  col0   col1  col2   col3  col4
row                                                                  
row0  0.77  0.605  0.000  0.860  0.65  0.01  0.745  0.00  0.010  0.02
row2  0.13  0.000  0.395  0.500  0.25  0.45  0.000  0.34  0.440  0.79
row3  0.00  0.310  0.000  0.545  0.00  0.00  0.230  0.00  0.075  0.00
row4  0.00  0.100  0.395  0.760  0.24  0.00  0.070  0.42  0.300  0.46

可以细分为多列吗？

item item0             item1                         item2                   
col   col2  col3  col4  col0  col1  col2  col3  col4  col0   col1  col3  col4
row                                                                          
row0  0.00  0.00  0.00  0.77  0.00  0.00  0.00  0.00  0.00  0.605  0.86  0.65
row2  0.35  0.00  0.37  0.00  0.00  0.44  0.00  0.00  0.13  0.000  0.50  0.13
row3  0.00  0.00  0.00  0.00  0.31  0.00  0.81  0.00  0.00  0.000  0.28  0.00
row4  0.15  0.64  0.00  0.00  0.10  0.64  0.88  0.24  0.00  0.000  0.00  0.00

要么

item      item0             item1                         item2                  
col        col2  col3  col4  col0  col1  col2  col3  col4  col0  col1  col3  col4
key  row                                                                         
key0 row0  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.86  0.00
     row2  0.00  0.00  0.37  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.50  0.00
     row3  0.00  0.00  0.00  0.00  0.31  0.00  0.81  0.00  0.00  0.00  0.00  0.00
     row4  0.15  0.64  0.00  0.00  0.00  0.00  0.00  0.24  0.00  0.00  0.00  0.00
key1 row0  0.00  0.00  0.00  0.77  0.00  0.00  0.00  0.00  0.00  0.81  0.00  0.65
     row2  0.35  0.00  0.00  0.00  0.00  0.44  0.00  0.00  0.00  0.00  0.00  0.13
     row3  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.28  0.00
     row4  0.00  0.00  0.00  0.00  0.10  0.00  0.00  0.00  0.00  0.00  0.00  0.00
key2 row0  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.40  0.00  0.00
     row2  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.13  0.00  0.00  0.00
     row4  0.00  0.00  0.00  0.00  0.00  0.64  0.88  0.00  0.00  0.00  0.00  0.00

我可以汇总列和行一起出现的频率，又称为“交叉表”吗？

col   col0  col1  col2  col3  col4
row                               
row0     1     2     0     1     1
row2     1     0     2     1     2
row3     0     1     0     2     0
row4     0     1     2     2     1

如何通过仅旋转两列将DataFrame从长转换为宽？鉴于

np.random.seed([3, 1415])
df2 = pd.DataFrame({'A': list('aaaabbbc'), 'B': np.random.choice(15, 8)})        
df2        
   A   B
0  a   0
1  a  11
2  a   2
3  a  11
4  b  10
5  b  10
6  b  14
7  c   7

预期应该看起来像

      a     b    c
0   0.0  10.0  7.0
1  11.0  10.0  NaN
2   2.0  14.0  NaN
3  11.0   NaN  NaN

我如何将多重索引展平为单索引 pivot

从

至

   1|1  2|1  2|2               
a    2    1    1
b    2    1    0
c    1    0    0

What is pivot?
How do I pivot?
Is this a pivot?
Long format to wide format?

I’ve seen a lot of questions that ask about pivot tables. Even if they don’t know that they are asking about pivot tables, they usually are. It is virtually impossible to write a canonical question and answer that encompasses all aspects of pivoting….

… But I’m going to give it a go.

The problem with existing questions and answers is that often the question is focused on a nuance that the OP has trouble generalizing in order to use a number of the existing good answers. However, none of the answers attempt to give a comprehensive explanation (because it’s a daunting task)

Look a few examples from my google search

How to pivot a dataframe in Pandas?
- Good question and answer. But the answer only answers the specific question with little explanation.
pandas pivot table to data frame
- In this question, the OP is concerned with the output of the pivot. Namely how the columns look. OP wanted it to look like R. This isn’t very helpful for pandas users.
pandas pivoting a dataframe, duplicate rows
- Another decent question but the answer focuses on one method, namely pd.DataFrame.pivot

So whenever someone searches for pivot they get sporadic results that are likely not going to answer their specific question.

Setup

You may notice that I conspicuously named my columns and relevant column values to correspond with how I’m going to pivot in the answers below.

import numpy as np
import pandas as pd
from numpy.core.defchararray import add

np.random.seed([3,1415])
n = 20

cols = np.array(['key', 'row', 'item', 'col'])
arr1 = (np.random.randint(5, size=(n, 4)) // [2, 1, 2, 1]).astype(str)

df = pd.DataFrame(
    add(cols, arr1), columns=cols
).join(
    pd.DataFrame(np.random.rand(n, 2).round(2)).add_prefix('val')
)
print(df)

     key   row   item   col  val0  val1
0   key0  row3  item1  col3  0.81  0.04
1   key1  row2  item1  col2  0.44  0.07
2   key1  row0  item1  col0  0.77  0.01
3   key0  row4  item0  col2  0.15  0.59
4   key1  row0  item2  col1  0.81  0.64
5   key1  row2  item2  col4  0.13  0.88
6   key2  row4  item1  col3  0.88  0.39
7   key1  row4  item1  col1  0.10  0.07
8   key1  row0  item2  col4  0.65  0.02
9   key1  row2  item0  col2  0.35  0.61
10  key2  row0  item2  col1  0.40  0.85
11  key2  row4  item1  col2  0.64  0.25
12  key0  row2  item2  col3  0.50  0.44
13  key0  row4  item1  col4  0.24  0.46
14  key1  row3  item2  col3  0.28  0.11
15  key0  row3  item1  col1  0.31  0.23
16  key0  row0  item2  col3  0.86  0.01
17  key0  row4  item0  col3  0.64  0.21
18  key2  row2  item2  col0  0.13  0.45
19  key0  row2  item0  col4  0.37  0.70

Question(s)

Why do I get ValueError: Index contains duplicate entries, cannot reshape

How do I pivot df such that the col values are columns, row values are the index, and mean of val0 are the values?

col   col0   col1   col2   col3  col4
row                                  
row0  0.77  0.605    NaN  0.860  0.65
row2  0.13    NaN  0.395  0.500  0.25
row3   NaN  0.310    NaN  0.545   NaN
row4   NaN  0.100  0.395  0.760  0.24

How do I pivot df such that the col values are columns, row values are the index, mean of val0 are the values, and missing values are 0?

col   col0   col1   col2   col3  col4
row                                  
row0  0.77  0.605  0.000  0.860  0.65
row2  0.13  0.000  0.395  0.500  0.25
row3  0.00  0.310  0.000  0.545  0.00
row4  0.00  0.100  0.395  0.760  0.24

Can I get something other than mean, like maybe sum?

col   col0  col1  col2  col3  col4
row                               
row0  0.77  1.21  0.00  0.86  0.65
row2  0.13  0.00  0.79  0.50  0.50
row3  0.00  0.31  0.00  1.09  0.00
row4  0.00  0.10  0.79  1.52  0.24

Can I do more that one aggregation at a time?

       sum                          mean                           
col   col0  col1  col2  col3  col4  col0   col1   col2   col3  col4
row                                                                
row0  0.77  1.21  0.00  0.86  0.65  0.77  0.605  0.000  0.860  0.65
row2  0.13  0.00  0.79  0.50  0.50  0.13  0.000  0.395  0.500  0.25
row3  0.00  0.31  0.00  1.09  0.00  0.00  0.310  0.000  0.545  0.00
row4  0.00  0.10  0.79  1.52  0.24  0.00  0.100  0.395  0.760  0.24

Can I aggregate over multiple value columns?

      val0                             val1                          
col   col0   col1   col2   col3  col4  col0   col1  col2   col3  col4
row                                                                  
row0  0.77  0.605  0.000  0.860  0.65  0.01  0.745  0.00  0.010  0.02
row2  0.13  0.000  0.395  0.500  0.25  0.45  0.000  0.34  0.440  0.79
row3  0.00  0.310  0.000  0.545  0.00  0.00  0.230  0.00  0.075  0.00
row4  0.00  0.100  0.395  0.760  0.24  0.00  0.070  0.42  0.300  0.46

Can Subdivide by multiple columns?

item item0             item1                         item2                   
col   col2  col3  col4  col0  col1  col2  col3  col4  col0   col1  col3  col4
row                                                                          
row0  0.00  0.00  0.00  0.77  0.00  0.00  0.00  0.00  0.00  0.605  0.86  0.65
row2  0.35  0.00  0.37  0.00  0.00  0.44  0.00  0.00  0.13  0.000  0.50  0.13
row3  0.00  0.00  0.00  0.00  0.31  0.00  0.81  0.00  0.00  0.000  0.28  0.00
row4  0.15  0.64  0.00  0.00  0.10  0.64  0.88  0.24  0.00  0.000  0.00  0.00

item      item0             item1                         item2                  
col        col2  col3  col4  col0  col1  col2  col3  col4  col0  col1  col3  col4
key  row                                                                         
key0 row0  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.86  0.00
     row2  0.00  0.00  0.37  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.50  0.00
     row3  0.00  0.00  0.00  0.00  0.31  0.00  0.81  0.00  0.00  0.00  0.00  0.00
     row4  0.15  0.64  0.00  0.00  0.00  0.00  0.00  0.24  0.00  0.00  0.00  0.00
key1 row0  0.00  0.00  0.00  0.77  0.00  0.00  0.00  0.00  0.00  0.81  0.00  0.65
     row2  0.35  0.00  0.00  0.00  0.00  0.44  0.00  0.00  0.00  0.00  0.00  0.13
     row3  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.28  0.00
     row4  0.00  0.00  0.00  0.00  0.10  0.00  0.00  0.00  0.00  0.00  0.00  0.00
key2 row0  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.40  0.00  0.00
     row2  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.13  0.00  0.00  0.00
     row4  0.00  0.00  0.00  0.00  0.00  0.64  0.88  0.00  0.00  0.00  0.00  0.00

Can I aggregate the frequency in which the column and rows occur together, aka “cross tabulation”?

col   col0  col1  col2  col3  col4
row                               
row0     1     2     0     1     1
row2     1     0     2     1     2
row3     0     1     0     2     0
row4     0     1     2     2     1

How do I convert a DataFrame from long to wide by pivoting on ONLY two columns? Given,

np.random.seed([3, 1415])
df2 = pd.DataFrame({'A': list('aaaabbbc'), 'B': np.random.choice(15, 8)})        
df2        
   A   B
0  a   0
1  a  11
2  a   2
3  a  11
4  b  10
5  b  10
6  b  14
7  c   7

The expected should would look something like

      a     b    c
0   0.0  10.0  7.0
1  11.0  10.0  NaN
2   2.0  14.0  NaN
3  11.0   NaN  NaN

How do I flatten the multiple index to single index after pivot

From

   1|1  2|1  2|2               
a    2    1    1
b    2    1    0
c    1    0    0

回答 0

我们首先回答第一个问题：

问题1

我为什么得到 ValueError: Index contains duplicate entries, cannot reshape

发生这种情况是因为pandas试图为a columns或index具有重复条目的对象重新编制索引。有多种方法可以执行数据透视。当有人要求重复输入密钥时，其中某些方法不太适合。例如。考虑一下pd.DataFrame.pivot。我知道有重复的条目共享row和col值：

df.duplicated(['row', 'col']).any()

True

所以当我pivot使用

df.pivot(index='row', columns='col', values='val0')

我收到上述错误。实际上，当我尝试使用以下命令执行相同的任务时，会出现相同的错误：

df.set_index(['row', 'col'])['val0'].unstack()

这是我们可以用来透视的成语列表

pd.DataFrame.groupby + pd.DataFrame.unstack
- 进行几乎所有类型的数据透视的良好通用方法
- 您可以指定一组将构成枢轴行级别和列级别的所有列。通过选择要聚合的其余列以及要执行聚合的功能，可以做到这一点。最后，您unstack要在列索引中包含的级别。
pd.DataFrame.pivot_table
- groupby具有更直观API的美化版本。对于许多人来说，这是首选方法。并且是开发人员的预期方法。
- 指定行级别，列级别，要聚合的值以及执行聚合的函数。
pd.DataFrame.set_index + pd.DataFrame.unstack
- 方便和直观的一些（包括我自己）。无法处理重复的分组密钥。
- 与groupby范式类似，我们指定最终将成为行或列级别的所有列，并将其设置为索引。然后unstack，我们在各列中选择所需的级别。如果其余索引级别或列级别都不唯一，则此方法将失败。
pd.DataFrame.pivot
- 非常相似之处set_index在于它共享重复的密钥限制。该API也非常有限。只需要对标量值index，columns，values。
- 与pivot_table方法类似，我们选择要在其上旋转的行，列和值。但是，我们无法聚合，并且如果行或列都不唯一，则此方法将失败。
pd.crosstab
- 这是pivot_table最纯粹的形式的一种专门版本，是执行多个任务的最直观的方法。
pd.factorize + np.bincount
- 这是一种非常先进的技术，它非常晦涩，但是速度却很快。并非在所有情况下都可以使用它，但是只要您可以使用它并且感觉舒适，就会获得性能上的回报。
pd.get_dummies + pd.DataFrame.dot
- 我用它来巧妙地执行交叉制表。

例子

对于接下来的每个答案和问题，我将使用进行回答pd.DataFrame.pivot_table。然后，我将提供替代方法来执行相同的任务。

问题3

如何旋转df以使col值是列，row值是索引，值的均值val0是和缺少值是0？

pd.DataFrame.pivot_table

fill_value默认情况下未设置。我倾向于适当地设置它。在这种情况下，我将其设置为0。请注意，我跳过了问题2，因为它与此答案相同，但没有fill_value

aggfunc='mean'是默认设置，我无需设置。我将其包括在内是为了明确。

df.pivot_table(
    values='val0', index='row', columns='col',
    fill_value=0, aggfunc='mean')

col   col0   col1   col2   col3  col4
row                                  
row0  0.77  0.605  0.000  0.860  0.65
row2  0.13  0.000  0.395  0.500  0.25
row3  0.00  0.310  0.000  0.545  0.00
row4  0.00  0.100  0.395  0.760  0.24

pd.DataFrame.groupby

df.groupby(['row', 'col'])['val0'].mean().unstack(fill_value=0)

pd.crosstab

pd.crosstab(
    index=df['row'], columns=df['col'],
    values=df['val0'], aggfunc='mean').fillna(0)

问题4

我可以得到比其他的东西mean，如可能sum？

pd.DataFrame.pivot_table

df.pivot_table(
    values='val0', index='row', columns='col',
    fill_value=0, aggfunc='sum')

col   col0  col1  col2  col3  col4
row                               
row0  0.77  1.21  0.00  0.86  0.65
row2  0.13  0.00  0.79  0.50  0.50
row3  0.00  0.31  0.00  1.09  0.00
row4  0.00  0.10  0.79  1.52  0.24

pd.DataFrame.groupby

df.groupby(['row', 'col'])['val0'].sum().unstack(fill_value=0)

pd.crosstab

pd.crosstab(
    index=df['row'], columns=df['col'],
    values=df['val0'], aggfunc='sum').fillna(0)

问题5

我可以一次做多个聚合吗？

请注意，for pivot_table和crosstab我需要传递可调用对象列表。另一方面，groupby.agg能够为有限数量的特殊功能使用字符串。 groupby.agg也会采用我们传递给其他对象的相同的可调用对象，但是利用字符串函数名称通常会更有效，因为可以提高效率。

pd.DataFrame.pivot_table

df.pivot_table(
    values='val0', index='row', columns='col',
    fill_value=0, aggfunc=[np.size, np.mean])

     size                      mean                           
col  col0 col1 col2 col3 col4  col0   col1   col2   col3  col4
row                                                           
row0    1    2    0    1    1  0.77  0.605  0.000  0.860  0.65
row2    1    0    2    1    2  0.13  0.000  0.395  0.500  0.25
row3    0    1    0    2    0  0.00  0.310  0.000  0.545  0.00
row4    0    1    2    2    1  0.00  0.100  0.395  0.760  0.24

pd.DataFrame.groupby

df.groupby(['row', 'col'])['val0'].agg(['size', 'mean']).unstack(fill_value=0)

pd.crosstab

pd.crosstab(
    index=df['row'], columns=df['col'],
    values=df['val0'], aggfunc=[np.size, np.mean]).fillna(0, downcast='infer')

问题6

我可以汇总多个值列吗？

pd.DataFrame.pivot_table我们通过了，values=['val0', 'val1']但我们可以完全取消

df.pivot_table(
    values=['val0', 'val1'], index='row', columns='col',
    fill_value=0, aggfunc='mean')

      val0                             val1                          
col   col0   col1   col2   col3  col4  col0   col1  col2   col3  col4
row                                                                  
row0  0.77  0.605  0.000  0.860  0.65  0.01  0.745  0.00  0.010  0.02
row2  0.13  0.000  0.395  0.500  0.25  0.45  0.000  0.34  0.440  0.79
row3  0.00  0.310  0.000  0.545  0.00  0.00  0.230  0.00  0.075  0.00
row4  0.00  0.100  0.395  0.760  0.24  0.00  0.070  0.42  0.300  0.46

pd.DataFrame.groupby

df.groupby(['row', 'col'])['val0', 'val1'].mean().unstack(fill_value=0)

问题7

可以细分为多列吗？

pd.DataFrame.pivot_table

df.pivot_table(
    values='val0', index='row', columns=['item', 'col'],
    fill_value=0, aggfunc='mean')

item item0             item1                         item2                   
col   col2  col3  col4  col0  col1  col2  col3  col4  col0   col1  col3  col4
row                                                                          
row0  0.00  0.00  0.00  0.77  0.00  0.00  0.00  0.00  0.00  0.605  0.86  0.65
row2  0.35  0.00  0.37  0.00  0.00  0.44  0.00  0.00  0.13  0.000  0.50  0.13
row3  0.00  0.00  0.00  0.00  0.31  0.00  0.81  0.00  0.00  0.000  0.28  0.00
row4  0.15  0.64  0.00  0.00  0.10  0.64  0.88  0.24  0.00  0.000  0.00  0.00

pd.DataFrame.groupby

df.groupby(
    ['row', 'item', 'col']
)['val0'].mean().unstack(['item', 'col']).fillna(0).sort_index(1)

问题8

可以细分为多列吗？

pd.DataFrame.pivot_table

df.pivot_table(
    values='val0', index=['key', 'row'], columns=['item', 'col'],
    fill_value=0, aggfunc='mean')

item      item0             item1                         item2                  
col        col2  col3  col4  col0  col1  col2  col3  col4  col0  col1  col3  col4
key  row                                                                         
key0 row0  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.86  0.00
     row2  0.00  0.00  0.37  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.50  0.00
     row3  0.00  0.00  0.00  0.00  0.31  0.00  0.81  0.00  0.00  0.00  0.00  0.00
     row4  0.15  0.64  0.00  0.00  0.00  0.00  0.00  0.24  0.00  0.00  0.00  0.00
key1 row0  0.00  0.00  0.00  0.77  0.00  0.00  0.00  0.00  0.00  0.81  0.00  0.65
     row2  0.35  0.00  0.00  0.00  0.00  0.44  0.00  0.00  0.00  0.00  0.00  0.13
     row3  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.28  0.00
     row4  0.00  0.00  0.00  0.00  0.10  0.00  0.00  0.00  0.00  0.00  0.00  0.00
key2 row0  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.40  0.00  0.00
     row2  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.13  0.00  0.00  0.00
     row4  0.00  0.00  0.00  0.00  0.00  0.64  0.88  0.00  0.00  0.00  0.00  0.00

pd.DataFrame.groupby

df.groupby(
    ['key', 'row', 'item', 'col']
)['val0'].mean().unstack(['item', 'col']).fillna(0).sort_index(1)

pd.DataFrame.set_index 因为键集对于行和列都是唯一的

df.set_index(
    ['key', 'row', 'item', 'col']
)['val0'].unstack(['item', 'col']).fillna(0).sort_index(1)

问题9

我可以汇总列和行一起出现的频率，又称为“交叉表”吗？

pd.DataFrame.pivot_table

df.pivot_table(index='row', columns='col', fill_value=0, aggfunc='size')

    col   col0  col1  col2  col3  col4
row                               
row0     1     2     0     1     1
row2     1     0     2     1     2
row3     0     1     0     2     0
row4     0     1     2     2     1

pd.DataFrame.groupby

df.groupby(['row', 'col'])['val0'].size().unstack(fill_value=0)

pd.crosstab
```
pd.crosstab(df['row'], df['col'])
```

pd.factorize + np.bincount

# get integer factorization `i` and unique values `r`
# for column `'row'`
i, r = pd.factorize(df['row'].values)
# get integer factorization `j` and unique values `c`
# for column `'col'`
j, c = pd.factorize(df['col'].values)
# `n` will be the number of rows
# `m` will be the number of columns
n, m = r.size, c.size
# `i * m + j` is a clever way of counting the 
# factorization bins assuming a flat array of length
# `n * m`.  Which is why we subsequently reshape as `(n, m)`
b = np.bincount(i * m + j, minlength=n * m).reshape(n, m)
# BTW, whenever I read this, I think 'Bean, Rice, and Cheese'
pd.DataFrame(b, r, c)

      col3  col2  col0  col1  col4
row3     2     0     0     1     0
row2     1     2     1     0     2
row0     1     0     1     2     1
row4     2     2     0     1     1

pd.get_dummies

pd.get_dummies(df['row']).T.dot(pd.get_dummies(df['col']))

      col0  col1  col2  col3  col4
row0     1     2     0     1     1
row2     1     0     2     1     2
row3     0     1     0     2     0
row4     0     1     2     2     1

问题10

如何通过仅旋转两列将DataFrame从长转换为宽？

第一步是为每行分配一个数字-该数字将成为透视结果中该值的行索引。使用GroupBy.cumcount以下命令完成此操作：

df2.insert(0, 'count', df.groupby('A').cumcount())
df2

   count  A   B
0      0  a   0
1      1  a  11
2      2  a   2
3      3  a  11
4      0  b  10
5      1  b  10
6      2  b  14
7      0  c   7

第二步是使用新创建的列作为要调用的索引DataFrame.pivot。

df2.pivot(*df)
# df.pivot(index='count', columns='A', values='B')

A         a     b    c
count                 
0       0.0  10.0  7.0
1      11.0  10.0  NaN
2       2.0  14.0  NaN
3      11.0   NaN  NaN

问题11

我如何将多重索引展平为单索引 pivot

如果columns输入object字符串join

df.columns = df.columns.map('|'.join)

其他 format

df.columns = df.columns.map('{0[0]}|{0[1]}'.format)

We start by answering the first question:

Question 1

Why do I get ValueError: Index contains duplicate entries, cannot reshape

This occurs because pandas is attempting to reindex either a columns or index object with duplicate entries. There are varying methods to use that can perform a pivot. Some of them are not well suited to when there are duplicates of the keys in which it is being asked to pivot on. For example. Consider pd.DataFrame.pivot. I know there are duplicate entries that share the row and col values:

df.duplicated(['row', 'col']).any()

True

So when I pivot using

df.pivot(index='row', columns='col', values='val0')

I get the error mentioned above. In fact, I get the same error when I try to perform the same task with:

df.set_index(['row', 'col'])['val0'].unstack()

Here is a list of idioms we can use to pivot

pd.DataFrame.groupby + pd.DataFrame.unstack
- Good general approach for doing just about any type of pivot
- You specify all columns that will constitute the pivoted row levels and column levels in one group by. You follow that by selecting the remaining columns you want to aggregate and the function(s) you want to perform the aggregation. Finally, you unstack the levels that you want to be in the column index.
pd.DataFrame.pivot_table
- A glorified version of groupby with more intuitive API. For many people, this is the preferred approach. And is the intended approach by the developers.
- Specify row level, column levels, values to be aggregated, and function(s) to perform aggregations.
pd.DataFrame.set_index + pd.DataFrame.unstack
- Convenient and intuitive for some (myself included). Cannot handle duplicate grouped keys.
- Similar to the groupby paradigm, we specify all columns that will eventually be either row or column levels and set those to be the index. We then unstack the levels we want in the columns. If either the remaining index levels or column levels are not unique, this method will fail.
pd.DataFrame.pivot
- Very similar to set_index in that it shares the duplicate key limitation. The API is very limited as well. It only takes scalar values for index, columns, values.
- Similar to the pivot_table method in that we select rows, columns, and values on which to pivot. However, we cannot aggregate and if either rows or columns are not unique, this method will fail.
pd.crosstab
- This a specialized version of pivot_table and in it’s purest form is the most intuitive way to perform several tasks.
pd.factorize + np.bincount
- This is a highly advanced technique that is very obscure but is very fast. It cannot be used in all circumstances, but when it can be used and you are comfortable using it, you will reap the performance rewards.
pd.get_dummies + pd.DataFrame.dot
- I use this for cleverly performing cross tabulation.

Examples

What I’m going to do for each subsequent answer and question is to answer it using pd.DataFrame.pivot_table. Then I’ll provide alternatives to perform the same task.

Question 3

How do I pivot df such that the col values are columns, row values are the index, mean of val0 are the values, and missing values are 0?

pd.DataFrame.pivot_table

fill_value is not set by default. I tend to set it appropriately. In this case I set it to 0. Notice I skipped question 2 as it’s the same as this answer without the fill_value

aggfunc='mean' is the default and I didn’t have to set it. I included it to be explicit.

df.pivot_table(
    values='val0', index='row', columns='col',
    fill_value=0, aggfunc='mean')

col   col0   col1   col2   col3  col4
row                                  
row0  0.77  0.605  0.000  0.860  0.65
row2  0.13  0.000  0.395  0.500  0.25
row3  0.00  0.310  0.000  0.545  0.00
row4  0.00  0.100  0.395  0.760  0.24

pd.DataFrame.groupby

df.groupby(['row', 'col'])['val0'].mean().unstack(fill_value=0)

pd.crosstab

pd.crosstab(
    index=df['row'], columns=df['col'],
    values=df['val0'], aggfunc='mean').fillna(0)

Question 4

Can I get something other than mean, like maybe sum?

pd.DataFrame.pivot_table

df.pivot_table(
    values='val0', index='row', columns='col',
    fill_value=0, aggfunc='sum')

col   col0  col1  col2  col3  col4
row                               
row0  0.77  1.21  0.00  0.86  0.65
row2  0.13  0.00  0.79  0.50  0.50
row3  0.00  0.31  0.00  1.09  0.00
row4  0.00  0.10  0.79  1.52  0.24

pd.DataFrame.groupby

df.groupby(['row', 'col'])['val0'].sum().unstack(fill_value=0)

pd.crosstab

pd.crosstab(
    index=df['row'], columns=df['col'],
    values=df['val0'], aggfunc='sum').fillna(0)

Question 5

Can I do more that one aggregation at a time?

Notice that for pivot_table and crosstab I needed to pass list of callables. On the other hand, groupby.agg is able to take strings for a limited number of special functions. groupby.agg would also have taken the same callables we passed to the others, but it is often more efficient to leverage the string function names as there are efficiencies to be gained.

pd.DataFrame.pivot_table

df.pivot_table(
    values='val0', index='row', columns='col',
    fill_value=0, aggfunc=[np.size, np.mean])

     size                      mean                           
col  col0 col1 col2 col3 col4  col0   col1   col2   col3  col4
row                                                           
row0    1    2    0    1    1  0.77  0.605  0.000  0.860  0.65
row2    1    0    2    1    2  0.13  0.000  0.395  0.500  0.25
row3    0    1    0    2    0  0.00  0.310  0.000  0.545  0.00
row4    0    1    2    2    1  0.00  0.100  0.395  0.760  0.24

pd.DataFrame.groupby

df.groupby(['row', 'col'])['val0'].agg(['size', 'mean']).unstack(fill_value=0)

pd.crosstab

pd.crosstab(
    index=df['row'], columns=df['col'],
    values=df['val0'], aggfunc=[np.size, np.mean]).fillna(0, downcast='infer')

Question 6

Can I aggregate over multiple value columns?

pd.DataFrame.pivot_table we pass values=['val0', 'val1'] but we could’ve left that off completely

df.pivot_table(
    values=['val0', 'val1'], index='row', columns='col',
    fill_value=0, aggfunc='mean')

      val0                             val1                          
col   col0   col1   col2   col3  col4  col0   col1  col2   col3  col4
row                                                                  
row0  0.77  0.605  0.000  0.860  0.65  0.01  0.745  0.00  0.010  0.02
row2  0.13  0.000  0.395  0.500  0.25  0.45  0.000  0.34  0.440  0.79
row3  0.00  0.310  0.000  0.545  0.00  0.00  0.230  0.00  0.075  0.00
row4  0.00  0.100  0.395  0.760  0.24  0.00  0.070  0.42  0.300  0.46

pd.DataFrame.groupby

df.groupby(['row', 'col'])['val0', 'val1'].mean().unstack(fill_value=0)

Question 7

Can Subdivide by multiple columns?

pd.DataFrame.pivot_table

df.pivot_table(
    values='val0', index='row', columns=['item', 'col'],
    fill_value=0, aggfunc='mean')

item item0             item1                         item2                   
col   col2  col3  col4  col0  col1  col2  col3  col4  col0   col1  col3  col4
row                                                                          
row0  0.00  0.00  0.00  0.77  0.00  0.00  0.00  0.00  0.00  0.605  0.86  0.65
row2  0.35  0.00  0.37  0.00  0.00  0.44  0.00  0.00  0.13  0.000  0.50  0.13
row3  0.00  0.00  0.00  0.00  0.31  0.00  0.81  0.00  0.00  0.000  0.28  0.00
row4  0.15  0.64  0.00  0.00  0.10  0.64  0.88  0.24  0.00  0.000  0.00  0.00

pd.DataFrame.groupby

df.groupby(
    ['row', 'item', 'col']
)['val0'].mean().unstack(['item', 'col']).fillna(0).sort_index(1)

Question 8

Can Subdivide by multiple columns?

pd.DataFrame.pivot_table

df.pivot_table(
    values='val0', index=['key', 'row'], columns=['item', 'col'],
    fill_value=0, aggfunc='mean')

item      item0             item1                         item2                  
col        col2  col3  col4  col0  col1  col2  col3  col4  col0  col1  col3  col4
key  row                                                                         
key0 row0  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.86  0.00
     row2  0.00  0.00  0.37  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.50  0.00
     row3  0.00  0.00  0.00  0.00  0.31  0.00  0.81  0.00  0.00  0.00  0.00  0.00
     row4  0.15  0.64  0.00  0.00  0.00  0.00  0.00  0.24  0.00  0.00  0.00  0.00
key1 row0  0.00  0.00  0.00  0.77  0.00  0.00  0.00  0.00  0.00  0.81  0.00  0.65
     row2  0.35  0.00  0.00  0.00  0.00  0.44  0.00  0.00  0.00  0.00  0.00  0.13
     row3  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.28  0.00
     row4  0.00  0.00  0.00  0.00  0.10  0.00  0.00  0.00  0.00  0.00  0.00  0.00
key2 row0  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.40  0.00  0.00
     row2  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.00  0.13  0.00  0.00  0.00
     row4  0.00  0.00  0.00  0.00  0.00  0.64  0.88  0.00  0.00  0.00  0.00  0.00

pd.DataFrame.groupby

df.groupby(
    ['key', 'row', 'item', 'col']
)['val0'].mean().unstack(['item', 'col']).fillna(0).sort_index(1)

pd.DataFrame.set_index because the set of keys are unique for both rows and columns

df.set_index(
    ['key', 'row', 'item', 'col']
)['val0'].unstack(['item', 'col']).fillna(0).sort_index(1)

Question 9

Can I aggregate the frequency in which the column and rows occur together, aka “cross tabulation”?

pd.DataFrame.pivot_table

df.pivot_table(index='row', columns='col', fill_value=0, aggfunc='size')

    col   col0  col1  col2  col3  col4
row                               
row0     1     2     0     1     1
row2     1     0     2     1     2
row3     0     1     0     2     0
row4     0     1     2     2     1

pd.DataFrame.groupby

df.groupby(['row', 'col'])['val0'].size().unstack(fill_value=0)

pd.crosstab
```
pd.crosstab(df['row'], df['col'])
```

pd.factorize + np.bincount

# get integer factorization `i` and unique values `r`
# for column `'row'`
i, r = pd.factorize(df['row'].values)
# get integer factorization `j` and unique values `c`
# for column `'col'`
j, c = pd.factorize(df['col'].values)
# `n` will be the number of rows
# `m` will be the number of columns
n, m = r.size, c.size
# `i * m + j` is a clever way of counting the 
# factorization bins assuming a flat array of length
# `n * m`.  Which is why we subsequently reshape as `(n, m)`
b = np.bincount(i * m + j, minlength=n * m).reshape(n, m)
# BTW, whenever I read this, I think 'Bean, Rice, and Cheese'
pd.DataFrame(b, r, c)

      col3  col2  col0  col1  col4
row3     2     0     0     1     0
row2     1     2     1     0     2
row0     1     0     1     2     1
row4     2     2     0     1     1

pd.get_dummies

pd.get_dummies(df['row']).T.dot(pd.get_dummies(df['col']))

      col0  col1  col2  col3  col4
row0     1     2     0     1     1
row2     1     0     2     1     2
row3     0     1     0     2     0
row4     0     1     2     2     1

Question 10

How do I convert a DataFrame from long to wide by pivoting on ONLY two columns?

The first step is to assign a number to each row – this number will be the row index of that value in the pivoted result. This is done using GroupBy.cumcount:

df2.insert(0, 'count', df.groupby('A').cumcount())
df2

   count  A   B
0      0  a   0
1      1  a  11
2      2  a   2
3      3  a  11
4      0  b  10
5      1  b  10
6      2  b  14
7      0  c   7

The second step is to use the newly created column as the index to call DataFrame.pivot.

df2.pivot(*df)
# df.pivot(index='count', columns='A', values='B')

A         a     b    c
count                 
0       0.0  10.0  7.0
1      11.0  10.0  NaN
2       2.0  14.0  NaN
3      11.0   NaN  NaN

Question 11

How do I flatten the multiple index to single index after pivot

If columns type object with string join

df.columns = df.columns.map('|'.join)

else format

df.columns = df.columns.map('{0[0]}|{0[1]}'.format)

回答 1

扩展@piRSquared的答案的另一个版本的问题10

问题10.1

数据框：

d = data = {'A': {0: 1, 1: 1, 2: 1, 3: 2, 4: 2, 5: 3, 6: 5},
 'B': {0: 'a', 1: 'b', 2: 'c', 3: 'a', 4: 'b', 5: 'a', 6: 'c'}}
df = pd.DataFrame(d)

   A  B
0  1  a
1  1  b
2  1  c
3  2  a
4  2  b
5  3  a
6  5  c

输出：

   0     1     2
A
1  a     b     c
2  a     b  None
3  a  None  None
5  c  None  None

使用df.groupby和pd.Series.tolist

t = df.groupby('A')['B'].apply(list)
out = pd.DataFrame(t.tolist(),index=t.index)
out
   0     1     2
A
1  a     b     c
2  a     b  None
3  a  None  None
5  c  None  None

或者使用更好的替代方法 pd.pivot_table与df.squeeze.

t = df.pivot_table(index='A',values='B',aggfunc=list).squeeze()
out = pd.DataFrame(t.tolist(),index=t.index)

To extend @piRSquared’s answer another version of Question 10

Question 10.1

DataFrame:

d = data = {'A': {0: 1, 1: 1, 2: 1, 3: 2, 4: 2, 5: 3, 6: 5},
 'B': {0: 'a', 1: 'b', 2: 'c', 3: 'a', 4: 'b', 5: 'a', 6: 'c'}}
df = pd.DataFrame(d)

   A  B
0  1  a
1  1  b
2  1  c
3  2  a
4  2  b
5  3  a
6  5  c

Output:

   0     1     2
A
1  a     b     c
2  a     b  None
3  a  None  None
5  c  None  None

Using df.groupby and pd.Series.tolist

t = df.groupby('A')['B'].apply(list)
out = pd.DataFrame(t.tolist(),index=t.index)
out
   0     1     2
A
1  a     b     c
2  a     b  None
3  a  None  None
5  c  None  None

Or A much better alternative using pd.pivot_table with df.squeeze.

t = df.pivot_table(index='A',values='B',aggfunc=list).squeeze()
out = pd.DataFrame(t.tolist(),index=t.index)

知识问答

列出N以下所有素数的最快方法

2021年7月27日 Python实用宝典

问题：列出N以下所有素数的最快方法

这是我能想到的最好的算法。

def get_primes(n):
    numbers = set(range(n, 1, -1))
    primes = []
    while numbers:
        p = numbers.pop()
        primes.append(p)
        numbers.difference_update(set(range(p*2, n+1, p)))
    return primes

>>> timeit.Timer(stmt='get_primes.get_primes(1000000)', setup='import   get_primes').timeit(1)
1.1499958793645562

可以使它更快吗？

此代码有一个缺陷：由于numbers是无序集合，因此不能保证numbers.pop()从集合中删除最低的数字。但是，它对某些输入数字有效（至少对我而言）：

>>> sum(get_primes(2000000))
142913828922L
#That's the correct sum of all numbers below 2 million
>>> 529 in get_primes(1000)
False
>>> 529 in get_primes(530)
True

This is the best algorithm I could come up.

def get_primes(n):
    numbers = set(range(n, 1, -1))
    primes = []
    while numbers:
        p = numbers.pop()
        primes.append(p)
        numbers.difference_update(set(range(p*2, n+1, p)))
    return primes

>>> timeit.Timer(stmt='get_primes.get_primes(1000000)', setup='import   get_primes').timeit(1)
1.1499958793645562

Can it be made even faster?

This code has a flaw: Since numbers is an unordered set, there is no guarantee that numbers.pop() will remove the lowest number from the set. Nevertheless, it works (at least for me) for some input numbers:

>>> sum(get_primes(2000000))
142913828922L
#That's the correct sum of all numbers below 2 million
>>> 529 in get_primes(1000)
False
>>> 529 in get_primes(530)
True

回答 0

警告： timeit由于硬件或Python版本的差异，结果可能会有所不同。

下面是一个脚本，比较了许多实现：

ambi_sieve_plain，
rwh_primes，
rwh_primes1，
rwh_primes2，
sieveOfAtkin，
埃拉托斯特尼筛法，
孙达拉姆3，
sieve_wheel_30，
ambi_sieve（需要numpy）
primesfrom3to（需要numpy）
primesfrom2to（需要numpy）

非常感谢斯蒂芬为使sieve_wheel_30引起我的注意。幸得罗伯特·威廉·汉克斯为primesfrom2to，primesfrom3to，rwh_primes，rwh_primes1和rwh_primes2。

在使用psyco测试的简单Python方法中，对于n = 1000000， rwh_primes1是测试最快的方法。

+---------------------+-------+
| Method              | ms    |
+---------------------+-------+
| rwh_primes1         | 43.0  |
| sieveOfAtkin        | 46.4  |
| rwh_primes          | 57.4  |
| sieve_wheel_30      | 63.0  |
| rwh_primes2         | 67.8  |    
| sieveOfEratosthenes | 147.0 |
| ambi_sieve_plain    | 152.0 |
| sundaram3           | 194.0 |
+---------------------+-------+

在没有psyco的情况下经过测试的普通Python方法中，对于n = 1000000， rwh_primes2是最快的。

+---------------------+-------+
| Method              | ms    |
+---------------------+-------+
| rwh_primes2         | 68.1  |
| rwh_primes1         | 93.7  |
| rwh_primes          | 94.6  |
| sieve_wheel_30      | 97.4  |
| sieveOfEratosthenes | 178.0 |
| ambi_sieve_plain    | 286.0 |
| sieveOfAtkin        | 314.0 |
| sundaram3           | 416.0 |
+---------------------+-------+

在所有测试的方法中，允许numpy，对于n = 1000000， primesfrom2to是测试最快的方法。

+---------------------+-------+
| Method              | ms    |
+---------------------+-------+
| primesfrom2to       | 15.9  |
| primesfrom3to       | 18.4  |
| ambi_sieve          | 29.3  |
+---------------------+-------+

使用以下命令测量时间：

python -mtimeit -s"import primes" "primes.{method}(1000000)"

用{method}每个方法名称替换。

primes.py：

#!/usr/bin/env python
import psyco; psyco.full()
from math import sqrt, ceil
import numpy as np

def rwh_primes(n):
    # /programming/2068372/fastest-way-to-list-all-primes-below-n-in-python/3035188#3035188
    """ Returns  a list of primes < n """
    sieve = [True] * n
    for i in xrange(3,int(n**0.5)+1,2):
        if sieve[i]:
            sieve[i*i::2*i]=[False]*((n-i*i-1)/(2*i)+1)
    return [2] + [i for i in xrange(3,n,2) if sieve[i]]

def rwh_primes1(n):
    # /programming/2068372/fastest-way-to-list-all-primes-below-n-in-python/3035188#3035188
    """ Returns  a list of primes < n """
    sieve = [True] * (n/2)
    for i in xrange(3,int(n**0.5)+1,2):
        if sieve[i/2]:
            sieve[i*i/2::i] = [False] * ((n-i*i-1)/(2*i)+1)
    return [2] + [2*i+1 for i in xrange(1,n/2) if sieve[i]]

def rwh_primes2(n):
    # /programming/2068372/fastest-way-to-list-all-primes-below-n-in-python/3035188#3035188
    """ Input n>=6, Returns a list of primes, 2 <= p < n """
    correction = (n%6>1)
    n = {0:n,1:n-1,2:n+4,3:n+3,4:n+2,5:n+1}[n%6]
    sieve = [True] * (n/3)
    sieve[0] = False
    for i in xrange(int(n**0.5)/3+1):
      if sieve[i]:
        k=3*i+1|1
        sieve[      ((k*k)/3)      ::2*k]=[False]*((n/6-(k*k)/6-1)/k+1)
        sieve[(k*k+4*k-2*k*(i&1))/3::2*k]=[False]*((n/6-(k*k+4*k-2*k*(i&1))/6-1)/k+1)
    return [2,3] + [3*i+1|1 for i in xrange(1,n/3-correction) if sieve[i]]

def sieve_wheel_30(N):
    # http://zerovolt.com/?p=88
    ''' Returns a list of primes <= N using wheel criterion 2*3*5 = 30

Copyright 2009 by zerovolt.com
This code is free for non-commercial purposes, in which case you can just leave this comment as a credit for my work.
If you need this code for commercial purposes, please contact me by sending an email to: info [at] zerovolt [dot] com.'''
    __smallp = ( 2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59,
    61, 67, 71, 73, 79, 83, 89, 97, 101, 103, 107, 109, 113, 127, 131, 137, 139,
    149, 151, 157, 163, 167, 173, 179, 181, 191, 193, 197, 199, 211, 223, 227,
    229, 233, 239, 241, 251, 257, 263, 269, 271, 277, 281, 283, 293, 307, 311,
    313, 317, 331, 337, 347, 349, 353, 359, 367, 373, 379, 383, 389, 397, 401,
    409, 419, 421, 431, 433, 439, 443, 449, 457, 461, 463, 467, 479, 487, 491,
    499, 503, 509, 521, 523, 541, 547, 557, 563, 569, 571, 577, 587, 593, 599,
    601, 607, 613, 617, 619, 631, 641, 643, 647, 653, 659, 661, 673, 677, 683,
    691, 701, 709, 719, 727, 733, 739, 743, 751, 757, 761, 769, 773, 787, 797,
    809, 811, 821, 823, 827, 829, 839, 853, 857, 859, 863, 877, 881, 883, 887,
    907, 911, 919, 929, 937, 941, 947, 953, 967, 971, 977, 983, 991, 997)

    wheel = (2, 3, 5)
    const = 30
    if N < 2:
        return []
    if N <= const:
        pos = 0
        while __smallp[pos] <= N:
            pos += 1
        return list(__smallp[:pos])
    # make the offsets list
    offsets = (7, 11, 13, 17, 19, 23, 29, 1)
    # prepare the list
    p = [2, 3, 5]
    dim = 2 + N // const
    tk1  = [True] * dim
    tk7  = [True] * dim
    tk11 = [True] * dim
    tk13 = [True] * dim
    tk17 = [True] * dim
    tk19 = [True] * dim
    tk23 = [True] * dim
    tk29 = [True] * dim
    tk1[0] = False
    # help dictionary d
    # d[a , b] = c  ==> if I want to find the smallest useful multiple of (30*pos)+a
    # on tkc, then I need the index given by the product of [(30*pos)+a][(30*pos)+b]
    # in general. If b < a, I need [(30*pos)+a][(30*(pos+1))+b]
    d = {}
    for x in offsets:
        for y in offsets:
            res = (x*y) % const
            if res in offsets:
                d[(x, res)] = y
    # another help dictionary: gives tkx calling tmptk[x]
    tmptk = {1:tk1, 7:tk7, 11:tk11, 13:tk13, 17:tk17, 19:tk19, 23:tk23, 29:tk29}
    pos, prime, lastadded, stop = 0, 0, 0, int(ceil(sqrt(N)))
    # inner functions definition
    def del_mult(tk, start, step):
        for k in xrange(start, len(tk), step):
            tk[k] = False
    # end of inner functions definition
    cpos = const * pos
    while prime < stop:
        # 30k + 7
        if tk7[pos]:
            prime = cpos + 7
            p.append(prime)
            lastadded = 7
            for off in offsets:
                tmp = d[(7, off)]
                start = (pos + prime) if off == 7 else (prime * (const * (pos + 1 if tmp < 7 else 0) + tmp) )//const
                del_mult(tmptk[off], start, prime)
        # 30k + 11
        if tk11[pos]:
            prime = cpos + 11
            p.append(prime)
            lastadded = 11
            for off in offsets:
                tmp = d[(11, off)]
                start = (pos + prime) if off == 11 else (prime * (const * (pos + 1 if tmp < 11 else 0) + tmp) )//const
                del_mult(tmptk[off], start, prime)
        # 30k + 13
        if tk13[pos]:
            prime = cpos + 13
            p.append(prime)
            lastadded = 13
            for off in offsets:
                tmp = d[(13, off)]
                start = (pos + prime) if off == 13 else (prime * (const * (pos + 1 if tmp < 13 else 0) + tmp) )//const
                del_mult(tmptk[off], start, prime)
        # 30k + 17
        if tk17[pos]:
            prime = cpos + 17
            p.append(prime)
            lastadded = 17
            for off in offsets:
                tmp = d[(17, off)]
                start = (pos + prime) if off == 17 else (prime * (const * (pos + 1 if tmp < 17 else 0) + tmp) )//const
                del_mult(tmptk[off], start, prime)
        # 30k + 19
        if tk19[pos]:
            prime = cpos + 19
            p.append(prime)
            lastadded = 19
            for off in offsets:
                tmp = d[(19, off)]
                start = (pos + prime) if off == 19 else (prime * (const * (pos + 1 if tmp < 19 else 0) + tmp) )//const
                del_mult(tmptk[off], start, prime)
        # 30k + 23
        if tk23[pos]:
            prime = cpos + 23
            p.append(prime)
            lastadded = 23
            for off in offsets:
                tmp = d[(23, off)]
                start = (pos + prime) if off == 23 else (prime * (const * (pos + 1 if tmp < 23 else 0) + tmp) )//const
                del_mult(tmptk[off], start, prime)
        # 30k + 29
        if tk29[pos]:
            prime = cpos + 29
            p.append(prime)
            lastadded = 29
            for off in offsets:
                tmp = d[(29, off)]
                start = (pos + prime) if off == 29 else (prime * (const * (pos + 1 if tmp < 29 else 0) + tmp) )//const
                del_mult(tmptk[off], start, prime)
        # now we go back to top tk1, so we need to increase pos by 1
        pos += 1
        cpos = const * pos
        # 30k + 1
        if tk1[pos]:
            prime = cpos + 1
            p.append(prime)
            lastadded = 1
            for off in offsets:
                tmp = d[(1, off)]
                start = (pos + prime) if off == 1 else (prime * (const * pos + tmp) )//const
                del_mult(tmptk[off], start, prime)
    # time to add remaining primes
    # if lastadded == 1, remove last element and start adding them from tk1
    # this way we don't need an "if" within the last while
    if lastadded == 1:
        p.pop()
    # now complete for every other possible prime
    while pos < len(tk1):
        cpos = const * pos
        if tk1[pos]: p.append(cpos + 1)
        if tk7[pos]: p.append(cpos + 7)
        if tk11[pos]: p.append(cpos + 11)
        if tk13[pos]: p.append(cpos + 13)
        if tk17[pos]: p.append(cpos + 17)
        if tk19[pos]: p.append(cpos + 19)
        if tk23[pos]: p.append(cpos + 23)
        if tk29[pos]: p.append(cpos + 29)
        pos += 1
    # remove exceeding if present
    pos = len(p) - 1
    while p[pos] > N:
        pos -= 1
    if pos < len(p) - 1:
        del p[pos+1:]
    # return p list
    return p

def sieveOfEratosthenes(n):
    """sieveOfEratosthenes(n): return the list of the primes < n."""
    # Code from: <dickinsm@gmail.com>, Nov 30 2006
    # http://groups.google.com/group/comp.lang.python/msg/f1f10ced88c68c2d
    if n <= 2:
        return []
    sieve = range(3, n, 2)
    top = len(sieve)
    for si in sieve:
        if si:
            bottom = (si*si - 3) // 2
            if bottom >= top:
                break
            sieve[bottom::si] = [0] * -((bottom - top) // si)
    return [2] + [el for el in sieve if el]

def sieveOfAtkin(end):
    """sieveOfAtkin(end): return a list of all the prime numbers <end
    using the Sieve of Atkin."""
    # Code by Steve Krenzel, <Sgk284@gmail.com>, improved
    # Code: https://web.archive.org/web/20080324064651/http://krenzel.info/?p=83
    # Info: http://en.wikipedia.org/wiki/Sieve_of_Atkin
    assert end > 0
    lng = ((end-1) // 2)
    sieve = [False] * (lng + 1)

    x_max, x2, xd = int(sqrt((end-1)/4.0)), 0, 4
    for xd in xrange(4, 8*x_max + 2, 8):
        x2 += xd
        y_max = int(sqrt(end-x2))
        n, n_diff = x2 + y_max*y_max, (y_max << 1) - 1
        if not (n & 1):
            n -= n_diff
            n_diff -= 2
        for d in xrange((n_diff - 1) << 1, -1, -8):
            m = n % 12
            if m == 1 or m == 5:
                m = n >> 1
                sieve[m] = not sieve[m]
            n -= d

    x_max, x2, xd = int(sqrt((end-1) / 3.0)), 0, 3
    for xd in xrange(3, 6 * x_max + 2, 6):
        x2 += xd
        y_max = int(sqrt(end-x2))
        n, n_diff = x2 + y_max*y_max, (y_max << 1) - 1
        if not(n & 1):
            n -= n_diff
            n_diff -= 2
        for d in xrange((n_diff - 1) << 1, -1, -8):
            if n % 12 == 7:
                m = n >> 1
                sieve[m] = not sieve[m]
            n -= d

    x_max, y_min, x2, xd = int((2 + sqrt(4-8*(1-end)))/4), -1, 0, 3
    for x in xrange(1, x_max + 1):
        x2 += xd
        xd += 6
        if x2 >= end: y_min = (((int(ceil(sqrt(x2 - end))) - 1) << 1) - 2) << 1
        n, n_diff = ((x*x + x) << 1) - 1, (((x-1) << 1) - 2) << 1
        for d in xrange(n_diff, y_min, -8):
            if n % 12 == 11:
                m = n >> 1
                sieve[m] = not sieve[m]
            n += d

    primes = [2, 3]
    if end <= 3:
        return primes[:max(0,end-2)]

    for n in xrange(5 >> 1, (int(sqrt(end))+1) >> 1):
        if sieve[n]:
            primes.append((n << 1) + 1)
            aux = (n << 1) + 1
            aux *= aux
            for k in xrange(aux, end, 2 * aux):
                sieve[k >> 1] = False

    s  = int(sqrt(end)) + 1
    if s  % 2 == 0:
        s += 1
    primes.extend([i for i in xrange(s, end, 2) if sieve[i >> 1]])

    return primes

def ambi_sieve_plain(n):
    s = range(3, n, 2)
    for m in xrange(3, int(n**0.5)+1, 2): 
        if s[(m-3)/2]: 
            for t in xrange((m*m-3)/2,(n>>1)-1,m):
                s[t]=0
    return [2]+[t for t in s if t>0]

def sundaram3(max_n):
    # /programming/2068372/fastest-way-to-list-all-primes-below-n-in-python/2073279#2073279
    numbers = range(3, max_n+1, 2)
    half = (max_n)//2
    initial = 4

    for step in xrange(3, max_n+1, 2):
        for i in xrange(initial, half, step):
            numbers[i-1] = 0
        initial += 2*(step+1)

        if initial > half:
            return [2] + filter(None, numbers)

################################################################################
# Using Numpy:
def ambi_sieve(n):
    # http://tommih.blogspot.com/2009/04/fast-prime-number-generator.html
    s = np.arange(3, n, 2)
    for m in xrange(3, int(n ** 0.5)+1, 2): 
        if s[(m-3)/2]: 
            s[(m*m-3)/2::m]=0
    return np.r_[2, s[s>0]]

def primesfrom3to(n):
    # /programming/2068372/fastest-way-to-list-all-primes-below-n-in-python/3035188#3035188
    """ Returns a array of primes, p < n """
    assert n>=2
    sieve = np.ones(n/2, dtype=np.bool)
    for i in xrange(3,int(n**0.5)+1,2):
        if sieve[i/2]:
            sieve[i*i/2::i] = False
    return np.r_[2, 2*np.nonzero(sieve)[0][1::]+1]    

def primesfrom2to(n):
    # /programming/2068372/fastest-way-to-list-all-primes-below-n-in-python/3035188#3035188
    """ Input n>=6, Returns a array of primes, 2 <= p < n """
    sieve = np.ones(n/3 + (n%6==2), dtype=np.bool)
    sieve[0] = False
    for i in xrange(int(n**0.5)/3+1):
        if sieve[i]:
            k=3*i+1|1
            sieve[      ((k*k)/3)      ::2*k] = False
            sieve[(k*k+4*k-2*k*(i&1))/3::2*k] = False
    return np.r_[2,3,((3*np.nonzero(sieve)[0]+1)|1)]

if __name__=='__main__':
    import itertools
    import sys

    def test(f1,f2,num):
        print('Testing {f1} and {f2} return same results'.format(
            f1=f1.func_name,
            f2=f2.func_name))
        if not all([a==b for a,b in itertools.izip_longest(f1(num),f2(num))]):
            sys.exit("Error: %s(%s) != %s(%s)"%(f1.func_name,num,f2.func_name,num))

    n=1000000
    test(sieveOfAtkin,sieveOfEratosthenes,n)
    test(sieveOfAtkin,ambi_sieve,n)
    test(sieveOfAtkin,ambi_sieve_plain,n) 
    test(sieveOfAtkin,sundaram3,n)
    test(sieveOfAtkin,sieve_wheel_30,n)
    test(sieveOfAtkin,primesfrom3to,n)
    test(sieveOfAtkin,primesfrom2to,n)
    test(sieveOfAtkin,rwh_primes,n)
    test(sieveOfAtkin,rwh_primes1,n)         
    test(sieveOfAtkin,rwh_primes2,n)

运行脚本会测试所有实现都给出相同的结果。

Warning: timeit results may vary due to differences in hardware or version of Python.

Below is a script which compares a number of implementations:

ambi_sieve_plain,
rwh_primes,
rwh_primes1,
rwh_primes2,
sieveOfAtkin,
sieveOfEratosthenes,
sundaram3,
sieve_wheel_30,
ambi_sieve (requires numpy)
primesfrom3to (requires numpy)
primesfrom2to (requires numpy)

Many thanks to stephan for bringing sieve_wheel_30 to my attention. Credit goes to Robert William Hanks for primesfrom2to, primesfrom3to, rwh_primes, rwh_primes1, and rwh_primes2.

Of the plain Python methods tested, with psyco, for n=1000000, rwh_primes1 was the fastest tested.

+---------------------+-------+
| Method              | ms    |
+---------------------+-------+
| rwh_primes1         | 43.0  |
| sieveOfAtkin        | 46.4  |
| rwh_primes          | 57.4  |
| sieve_wheel_30      | 63.0  |
| rwh_primes2         | 67.8  |    
| sieveOfEratosthenes | 147.0 |
| ambi_sieve_plain    | 152.0 |
| sundaram3           | 194.0 |
+---------------------+-------+

Of the plain Python methods tested, without psyco, for n=1000000, rwh_primes2 was the fastest.

+---------------------+-------+
| Method              | ms    |
+---------------------+-------+
| rwh_primes2         | 68.1  |
| rwh_primes1         | 93.7  |
| rwh_primes          | 94.6  |
| sieve_wheel_30      | 97.4  |
| sieveOfEratosthenes | 178.0 |
| ambi_sieve_plain    | 286.0 |
| sieveOfAtkin        | 314.0 |
| sundaram3           | 416.0 |
+---------------------+-------+

Of all the methods tested, allowing numpy, for n=1000000, primesfrom2to was the fastest tested.

+---------------------+-------+
| Method              | ms    |
+---------------------+-------+
| primesfrom2to       | 15.9  |
| primesfrom3to       | 18.4  |
| ambi_sieve          | 29.3  |
+---------------------+-------+

Timings were measured using the command:

python -mtimeit -s"import primes" "primes.{method}(1000000)"

with {method} replaced by each of the method names.

primes.py:

#!/usr/bin/env python
import psyco; psyco.full()
from math import sqrt, ceil
import numpy as np

def rwh_primes(n):
    # https://stackoverflow.com/questions/2068372/fastest-way-to-list-all-primes-below-n-in-python/3035188#3035188
    """ Returns  a list of primes < n """
    sieve = [True] * n
    for i in xrange(3,int(n**0.5)+1,2):
        if sieve[i]:
            sieve[i*i::2*i]=[False]*((n-i*i-1)/(2*i)+1)
    return [2] + [i for i in xrange(3,n,2) if sieve[i]]

def rwh_primes1(n):
    # https://stackoverflow.com/questions/2068372/fastest-way-to-list-all-primes-below-n-in-python/3035188#3035188
    """ Returns  a list of primes < n """
    sieve = [True] * (n/2)
    for i in xrange(3,int(n**0.5)+1,2):
        if sieve[i/2]:
            sieve[i*i/2::i] = [False] * ((n-i*i-1)/(2*i)+1)
    return [2] + [2*i+1 for i in xrange(1,n/2) if sieve[i]]

def rwh_primes2(n):
    # https://stackoverflow.com/questions/2068372/fastest-way-to-list-all-primes-below-n-in-python/3035188#3035188
    """ Input n>=6, Returns a list of primes, 2 <= p < n """
    correction = (n%6>1)
    n = {0:n,1:n-1,2:n+4,3:n+3,4:n+2,5:n+1}[n%6]
    sieve = [True] * (n/3)
    sieve[0] = False
    for i in xrange(int(n**0.5)/3+1):
      if sieve[i]:
        k=3*i+1|1
        sieve[      ((k*k)/3)      ::2*k]=[False]*((n/6-(k*k)/6-1)/k+1)
        sieve[(k*k+4*k-2*k*(i&1))/3::2*k]=[False]*((n/6-(k*k+4*k-2*k*(i&1))/6-1)/k+1)
    return [2,3] + [3*i+1|1 for i in xrange(1,n/3-correction) if sieve[i]]

def sieve_wheel_30(N):
    # http://zerovolt.com/?p=88
    ''' Returns a list of primes <= N using wheel criterion 2*3*5 = 30

Copyright 2009 by zerovolt.com
This code is free for non-commercial purposes, in which case you can just leave this comment as a credit for my work.
If you need this code for commercial purposes, please contact me by sending an email to: info [at] zerovolt [dot] com.'''
    __smallp = ( 2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59,
    61, 67, 71, 73, 79, 83, 89, 97, 101, 103, 107, 109, 113, 127, 131, 137, 139,
    149, 151, 157, 163, 167, 173, 179, 181, 191, 193, 197, 199, 211, 223, 227,
    229, 233, 239, 241, 251, 257, 263, 269, 271, 277, 281, 283, 293, 307, 311,
    313, 317, 331, 337, 347, 349, 353, 359, 367, 373, 379, 383, 389, 397, 401,
    409, 419, 421, 431, 433, 439, 443, 449, 457, 461, 463, 467, 479, 487, 491,
    499, 503, 509, 521, 523, 541, 547, 557, 563, 569, 571, 577, 587, 593, 599,
    601, 607, 613, 617, 619, 631, 641, 643, 647, 653, 659, 661, 673, 677, 683,
    691, 701, 709, 719, 727, 733, 739, 743, 751, 757, 761, 769, 773, 787, 797,
    809, 811, 821, 823, 827, 829, 839, 853, 857, 859, 863, 877, 881, 883, 887,
    907, 911, 919, 929, 937, 941, 947, 953, 967, 971, 977, 983, 991, 997)

    wheel = (2, 3, 5)
    const = 30
    if N < 2:
        return []
    if N <= const:
        pos = 0
        while __smallp[pos] <= N:
            pos += 1
        return list(__smallp[:pos])
    # make the offsets list
    offsets = (7, 11, 13, 17, 19, 23, 29, 1)
    # prepare the list
    p = [2, 3, 5]
    dim = 2 + N // const
    tk1  = [True] * dim
    tk7  = [True] * dim
    tk11 = [True] * dim
    tk13 = [True] * dim
    tk17 = [True] * dim
    tk19 = [True] * dim
    tk23 = [True] * dim
    tk29 = [True] * dim
    tk1[0] = False
    # help dictionary d
    # d[a , b] = c  ==> if I want to find the smallest useful multiple of (30*pos)+a
    # on tkc, then I need the index given by the product of [(30*pos)+a][(30*pos)+b]
    # in general. If b < a, I need [(30*pos)+a][(30*(pos+1))+b]
    d = {}
    for x in offsets:
        for y in offsets:
            res = (x*y) % const
            if res in offsets:
                d[(x, res)] = y
    # another help dictionary: gives tkx calling tmptk[x]
    tmptk = {1:tk1, 7:tk7, 11:tk11, 13:tk13, 17:tk17, 19:tk19, 23:tk23, 29:tk29}
    pos, prime, lastadded, stop = 0, 0, 0, int(ceil(sqrt(N)))
    # inner functions definition
    def del_mult(tk, start, step):
        for k in xrange(start, len(tk), step):
            tk[k] = False
    # end of inner functions definition
    cpos = const * pos
    while prime < stop:
        # 30k + 7
        if tk7[pos]:
            prime = cpos + 7
            p.append(prime)
            lastadded = 7
            for off in offsets:
                tmp = d[(7, off)]
                start = (pos + prime) if off == 7 else (prime * (const * (pos + 1 if tmp < 7 else 0) + tmp) )//const
                del_mult(tmptk[off], start, prime)
        # 30k + 11
        if tk11[pos]:
            prime = cpos + 11
            p.append(prime)
            lastadded = 11
            for off in offsets:
                tmp = d[(11, off)]
                start = (pos + prime) if off == 11 else (prime * (const * (pos + 1 if tmp < 11 else 0) + tmp) )//const
                del_mult(tmptk[off], start, prime)
        # 30k + 13
        if tk13[pos]:
            prime = cpos + 13
            p.append(prime)
            lastadded = 13
            for off in offsets:
                tmp = d[(13, off)]
                start = (pos + prime) if off == 13 else (prime * (const * (pos + 1 if tmp < 13 else 0) + tmp) )//const
                del_mult(tmptk[off], start, prime)
        # 30k + 17
        if tk17[pos]:
            prime = cpos + 17
            p.append(prime)
            lastadded = 17
            for off in offsets:
                tmp = d[(17, off)]
                start = (pos + prime) if off == 17 else (prime * (const * (pos + 1 if tmp < 17 else 0) + tmp) )//const
                del_mult(tmptk[off], start, prime)
        # 30k + 19
        if tk19[pos]:
            prime = cpos + 19
            p.append(prime)
            lastadded = 19
            for off in offsets:
                tmp = d[(19, off)]
                start = (pos + prime) if off == 19 else (prime * (const * (pos + 1 if tmp < 19 else 0) + tmp) )//const
                del_mult(tmptk[off], start, prime)
        # 30k + 23
        if tk23[pos]:
            prime = cpos + 23
            p.append(prime)
            lastadded = 23
            for off in offsets:
                tmp = d[(23, off)]
                start = (pos + prime) if off == 23 else (prime * (const * (pos + 1 if tmp < 23 else 0) + tmp) )//const
                del_mult(tmptk[off], start, prime)
        # 30k + 29
        if tk29[pos]:
            prime = cpos + 29
            p.append(prime)
            lastadded = 29
            for off in offsets:
                tmp = d[(29, off)]
                start = (pos + prime) if off == 29 else (prime * (const * (pos + 1 if tmp < 29 else 0) + tmp) )//const
                del_mult(tmptk[off], start, prime)
        # now we go back to top tk1, so we need to increase pos by 1
        pos += 1
        cpos = const * pos
        # 30k + 1
        if tk1[pos]:
            prime = cpos + 1
            p.append(prime)
            lastadded = 1
            for off in offsets:
                tmp = d[(1, off)]
                start = (pos + prime) if off == 1 else (prime * (const * pos + tmp) )//const
                del_mult(tmptk[off], start, prime)
    # time to add remaining primes
    # if lastadded == 1, remove last element and start adding them from tk1
    # this way we don't need an "if" within the last while
    if lastadded == 1:
        p.pop()
    # now complete for every other possible prime
    while pos < len(tk1):
        cpos = const * pos
        if tk1[pos]: p.append(cpos + 1)
        if tk7[pos]: p.append(cpos + 7)
        if tk11[pos]: p.append(cpos + 11)
        if tk13[pos]: p.append(cpos + 13)
        if tk17[pos]: p.append(cpos + 17)
        if tk19[pos]: p.append(cpos + 19)
        if tk23[pos]: p.append(cpos + 23)
        if tk29[pos]: p.append(cpos + 29)
        pos += 1
    # remove exceeding if present
    pos = len(p) - 1
    while p[pos] > N:
        pos -= 1
    if pos < len(p) - 1:
        del p[pos+1:]
    # return p list
    return p

def sieveOfEratosthenes(n):
    """sieveOfEratosthenes(n): return the list of the primes < n."""
    # Code from: <dickinsm@gmail.com>, Nov 30 2006
    # http://groups.google.com/group/comp.lang.python/msg/f1f10ced88c68c2d
    if n <= 2:
        return []
    sieve = range(3, n, 2)
    top = len(sieve)
    for si in sieve:
        if si:
            bottom = (si*si - 3) // 2
            if bottom >= top:
                break
            sieve[bottom::si] = [0] * -((bottom - top) // si)
    return [2] + [el for el in sieve if el]

def sieveOfAtkin(end):
    """sieveOfAtkin(end): return a list of all the prime numbers <end
    using the Sieve of Atkin."""
    # Code by Steve Krenzel, <Sgk284@gmail.com>, improved
    # Code: https://web.archive.org/web/20080324064651/http://krenzel.info/?p=83
    # Info: http://en.wikipedia.org/wiki/Sieve_of_Atkin
    assert end > 0
    lng = ((end-1) // 2)
    sieve = [False] * (lng + 1)

    x_max, x2, xd = int(sqrt((end-1)/4.0)), 0, 4
    for xd in xrange(4, 8*x_max + 2, 8):
        x2 += xd
        y_max = int(sqrt(end-x2))
        n, n_diff = x2 + y_max*y_max, (y_max << 1) - 1
        if not (n & 1):
            n -= n_diff
            n_diff -= 2
        for d in xrange((n_diff - 1) << 1, -1, -8):
            m = n % 12
            if m == 1 or m == 5:
                m = n >> 1
                sieve[m] = not sieve[m]
            n -= d

    x_max, x2, xd = int(sqrt((end-1) / 3.0)), 0, 3
    for xd in xrange(3, 6 * x_max + 2, 6):
        x2 += xd
        y_max = int(sqrt(end-x2))
        n, n_diff = x2 + y_max*y_max, (y_max << 1) - 1
        if not(n & 1):
            n -= n_diff
            n_diff -= 2
        for d in xrange((n_diff - 1) << 1, -1, -8):
            if n % 12 == 7:
                m = n >> 1
                sieve[m] = not sieve[m]
            n -= d

    x_max, y_min, x2, xd = int((2 + sqrt(4-8*(1-end)))/4), -1, 0, 3
    for x in xrange(1, x_max + 1):
        x2 += xd
        xd += 6
        if x2 >= end: y_min = (((int(ceil(sqrt(x2 - end))) - 1) << 1) - 2) << 1
        n, n_diff = ((x*x + x) << 1) - 1, (((x-1) << 1) - 2) << 1
        for d in xrange(n_diff, y_min, -8):
            if n % 12 == 11:
                m = n >> 1
                sieve[m] = not sieve[m]
            n += d

    primes = [2, 3]
    if end <= 3:
        return primes[:max(0,end-2)]

    for n in xrange(5 >> 1, (int(sqrt(end))+1) >> 1):
        if sieve[n]:
            primes.append((n << 1) + 1)
            aux = (n << 1) + 1
            aux *= aux
            for k in xrange(aux, end, 2 * aux):
                sieve[k >> 1] = False

    s  = int(sqrt(end)) + 1
    if s  % 2 == 0:
        s += 1
    primes.extend([i for i in xrange(s, end, 2) if sieve[i >> 1]])

    return primes

def ambi_sieve_plain(n):
    s = range(3, n, 2)
    for m in xrange(3, int(n**0.5)+1, 2): 
        if s[(m-3)/2]: 
            for t in xrange((m*m-3)/2,(n>>1)-1,m):
                s[t]=0
    return [2]+[t for t in s if t>0]

def sundaram3(max_n):
    # https://stackoverflow.com/questions/2068372/fastest-way-to-list-all-primes-below-n-in-python/2073279#2073279
    numbers = range(3, max_n+1, 2)
    half = (max_n)//2
    initial = 4

    for step in xrange(3, max_n+1, 2):
        for i in xrange(initial, half, step):
            numbers[i-1] = 0
        initial += 2*(step+1)

        if initial > half:
            return [2] + filter(None, numbers)

################################################################################
# Using Numpy:
def ambi_sieve(n):
    # http://tommih.blogspot.com/2009/04/fast-prime-number-generator.html
    s = np.arange(3, n, 2)
    for m in xrange(3, int(n ** 0.5)+1, 2): 
        if s[(m-3)/2]: 
            s[(m*m-3)/2::m]=0
    return np.r_[2, s[s>0]]

def primesfrom3to(n):
    # https://stackoverflow.com/questions/2068372/fastest-way-to-list-all-primes-below-n-in-python/3035188#3035188
    """ Returns a array of primes, p < n """
    assert n>=2
    sieve = np.ones(n/2, dtype=np.bool)
    for i in xrange(3,int(n**0.5)+1,2):
        if sieve[i/2]:
            sieve[i*i/2::i] = False
    return np.r_[2, 2*np.nonzero(sieve)[0][1::]+1]    

def primesfrom2to(n):
    # https://stackoverflow.com/questions/2068372/fastest-way-to-list-all-primes-below-n-in-python/3035188#3035188
    """ Input n>=6, Returns a array of primes, 2 <= p < n """
    sieve = np.ones(n/3 + (n%6==2), dtype=np.bool)
    sieve[0] = False
    for i in xrange(int(n**0.5)/3+1):
        if sieve[i]:
            k=3*i+1|1
            sieve[      ((k*k)/3)      ::2*k] = False
            sieve[(k*k+4*k-2*k*(i&1))/3::2*k] = False
    return np.r_[2,3,((3*np.nonzero(sieve)[0]+1)|1)]

if __name__=='__main__':
    import itertools
    import sys

    def test(f1,f2,num):
        print('Testing {f1} and {f2} return same results'.format(
            f1=f1.func_name,
            f2=f2.func_name))
        if not all([a==b for a,b in itertools.izip_longest(f1(num),f2(num))]):
            sys.exit("Error: %s(%s) != %s(%s)"%(f1.func_name,num,f2.func_name,num))

    n=1000000
    test(sieveOfAtkin,sieveOfEratosthenes,n)
    test(sieveOfAtkin,ambi_sieve,n)
    test(sieveOfAtkin,ambi_sieve_plain,n) 
    test(sieveOfAtkin,sundaram3,n)
    test(sieveOfAtkin,sieve_wheel_30,n)
    test(sieveOfAtkin,primesfrom3to,n)
    test(sieveOfAtkin,primesfrom2to,n)
    test(sieveOfAtkin,rwh_primes,n)
    test(sieveOfAtkin,rwh_primes1,n)         
    test(sieveOfAtkin,rwh_primes2,n)

Running the script tests that all implementations give the same result.

回答 1

更快，更明智的纯Python代码：

def primes(n):
    """ Returns  a list of primes < n """
    sieve = [True] * n
    for i in range(3,int(n**0.5)+1,2):
        if sieve[i]:
            sieve[i*i::2*i]=[False]*((n-i*i-1)//(2*i)+1)
    return [2] + [i for i in range(3,n,2) if sieve[i]]

或从半筛开始

def primes1(n):
    """ Returns  a list of primes < n """
    sieve = [True] * (n//2)
    for i in range(3,int(n**0.5)+1,2):
        if sieve[i//2]:
            sieve[i*i//2::i] = [False] * ((n-i*i-1)//(2*i)+1)
    return [2] + [2*i+1 for i in range(1,n//2) if sieve[i]]

更快，更明智的内存numpy代码：

import numpy
def primesfrom3to(n):
    """ Returns a array of primes, 3 <= p < n """
    sieve = numpy.ones(n//2, dtype=numpy.bool)
    for i in range(3,int(n**0.5)+1,2):
        if sieve[i//2]:
            sieve[i*i//2::i] = False
    return 2*numpy.nonzero(sieve)[0][1::]+1

从三分之一的筛子开始的更快的变化：

import numpy
def primesfrom2to(n):
    """ Input n>=6, Returns a array of primes, 2 <= p < n """
    sieve = numpy.ones(n//3 + (n%6==2), dtype=numpy.bool)
    for i in range(1,int(n**0.5)//3+1):
        if sieve[i]:
            k=3*i+1|1
            sieve[       k*k//3     ::2*k] = False
            sieve[k*(k-2*(i&1)+4)//3::2*k] = False
    return numpy.r_[2,3,((3*numpy.nonzero(sieve)[0][1:]+1)|1)]

上述代码的（难编码）纯python版本为：

def primes2(n):
    """ Input n>=6, Returns a list of primes, 2 <= p < n """
    n, correction = n-n%6+6, 2-(n%6>1)
    sieve = [True] * (n//3)
    for i in range(1,int(n**0.5)//3+1):
      if sieve[i]:
        k=3*i+1|1
        sieve[      k*k//3      ::2*k] = [False] * ((n//6-k*k//6-1)//k+1)
        sieve[k*(k-2*(i&1)+4)//3::2*k] = [False] * ((n//6-k*(k-2*(i&1)+4)//6-1)//k+1)
    return [2,3] + [3*i+1|1 for i in range(1,n//3-correction) if sieve[i]]

不幸的是，纯python并没有采用更简单，更快速的numpy方式进行赋值，并且len()像in [False]*len(sieve[((k*k)//3)::2*k])中那样在循环内调用太慢了。因此，我不得不即兴改正输入（避免更多的数学运算），并做一些极端的（令人痛苦的）数学魔术。

我个人认为numpy（已被广泛使用）不是Python标准库的一部分，而Python开发人员似乎完全忽略了语法和速度方面的改进，这是一个遗憾。

Faster & more memory-wise pure Python code:

def primes(n):
    """ Returns  a list of primes < n """
    sieve = [True] * n
    for i in range(3,int(n**0.5)+1,2):
        if sieve[i]:
            sieve[i*i::2*i]=[False]*((n-i*i-1)//(2*i)+1)
    return [2] + [i for i in range(3,n,2) if sieve[i]]

or starting with half sieve

def primes1(n):
    """ Returns  a list of primes < n """
    sieve = [True] * (n//2)
    for i in range(3,int(n**0.5)+1,2):
        if sieve[i//2]:
            sieve[i*i//2::i] = [False] * ((n-i*i-1)//(2*i)+1)
    return [2] + [2*i+1 for i in range(1,n//2) if sieve[i]]

Faster & more memory-wise numpy code:

import numpy
def primesfrom3to(n):
    """ Returns a array of primes, 3 <= p < n """
    sieve = numpy.ones(n//2, dtype=numpy.bool)
    for i in range(3,int(n**0.5)+1,2):
        if sieve[i//2]:
            sieve[i*i//2::i] = False
    return 2*numpy.nonzero(sieve)[0][1::]+1

a faster variation starting with a third of a sieve:

import numpy
def primesfrom2to(n):
    """ Input n>=6, Returns a array of primes, 2 <= p < n """
    sieve = numpy.ones(n//3 + (n%6==2), dtype=numpy.bool)
    for i in range(1,int(n**0.5)//3+1):
        if sieve[i]:
            k=3*i+1|1
            sieve[       k*k//3     ::2*k] = False
            sieve[k*(k-2*(i&1)+4)//3::2*k] = False
    return numpy.r_[2,3,((3*numpy.nonzero(sieve)[0][1:]+1)|1)]

A (hard-to-code) pure-python version of the above code would be:

def primes2(n):
    """ Input n>=6, Returns a list of primes, 2 <= p < n """
    n, correction = n-n%6+6, 2-(n%6>1)
    sieve = [True] * (n//3)
    for i in range(1,int(n**0.5)//3+1):
      if sieve[i]:
        k=3*i+1|1
        sieve[      k*k//3      ::2*k] = [False] * ((n//6-k*k//6-1)//k+1)
        sieve[k*(k-2*(i&1)+4)//3::2*k] = [False] * ((n//6-k*(k-2*(i&1)+4)//6-1)//k+1)
    return [2,3] + [3*i+1|1 for i in range(1,n//3-correction) if sieve[i]]

Unfortunately pure-python don’t adopt the simpler and faster numpy way of doing assignment, and calling len() inside the loop as in [False]*len(sieve[((k*k)//3)::2*k]) is too slow. So I had to improvise to correct input (& avoid more math) and do some extreme (& painful) math-magic.

Personally I think it is a shame that numpy (which is so widely used) is not part of Python standard library, and that the improvements in syntax and speed seem to be completely overlooked by Python developers.

回答 2

有从Python食谱一个漂亮整洁的样品这里 -建议对URL最快的版本是：

import itertools
def erat2( ):
    D = {  }
    yield 2
    for q in itertools.islice(itertools.count(3), 0, None, 2):
        p = D.pop(q, None)
        if p is None:
            D[q*q] = q
            yield q
        else:
            x = p + q
            while x in D or not (x&1):
                x += p
            D[x] = p

这样就可以

def get_primes_erat(n):
  return list(itertools.takewhile(lambda p: p<n, erat2()))

使用pri.py中的这段代码在shell提示符下（我喜欢这样做）进行测量，我观察到：

$ python2.5 -mtimeit -s'import pri' 'pri.get_primes(1000000)'
10 loops, best of 3: 1.69 sec per loop
$ python2.5 -mtimeit -s'import pri' 'pri.get_primes_erat(1000000)'
10 loops, best of 3: 673 msec per loop

因此，看起来Cookbook解决方案的速度是以前的两倍。

There’s a pretty neat sample from the Python Cookbook here — the fastest version proposed on that URL is:

import itertools
def erat2( ):
    D = {  }
    yield 2
    for q in itertools.islice(itertools.count(3), 0, None, 2):
        p = D.pop(q, None)
        if p is None:
            D[q*q] = q
            yield q
        else:
            x = p + q
            while x in D or not (x&1):
                x += p
            D[x] = p

so that would give

def get_primes_erat(n):
  return list(itertools.takewhile(lambda p: p<n, erat2()))

Measuring at the shell prompt (as I prefer to do) with this code in pri.py, I observe:

$ python2.5 -mtimeit -s'import pri' 'pri.get_primes(1000000)'
10 loops, best of 3: 1.69 sec per loop
$ python2.5 -mtimeit -s'import pri' 'pri.get_primes_erat(1000000)'
10 loops, best of 3: 673 msec per loop

so it looks like the Cookbook solution is over twice as fast.

回答 3

我认为使用Sundaram的Sieve打破了纯Python的记录：

def sundaram3(max_n):
    numbers = range(3, max_n+1, 2)
    half = (max_n)//2
    initial = 4

    for step in xrange(3, max_n+1, 2):
        for i in xrange(initial, half, step):
            numbers[i-1] = 0
        initial += 2*(step+1)

        if initial > half:
            return [2] + filter(None, numbers)

对比：

C:\USERS>python -m timeit -n10 -s "import get_primes" "get_primes.get_primes_erat(1000000)"
10 loops, best of 3: 710 msec per loop

C:\USERS>python -m timeit -n10 -s "import get_primes" "get_primes.daniel_sieve_2(1000000)"
10 loops, best of 3: 435 msec per loop

C:\USERS>python -m timeit -n10 -s "import get_primes" "get_primes.sundaram3(1000000)"
10 loops, best of 3: 327 msec per loop

Using Sundaram’s Sieve, I think I broke pure-Python’s record:

def sundaram3(max_n):
    numbers = range(3, max_n+1, 2)
    half = (max_n)//2
    initial = 4

    for step in xrange(3, max_n+1, 2):
        for i in xrange(initial, half, step):
            numbers[i-1] = 0
        initial += 2*(step+1)

        if initial > half:
            return [2] + filter(None, numbers)

Comparasion:

C:\USERS>python -m timeit -n10 -s "import get_primes" "get_primes.get_primes_erat(1000000)"
10 loops, best of 3: 710 msec per loop

C:\USERS>python -m timeit -n10 -s "import get_primes" "get_primes.daniel_sieve_2(1000000)"
10 loops, best of 3: 435 msec per loop

C:\USERS>python -m timeit -n10 -s "import get_primes" "get_primes.sundaram3(1000000)"
10 loops, best of 3: 327 msec per loop

回答 4

该算法速度很快，但存在严重缺陷：

>>> sorted(get_primes(530))
[2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73,
79, 83, 89, 97, 101, 103, 107, 109, 113, 127, 131, 137, 139, 149, 151, 157, 163,
167, 173, 179, 181, 191, 193, 197, 199, 211, 223, 227, 229, 233, 239, 241, 251,
257, 263, 269, 271, 277, 281, 283, 293, 307, 311, 313, 317, 331, 337, 347, 349,
353, 359, 367, 373, 379, 383, 389, 397, 401, 409, 419, 421, 431, 433, 439, 443,
449, 457, 461, 463, 467, 479, 487, 491, 499, 503, 509, 521, 523, 527, 529]
>>> 17*31
527
>>> 23*23
529

您假设这numbers.pop()将返回集合中的最小数字，但是完全不能保证。集是无序的，并且pop()删除并返回任意元素，因此不能用于从其余数字中选择下一个素数。

The algorithm is fast, but it has a serious flaw:

>>> sorted(get_primes(530))
[2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47, 53, 59, 61, 67, 71, 73,
79, 83, 89, 97, 101, 103, 107, 109, 113, 127, 131, 137, 139, 149, 151, 157, 163,
167, 173, 179, 181, 191, 193, 197, 199, 211, 223, 227, 229, 233, 239, 241, 251,
257, 263, 269, 271, 277, 281, 283, 293, 307, 311, 313, 317, 331, 337, 347, 349,
353, 359, 367, 373, 379, 383, 389, 397, 401, 409, 419, 421, 431, 433, 439, 443,
449, 457, 461, 463, 467, 479, 487, 491, 499, 503, 509, 521, 523, 527, 529]
>>> 17*31
527
>>> 23*23
529

You assume that numbers.pop() would return the smallest number in the set, but this is not guaranteed at all. Sets are unordered and pop() removes and returns an arbitrary element, so it cannot be used to select the next prime from the remaining numbers.

回答 5

对于具有足够大N的真正最快的解决方案，将是下载预先计算的素数列表，将其存储为元组，然后执行以下操作：

for pos,i in enumerate(primes):
    if i > N:
        print primes[:pos]

如果N > primes[-1] 只是这样，则可以计算更多的质数并将新列表保存在您的代码中，因此下一次同样快。

始终在框外思考。

For truly fastest solution with sufficiently large N would be to download a pre-calculated list of primes, store it as a tuple and do something like:

for pos,i in enumerate(primes):
    if i > N:
        print primes[:pos]

If N > primes[-1] only then calculate more primes and save the new list in your code, so next time it is equally as fast.

Always think outside the box.

回答 6

如果您不想重新发明轮子，可以安装符号数学库sympy（是的，它与Python 3兼容）

pip install sympy

并使用primerange功能

from sympy import sieve
primes = list(sieve.primerange(1, 10**6))

If you don’t want to reinvent the wheel, you can install the symbolic maths library sympy (yes it’s Python 3 compatible)

pip install sympy

And use the primerange function

from sympy import sieve
primes = list(sieve.primerange(1, 10**6))

回答 7

如果您接受itertools但不接受numpy，则这是针对Python 3的rwh_primes2的改编版，其在我的计算机上的运行速度约为以前的两倍。唯一的实质性更改是使用字节数组而不是布尔值列表，并使用compress而不是列表推导来构建最终列表。（如果可以的话，我将其添加为类似moarningsun的评论。）

import itertools
izip = itertools.zip_longest
chain = itertools.chain.from_iterable
compress = itertools.compress
def rwh_primes2_python3(n):
    """ Input n>=6, Returns a list of primes, 2 <= p < n """
    zero = bytearray([False])
    size = n//3 + (n % 6 == 2)
    sieve = bytearray([True]) * size
    sieve[0] = False
    for i in range(int(n**0.5)//3+1):
      if sieve[i]:
        k=3*i+1|1
        start = (k*k+4*k-2*k*(i&1))//3
        sieve[(k*k)//3::2*k]=zero*((size - (k*k)//3 - 1) // (2 * k) + 1)
        sieve[  start ::2*k]=zero*((size -   start  - 1) // (2 * k) + 1)
    ans = [2,3]
    poss = chain(izip(*[range(i, n, 6) for i in (1,5)]))
    ans.extend(compress(poss, sieve))
    return ans

比较：

>>> timeit.timeit('primes.rwh_primes2(10**6)', setup='import primes', number=1)
0.0652179726976101
>>> timeit.timeit('primes.rwh_primes2_python3(10**6)', setup='import primes', number=1)
0.03267321276325674

和

>>> timeit.timeit('primes.rwh_primes2(10**8)', setup='import primes', number=1)
6.394284538007014
>>> timeit.timeit('primes.rwh_primes2_python3(10**8)', setup='import primes', number=1)
3.833829450302801

If you accept itertools but not numpy, here is an adaptation of rwh_primes2 for Python 3 that runs about twice as fast on my machine. The only substantial change is using a bytearray instead of a list for the boolean, and using compress instead of a list comprehension to build the final list. (I’d add this as a comment like moarningsun if I were able.)

import itertools
izip = itertools.zip_longest
chain = itertools.chain.from_iterable
compress = itertools.compress
def rwh_primes2_python3(n):
    """ Input n>=6, Returns a list of primes, 2 <= p < n """
    zero = bytearray([False])
    size = n//3 + (n % 6 == 2)
    sieve = bytearray([True]) * size
    sieve[0] = False
    for i in range(int(n**0.5)//3+1):
      if sieve[i]:
        k=3*i+1|1
        start = (k*k+4*k-2*k*(i&1))//3
        sieve[(k*k)//3::2*k]=zero*((size - (k*k)//3 - 1) // (2 * k) + 1)
        sieve[  start ::2*k]=zero*((size -   start  - 1) // (2 * k) + 1)
    ans = [2,3]
    poss = chain(izip(*[range(i, n, 6) for i in (1,5)]))
    ans.extend(compress(poss, sieve))
    return ans

Comparisons:

>>> timeit.timeit('primes.rwh_primes2(10**6)', setup='import primes', number=1)
0.0652179726976101
>>> timeit.timeit('primes.rwh_primes2_python3(10**6)', setup='import primes', number=1)
0.03267321276325674

and

>>> timeit.timeit('primes.rwh_primes2(10**8)', setup='import primes', number=1)
6.394284538007014
>>> timeit.timeit('primes.rwh_primes2_python3(10**8)', setup='import primes', number=1)
3.833829450302801

回答 8

编写自己的主要发现代码很有启发性，但是手头有一个快速可靠的库也很有用。我围绕C ++库primesieve编写了一个包装器，将其命名为primesieve-python

尝试一下 pip install primesieve

import primesieve
primes = primesieve.generate_primes(10**8)

我很好奇看到速度比较。

It’s instructive to write your own prime finding code, but it’s also useful to have a fast reliable library at hand. I wrote a wrapper around the C++ library primesieve, named it primesieve-python

Try it pip install primesieve

import primesieve
primes = primesieve.generate_primes(10**8)

I’d be curious to see the speed compared.

回答 9

这是最快的功能之一的两个更新版本（纯Python 3.6），

from itertools import compress

def rwh_primes1v1(n):
    """ Returns  a list of primes < n for n > 2 """
    sieve = bytearray([True]) * (n//2)
    for i in range(3,int(n**0.5)+1,2):
        if sieve[i//2]:
            sieve[i*i//2::i] = bytearray((n-i*i-1)//(2*i)+1)
    return [2,*compress(range(3,n,2), sieve[1:])]

def rwh_primes1v2(n):
    """ Returns a list of primes < n for n > 2 """
    sieve = bytearray([True]) * (n//2+1)
    for i in range(1,int(n**0.5)//2+1):
        if sieve[i]:
            sieve[2*i*(i+1)::2*i+1] = bytearray((n//2-2*i*(i+1))//(2*i+1)+1)
    return [2,*compress(range(3,n,2), sieve[1:])]

Here is two updated (pure Python 3.6) versions of one of the fastest functions,

from itertools import compress

def rwh_primes1v1(n):
    """ Returns  a list of primes < n for n > 2 """
    sieve = bytearray([True]) * (n//2)
    for i in range(3,int(n**0.5)+1,2):
        if sieve[i//2]:
            sieve[i*i//2::i] = bytearray((n-i*i-1)//(2*i)+1)
    return [2,*compress(range(3,n,2), sieve[1:])]

def rwh_primes1v2(n):
    """ Returns a list of primes < n for n > 2 """
    sieve = bytearray([True]) * (n//2+1)
    for i in range(1,int(n**0.5)//2+1):
        if sieve[i]:
            sieve[2*i*(i+1)::2*i+1] = bytearray((n//2-2*i*(i+1))//(2*i+1)+1)
    return [2,*compress(range(3,n,2), sieve[1:])]

回答 10

在N <9,080,191的假设下确定性实施Miller-Rabin素数检验

import sys
import random

def miller_rabin_pass(a, n):
    d = n - 1
    s = 0
    while d % 2 == 0:
        d >>= 1
        s += 1

    a_to_power = pow(a, d, n)
    if a_to_power == 1:
        return True
    for i in xrange(s-1):
        if a_to_power == n - 1:
            return True
        a_to_power = (a_to_power * a_to_power) % n
    return a_to_power == n - 1


def miller_rabin(n):
    for a in [2, 3, 37, 73]:
      if not miller_rabin_pass(a, n):
        return False
    return True


n = int(sys.argv[1])
primes = [2]
for p in range(3,n,2):
  if miller_rabin(p):
    primes.append(p)
print len(primes)

根据Wikipedia上的文章（http://en.wikipedia.org/wiki/Miller–Rabin_primality_test），测试N <9,080,191的a = 2,3,37和73足以确定N是否为复合值。

然后，我从此处找到的原始Miller-Rabin测试的概率实现中改编了源代码：http : //en.literateprograms.org/Miller-Rabin_primality_test_(Python）

A deterministic implementation of Miller-Rabin’s Primality test on the assumption that N < 9,080,191

import sys
import random

def miller_rabin_pass(a, n):
    d = n - 1
    s = 0
    while d % 2 == 0:
        d >>= 1
        s += 1

    a_to_power = pow(a, d, n)
    if a_to_power == 1:
        return True
    for i in xrange(s-1):
        if a_to_power == n - 1:
            return True
        a_to_power = (a_to_power * a_to_power) % n
    return a_to_power == n - 1


def miller_rabin(n):
    for a in [2, 3, 37, 73]:
      if not miller_rabin_pass(a, n):
        return False
    return True


n = int(sys.argv[1])
primes = [2]
for p in range(3,n,2):
  if miller_rabin(p):
    primes.append(p)
print len(primes)

According to the article on Wikipedia (http://en.wikipedia.org/wiki/Miller–Rabin_primality_test) testing N < 9,080,191 for a = 2,3,37, and 73 is enough to decide whether N is composite or not.

And I adapted the source code from the probabilistic implementation of original Miller-Rabin’s test found here: http://en.literateprograms.org/Miller-Rabin_primality_test_(Python)

回答 11

如果您可以控制N，则列出所有素数的最快方法是预先计算它们。说真的预计算是一种被忽略的优化方法。

If you have control over N, the very fastest way to list all primes is to precompute them. Seriously. Precomputing is a way overlooked optimization.

回答 12

这是我通常用于在Python中生成素数的代码：

$ python -mtimeit -s'import sieve' 'sieve.sieve(1000000)' 
10 loops, best of 3: 445 msec per loop
$ cat sieve.py
from math import sqrt

def sieve(size):
 prime=[True]*size
 rng=xrange
 limit=int(sqrt(size))

 for i in rng(3,limit+1,+2):
  if prime[i]:
   prime[i*i::+i]=[False]*len(prime[i*i::+i])

 return [2]+[i for i in rng(3,size,+2) if prime[i]]

if __name__=='__main__':
 print sieve(100)

它无法与此处发布的更快的解决方案竞争，但至少它是纯python。

感谢您发布此问题。今天我真的学到了很多东西。

Here’s the code I normally use to generate primes in Python:

$ python -mtimeit -s'import sieve' 'sieve.sieve(1000000)' 
10 loops, best of 3: 445 msec per loop
$ cat sieve.py
from math import sqrt

def sieve(size):
 prime=[True]*size
 rng=xrange
 limit=int(sqrt(size))

 for i in rng(3,limit+1,+2):
  if prime[i]:
   prime[i*i::+i]=[False]*len(prime[i*i::+i])

 return [2]+[i for i in rng(3,size,+2) if prime[i]]

if __name__=='__main__':
 print sieve(100)

It can’t compete with the faster solutions posted here, but at least it is pure python.

Thanks for posting this question. I really learnt a lot today.

回答 13

对于最快的代码，numpy解决方案是最好的。不过，出于纯粹的学术原因，我要发布我的纯python版本，该版本比上面发布的食谱版本快50％。由于我将整个列表存储在内存中，因此您需要足够的空间来容纳所有内容，但它似乎可以很好地扩展。

def daniel_sieve_2(maxNumber):
    """
    Given a number, returns all numbers less than or equal to
    that number which are prime.
    """
    allNumbers = range(3, maxNumber+1, 2)
    for mIndex, number in enumerate(xrange(3, maxNumber+1, 2)):
        if allNumbers[mIndex] == 0:
            continue
        # now set all multiples to 0
        for index in xrange(mIndex+number, (maxNumber-3)/2+1, number):
            allNumbers[index] = 0
    return [2] + filter(lambda n: n!=0, allNumbers)

结果：

>>>mine = timeit.Timer("daniel_sieve_2(1000000)",
...                    "from sieves import daniel_sieve_2")
>>>prev = timeit.Timer("get_primes_erat(1000000)",
...                    "from sieves import get_primes_erat")
>>>print "Mine: {0:0.4f} ms".format(min(mine.repeat(3, 1))*1000)
Mine: 428.9446 ms
>>>print "Previous Best {0:0.4f} ms".format(min(prev.repeat(3, 1))*1000)
Previous Best 621.3581 ms

For the fastest code, the numpy solution is the best. For purely academic reasons, though, I’m posting my pure python version, which is a bit less than 50% faster than the cookbook version posted above. Since I make the entire list in memory, you need enough space to hold everything, but it seems to scale fairly well.

def daniel_sieve_2(maxNumber):
    """
    Given a number, returns all numbers less than or equal to
    that number which are prime.
    """
    allNumbers = range(3, maxNumber+1, 2)
    for mIndex, number in enumerate(xrange(3, maxNumber+1, 2)):
        if allNumbers[mIndex] == 0:
            continue
        # now set all multiples to 0
        for index in xrange(mIndex+number, (maxNumber-3)/2+1, number):
            allNumbers[index] = 0
    return [2] + filter(lambda n: n!=0, allNumbers)

And the results:

>>>mine = timeit.Timer("daniel_sieve_2(1000000)",
...                    "from sieves import daniel_sieve_2")
>>>prev = timeit.Timer("get_primes_erat(1000000)",
...                    "from sieves import get_primes_erat")
>>>print "Mine: {0:0.4f} ms".format(min(mine.repeat(3, 1))*1000)
Mine: 428.9446 ms
>>>print "Previous Best {0:0.4f} ms".format(min(prev.repeat(3, 1))*1000)
Previous Best 621.3581 ms

回答 14

使用Numpy的半筛的实现略有不同：

http://rebrained.com/?p=458

导入数学
导入numpy
def prime6（最高）：
    primes = numpy.arange（3，最高+1,2）
    isprime = numpy.ones（（最多1）/ 2，dtype = bool）
    对于素数[：int（math.sqrt（upto））]：
        如果isprime [（factor-2）/ 2]：isprime [（factor * 3-2）/ 2：（upto-1）/ 2：factor] = 0
    返回numpy.insert（primes [isprime]，0,2）

有人可以将此与其他时间进行比较吗？在我的机器上，它看起来可以与其他Numpy半筛媲美。

A slightly different implementation of a half sieve using Numpy:

http://rebrained.com/?p=458

import math
import numpy
def prime6(upto):
    primes=numpy.arange(3,upto+1,2)
    isprime=numpy.ones((upto-1)/2,dtype=bool)
    for factor in primes[:int(math.sqrt(upto))]:
        if isprime[(factor-2)/2]: isprime[(factor*3-2)/2:(upto-1)/2:factor]=0
    return numpy.insert(primes[isprime],0,2)

Can someone compare this with the other timings? On my machine it seems pretty comparable to the other Numpy half-sieve.

回答 15

全部都经过编写和测试。因此，无需重新发明轮子。

python -m timeit -r10 -s"from sympy import sieve" "primes = list(sieve.primerange(1, 10**6))"

给我们打破了12.2毫秒的记录！

10 loops, best of 10: 12.2 msec per loop

如果这还不够快，您可以尝试PyPy：

pypy -m timeit -r10 -s"from sympy import sieve" "primes = list(sieve.primerange(1, 10**6))"

结果是：

10 loops, best of 10: 2.03 msec per loop

247次投票的答案列出了15.9毫秒的最佳解决方案。比较一下！！！

It’s all written and tested. So there is no need to reinvent the wheel.

python -m timeit -r10 -s"from sympy import sieve" "primes = list(sieve.primerange(1, 10**6))"

gives us a record breaking 12.2 msec!

10 loops, best of 10: 12.2 msec per loop

If this is not fast enough, you can try PyPy:

pypy -m timeit -r10 -s"from sympy import sieve" "primes = list(sieve.primerange(1, 10**6))"

which results in:

10 loops, best of 10: 2.03 msec per loop

The answer with 247 up-votes lists 15.9 ms for the best solution. Compare this!!!

回答 16

我测试了一些unutbu的函数，用饥饿的数百万来计算

获奖者是使用numpy库的函数，

注意：进行内存利用率测试也很有趣：)

样例代码

我的github存储库上的完整代码

#!/usr/bin/env python

import lib
import timeit
import sys
import math
import datetime

import prettyplotlib as ppl
import numpy as np

import matplotlib.pyplot as plt
from prettyplotlib import brewer2mpl

primenumbers_gen = [
    'sieveOfEratosthenes',
    'ambi_sieve',
    'ambi_sieve_plain',
    'sundaram3',
    'sieve_wheel_30',
    'primesfrom3to',
    'primesfrom2to',
    'rwh_primes',
    'rwh_primes1',
    'rwh_primes2',
]

def human_format(num):
    # /programming/579310/formatting-long-numbers-as-strings-in-python?answertab=active#tab-top
    magnitude = 0
    while abs(num) >= 1000:
        magnitude += 1
        num /= 1000.0
    # add more suffixes if you need them
    return '%.2f%s' % (num, ['', 'K', 'M', 'G', 'T', 'P'][magnitude])


if __name__=='__main__':

    # Vars
    n = 10000000 # number itereration generator
    nbcol = 5 # For decompose prime number generator
    nb_benchloop = 3 # Eliminate false positive value during the test (bench average time)
    datetimeformat = '%Y-%m-%d %H:%M:%S.%f'
    config = 'from __main__ import n; import lib'
    primenumbers_gen = {
        'sieveOfEratosthenes': {'color': 'b'},
        'ambi_sieve': {'color': 'b'},
        'ambi_sieve_plain': {'color': 'b'},
         'sundaram3': {'color': 'b'},
        'sieve_wheel_30': {'color': 'b'},
# # #        'primesfrom2to': {'color': 'b'},
        'primesfrom3to': {'color': 'b'},
        # 'rwh_primes': {'color': 'b'},
        # 'rwh_primes1': {'color': 'b'},
        'rwh_primes2': {'color': 'b'},
    }


    # Get n in command line
    if len(sys.argv)>1:
        n = int(sys.argv[1])

    step = int(math.ceil(n / float(nbcol)))
    nbs = np.array([i * step for i in range(1, int(nbcol) + 1)])
    set2 = brewer2mpl.get_map('Paired', 'qualitative', 12).mpl_colors

    print datetime.datetime.now().strftime(datetimeformat)
    print("Compute prime number to %(n)s" % locals())
    print("")

    results = dict()
    for pgen in primenumbers_gen:
        results[pgen] = dict()
        benchtimes = list()
        for n in nbs:
            t = timeit.Timer("lib.%(pgen)s(n)" % locals(), setup=config)
            execute_times = t.repeat(repeat=nb_benchloop,number=1)
            benchtime = np.mean(execute_times)
            benchtimes.append(benchtime)
        results[pgen] = {'benchtimes':np.array(benchtimes)}

fig, ax = plt.subplots(1)
plt.ylabel('Computation time (in second)')
plt.xlabel('Numbers computed')
i = 0
for pgen in primenumbers_gen:

    bench = results[pgen]['benchtimes']
    avgs = np.divide(bench,nbs)
    avg = np.average(bench, weights=nbs)

    # Compute linear regression
    A = np.vstack([nbs, np.ones(len(nbs))]).T
    a, b = np.linalg.lstsq(A, nbs*avgs)[0]

    # Plot
    i += 1
    #label="%(pgen)s" % locals()
    #ppl.plot(nbs, nbs*avgs, label=label, lw=1, linestyle='--', color=set2[i % 12])
    label="%(pgen)s avg" % locals()
    ppl.plot(nbs, a * nbs + b, label=label, lw=2, color=set2[i % 12])
print datetime.datetime.now().strftime(datetimeformat)

ppl.legend(ax, loc='upper left', ncol=4)

# Change x axis label
ax.get_xaxis().get_major_formatter().set_scientific(False)
fig.canvas.draw()
labels = [human_format(int(item.get_text())) for item in ax.get_xticklabels()]

ax.set_xticklabels(labels)
ax = plt.gca()

plt.show()

I tested some unutbu’s functions, i computed it with hungred millions number

The winners are the functions that use numpy library,

Note: It would also interesting make a memory utilization test :)

Sample code

Complete code on my github repository

#!/usr/bin/env python

import lib
import timeit
import sys
import math
import datetime

import prettyplotlib as ppl
import numpy as np

import matplotlib.pyplot as plt
from prettyplotlib import brewer2mpl

primenumbers_gen = [
    'sieveOfEratosthenes',
    'ambi_sieve',
    'ambi_sieve_plain',
    'sundaram3',
    'sieve_wheel_30',
    'primesfrom3to',
    'primesfrom2to',
    'rwh_primes',
    'rwh_primes1',
    'rwh_primes2',
]

def human_format(num):
    # https://stackoverflow.com/questions/579310/formatting-long-numbers-as-strings-in-python?answertab=active#tab-top
    magnitude = 0
    while abs(num) >= 1000:
        magnitude += 1
        num /= 1000.0
    # add more suffixes if you need them
    return '%.2f%s' % (num, ['', 'K', 'M', 'G', 'T', 'P'][magnitude])


if __name__=='__main__':

    # Vars
    n = 10000000 # number itereration generator
    nbcol = 5 # For decompose prime number generator
    nb_benchloop = 3 # Eliminate false positive value during the test (bench average time)
    datetimeformat = '%Y-%m-%d %H:%M:%S.%f'
    config = 'from __main__ import n; import lib'
    primenumbers_gen = {
        'sieveOfEratosthenes': {'color': 'b'},
        'ambi_sieve': {'color': 'b'},
        'ambi_sieve_plain': {'color': 'b'},
         'sundaram3': {'color': 'b'},
        'sieve_wheel_30': {'color': 'b'},
# # #        'primesfrom2to': {'color': 'b'},
        'primesfrom3to': {'color': 'b'},
        # 'rwh_primes': {'color': 'b'},
        # 'rwh_primes1': {'color': 'b'},
        'rwh_primes2': {'color': 'b'},
    }


    # Get n in command line
    if len(sys.argv)>1:
        n = int(sys.argv[1])

    step = int(math.ceil(n / float(nbcol)))
    nbs = np.array([i * step for i in range(1, int(nbcol) + 1)])
    set2 = brewer2mpl.get_map('Paired', 'qualitative', 12).mpl_colors

    print datetime.datetime.now().strftime(datetimeformat)
    print("Compute prime number to %(n)s" % locals())
    print("")

    results = dict()
    for pgen in primenumbers_gen:
        results[pgen] = dict()
        benchtimes = list()
        for n in nbs:
            t = timeit.Timer("lib.%(pgen)s(n)" % locals(), setup=config)
            execute_times = t.repeat(repeat=nb_benchloop,number=1)
            benchtime = np.mean(execute_times)
            benchtimes.append(benchtime)
        results[pgen] = {'benchtimes':np.array(benchtimes)}

fig, ax = plt.subplots(1)
plt.ylabel('Computation time (in second)')
plt.xlabel('Numbers computed')
i = 0
for pgen in primenumbers_gen:

    bench = results[pgen]['benchtimes']
    avgs = np.divide(bench,nbs)
    avg = np.average(bench, weights=nbs)

    # Compute linear regression
    A = np.vstack([nbs, np.ones(len(nbs))]).T
    a, b = np.linalg.lstsq(A, nbs*avgs)[0]

    # Plot
    i += 1
    #label="%(pgen)s" % locals()
    #ppl.plot(nbs, nbs*avgs, label=label, lw=1, linestyle='--', color=set2[i % 12])
    label="%(pgen)s avg" % locals()
    ppl.plot(nbs, a * nbs + b, label=label, lw=2, color=set2[i % 12])
print datetime.datetime.now().strftime(datetimeformat)

ppl.legend(ax, loc='upper left', ncol=4)

# Change x axis label
ax.get_xaxis().get_major_formatter().set_scientific(False)
fig.canvas.draw()
labels = [human_format(int(item.get_text())) for item in ax.get_xticklabels()]

ax.set_xticklabels(labels)
ax = plt.gca()

plt.show()

回答 17

对于Python 3

def rwh_primes2(n):
    correction = (n%6>1)
    n = {0:n,1:n-1,2:n+4,3:n+3,4:n+2,5:n+1}[n%6]
    sieve = [True] * (n//3)
    sieve[0] = False
    for i in range(int(n**0.5)//3+1):
      if sieve[i]:
        k=3*i+1|1
        sieve[      ((k*k)//3)      ::2*k]=[False]*((n//6-(k*k)//6-1)//k+1)
        sieve[(k*k+4*k-2*k*(i&1))//3::2*k]=[False]*((n//6-(k*k+4*k-2*k*(i&1))//6-1)//k+1)
    return [2,3] + [3*i+1|1 for i in range(1,n//3-correction) if sieve[i]]

For Python 3

def rwh_primes2(n):
    correction = (n%6>1)
    n = {0:n,1:n-1,2:n+4,3:n+3,4:n+2,5:n+1}[n%6]
    sieve = [True] * (n//3)
    sieve[0] = False
    for i in range(int(n**0.5)//3+1):
      if sieve[i]:
        k=3*i+1|1
        sieve[      ((k*k)//3)      ::2*k]=[False]*((n//6-(k*k)//6-1)//k+1)
        sieve[(k*k+4*k-2*k*(i&1))//3::2*k]=[False]*((n//6-(k*k+4*k-2*k*(i&1))//6-1)//k+1)
    return [2,3] + [3*i+1|1 for i in range(1,n//3-correction) if sieve[i]]

回答 18

纯Python中最快的基本筛：

from itertools import compress

def half_sieve(n):
    """
    Returns a list of prime numbers less than `n`.
    """
    if n <= 2:
        return []
    sieve = bytearray([True]) * (n // 2)
    for i in range(3, int(n ** 0.5) + 1, 2):
        if sieve[i // 2]:
            sieve[i * i // 2::i] = bytearray((n - i * i - 1) // (2 * i) + 1)
    primes = list(compress(range(1, n, 2), sieve))
    primes[0] = 2
    return primes

我优化了Eratosthenes筛的速度和内存。

基准测试

from time import clock
import platform

def benchmark(iterations, limit):
    start = clock()
    for x in range(iterations):
        half_sieve(limit)
    end = clock() - start
    print(f'{end/iterations:.4f} seconds for primes < {limit}')

if __name__ == '__main__':
    print(platform.python_version())
    print(platform.platform())
    print(platform.processor())
    it = 10
    for pw in range(4, 9):
        benchmark(it, 10**pw)

输出量

>>> 3.6.7
>>> Windows-10-10.0.17763-SP0
>>> Intel64 Family 6 Model 78 Stepping 3, GenuineIntel
>>> 0.0003 seconds for primes < 10000
>>> 0.0021 seconds for primes < 100000
>>> 0.0204 seconds for primes < 1000000
>>> 0.2389 seconds for primes < 10000000
>>> 2.6702 seconds for primes < 100000000

Fastest prime sieve in Pure Python:

from itertools import compress

def half_sieve(n):
    """
    Returns a list of prime numbers less than `n`.
    """
    if n <= 2:
        return []
    sieve = bytearray([True]) * (n // 2)
    for i in range(3, int(n ** 0.5) + 1, 2):
        if sieve[i // 2]:
            sieve[i * i // 2::i] = bytearray((n - i * i - 1) // (2 * i) + 1)
    primes = list(compress(range(1, n, 2), sieve))
    primes[0] = 2
    return primes

I optimised Sieve of Eratosthenes for speed and memory.

Benchmark

from time import clock
import platform

def benchmark(iterations, limit):
    start = clock()
    for x in range(iterations):
        half_sieve(limit)
    end = clock() - start
    print(f'{end/iterations:.4f} seconds for primes < {limit}')

if __name__ == '__main__':
    print(platform.python_version())
    print(platform.platform())
    print(platform.processor())
    it = 10
    for pw in range(4, 9):
        benchmark(it, 10**pw)

Output

>>> 3.6.7
>>> Windows-10-10.0.17763-SP0
>>> Intel64 Family 6 Model 78 Stepping 3, GenuineIntel
>>> 0.0003 seconds for primes < 10000
>>> 0.0021 seconds for primes < 100000
>>> 0.0204 seconds for primes < 1000000
>>> 0.2389 seconds for primes < 10000000
>>> 2.6702 seconds for primes < 100000000

回答 19

第一次使用python，因此我在其中使用的某些方法似乎有点麻烦。我只是将我的c ++代码直接转换为python，这就是我所拥有的（尽管在python中有点slowww）

#!/usr/bin/env python
import time

def GetPrimes(n):

    Sieve = [1 for x in xrange(n)]

    Done = False
    w = 3

    while not Done:

        for q in xrange (3, n, 2):
            Prod = w*q
            if Prod < n:
                Sieve[Prod] = 0
            else:
                break

        if w > (n/2):
            Done = True
        w += 2

    return Sieve



start = time.clock()

d = 10000000
Primes = GetPrimes(d)

count = 1 #This is for 2

for x in xrange (3, d, 2):
    if Primes[x]:
        count+=1

elapsed = (time.clock() - start)
print "\nFound", count, "primes in", elapsed, "seconds!\n"

pythonw Primes.py

在12.799119秒内找到664579质数！

#!/usr/bin/env python
import time

def GetPrimes2(n):

    Sieve = [1 for x in xrange(n)]

    for q in xrange (3, n, 2):
        k = q
        for y in xrange(k*3, n, k*2):
            Sieve[y] = 0

    return Sieve



start = time.clock()

d = 10000000
Primes = GetPrimes2(d)

count = 1 #This is for 2

for x in xrange (3, d, 2):
    if Primes[x]:
        count+=1

elapsed = (time.clock() - start)
print "\nFound", count, "primes in", elapsed, "seconds!\n"

pythonw Primes2.py

在10.230172秒内找到664579质数！

#!/usr/bin/env python
import time

def GetPrimes3(n):

    Sieve = [1 for x in xrange(n)]

    for q in xrange (3, n, 2):
        k = q
        for y in xrange(k*k, n, k << 1):
            Sieve[y] = 0

    return Sieve



start = time.clock()

d = 10000000
Primes = GetPrimes3(d)

count = 1 #This is for 2

for x in xrange (3, d, 2):
    if Primes[x]:
        count+=1

elapsed = (time.clock() - start)
print "\nFound", count, "primes in", elapsed, "seconds!\n"

Python Primes2.py

在7.113776秒内发现664579质数！

First time using python, so some of the methods I use in this might seem a bit cumbersome. I just straight converted my c++ code to python and this is what I have (albeit a tad bit slowww in python)

#!/usr/bin/env python
import time

def GetPrimes(n):

    Sieve = [1 for x in xrange(n)]

    Done = False
    w = 3

    while not Done:

        for q in xrange (3, n, 2):
            Prod = w*q
            if Prod < n:
                Sieve[Prod] = 0
            else:
                break

        if w > (n/2):
            Done = True
        w += 2

    return Sieve



start = time.clock()

d = 10000000
Primes = GetPrimes(d)

count = 1 #This is for 2

for x in xrange (3, d, 2):
    if Primes[x]:
        count+=1

elapsed = (time.clock() - start)
print "\nFound", count, "primes in", elapsed, "seconds!\n"

pythonw Primes.py

Found 664579 primes in 12.799119 seconds!

#!/usr/bin/env python
import time

def GetPrimes2(n):

    Sieve = [1 for x in xrange(n)]

    for q in xrange (3, n, 2):
        k = q
        for y in xrange(k*3, n, k*2):
            Sieve[y] = 0

    return Sieve



start = time.clock()

d = 10000000
Primes = GetPrimes2(d)

count = 1 #This is for 2

for x in xrange (3, d, 2):
    if Primes[x]:
        count+=1

elapsed = (time.clock() - start)
print "\nFound", count, "primes in", elapsed, "seconds!\n"

pythonw Primes2.py

Found 664579 primes in 10.230172 seconds!

#!/usr/bin/env python
import time

def GetPrimes3(n):

    Sieve = [1 for x in xrange(n)]

    for q in xrange (3, n, 2):
        k = q
        for y in xrange(k*k, n, k << 1):
            Sieve[y] = 0

    return Sieve



start = time.clock()

d = 10000000
Primes = GetPrimes3(d)

count = 1 #This is for 2

for x in xrange (3, d, 2):
    if Primes[x]:
        count+=1

elapsed = (time.clock() - start)
print "\nFound", count, "primes in", elapsed, "seconds!\n"

python Primes2.py

Found 664579 primes in 7.113776 seconds!

回答 20

我知道比赛已经结束了几年。…

尽管如此，这是我对纯python主筛的建议，它基于在处理向前的筛子时通过使用适当的步骤来省略2、3和5的倍数。但是，对于N <10 ^ 9而言，它实际上要比@Robert William Hanks高级解决方案rwh_primes2和rwh_primes1慢。通过使用高于1.5 * 10 ^ 8的ctypes.c_ushort筛子数组，可以以某种方式适应内存限制。

10 ^ 6

$ python -mtimeit -s“导入primeSieveSpeedComp”“ primeSieveSpeedComp.primeSieveSeq（1000000）” 10个循环，每个循环最好3：46.7毫秒

比较：$ python -mtimeit -s“导入primeSieveSpeedComp”“ primeSieveSpeedComp.rwh_primes1（1000000）” 10个循环，最好是3个循环：每个循环要进行43.2毫秒比较：$ python -m timeit -s“ import primeSieveSpeedComp”“ primeSieveSpeedComp.rwh_primes2 （1000000）“ 10个循环，最好为3：每个循环34.5毫秒

10 ^ 7

$ python -mtimeit -s“导入primeSieveSpeedComp”“ primeSieveSpeedComp.primeSieveSeq（10000000）” 10个循环，每循环最好530毫秒

比较：$ python -mtimeit -s“导入primeSieveSpeedComp”“ primeSieveSpeedComp.rwh_primes1（10000000）” 10个循环，最好是3个循环：494毫秒比较：$ python -m timeit -s“ import primeSieveSpeedComp”“ primeSieveSpeedComp.rwh_primes2 （10000000）“ 10个循环，每个循环最好3：375毫秒

10 ^ 8

$ python -mtimeit -s“ import primeSieveSpeedComp”“ primeSieveSpeedComp.primeSieveSeq（100000000）” 10个循环，每循环最好3：5.55秒

比较：$ python -mtimeit -s“导入primeSieveSpeedComp”“ primeSieveSpeedComp.rwh_primes1（100000000）” 10个循环，最好每个循环进行3：5.33秒进行比较：$ python -m timeit -s“ import primeSieveSpeedComp”“ primeSieveSpeedComp.rwh_primes2 （100000000）“ 10个循环，每循环3：3.95秒的最佳时间

10 ^ 9

$ python -mtimeit -s“ import primeSieveSpeedComp”“ primeSieveSpeedComp.primeSieveSeq（1000000000）” 10个循环，最好3个循环：每个循环61.2秒

比较：$ python -mtimeit -n 3 -s“ import primeSieveSpeedComp”“ primeSieveSpeedComp.rwh_primes1（1000000000）” 3个循环，最佳3：97.8 每个循环秒

比较：$ python -m timeit -s“ import primeSieveSpeedComp”“ primeSieveSpeedComp.rwh_primes2（1000000000）” 10个循环，最好3个循环：每个循环41.9秒

您可以将以下代码复制到ubuntus primeSieveSpeedComp中，以查看此测试。

def primeSieveSeq(MAX_Int):
    if MAX_Int > 5*10**8:
        import ctypes
        int16Array = ctypes.c_ushort * (MAX_Int >> 1)
        sieve = int16Array()
        #print 'uses ctypes "unsigned short int Array"'
    else:
        sieve = (MAX_Int >> 1) * [False]
        #print 'uses python list() of long long int'
    if MAX_Int < 10**8:
        sieve[4::3] = [True]*((MAX_Int - 8)/6+1)
        sieve[12::5] = [True]*((MAX_Int - 24)/10+1)
    r = [2, 3, 5]
    n = 0
    for i in xrange(int(MAX_Int**0.5)/30+1):
        n += 3
        if not sieve[n]:
            n2 = (n << 1) + 1
            r.append(n2)
            n2q = (n2**2) >> 1
            sieve[n2q::n2] = [True]*(((MAX_Int >> 1) - n2q - 1) / n2 + 1)
        n += 2
        if not sieve[n]:
            n2 = (n << 1) + 1
            r.append(n2)
            n2q = (n2**2) >> 1
            sieve[n2q::n2] = [True]*(((MAX_Int >> 1) - n2q - 1) / n2 + 1)
        n += 1
        if not sieve[n]:
            n2 = (n << 1) + 1
            r.append(n2)
            n2q = (n2**2) >> 1
            sieve[n2q::n2] = [True]*(((MAX_Int >> 1) - n2q - 1) / n2 + 1)
        n += 2
        if not sieve[n]:
            n2 = (n << 1) + 1
            r.append(n2)
            n2q = (n2**2) >> 1
            sieve[n2q::n2] = [True]*(((MAX_Int >> 1) - n2q - 1) / n2 + 1)
        n += 1
        if not sieve[n]:
            n2 = (n << 1) + 1
            r.append(n2)
            n2q = (n2**2) >> 1
            sieve[n2q::n2] = [True]*(((MAX_Int >> 1) - n2q - 1) / n2 + 1)
        n += 2
        if not sieve[n]:
            n2 = (n << 1) + 1
            r.append(n2)
            n2q = (n2**2) >> 1
            sieve[n2q::n2] = [True]*(((MAX_Int >> 1) - n2q - 1) / n2 + 1)
        n += 3
        if not sieve[n]:
            n2 = (n << 1) + 1
            r.append(n2)
            n2q = (n2**2) >> 1
            sieve[n2q::n2] = [True]*(((MAX_Int >> 1) - n2q - 1) / n2 + 1)
        n += 1
        if not sieve[n]:
            n2 = (n << 1) + 1
            r.append(n2)
            n2q = (n2**2) >> 1
            sieve[n2q::n2] = [True]*(((MAX_Int >> 1) - n2q - 1) / n2 + 1)
    if MAX_Int < 10**8:
        return [2, 3, 5]+[(p << 1) + 1 for p in [n for n in xrange(3, MAX_Int >> 1) if not sieve[n]]]
    n = n >> 1
    try:
        for i in xrange((MAX_Int-2*n)/30 + 1):
            n += 3
            if not sieve[n]:
                r.append((n << 1) + 1)
            n += 2
            if not sieve[n]:
                r.append((n << 1) + 1)
            n += 1
            if not sieve[n]:
                r.append((n << 1) + 1)
            n += 2
            if not sieve[n]:
                r.append((n << 1) + 1)
            n += 1
            if not sieve[n]:
                r.append((n << 1) + 1)
            n += 2
            if not sieve[n]:
                r.append((n << 1) + 1)
            n += 3
            if not sieve[n]:
                r.append((n << 1) + 1)
            n += 1
            if not sieve[n]:
                r.append((n << 1) + 1)
    except:
        pass
    return r

I know the competition is closed for some years. …

Nonetheless this is my suggestion for a pure python prime sieve, based on omitting the multiples of 2, 3 and 5 by using appropriate steps while processing the sieve forward. Nonetheless it is actually slower for N<10^9 than @Robert William Hanks superior solutions rwh_primes2 and rwh_primes1. By using a ctypes.c_ushort sieve array above 1.5* 10^8 it is somehow adaptive to memory limits.

10^6

$ python -mtimeit -s”import primeSieveSpeedComp” “primeSieveSpeedComp.primeSieveSeq(1000000)” 10 loops, best of 3: 46.7 msec per loop

to compare:$ python -mtimeit -s”import primeSieveSpeedComp” “primeSieveSpeedComp.rwh_primes1(1000000)” 10 loops, best of 3: 43.2 msec per loop to compare: $ python -m timeit -s”import primeSieveSpeedComp” “primeSieveSpeedComp.rwh_primes2(1000000)” 10 loops, best of 3: 34.5 msec per loop

10^7

$ python -mtimeit -s”import primeSieveSpeedComp” “primeSieveSpeedComp.primeSieveSeq(10000000)” 10 loops, best of 3: 530 msec per loop

to compare:$ python -mtimeit -s”import primeSieveSpeedComp” “primeSieveSpeedComp.rwh_primes1(10000000)” 10 loops, best of 3: 494 msec per loop to compare: $ python -m timeit -s”import primeSieveSpeedComp” “primeSieveSpeedComp.rwh_primes2(10000000)” 10 loops, best of 3: 375 msec per loop

10^8

$ python -mtimeit -s”import primeSieveSpeedComp” “primeSieveSpeedComp.primeSieveSeq(100000000)” 10 loops, best of 3: 5.55 sec per loop

to compare: $ python -mtimeit -s”import primeSieveSpeedComp” “primeSieveSpeedComp.rwh_primes1(100000000)” 10 loops, best of 3: 5.33 sec per loop to compare: $ python -m timeit -s”import primeSieveSpeedComp” “primeSieveSpeedComp.rwh_primes2(100000000)” 10 loops, best of 3: 3.95 sec per loop

10^9

$ python -mtimeit -s”import primeSieveSpeedComp” “primeSieveSpeedComp.primeSieveSeq(1000000000)” 10 loops, best of 3: 61.2 sec per loop

to compare: $ python -mtimeit -n 3 -s”import primeSieveSpeedComp” “primeSieveSpeedComp.rwh_primes1(1000000000)” 3 loops, best of 3: 97.8 sec per loop

to compare: $ python -m timeit -s”import primeSieveSpeedComp” “primeSieveSpeedComp.rwh_primes2(1000000000)” 10 loops, best of 3: 41.9 sec per loop

You may copy the code below into ubuntus primeSieveSpeedComp to review this tests.

def primeSieveSeq(MAX_Int):
    if MAX_Int > 5*10**8:
        import ctypes
        int16Array = ctypes.c_ushort * (MAX_Int >> 1)
        sieve = int16Array()
        #print 'uses ctypes "unsigned short int Array"'
    else:
        sieve = (MAX_Int >> 1) * [False]
        #print 'uses python list() of long long int'
    if MAX_Int < 10**8:
        sieve[4::3] = [True]*((MAX_Int - 8)/6+1)
        sieve[12::5] = [True]*((MAX_Int - 24)/10+1)
    r = [2, 3, 5]
    n = 0
    for i in xrange(int(MAX_Int**0.5)/30+1):
        n += 3
        if not sieve[n]:
            n2 = (n << 1) + 1
            r.append(n2)
            n2q = (n2**2) >> 1
            sieve[n2q::n2] = [True]*(((MAX_Int >> 1) - n2q - 1) / n2 + 1)
        n += 2
        if not sieve[n]:
            n2 = (n << 1) + 1
            r.append(n2)
            n2q = (n2**2) >> 1
            sieve[n2q::n2] = [True]*(((MAX_Int >> 1) - n2q - 1) / n2 + 1)
        n += 1
        if not sieve[n]:
            n2 = (n << 1) + 1
            r.append(n2)
            n2q = (n2**2) >> 1
            sieve[n2q::n2] = [True]*(((MAX_Int >> 1) - n2q - 1) / n2 + 1)
        n += 2
        if not sieve[n]:
            n2 = (n << 1) + 1
            r.append(n2)
            n2q = (n2**2) >> 1
            sieve[n2q::n2] = [True]*(((MAX_Int >> 1) - n2q - 1) / n2 + 1)
        n += 1
        if not sieve[n]:
            n2 = (n << 1) + 1
            r.append(n2)
            n2q = (n2**2) >> 1
            sieve[n2q::n2] = [True]*(((MAX_Int >> 1) - n2q - 1) / n2 + 1)
        n += 2
        if not sieve[n]:
            n2 = (n << 1) + 1
            r.append(n2)
            n2q = (n2**2) >> 1
            sieve[n2q::n2] = [True]*(((MAX_Int >> 1) - n2q - 1) / n2 + 1)
        n += 3
        if not sieve[n]:
            n2 = (n << 1) + 1
            r.append(n2)
            n2q = (n2**2) >> 1
            sieve[n2q::n2] = [True]*(((MAX_Int >> 1) - n2q - 1) / n2 + 1)
        n += 1
        if not sieve[n]:
            n2 = (n << 1) + 1
            r.append(n2)
            n2q = (n2**2) >> 1
            sieve[n2q::n2] = [True]*(((MAX_Int >> 1) - n2q - 1) / n2 + 1)
    if MAX_Int < 10**8:
        return [2, 3, 5]+[(p << 1) + 1 for p in [n for n in xrange(3, MAX_Int >> 1) if not sieve[n]]]
    n = n >> 1
    try:
        for i in xrange((MAX_Int-2*n)/30 + 1):
            n += 3
            if not sieve[n]:
                r.append((n << 1) + 1)
            n += 2
            if not sieve[n]:
                r.append((n << 1) + 1)
            n += 1
            if not sieve[n]:
                r.append((n << 1) + 1)
            n += 2
            if not sieve[n]:
                r.append((n << 1) + 1)
            n += 1
            if not sieve[n]:
                r.append((n << 1) + 1)
            n += 2
            if not sieve[n]:
                r.append((n << 1) + 1)
            n += 3
            if not sieve[n]:
                r.append((n << 1) + 1)
            n += 1
            if not sieve[n]:
                r.append((n << 1) + 1)
    except:
        pass
    return r

回答 21

这是Eratosthenes筛子的numpy版本，具有良好的复杂性（比对长度为n的数组排序要低）和矢量化。与@unutbu相比，此速度与使用46微微秒的软件包查找所有小于一百万的质数的速度一样快。

import numpy as np 
def generate_primes(n):
    is_prime = np.ones(n+1,dtype=bool)
    is_prime[0:2] = False
    for i in range(int(n**0.5)+1):
        if is_prime[i]:
            is_prime[i**2::i]=False
    return np.where(is_prime)[0]

时间：

import time    
for i in range(2,10):
    timer =time.time()
    generate_primes(10**i)
    print('n = 10^',i,' time =', round(time.time()-timer,6))

>> n = 10^ 2  time = 5.6e-05
>> n = 10^ 3  time = 6.4e-05
>> n = 10^ 4  time = 0.000114
>> n = 10^ 5  time = 0.000593
>> n = 10^ 6  time = 0.00467
>> n = 10^ 7  time = 0.177758
>> n = 10^ 8  time = 1.701312
>> n = 10^ 9  time = 19.322478

Here is a numpy version of Sieve of Eratosthenes having both good complexity (lower than sorting an array of length n) and vectorization. Compared to @unutbu times this just as fast as the packages with 46 microsecons to find all primes below a million.

import numpy as np 
def generate_primes(n):
    is_prime = np.ones(n+1,dtype=bool)
    is_prime[0:2] = False
    for i in range(int(n**0.5)+1):
        if is_prime[i]:
            is_prime[i**2::i]=False
    return np.where(is_prime)[0]

Timings:

import time    
for i in range(2,10):
    timer =time.time()
    generate_primes(10**i)
    print('n = 10^',i,' time =', round(time.time()-timer,6))

>> n = 10^ 2  time = 5.6e-05
>> n = 10^ 3  time = 6.4e-05
>> n = 10^ 4  time = 0.000114
>> n = 10^ 5  time = 0.000593
>> n = 10^ 6  time = 0.00467
>> n = 10^ 7  time = 0.177758
>> n = 10^ 8  time = 1.701312
>> n = 10^ 9  time = 19.322478

回答 22

我已经更新了许多Python 3代码，并将其扔到了perfplot（我的一个项目）上，以查看哪个实际上最快。事实证明，对于大蛋糕n，primesfrom{2,3}to可以选择蛋糕：

复制剧情的代码：

import perfplot
from math import sqrt, ceil
import numpy as np
import sympy


def rwh_primes(n):
    # /programming/2068372/fastest-way-to-list-all-primes-below-n-in-python/3035188#3035188
    """ Returns  a list of primes < n """
    sieve = [True] * n
    for i in range(3, int(n ** 0.5) + 1, 2):
        if sieve[i]:
            sieve[i * i::2 * i] = [False] * ((n - i * i - 1) // (2 * i) + 1)
    return [2] + [i for i in range(3, n, 2) if sieve[i]]


def rwh_primes1(n):
    # /programming/2068372/fastest-way-to-list-all-primes-below-n-in-python/3035188#3035188
    """ Returns  a list of primes < n """
    sieve = [True] * (n // 2)
    for i in range(3, int(n ** 0.5) + 1, 2):
        if sieve[i // 2]:
            sieve[i * i // 2::i] = [False] * ((n - i * i - 1) // (2 * i) + 1)
    return [2] + [2 * i + 1 for i in range(1, n // 2) if sieve[i]]


def rwh_primes2(n):
    # /programming/2068372/fastest-way-to-list-all-primes-below-n-in-python/3035188#3035188
    """Input n>=6, Returns a list of primes, 2 <= p < n"""
    assert n >= 6
    correction = n % 6 > 1
    n = {0: n, 1: n - 1, 2: n + 4, 3: n + 3, 4: n + 2, 5: n + 1}[n % 6]
    sieve = [True] * (n // 3)
    sieve[0] = False
    for i in range(int(n ** 0.5) // 3 + 1):
        if sieve[i]:
            k = 3 * i + 1 | 1
            sieve[((k * k) // 3)::2 * k] = [False] * (
                (n // 6 - (k * k) // 6 - 1) // k + 1
            )
            sieve[(k * k + 4 * k - 2 * k * (i & 1)) // 3::2 * k] = [False] * (
                (n // 6 - (k * k + 4 * k - 2 * k * (i & 1)) // 6 - 1) // k + 1
            )
    return [2, 3] + [3 * i + 1 | 1 for i in range(1, n // 3 - correction) if sieve[i]]


def sieve_wheel_30(N):
    # http://zerovolt.com/?p=88
    """ Returns a list of primes <= N using wheel criterion 2*3*5 = 30

Copyright 2009 by zerovolt.com
This code is free for non-commercial purposes, in which case you can just leave this comment as a credit for my work.
If you need this code for commercial purposes, please contact me by sending an email to: info [at] zerovolt [dot] com."""
    __smallp = (
        2,
        3,
        5,
        7,
        11,
        13,
        17,
        19,
        23,
        29,
        31,
        37,
        41,
        43,
        47,
        53,
        59,
        61,
        67,
        71,
        73,
        79,
        83,
        89,
        97,
        101,
        103,
        107,
        109,
        113,
        127,
        131,
        137,
        139,
        149,
        151,
        157,
        163,
        167,
        173,
        179,
        181,
        191,
        193,
        197,
        199,
        211,
        223,
        227,
        229,
        233,
        239,
        241,
        251,
        257,
        263,
        269,
        271,
        277,
        281,
        283,
        293,
        307,
        311,
        313,
        317,
        331,
        337,
        347,
        349,
        353,
        359,
        367,
        373,
        379,
        383,
        389,
        397,
        401,
        409,
        419,
        421,
        431,
        433,
        439,
        443,
        449,
        457,
        461,
        463,
        467,
        479,
        487,
        491,
        499,
        503,
        509,
        521,
        523,
        541,
        547,
        557,
        563,
        569,
        571,
        577,
        587,
        593,
        599,
        601,
        607,
        613,
        617,
        619,
        631,
        641,
        643,
        647,
        653,
        659,
        661,
        673,
        677,
        683,
        691,
        701,
        709,
        719,
        727,
        733,
        739,
        743,
        751,
        757,
        761,
        769,
        773,
        787,
        797,
        809,
        811,
        821,
        823,
        827,
        829,
        839,
        853,
        857,
        859,
        863,
        877,
        881,
        883,
        887,
        907,
        911,
        919,
        929,
        937,
        941,
        947,
        953,
        967,
        971,
        977,
        983,
        991,
        997,
    )
    # wheel = (2, 3, 5)
    const = 30
    if N < 2:
        return []
    if N <= const:
        pos = 0
        while __smallp[pos] <= N:
            pos += 1
        return list(__smallp[:pos])
    # make the offsets list
    offsets = (7, 11, 13, 17, 19, 23, 29, 1)
    # prepare the list
    p = [2, 3, 5]
    dim = 2 + N // const
    tk1 = [True] * dim
    tk7 = [True] * dim
    tk11 = [True] * dim
    tk13 = [True] * dim
    tk17 = [True] * dim
    tk19 = [True] * dim
    tk23 = [True] * dim
    tk29 = [True] * dim
    tk1[0] = False
    # help dictionary d
    # d[a , b] = c  ==> if I want to find the smallest useful multiple of (30*pos)+a
    # on tkc, then I need the index given by the product of [(30*pos)+a][(30*pos)+b]
    # in general. If b < a, I need [(30*pos)+a][(30*(pos+1))+b]
    d = {}
    for x in offsets:
        for y in offsets:
            res = (x * y) % const
            if res in offsets:
                d[(x, res)] = y
    # another help dictionary: gives tkx calling tmptk[x]
    tmptk = {1: tk1, 7: tk7, 11: tk11, 13: tk13, 17: tk17, 19: tk19, 23: tk23, 29: tk29}
    pos, prime, lastadded, stop = 0, 0, 0, int(ceil(sqrt(N)))

    # inner functions definition
    def del_mult(tk, start, step):
        for k in range(start, len(tk), step):
            tk[k] = False

    # end of inner functions definition
    cpos = const * pos
    while prime < stop:
        # 30k + 7
        if tk7[pos]:
            prime = cpos + 7
            p.append(prime)
            lastadded = 7
            for off in offsets:
                tmp = d[(7, off)]
                start = (
                    (pos + prime)
                    if off == 7
                    else (prime * (const * (pos + 1 if tmp < 7 else 0) + tmp)) // const
                )
                del_mult(tmptk[off], start, prime)
        # 30k + 11
        if tk11[pos]:
            prime = cpos + 11
            p.append(prime)
            lastadded = 11
            for off in offsets:
                tmp = d[(11, off)]
                start = (
                    (pos + prime)
                    if off == 11
                    else (prime * (const * (pos + 1 if tmp < 11 else 0) + tmp)) // const
                )
                del_mult(tmptk[off], start, prime)
        # 30k + 13
        if tk13[pos]:
            prime = cpos + 13
            p.append(prime)
            lastadded = 13
            for off in offsets:
                tmp = d[(13, off)]
                start = (
                    (pos + prime)
                    if off == 13
                    else (prime * (const * (pos + 1 if tmp < 13 else 0) + tmp)) // const
                )
                del_mult(tmptk[off], start, prime)
        # 30k + 17
        if tk17[pos]:
            prime = cpos + 17
            p.append(prime)
            lastadded = 17
            for off in offsets:
                tmp = d[(17, off)]
                start = (
                    (pos + prime)
                    if off == 17
                    else (prime * (const * (pos + 1 if tmp < 17 else 0) + tmp)) // const
                )
                del_mult(tmptk[off], start, prime)
        # 30k + 19
        if tk19[pos]:
            prime = cpos + 19
            p.append(prime)
            lastadded = 19
            for off in offsets:
                tmp = d[(19, off)]
                start = (
                    (pos + prime)
                    if off == 19
                    else (prime * (const * (pos + 1 if tmp < 19 else 0) + tmp)) // const
                )
                del_mult(tmptk[off], start, prime)
        # 30k + 23
        if tk23[pos]:
            prime = cpos + 23
            p.append(prime)
            lastadded = 23
            for off in offsets:
                tmp = d[(23, off)]
                start = (
                    (pos + prime)
                    if off == 23
                    else (prime * (const * (pos + 1 if tmp < 23 else 0) + tmp)) // const
                )
                del_mult(tmptk[off], start, prime)
        # 30k + 29
        if tk29[pos]:
            prime = cpos + 29
            p.append(prime)
            lastadded = 29
            for off in offsets:
                tmp = d[(29, off)]
                start = (
                    (pos + prime)
                    if off == 29
                    else (prime * (const * (pos + 1 if tmp < 29 else 0) + tmp)) // const
                )
                del_mult(tmptk[off], start, prime)
        # now we go back to top tk1, so we need to increase pos by 1
        pos += 1
        cpos = const * pos
        # 30k + 1
        if tk1[pos]:
            prime = cpos + 1
            p.append(prime)
            lastadded = 1
            for off in offsets:
                tmp = d[(1, off)]
                start = (
                    (pos + prime)
                    if off == 1
                    else (prime * (const * pos + tmp)) // const
                )
                del_mult(tmptk[off], start, prime)
    # time to add remaining primes
    # if lastadded == 1, remove last element and start adding them from tk1
    # this way we don't need an "if" within the last while
    if lastadded == 1:
        p.pop()
    # now complete for every other possible prime
    while pos < len(tk1):
        cpos = const * pos
        if tk1[pos]:
            p.append(cpos + 1)
        if tk7[pos]:
            p.append(cpos + 7)
        if tk11[pos]:
            p.append(cpos + 11)
        if tk13[pos]:
            p.append(cpos + 13)
        if tk17[pos]:
            p.append(cpos + 17)
        if tk19[pos]:
            p.append(cpos + 19)
        if tk23[pos]:
            p.append(cpos + 23)
        if tk29[pos]:
            p.append(cpos + 29)
        pos += 1
    # remove exceeding if present
    pos = len(p) - 1
    while p[pos] > N:
        pos -= 1
    if pos < len(p) - 1:
        del p[pos + 1 :]
    # return p list
    return p


def sieve_of_eratosthenes(n):
    """sieveOfEratosthenes(n): return the list of the primes < n."""
    # Code from: <dickinsm@gmail.com>, Nov 30 2006
    # http://groups.google.com/group/comp.lang.python/msg/f1f10ced88c68c2d
    if n <= 2:
        return []
    sieve = list(range(3, n, 2))
    top = len(sieve)
    for si in sieve:
        if si:
            bottom = (si * si - 3) // 2
            if bottom >= top:
                break
            sieve[bottom::si] = [0] * -((bottom - top) // si)
    return [2] + [el for el in sieve if el]


def sieve_of_atkin(end):
    """return a list of all the prime numbers <end using the Sieve of Atkin."""
    # Code by Steve Krenzel, <Sgk284@gmail.com>, improved
    # Code: https://web.archive.org/web/20080324064651/http://krenzel.info/?p=83
    # Info: http://en.wikipedia.org/wiki/Sieve_of_Atkin
    assert end > 0
    lng = (end - 1) // 2
    sieve = [False] * (lng + 1)

    x_max, x2, xd = int(sqrt((end - 1) / 4.0)), 0, 4
    for xd in range(4, 8 * x_max + 2, 8):
        x2 += xd
        y_max = int(sqrt(end - x2))
        n, n_diff = x2 + y_max * y_max, (y_max << 1) - 1
        if not (n & 1):
            n -= n_diff
            n_diff -= 2
        for d in range((n_diff - 1) << 1, -1, -8):
            m = n % 12
            if m == 1 or m == 5:
                m = n >> 1
                sieve[m] = not sieve[m]
            n -= d

    x_max, x2, xd = int(sqrt((end - 1) / 3.0)), 0, 3
    for xd in range(3, 6 * x_max + 2, 6):
        x2 += xd
        y_max = int(sqrt(end - x2))
        n, n_diff = x2 + y_max * y_max, (y_max << 1) - 1
        if not (n & 1):
            n -= n_diff
            n_diff -= 2
        for d in range((n_diff - 1) << 1, -1, -8):
            if n % 12 == 7:
                m = n >> 1
                sieve[m] = not sieve[m]
            n -= d

    x_max, y_min, x2, xd = int((2 + sqrt(4 - 8 * (1 - end))) / 4), -1, 0, 3
    for x in range(1, x_max + 1):
        x2 += xd
        xd += 6
        if x2 >= end:
            y_min = (((int(ceil(sqrt(x2 - end))) - 1) << 1) - 2) << 1
        n, n_diff = ((x * x + x) << 1) - 1, (((x - 1) << 1) - 2) << 1
        for d in range(n_diff, y_min, -8):
            if n % 12 == 11:
                m = n >> 1
                sieve[m] = not sieve[m]
            n += d

    primes = [2, 3]
    if end <= 3:
        return primes[: max(0, end - 2)]

    for n in range(5 >> 1, (int(sqrt(end)) + 1) >> 1):
        if sieve[n]:
            primes.append((n << 1) + 1)
            aux = (n << 1) + 1
            aux *= aux
            for k in range(aux, end, 2 * aux):
                sieve[k >> 1] = False

    s = int(sqrt(end)) + 1
    if s % 2 == 0:
        s += 1
    primes.extend([i for i in range(s, end, 2) if sieve[i >> 1]])

    return primes


def ambi_sieve_plain(n):
    s = list(range(3, n, 2))
    for m in range(3, int(n ** 0.5) + 1, 2):
        if s[(m - 3) // 2]:
            for t in range((m * m - 3) // 2, (n >> 1) - 1, m):
                s[t] = 0
    return [2] + [t for t in s if t > 0]


def sundaram3(max_n):
    # /programming/2068372/fastest-way-to-list-all-primes-below-n-in-python/2073279#2073279
    numbers = range(3, max_n + 1, 2)
    half = (max_n) // 2
    initial = 4

    for step in range(3, max_n + 1, 2):
        for i in range(initial, half, step):
            numbers[i - 1] = 0
        initial += 2 * (step + 1)

        if initial > half:
            return [2] + filter(None, numbers)


# Using Numpy:
def ambi_sieve(n):
    # http://tommih.blogspot.com/2009/04/fast-prime-number-generator.html
    s = np.arange(3, n, 2)
    for m in range(3, int(n ** 0.5) + 1, 2):
        if s[(m - 3) // 2]:
            s[(m * m - 3) // 2::m] = 0
    return np.r_[2, s[s > 0]]


def primesfrom3to(n):
    # /programming/2068372/fastest-way-to-list-all-primes-below-n-in-python/3035188#3035188
    """ Returns an array of primes, p < n """
    assert n >= 2
    sieve = np.ones(n // 2, dtype=np.bool)
    for i in range(3, int(n ** 0.5) + 1, 2):
        if sieve[i // 2]:
            sieve[i * i // 2::i] = False
    return np.r_[2, 2 * np.nonzero(sieve)[0][1::] + 1]


def primesfrom2to(n):
    # /programming/2068372/fastest-way-to-list-all-primes-below-n-in-python/3035188#3035188
    """ Input n>=6, Returns an array of primes, 2 <= p < n """
    assert n >= 6
    sieve = np.ones(n // 3 + (n % 6 == 2), dtype=np.bool)
    sieve[0] = False
    for i in range(int(n ** 0.5) // 3 + 1):
        if sieve[i]:
            k = 3 * i + 1 | 1
            sieve[((k * k) // 3)::2 * k] = False
            sieve[(k * k + 4 * k - 2 * k * (i & 1)) // 3::2 * k] = False
    return np.r_[2, 3, ((3 * np.nonzero(sieve)[0] + 1) | 1)]


def sympy_sieve(n):
    return list(sympy.sieve.primerange(1, n))


perfplot.save(
    "prime.png",
    setup=lambda n: n,
    kernels=[
        rwh_primes,
        rwh_primes1,
        rwh_primes2,
        sieve_wheel_30,
        sieve_of_eratosthenes,
        sieve_of_atkin,
        # ambi_sieve_plain,
        # sundaram3,
        ambi_sieve,
        primesfrom3to,
        primesfrom2to,
        sympy_sieve,
    ],
    n_range=[2 ** k for k in range(3, 25)],
    logx=True,
    logy=True,
    xlabel="n",
)

I’ve updated much of the code for Python 3 and threw it at perfplot (a project of mine) to see which is actually fastest. Turns out that, for large n, primesfrom{2,3}to take the cake:

Code to reproduce the plot:

import perfplot
from math import sqrt, ceil
import numpy as np
import sympy


def rwh_primes(n):
    # https://stackoverflow.com/questions/2068372/fastest-way-to-list-all-primes-below-n-in-python/3035188#3035188
    """ Returns  a list of primes < n """
    sieve = [True] * n
    for i in range(3, int(n ** 0.5) + 1, 2):
        if sieve[i]:
            sieve[i * i::2 * i] = [False] * ((n - i * i - 1) // (2 * i) + 1)
    return [2] + [i for i in range(3, n, 2) if sieve[i]]


def rwh_primes1(n):
    # https://stackoverflow.com/questions/2068372/fastest-way-to-list-all-primes-below-n-in-python/3035188#3035188
    """ Returns  a list of primes < n """
    sieve = [True] * (n // 2)
    for i in range(3, int(n ** 0.5) + 1, 2):
        if sieve[i // 2]:
            sieve[i * i // 2::i] = [False] * ((n - i * i - 1) // (2 * i) + 1)
    return [2] + [2 * i + 1 for i in range(1, n // 2) if sieve[i]]


def rwh_primes2(n):
    # https://stackoverflow.com/questions/2068372/fastest-way-to-list-all-primes-below-n-in-python/3035188#3035188
    """Input n>=6, Returns a list of primes, 2 <= p < n"""
    assert n >= 6
    correction = n % 6 > 1
    n = {0: n, 1: n - 1, 2: n + 4, 3: n + 3, 4: n + 2, 5: n + 1}[n % 6]
    sieve = [True] * (n // 3)
    sieve[0] = False
    for i in range(int(n ** 0.5) // 3 + 1):
        if sieve[i]:
            k = 3 * i + 1 | 1
            sieve[((k * k) // 3)::2 * k] = [False] * (
                (n // 6 - (k * k) // 6 - 1) // k + 1
            )
            sieve[(k * k + 4 * k - 2 * k * (i & 1)) // 3::2 * k] = [False] * (
                (n // 6 - (k * k + 4 * k - 2 * k * (i & 1)) // 6 - 1) // k + 1
            )
    return [2, 3] + [3 * i + 1 | 1 for i in range(1, n // 3 - correction) if sieve[i]]


def sieve_wheel_30(N):
    # http://zerovolt.com/?p=88
    """ Returns a list of primes <= N using wheel criterion 2*3*5 = 30

Copyright 2009 by zerovolt.com
This code is free for non-commercial purposes, in which case you can just leave this comment as a credit for my work.
If you need this code for commercial purposes, please contact me by sending an email to: info [at] zerovolt [dot] com."""
    __smallp = (
        2,
        3,
        5,
        7,
        11,
        13,
        17,
        19,
        23,
        29,
        31,
        37,
        41,
        43,
        47,
        53,
        59,
        61,
        67,
        71,
        73,
        79,
        83,
        89,
        97,
        101,
        103,
        107,
        109,
        113,
        127,
        131,
        137,
        139,
        149,
        151,
        157,
        163,
        167,
        173,
        179,
        181,
        191,
        193,
        197,
        199,
        211,
        223,
        227,
        229,
        233,
        239,
        241,
        251,
        257,
        263,
        269,
        271,
        277,
        281,
        283,
        293,
        307,
        311,
        313,
        317,
        331,
        337,
        347,
        349,
        353,
        359,
        367,
        373,
        379,
        383,
        389,
        397,
        401,
        409,
        419,
        421,
        431,
        433,
        439,
        443,
        449,
        457,
        461,
        463,
        467,
        479,
        487,
        491,
        499,
        503,
        509,
        521,
        523,
        541,
        547,
        557,
        563,
        569,
        571,
        577,
        587,
        593,
        599,
        601,
        607,
        613,
        617,
        619,
        631,
        641,
        643,
        647,
        653,
        659,
        661,
        673,
        677,
        683,
        691,
        701,
        709,
        719,
        727,
        733,
        739,
        743,
        751,
        757,
        761,
        769,
        773,
        787,
        797,
        809,
        811,
        821,
        823,
        827,
        829,
        839,
        853,
        857,
        859,
        863,
        877,
        881,
        883,
        887,
        907,
        911,
        919,
        929,
        937,
        941,
        947,
        953,
        967,
        971,
        977,
        983,
        991,
        997,
    )
    # wheel = (2, 3, 5)
    const = 30
    if N < 2:
        return []
    if N <= const:
        pos = 0
        while __smallp[pos] <= N:
            pos += 1
        return list(__smallp[:pos])
    # make the offsets list
    offsets = (7, 11, 13, 17, 19, 23, 29, 1)
    # prepare the list
    p = [2, 3, 5]
    dim = 2 + N // const
    tk1 = [True] * dim
    tk7 = [True] * dim
    tk11 = [True] * dim
    tk13 = [True] * dim
    tk17 = [True] * dim
    tk19 = [True] * dim
    tk23 = [True] * dim
    tk29 = [True] * dim
    tk1[0] = False
    # help dictionary d
    # d[a , b] = c  ==> if I want to find the smallest useful multiple of (30*pos)+a
    # on tkc, then I need the index given by the product of [(30*pos)+a][(30*pos)+b]
    # in general. If b < a, I need [(30*pos)+a][(30*(pos+1))+b]
    d = {}
    for x in offsets:
        for y in offsets:
            res = (x * y) % const
            if res in offsets:
                d[(x, res)] = y
    # another help dictionary: gives tkx calling tmptk[x]
    tmptk = {1: tk1, 7: tk7, 11: tk11, 13: tk13, 17: tk17, 19: tk19, 23: tk23, 29: tk29}
    pos, prime, lastadded, stop = 0, 0, 0, int(ceil(sqrt(N)))

    # inner functions definition
    def del_mult(tk, start, step):
        for k in range(start, len(tk), step):
            tk[k] = False

    # end of inner functions definition
    cpos = const * pos
    while prime < stop:
        # 30k + 7
        if tk7[pos]:
            prime = cpos + 7
            p.append(prime)
            lastadded = 7
            for off in offsets:
                tmp = d[(7, off)]
                start = (
                    (pos + prime)
                    if off == 7
                    else (prime * (const * (pos + 1 if tmp < 7 else 0) + tmp)) // const
                )
                del_mult(tmptk[off], start, prime)
        # 30k + 11
        if tk11[pos]:
            prime = cpos + 11
            p.append(prime)
            lastadded = 11
            for off in offsets:
                tmp = d[(11, off)]
                start = (
                    (pos + prime)
                    if off == 11
                    else (prime * (const * (pos + 1 if tmp < 11 else 0) + tmp)) // const
                )
                del_mult(tmptk[off], start, prime)
        # 30k + 13
        if tk13[pos]:
            prime = cpos + 13
            p.append(prime)
            lastadded = 13
            for off in offsets:
                tmp = d[(13, off)]
                start = (
                    (pos + prime)
                    if off == 13
                    else (prime * (const * (pos + 1 if tmp < 13 else 0) + tmp)) // const
                )
                del_mult(tmptk[off], start, prime)
        # 30k + 17
        if tk17[pos]:
            prime = cpos + 17
            p.append(prime)
            lastadded = 17
            for off in offsets:
                tmp = d[(17, off)]
                start = (
                    (pos + prime)
                    if off == 17
                    else (prime * (const * (pos + 1 if tmp < 17 else 0) + tmp)) // const
                )
                del_mult(tmptk[off], start, prime)
        # 30k + 19
        if tk19[pos]:
            prime = cpos + 19
            p.append(prime)
            lastadded = 19
            for off in offsets:
                tmp = d[(19, off)]
                start = (
                    (pos + prime)
                    if off == 19
                    else (prime * (const * (pos + 1 if tmp < 19 else 0) + tmp)) // const
                )
                del_mult(tmptk[off], start, prime)
        # 30k + 23
        if tk23[pos]:
            prime = cpos + 23
            p.append(prime)
            lastadded = 23
            for off in offsets:
                tmp = d[(23, off)]
                start = (
                    (pos + prime)
                    if off == 23
                    else (prime * (const * (pos + 1 if tmp < 23 else 0) + tmp)) // const
                )
                del_mult(tmptk[off], start, prime)
        # 30k + 29
        if tk29[pos]:
            prime = cpos + 29
            p.append(prime)
            lastadded = 29
            for off in offsets:
                tmp = d[(29, off)]
                start = (
                    (pos + prime)
                    if off == 29
                    else (prime * (const * (pos + 1 if tmp < 29 else 0) + tmp)) // const
                )
                del_mult(tmptk[off], start, prime)
        # now we go back to top tk1, so we need to increase pos by 1
        pos += 1
        cpos = const * pos
        # 30k + 1
        if tk1[pos]:
            prime = cpos + 1
            p.append(prime)
            lastadded = 1
            for off in offsets:
                tmp = d[(1, off)]
                start = (
                    (pos + prime)
                    if off == 1
                    else (prime * (const * pos + tmp)) // const
                )
                del_mult(tmptk[off], start, prime)
    # time to add remaining primes
    # if lastadded == 1, remove last element and start adding them from tk1
    # this way we don't need an "if" within the last while
    if lastadded == 1:
        p.pop()
    # now complete for every other possible prime
    while pos < len(tk1):
        cpos = const * pos
        if tk1[pos]:
            p.append(cpos + 1)
        if tk7[pos]:
            p.append(cpos + 7)
        if tk11[pos]:
            p.append(cpos + 11)
        if tk13[pos]:
            p.append(cpos + 13)
        if tk17[pos]:
            p.append(cpos + 17)
        if tk19[pos]:
            p.append(cpos + 19)
        if tk23[pos]:
            p.append(cpos + 23)
        if tk29[pos]:
            p.append(cpos + 29)
        pos += 1
    # remove exceeding if present
    pos = len(p) - 1
    while p[pos] > N:
        pos -= 1
    if pos < len(p) - 1:
        del p[pos + 1 :]
    # return p list
    return p


def sieve_of_eratosthenes(n):
    """sieveOfEratosthenes(n): return the list of the primes < n."""
    # Code from: <dickinsm@gmail.com>, Nov 30 2006
    # http://groups.google.com/group/comp.lang.python/msg/f1f10ced88c68c2d
    if n <= 2:
        return []
    sieve = list(range(3, n, 2))
    top = len(sieve)
    for si in sieve:
        if si:
            bottom = (si * si - 3) // 2
            if bottom >= top:
                break
            sieve[bottom::si] = [0] * -((bottom - top) // si)
    return [2] + [el for el in sieve if el]


def sieve_of_atkin(end):
    """return a list of all the prime numbers <end using the Sieve of Atkin."""
    # Code by Steve Krenzel, <Sgk284@gmail.com>, improved
    # Code: https://web.archive.org/web/20080324064651/http://krenzel.info/?p=83
    # Info: http://en.wikipedia.org/wiki/Sieve_of_Atkin
    assert end > 0
    lng = (end - 1) // 2
    sieve = [False] * (lng + 1)

    x_max, x2, xd = int(sqrt((end - 1) / 4.0)), 0, 4
    for xd in range(4, 8 * x_max + 2, 8):
        x2 += xd
        y_max = int(sqrt(end - x2))
        n, n_diff = x2 + y_max * y_max, (y_max << 1) - 1
        if not (n & 1):
            n -= n_diff
            n_diff -= 2
        for d in range((n_diff - 1) << 1, -1, -8):
            m = n % 12
            if m == 1 or m == 5:
                m = n >> 1
                sieve[m] = not sieve[m]
            n -= d

    x_max, x2, xd = int(sqrt((end - 1) / 3.0)), 0, 3
    for xd in range(3, 6 * x_max + 2, 6):
        x2 += xd
        y_max = int(sqrt(end - x2))
        n, n_diff = x2 + y_max * y_max, (y_max << 1) - 1
        if not (n & 1):
            n -= n_diff
            n_diff -= 2
        for d in range((n_diff - 1) << 1, -1, -8):
            if n % 12 == 7:
                m = n >> 1
                sieve[m] = not sieve[m]
            n -= d

    x_max, y_min, x2, xd = int((2 + sqrt(4 - 8 * (1 - end))) / 4), -1, 0, 3
    for x in range(1, x_max + 1):
        x2 += xd
        xd += 6
        if x2 >= end:
            y_min = (((int(ceil(sqrt(x2 - end))) - 1) << 1) - 2) << 1
        n, n_diff = ((x * x + x) << 1) - 1, (((x - 1) << 1) - 2) << 1
        for d in range(n_diff, y_min, -8):
            if n % 12 == 11:
                m = n >> 1
                sieve[m] = not sieve[m]
            n += d

    primes = [2, 3]
    if end <= 3:
        return primes[: max(0, end - 2)]

    for n in range(5 >> 1, (int(sqrt(end)) + 1) >> 1):
        if sieve[n]:
            primes.append((n << 1) + 1)
            aux = (n << 1) + 1
            aux *= aux
            for k in range(aux, end, 2 * aux):
                sieve[k >> 1] = False

    s = int(sqrt(end)) + 1
    if s % 2 == 0:
        s += 1
    primes.extend([i for i in range(s, end, 2) if sieve[i >> 1]])

    return primes


def ambi_sieve_plain(n):
    s = list(range(3, n, 2))
    for m in range(3, int(n ** 0.5) + 1, 2):
        if s[(m - 3) // 2]:
            for t in range((m * m - 3) // 2, (n >> 1) - 1, m):
                s[t] = 0
    return [2] + [t for t in s if t > 0]


def sundaram3(max_n):
    # https://stackoverflow.com/questions/2068372/fastest-way-to-list-all-primes-below-n-in-python/2073279#2073279
    numbers = range(3, max_n + 1, 2)
    half = (max_n) // 2
    initial = 4

    for step in range(3, max_n + 1, 2):
        for i in range(initial, half, step):
            numbers[i - 1] = 0
        initial += 2 * (step + 1)

        if initial > half:
            return [2] + filter(None, numbers)


# Using Numpy:
def ambi_sieve(n):
    # http://tommih.blogspot.com/2009/04/fast-prime-number-generator.html
    s = np.arange(3, n, 2)
    for m in range(3, int(n ** 0.5) + 1, 2):
        if s[(m - 3) // 2]:
            s[(m * m - 3) // 2::m] = 0
    return np.r_[2, s[s > 0]]


def primesfrom3to(n):
    # https://stackoverflow.com/questions/2068372/fastest-way-to-list-all-primes-below-n-in-python/3035188#3035188
    """ Returns an array of primes, p < n """
    assert n >= 2
    sieve = np.ones(n // 2, dtype=np.bool)
    for i in range(3, int(n ** 0.5) + 1, 2):
        if sieve[i // 2]:
            sieve[i * i // 2::i] = False
    return np.r_[2, 2 * np.nonzero(sieve)[0][1::] + 1]


def primesfrom2to(n):
    # https://stackoverflow.com/questions/2068372/fastest-way-to-list-all-primes-below-n-in-python/3035188#3035188
    """ Input n>=6, Returns an array of primes, 2 <= p < n """
    assert n >= 6
    sieve = np.ones(n // 3 + (n % 6 == 2), dtype=np.bool)
    sieve[0] = False
    for i in range(int(n ** 0.5) // 3 + 1):
        if sieve[i]:
            k = 3 * i + 1 | 1
            sieve[((k * k) // 3)::2 * k] = False
            sieve[(k * k + 4 * k - 2 * k * (i & 1)) // 3::2 * k] = False
    return np.r_[2, 3, ((3 * np.nonzero(sieve)[0] + 1) | 1)]


def sympy_sieve(n):
    return list(sympy.sieve.primerange(1, n))


perfplot.save(
    "prime.png",
    setup=lambda n: n,
    kernels=[
        rwh_primes,
        rwh_primes1,
        rwh_primes2,
        sieve_wheel_30,
        sieve_of_eratosthenes,
        sieve_of_atkin,
        # ambi_sieve_plain,
        # sundaram3,
        ambi_sieve,
        primesfrom3to,
        primesfrom2to,
        sympy_sieve,
    ],
    n_range=[2 ** k for k in range(3, 25)],
    logx=True,
    logy=True,
    xlabel="n",
)

回答 23

我的猜测是最快所有方式中的就是对代码中的素数进行硬编码。

因此，为什么不编写一个缓慢的脚本来生成另一个包含所有数字的源文件，然后在运行实际程序时导入该源文件。

当然，仅当您在编译时知道N的上限时，此方法才有效，但是（几乎）所有项目Euler问题都是如此。

PS： 我可能错了，尽管使用硬连线素数解析源代码比首先计算它们要慢，但是据我所知Python运行于编译.pyc文件中，因此读取所有素数不超过N的二进制数组应该很血腥在这种情况下很快。

My guess is that the fastest of all ways is to hard code the primes in your code.

So why not just write a slow script that generates another source file that has all numbers hardwired in it, and then import that source file when you run your actual program.

Of course, this works only if you know the upper bound of N at compile time, but thus is the case for (almost) all project Euler problems.

PS: I might be wrong though iff parsing the source with hard-wired primes is slower than computing them in the first place, but as far I know Python runs from compiled .pyc files so reading a binary array with all primes up to N should be bloody fast in that case.

回答 24

很抱歉打扰，但是erat2（）在算法中存在严重缺陷。

在搜索下一个复合词时，我们仅需要测试奇数。q，p都是奇数；那么q + p是偶数，不需要测试，但是q + 2 * p总是奇数。这消除了while循环条件下的“ if even”测试，并节省了大约30％的运行时间。

当我们讨论它时：使用优雅的’D.pop（q，None）’获取和删除方法，而不是’if q in D：p = D [q]，del D [q]’，它快两倍！至少在我的机器上（P3-1Ghz）。所以我建议这种聪明算法的实现：

def erat3( ):
    from itertools import islice, count

    # q is the running integer that's checked for primeness.
    # yield 2 and no other even number thereafter
    yield 2
    D = {}
    # no need to mark D[4] as we will test odd numbers only
    for q in islice(count(3),0,None,2):
        if q in D:                  #  is composite
            p = D[q]
            del D[q]
            # q is composite. p=D[q] is the first prime that
            # divides it. Since we've reached q, we no longer
            # need it in the map, but we'll mark the next
            # multiple of its witnesses to prepare for larger
            # numbers.
            x = q + p+p        # next odd(!) multiple
            while x in D:      # skip composites
                x += p+p
            D[x] = p
        else:                  # is prime
            # q is a new prime.
            # Yield it and mark its first multiple that isn't
            # already marked in previous iterations.
            D[q*q] = q
            yield q

Sorry to bother but erat2() has a serious flaw in the algorithm.

While searching for the next composite, we need to test odd numbers only. q,p both are odd; then q+p is even and doesn’t need to be tested, but q+2*p is always odd. This eliminates the “if even” test in the while loop condition and saves about 30% of the runtime.

While we’re at it: instead of the elegant ‘D.pop(q,None)’ get and delete method use ‘if q in D: p=D[q],del D[q]’ which is twice as fast! At least on my machine (P3-1Ghz). So I suggest this implementation of this clever algorithm:

def erat3( ):
    from itertools import islice, count

    # q is the running integer that's checked for primeness.
    # yield 2 and no other even number thereafter
    yield 2
    D = {}
    # no need to mark D[4] as we will test odd numbers only
    for q in islice(count(3),0,None,2):
        if q in D:                  #  is composite
            p = D[q]
            del D[q]
            # q is composite. p=D[q] is the first prime that
            # divides it. Since we've reached q, we no longer
            # need it in the map, but we'll mark the next
            # multiple of its witnesses to prepare for larger
            # numbers.
            x = q + p+p        # next odd(!) multiple
            while x in D:      # skip composites
                x += p+p
            D[x] = p
        else:                  # is prime
            # q is a new prime.
            # Yield it and mark its first multiple that isn't
            # already marked in previous iterations.
            D[q*q] = q
            yield q

回答 25

到目前为止，我尝试过的最快方法是基于Python Cookbookerat2函数：

import itertools as it
def erat2a( ):
    D = {  }
    yield 2
    for q in it.islice(it.count(3), 0, None, 2):
        p = D.pop(q, None)
        if p is None:
            D[q*q] = q
            yield q
        else:
            x = q + 2*p
            while x in D:
                x += 2*p
            D[x] = p

有关加速的说明，请参见此答案。

The fastest method I’ve tried so far is based on the Python cookbook erat2 function:

import itertools as it
def erat2a( ):
    D = {  }
    yield 2
    for q in it.islice(it.count(3), 0, None, 2):
        p = D.pop(q, None)
        if p is None:
            D[q*q] = q
            yield q
        else:
            x = q + 2*p
            while x in D:
                x += 2*p
            D[x] = p

See this answer for an explanation of the speeding-up.

回答 26

我可能聚会晚了，但是必须为此添加我自己的代码。它占用的空间大约为n / 2，因为我们不需要存储偶数，而且我还使用了bitarray python模块，从而进一步大大减少了内存消耗，并使所有素数的计算能力高达1,000,000,000

from bitarray import bitarray
def primes_to(n):
    size = n//2
    sieve = bitarray(size)
    sieve.setall(1)
    limit = int(n**0.5)
    for i in range(1,limit):
        if sieve[i]:
            val = 2*i+1
            sieve[(i+i*val)::val] = 0
    return [2] + [2*i+1 for i, v in enumerate(sieve) if v and i > 0]

python -m timeit -n10 -s "import euler" "euler.primes_to(1000000000)"
10 loops, best of 3: 46.5 sec per loop

这是在64位2.4GHZ MAC OSX 10.8.3上运行的

I may be late to the party but will have to add my own code for this. It uses approximately n/2 in space because we don’t need to store even numbers and I also make use of the bitarray python module, further draStically cutting down on memory consumption and enabling computing all primes up to 1,000,000,000

from bitarray import bitarray
def primes_to(n):
    size = n//2
    sieve = bitarray(size)
    sieve.setall(1)
    limit = int(n**0.5)
    for i in range(1,limit):
        if sieve[i]:
            val = 2*i+1
            sieve[(i+i*val)::val] = 0
    return [2] + [2*i+1 for i, v in enumerate(sieve) if v and i > 0]

python -m timeit -n10 -s "import euler" "euler.primes_to(1000000000)"
10 loops, best of 3: 46.5 sec per loop

This was run on a 64bit 2.4GHZ MAC OSX 10.8.3

回答 27

随着时间的推移，我收集了几个质数筛。我的计算机上最快的是：

from time import time
# 175 ms for all the primes up to the value 10**6
def primes_sieve(limit):
    a = [True] * limit
    a[0] = a[1] = False
    #a[2] = True
    for n in xrange(4, limit, 2):
        a[n] = False
    root_limit = int(limit**.5)+1
    for i in xrange(3,root_limit):
        if a[i]:
            for n in xrange(i*i, limit, 2*i):
                a[n] = False
    return a

LIMIT = 10**6
s=time()
primes = primes_sieve(LIMIT)
print time()-s

I collected several prime number sieves over time. The fastest on my computer is this:

from time import time
# 175 ms for all the primes up to the value 10**6
def primes_sieve(limit):
    a = [True] * limit
    a[0] = a[1] = False
    #a[2] = True
    for n in xrange(4, limit, 2):
        a[n] = False
    root_limit = int(limit**.5)+1
    for i in xrange(3,root_limit):
        if a[i]:
            for n in xrange(i*i, limit, 2*i):
                a[n] = False
    return a

LIMIT = 10**6
s=time()
primes = primes_sieve(LIMIT)
print time()-s

回答 28

我对这个问题的回答很慢，但这似乎很有趣。我正在使用numpy，这可能会作弊，我怀疑这种方法是最快的，但应该清楚。它筛选仅引用其索引的布尔数组，并从所有True值的索引中引出质数。无需取模。

import numpy as np
def ajs_primes3a(upto):
    mat = np.ones((upto), dtype=bool)
    mat[0] = False
    mat[1] = False
    mat[4::2] = False
    for idx in range(3, int(upto ** 0.5)+1, 2):
        mat[idx*2::idx] = False
    return np.where(mat == True)[0]

I’m slow responding to this question but it seemed like a fun exercise. I’m using numpy which might be cheating and I doubt this method is the fastest but it should be clear. It sieves a Boolean array referring to its indices only and elicits prime numbers from the indices of all True values. No modulo needed.

import numpy as np
def ajs_primes3a(upto):
    mat = np.ones((upto), dtype=bool)
    mat[0] = False
    mat[1] = False
    mat[4::2] = False
    for idx in range(3, int(upto ** 0.5)+1, 2):
        mat[idx*2::idx] = False
    return np.where(mat == True)[0]

回答 29

这是一种有趣的技术，可以使用python的列表推导来生成质数（但不是最有效的）：

noprimes = [j for i in range(2, 8) for j in range(i*2, 50, i)]
primes = [x for x in range(2, 50) if x not in noprimes]

您可以在此处找到示例和一些说明

Here is an interesting technique to generate prime numbers (yet not the most efficient) using python’s list comprehensions:

noprimes = [j for i in range(2, 8) for j in range(i*2, 50, i)]
primes = [x for x in range(2, 50) if x not in noprimes]

You can find the example and some explanations right here

知识问答

如何在Python中执行包含Python代码的字符串？

2021年7月27日 Python实用宝典

问题：如何在Python中执行包含Python代码的字符串？

如何在Python中执行包含Python代码的字符串？

How do I execute a string containing Python code in Python?

回答 0

对于语句，请使用exec(string)（Python 2/3）或exec string（Python 2）：

>>> mycode = 'print "hello world"'
>>> exec(mycode)
Hello world

当需要表达式的值时，请使用eval(string)：

>>> x = eval("2+2")
>>> x
4

但是，第一步应该是问自己是否真的需要。通常，执行代码应该是最后的选择：如果代码中可能包含用户输入的代码，则它很慢，很丑陋而且很危险。您应该始终首先考虑替代项，例如高阶函数，以查看它们是否可以更好地满足您的需求。

For statements, use exec(string) (Python 2/3) or exec string (Python 2):

>>> mycode = 'print "hello world"'
>>> exec(mycode)
Hello world

When you need the value of an expression, use eval(string):

>>> x = eval("2+2")
>>> x
4

However, the first step should be to ask yourself if you really need to. Executing code should generally be the position of last resort: It’s slow, ugly and dangerous if it can contain user-entered code. You should always look at alternatives first, such as higher order functions, to see if these can better meet your needs.

回答 1

在示例中，使用exec函数将字符串作为代码执行。

import sys
import StringIO

# create file-like string to capture output
codeOut = StringIO.StringIO()
codeErr = StringIO.StringIO()

code = """
def f(x):
    x = x + 1
    return x

print 'This is my output.'
"""

# capture output and errors
sys.stdout = codeOut
sys.stderr = codeErr

exec code

# restore stdout and stderr
sys.stdout = sys.__stdout__
sys.stderr = sys.__stderr__

print f(4)

s = codeErr.getvalue()

print "error:\n%s\n" % s

s = codeOut.getvalue()

print "output:\n%s" % s

codeOut.close()
codeErr.close()

In the example a string is executed as code using the exec function.

import sys
import StringIO

# create file-like string to capture output
codeOut = StringIO.StringIO()
codeErr = StringIO.StringIO()

code = """
def f(x):
    x = x + 1
    return x

print 'This is my output.'
"""

# capture output and errors
sys.stdout = codeOut
sys.stderr = codeErr

exec code

# restore stdout and stderr
sys.stdout = sys.__stdout__
sys.stderr = sys.__stderr__

print f(4)

s = codeErr.getvalue()

print "error:\n%s\n" % s

s = codeOut.getvalue()

print "output:\n%s" % s

codeOut.close()
codeErr.close()

回答 2

eval并且exec是正确的解决方案，它们可以被用在更安全方式。

正如Python参考手册中所讨论并在本教程中明确说明的那样，eval和exec函数使用两个额外的参数，这些参数允许用户指定可用的全局和局部函数和变量。

例如：

public_variable = 10

private_variable = 2

def public_function():
    return "public information"

def private_function():
    return "super sensitive information"

# make a list of safe functions
safe_list = ['public_variable', 'public_function']
safe_dict = dict([ (k, locals().get(k, None)) for k in safe_list ])
# add any needed builtins back in
safe_dict['len'] = len

>>> eval("public_variable+2", {"__builtins__" : None }, safe_dict)
12

>>> eval("private_variable+2", {"__builtins__" : None }, safe_dict)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1, in <module>
NameError: name 'private_variable' is not defined

>>> exec("print \"'%s' has %i characters\" % (public_function(), len(public_function()))", {"__builtins__" : None}, safe_dict)
'public information' has 18 characters

>>> exec("print \"'%s' has %i characters\" % (private_function(), len(private_function()))", {"__builtins__" : None}, safe_dict)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1, in <module>
NameError: name 'private_function' is not defined

本质上，您是在定义将在其中执行代码的命名空间。

eval and exec are the correct solution, and they can be used in a safer manner.

As discussed in Python’s reference manual and clearly explained in this tutorial, the eval and exec functions take two extra parameters that allow a user to specify what global and local functions and variables are available.

For example:

public_variable = 10

private_variable = 2

def public_function():
    return "public information"

def private_function():
    return "super sensitive information"

# make a list of safe functions
safe_list = ['public_variable', 'public_function']
safe_dict = dict([ (k, locals().get(k, None)) for k in safe_list ])
# add any needed builtins back in
safe_dict['len'] = len

>>> eval("public_variable+2", {"__builtins__" : None }, safe_dict)
12

>>> eval("private_variable+2", {"__builtins__" : None }, safe_dict)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1, in <module>
NameError: name 'private_variable' is not defined

>>> exec("print \"'%s' has %i characters\" % (public_function(), len(public_function()))", {"__builtins__" : None}, safe_dict)
'public information' has 18 characters

>>> exec("print \"'%s' has %i characters\" % (private_function(), len(private_function()))", {"__builtins__" : None}, safe_dict)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1, in <module>
NameError: name 'private_function' is not defined

In essence you are defining the namespace in which the code will be executed.

回答 3

请记住，从版本3开始exec是一个功能！
因此请始终使用exec(mystring)代替exec mystring。

Remember that from version 3 exec is a function!
so always use exec(mystring) instead of exec mystring.

回答 4

eval()仅用于表达，虽然eval('x+1')有效，但eval('x=1')不会起作用。在这种情况下，最好使用exec，甚至更好：尝试找到更好的解决方案:)

eval() is just for expressions, while eval('x+1') works, eval('x=1') won’t work for example. In that case, it’s better to use exec, or even better: try to find a better solution :)

回答 5

避免`exec`和`eval`

在Python中使用`exec`和`eval`受到了极大的反对。

有更好的选择

从最上面的答案（强调我的）：

对于语句，请使用exec。

当需要表达式的值时，请使用eval。

但是，第一步应该是问自己是否真的需要。执行代码通常应该是万不得已的方法：如果它可以包含用户输入的代码，则它很慢，很丑陋而且很危险。您应该始终首先查看替代项，例如高阶函数，以查看它们是否可以更好地满足您的需求。

从替代方案到exec / eval？

使用字符串中的名称设置和获取变量的值

[而 eval ]有效，但通常不建议使用对程序本身有意义的变量名。

相反，最好使用字典。

这不是惯用的

来自http://lucumr.pocoo.org/2011/2/1/exec-in-python/（重点是我的）

Python不是PHP

不要试图绕过Python的习惯用法，因为其他一些语言会做不同的事情。在Python中使用命名空间是有原因的，仅仅是因为它为您提供了该工具，exec但这并不意味着您应该使用该工具。

有危险

来自http://nedbatchelder.com/blog/201206/eval_really_is_dangerous.html（重点是我的）

因此，即使您删除了所有的全局变量和内置函数，eval也不安全！

所有这些尝试保护eval（）的问题都在于它们是黑名单。他们明确删除了可能危险的内容。那是一场失败的战斗，因为如果只剩下一个项目，那么您可以攻击系统。

那么，可以使eval安全吗？很难说。在这一点上，我最好的猜测是，如果您不能使用任何双下划线，那么不会造成任何伤害，因此，如果您排除任何具有双下划线的字符串，那么您是安全的。也许…

很难阅读和理解

来自http://stupidpythonideas.blogspot.it/2013/05/why-evalexec-is-bad.html（重点是我）：

首先，exec使人类更难阅读您的代码。为了弄清楚发生了什么，我不仅要阅读您的代码，还必须阅读您的代码，弄清楚它将生成什么字符串，然后阅读该虚拟代码。因此，如果您在团队中工作，发布开源软件或在StackOverflow之类的地方寻求帮助，那么其他人将很难为您提供帮助。而且，如果您有机会在6个月后进行该代码的调试或扩展，那么直接给自己增加难度。

Avoid `exec` and `eval`

Using `exec` and `eval` in Python is highly frowned upon.

There are better alternatives

From the top answer (emphasis mine):

For statements, use exec.

When you need the value of an expression, use eval.

However, the first step should be to ask yourself if you really need to. Executing code should generally be the position of last resort: It’s slow, ugly and dangerous if it can contain user-entered code. You should always look at alternatives first, such as higher order functions, to see if these can better meet your needs.

From Alternatives to exec/eval?

set and get values of variables with the names in strings

[while eval] would work, it is generally not advised to use variable names bearing a meaning to the program itself.

Instead, better use a dict.

It is not idiomatic

From http://lucumr.pocoo.org/2011/2/1/exec-in-python/ (emphasis mine)

Python is not PHP

Don’t try to circumvent Python idioms because some other language does it differently. Namespaces are in Python for a reason and just because it gives you the tool exec it does not mean you should use that tool.

It is dangerous

From http://nedbatchelder.com/blog/201206/eval_really_is_dangerous.html (emphasis mine)

So eval is not safe, even if you remove all the globals and the builtins!

The problem with all of these attempts to protect eval() is that they are blacklists. They explicitly remove things that could be dangerous. That is a losing battle because if there’s just one item left off the list, you can attack the system.

So, can eval be made safe? Hard to say. At this point, my best guess is that you can’t do any harm if you can’t use any double underscores, so maybe if you exclude any string with double underscores you are safe. Maybe…

It is hard to read and understand

From http://stupidpythonideas.blogspot.it/2013/05/why-evalexec-is-bad.html (emphasis mine):

First, exec makes it harder to human beings to read your code. In order to figure out what’s happening, I don’t just have to read your code, I have to read your code, figure out what string it’s going to generate, then read that virtual code. So, if you’re working on a team, or publishing open source software, or asking for help somewhere like StackOverflow, you’re making it harder for other people to help you. And if there’s any chance that you’re going to be debugging or expanding on this code 6 months from now, you’re making it harder for yourself directly.

回答 6

您可以使用exec完成执行代码，就像下面的IDLE会话一样：

>>> kw = {}
>>> exec( "ret = 4" ) in kw
>>> kw['ret']

4

You accomplish executing code using exec, as with the following IDLE session:

>>> kw = {}
>>> exec( "ret = 4" ) in kw
>>> kw['ret']

4

回答 7

正如其他人提到的那样，它是“ exec” ..

但是，如果您的代码包含变量，则可以使用“全局”来访问它，还可以防止编译器引发以下错误：

NameError：名称“ p_variable”未定义

exec('p_variable = [1,2,3,4]')
global p_variable
print(p_variable)

As the others mentioned, it’s “exec” ..

but, in case your code contains variables, you can use “global” to access it, also to prevent the compiler to raise the following error:

NameError: name ‘p_variable’ is not defined

exec('p_variable = [1,2,3,4]')
global p_variable
print(p_variable)

回答 8

使用eval。

Use eval.

回答 9

值得一提的是，如果您要调用python文件，则该exec兄弟也存在execfile。如果您使用的第三方程序包中包含糟糕的IDE，并且您想在其程序包之外进行编码，那有时会很好。

例：

execfile('/path/to/source.py)'

要么：

exec(open("/path/to/source.py").read())

It’s worth mentioning, that’ exec‘s brother exist as well called execfile if you want to call a python file. That is sometimes good if you are working in a third party package which have terrible IDE’s included and you want to code outside of their package.

Example:

execfile('/path/to/source.py)'

or:

exec(open("/path/to/source.py").read())

回答 10

查看eval：

x = 1
print eval('x+1')
->2

Check out eval:

x = 1
print eval('x+1')
->2

回答 11

我尝试了很多事情，但是唯一可行的事情如下：

temp_dict = {}
exec("temp_dict['val'] = 10") 
print(temp_dict['val'])

输出：

10

I tried quite a few things, but the only thing that work was the following:

temp_dict = {}
exec("temp_dict['val'] = 10") 
print(temp_dict['val'])

output:

10

回答 12

最合乎逻辑的解决方案是使用内置的eval（）函数。另一种解决方案是将该字符串写入临时python文件并执行。

The most logical solution would be to use the built-in eval() function .Another solution is to write that string to a temporary python file and execute it.

回答 13

好吧..我知道这不是一个确切的答案，但可能是对像我一样看这个问题的人的注释。我想为不同的用户/客户执行特定的代码，但也想避免执行/评估。我最初希望将代码存储在每个用户的数据库中，然后执行上述操作。

我最终在“ customer_filters”文件夹中的文件系统上创建了文件，并使用了“ imp”模块，如果没有针对该客户应用的过滤器，它将继续进行

import imp


def get_customer_module(customerName='default', name='filter'):
    lm = None
    try:
        module_name = customerName+"_"+name;
        m = imp.find_module(module_name, ['customer_filters'])
        lm = imp.load_module(module_name, m[0], m[1], m[2])
    except:
        ''
        #ignore, if no module is found, 
    return lm

m = get_customer_module(customerName, "filter")
if m is not None:
    m.apply_address_filter(myobj)

因此，customerName =“ jj”将执行customer_filters \ jj_filter.py文件中的apply_address_filter

Ok .. I know this isn’t exactly an answer, but possibly a note for people looking at this as I was. I wanted to execute specific code for different users/customers but also wanted to avoid the exec/eval. I initially looked to storing the code in a database for each user and doing the above.

I ended up creating the files on the file system within a ‘customer_filters’ folder and using the ‘imp’ module, if no filter applied for that customer, it just carried on

import imp


def get_customer_module(customerName='default', name='filter'):
    lm = None
    try:
        module_name = customerName+"_"+name;
        m = imp.find_module(module_name, ['customer_filters'])
        lm = imp.load_module(module_name, m[0], m[1], m[2])
    except:
        ''
        #ignore, if no module is found, 
    return lm

m = get_customer_module(customerName, "filter")
if m is not None:
    m.apply_address_filter(myobj)

so customerName = “jj” would execute apply_address_filter from the customer_filters\jj_filter.py file

知识问答

如何清空列表？

2021年7月27日 Python实用宝典

问题：如何清空列表？

用这种方式清空列表似乎太“脏”了：

while len(alist) > 0 : alist.pop()

是否存在明确的方法？

It seems so “dirty” emptying a list in this way:

while len(alist) > 0 : alist.pop()

Does a clear way exist to do that?

回答 0

实际上，这将从列表中删除内容，但不会用新的空列表替换旧标签：

del lst[:]

这是一个例子：

lst1 = [1, 2, 3]
lst2 = lst1
del lst1[:]
print(lst2)

为了完整起见，切片分配具有相同的效果：

lst[:] = []

它也可以用于缩小列表的一部分，同时替换一部分（但这超出了问题的范围）。

请注意，这样lst = []做不会清空列表，只是创建一个新对象并将其绑定到变量lst，但是旧列表仍将具有相同的元素，并且如果具有其他变量绑定，效果将显而易见。

This actually removes the contents from the list, but doesn’t replace the old label with a new empty list:

del lst[:]

Here’s an example:

lst1 = [1, 2, 3]
lst2 = lst1
del lst1[:]
print(lst2)

For the sake of completeness, the slice assignment has the same effect:

lst[:] = []

It can also be used to shrink a part of the list while replacing a part at the same time (but that is out of the scope of the question).

Note that doing lst = [] does not empty the list, just creates a new object and binds it to the variable lst, but the old list will still have the same elements, and effect will be apparent if it had other variable bindings.

回答 1

如果你正在运行的Python 3.3或更好的，你可以使用clear()的方法list，这是平行clear()的dict，set，deque和其他易变的容器类型：

alist.clear()  # removes all items from alist (equivalent to del alist[:])

按照链接的文档页面，也可以使用来实现相同的目的alist *= 0。

总结起来，有四种等效的方法可以就地清除列表（与Python的Zen完全相反！）：

alist.clear() # Python 3.3+
del alist[:]
alist[:] = []
alist *= 0

If you’re running Python 3.3 or better, you can use the clear() method of list, which is parallel to clear() of dict, set, deque and other mutable container types:

alist.clear()  # removes all items from alist (equivalent to del alist[:])

As per the linked documentation page, the same can also be achieved with alist *= 0.

To sum up, there are four equivalent ways to clear a list in-place (quite contrary to the Zen of Python!):

alist.clear() # Python 3.3+
del alist[:]
alist[:] = []
alist *= 0

回答 2

您可以尝试：

alist[:] = []

这意味着：在[]位置[:]（从头到尾的所有索引）的列表（0个元素）中拼接

[：]是切片运算符。有关更多信息，请参见此问题。

You could try:

alist[:] = []

Which means: Splice in the list [] (0 elements) at the location [:] (all indexes from start to finish)

The [:] is the slice operator. See this question for more information.

回答 3

事实证明，使用python 2.5.2的del l[:]速度比l[:] = []使用1.1 usec的速度略慢。

$ python -mtimeit "l=list(range(1000))" "b=l[:];del b[:]"
10000 loops, best of 3: 29.8 usec per loop
$ python -mtimeit "l=list(range(1000))" "b=l[:];b[:] = []"
10000 loops, best of 3: 28.7 usec per loop
$ python -V
Python 2.5.2

it turns out that with python 2.5.2, del l[:] is slightly slower than l[:] = [] by 1.1 usec.

$ python -mtimeit "l=list(range(1000))" "b=l[:];del b[:]"
10000 loops, best of 3: 29.8 usec per loop
$ python -mtimeit "l=list(range(1000))" "b=l[:];b[:] = []"
10000 loops, best of 3: 28.7 usec per loop
$ python -V
Python 2.5.2

回答 4

lst *= 0

与…具有相同的效果

lst[:] = []

它稍微简单一些，也许更容易记住。除此之外，无话可说

效率似乎差不多

lst *= 0

has the same effect as

lst[:] = []

It’s a little simpler and maybe easier to remember. Other than that there’s not much to say

The efficiency seems to be about the same

回答 5

list = []

将重置list为空列表。

请注意，通常您不应该遮盖保留的函数名称，例如list，它是列表对象的构造函数- 例如，可以使用lst或list_代替。

list = []

will reset list to an empty list.

Note that you generally should not shadow reserved function names, such as list, which is the constructor for a list object — you could use lst or list_ instead, for instance.

回答 6

您可以使用的另一个简单代码（取决于您的情况）是：

index=len(list)-1

while index>=0:
    del list[index]
    index-=1

您必须在列表的长度处开始索引，然后相对于索引在0处向前，向后前进，因为这将使您最终获得的索引等于列表的长度，并且仅将其切成两半。

另外，请确保while行上有一个“大于或等于”符号。省略它会使您剩下list [0]。

Another simple code you could use (depending on your situation) is:

index=len(list)-1

while index>=0:
    del list[index]
    index-=1

You have to start index at the length of the list and go backwards versus index at 0, forwards because that would end you up with index equal to the length of the list with it only being cut in half.

Also, be sure that the while line has a “greater than or equal to” sign. Omitting it will leave you with list[0] remaining.

问题：如何在Python中访问命令行参数？

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

问题：熊猫合并101

回答 0

聊够了，只是告诉我如何使用merge！

其他联接-左排除，右排除和全排除/ ANTI连接

键列的不同名称

避免在输出中重复键列

仅合并其中一个的单个列 DataFrames

合并多列

其他有用的merge*操作和功能

基于索引的* -JOIN（+ index-column merges）

支持索引名称

合并一个索引，另一个列

通用化：merge处理多个DataFrame

多路合并唯一键（或索引）

多路合并重复项

Enough Talk, just show me how to use merge!

Other JOINs – LEFT-Excluding, RIGHT-Excluding, and FULL-Excluding/ANTI JOINs

Different names for key columns

Avoiding duplicate key column in output

Merging only a single column from one of the DataFrames

Merging on multiple columns

Other useful merge* operations and functions

Index-based *-JOIN (+ index-column merges)

Support for index names

Merging on index of one, column(s) of another

Generalizing: mergeing multiple DataFrames

Multiway merge on unique keys (or index)

Multiway merge on keys with duplicates

回答 1

问题：解释Python的’__enter__’和’__exit__’

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

问题：有没有办法在python的lambda中执行“如果”

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

回答 7

回答 8

回答 9

回答 10

回答 11

回答 12

问题：在Python中连接字符串的首选方法是什么？

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

回答 7

回答 8

回答 9

回答 10

回答 11

问题：如何使用Python重命名文件

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

回答 7

聊够了，只是告诉我如何使用`merge`！

仅合并其中一个的单个列 `DataFrames`

**其他有用的`merge*`操作和功能**

基于索引的* -JOIN（+ index-column `merge`s）

通用化：`merge`处理多个DataFrame

Enough Talk, just show me how to use `merge`!

Merging only a single column from one of the `DataFrames`

**Other useful `merge*` operations and functions**

Index-based *-JOIN (+ index-column `merge`s)

Generalizing: `merge`ing multiple DataFrames

问题：解释Python的’enter’和’exit’