知识问答

为什么在Python 3.5中str.translate比Python 3.4更快？

内容 隐藏
问题：为什么在Python 3.5中str.translate比Python 3.4更快？
回答 0
为什么str.translate()慢呢？
现在发生什么事？
Why was str.translate() slow?
What happened now?

问题：为什么在Python 3.5中str.translate比Python 3.4更快？

我试图使用text.translate()Python 3.4 从给定的字符串中删除不需要的字符。

最小的代码是：

import sys 
s = 'abcde12345@#@$#%$'
mapper = dict.fromkeys(i for i in range(sys.maxunicode) if chr(i) in '@#$')
print(s.translate(mapper))

它按预期工作。但是，在Python 3.4和Python 3.5中执行相同的程序会产生很大的不同。

计算时间的代码是

python3 -m timeit -s "import sys;s = 'abcde12345@#@$#%$'*1000 ; mapper = dict.fromkeys(i for i in range(sys.maxunicode) if chr(i) in '@#$'); "   "s.translate(mapper)"

Python 3.4程序花费1.3毫秒，而Python 3.5中的同一程序仅花费26.4μs。

Python 3.5中有哪些改进使其比Python 3.4更快？

点击查看英文原文

I was trying to remove unwanted characters from a given string using text.translate() in Python 3.4.

The minimal code is:

import sys 
s = 'abcde12345@#@$#%$'
mapper = dict.fromkeys(i for i in range(sys.maxunicode) if chr(i) in '@#$')
print(s.translate(mapper))

It works as expected. However the same program when executed in Python 3.4 and Python 3.5 gives a large difference.

The code to calculate timings is

python3 -m timeit -s "import sys;s = 'abcde12345@#@$#%$'*1000 ; mapper = dict.fromkeys(i for i in range(sys.maxunicode) if chr(i) in '@#$'); "   "s.translate(mapper)"

The Python 3.4 program takes 1.3ms whereas the same program in Python 3.5 takes only 26.4μs.

What has improved in Python 3.5 that makes it faster compared to Python 3.4?

回答 0

TL; DR- 问题21118

长篇故事

Josh Rosenberg发现str.translate()与相比，该功能非常慢bytes.translate，他提出了一个问题，并指出：

在Python 3中，str.translate()通常是性能悲观，而不是优化。

为什么str.translate()慢呢？

str.translate()速度很慢的主要原因是查找曾经在Python字典中进行。

使用maketrans此问题使情况变得更糟。类似的方法是使用bytes256个项目构建一个C数组以快速查找表。因此，较高级别的Python的使用dict使str.translate()Python 3.4中的速度非常慢。

现在发生什么事？

第一种方法是添加一个小的补丁，translate_writer，但是速度的提高并不令人满意。很快又测试了另一个补丁fast_translate，它产生了非常好的结果，加速了55％。

从文件中可以看到的主要变化是Python字典查找已更改为C级查找。

现在的速度几乎与 bytes

                                unpatched           patched

str.translate                   4.55125927699919    0.7898181750006188
str.translate from bytes trans  1.8910855210015143  0.779950579000797

这里需要注意的一点是，性能增强仅在ASCII字符串中突出。

正如JFSebastian在下面的注释中提到的，在3.5之前，对于ASCII和非ASCII情况，转换以前都以相同的方式工作。但是从3.5 ASCII起，大小写要快得多。

早期的ASCII与非ASCII几乎相同，但是现在我们可以看到性能有了很大的变化。

如答案所示，它可以从71.6μs改善到2.33μs 。

以下代码演示了这一点 

python3.5 -m timeit -s "text = 'mJssissippi'*100; d=dict(J='i')" "text.translate(d)"
100000 loops, best of 3: 2.3 usec per loop
python3.5 -m timeit -s "text = 'm\U0001F602ssissippi'*100; d={'\U0001F602': 'i'}" "text.translate(d)"
10000 loops, best of 3: 117 usec per loop

python3 -m timeit -s "text = 'm\U0001F602ssissippi'*100; d={'\U0001F602': 'i'}" "text.translate(d)"
10000 loops, best of 3: 91.2 usec per loop
python3 -m timeit -s "text = 'mJssissippi'*100; d=dict(J='i')" "text.translate(d)"
10000 loops, best of 3: 101 usec per loop

结果列表：

         Python 3.4    Python 3.5  
Ascii     91.2          2.3 
Unicode   101           117
点击查看英文原文

TL;DR – ISSUE 21118

The long Story

Josh Rosenberg found out that the str.translate() function is very slow compared to the bytes.translate, he raised an issue, stating that:

In Python 3, str.translate() is usually a performance pessimization, not optimization.

Why was str.translate() slow?

The main reason for str.translate() to be very slow was that the lookup used to be in a Python dictionary.

The usage of maketrans made this problem worse. The similar approach using bytes builds a C array of 256 items to fast table lookup. Hence the usage of higher level Python dict makes the str.translate() in Python 3.4 very slow.

What happened now?

The first approach was to add a small patch, translate_writer, However the speed increase was not that pleasing. Soon another patch fast_translate was tested and it yielded very nice results of up to 55% speedup.

The main change as can be seen from the file is that the Python dictionary lookup is changed into a C level lookup.

The speeds now are almost the same as bytes

                                unpatched           patched

str.translate                   4.55125927699919    0.7898181750006188
str.translate from bytes trans  1.8910855210015143  0.779950579000797

A small note here is that the performance enhancement is only prominent in ASCII strings.

As J.F.Sebastian mentions in a comment below, Before 3.5, translate used to work in the same way for both ASCII and non-ASCII cases. However from 3.5 ASCII case is much faster.

Earlier ASCII vs non-ascii used to be almost same, however now we can see a great change in the performance.

It can be an improvement from 71.6μs to 2.33μs as seen in this answer.

The following code demonstrates this 

python3.5 -m timeit -s "text = 'mJssissippi'*100; d=dict(J='i')" "text.translate(d)"
100000 loops, best of 3: 2.3 usec per loop
python3.5 -m timeit -s "text = 'm\U0001F602ssissippi'*100; d={'\U0001F602': 'i'}" "text.translate(d)"
10000 loops, best of 3: 117 usec per loop

python3 -m timeit -s "text = 'm\U0001F602ssissippi'*100; d={'\U0001F602': 'i'}" "text.translate(d)"
10000 loops, best of 3: 91.2 usec per loop
python3 -m timeit -s "text = 'mJssissippi'*100; d=dict(J='i')" "text.translate(d)"
10000 loops, best of 3: 101 usec per loop

Tabulation of the results:

         Python 3.4    Python 3.5  
Ascii     91.2          2.3 
Unicode   101           117