Python是否支持多线程?可以加快执行时间吗?

问题:Python是否支持多线程?可以加快执行时间吗?

我对多线程是否可以在Python中工作感到有些困惑。

我知道对此有很多疑问,我已经阅读了很多,但是我仍然很困惑。我从我自己的经验中知道,并且看到其他人在StackOverflow上发表了自己的答案和示例,在Python中确实可以实现多线程。那么,为什么每个人都在说Python被GIL锁定并且一次只能运行一个线程呢?显然可以。还是我不来这里有什么区别?

许多张贴者/受访者还不断提到线程是有限的,因为它不使用多个核心。但是我会说它们仍然有用,因为它们可以同时工作,因此可以更快地完成合并的工作量。我的意思是为什么还要有Python线程模块呢?

更新:

到目前为止,感谢您提供所有答案。据我了解,多线程只能并行运行某些IO任务,而一次只能运行一个CPU绑定的多个核心任务。

我并不完全确定这对我实际上意味着什么,所以我仅举一个我想进行多线程的任务示例。例如,假设我要遍历很长的字符串列表,并且希望对每个列表项执行一些基本的字符串操作。如果拆分列表,将每个要由循环/字符串代码处理的子列表发送到新线程中,然后将结果发送回队列中,这些工作负载是否会大致同时运行?最重要的是,从理论上讲,这会加快运行脚本的时间吗?

另一个例子可能是,如果我可以在四个不同的线程中使用PIL渲染和保存四张不同的图片,并且这比一张又一张地处理图片要快吗?我想这个速度要素是我真正想知道的,而不是正确的术语。

我也了解多处理模块,但是我现在的主要兴趣是中小型任务负载(10-30秒),因此我认为多线程将更合适,因为子进程的启动速度很慢。

I’m slightly confused about whether multithreading works in Python or not.

I know there has been a lot of questions about this and I’ve read many of them, but I’m still confused. I know from my own experience and have seen others post their own answers and examples here on StackOverflow that multithreading is indeed possible in Python. So why is it that everyone keep saying that Python is locked by the GIL and that only one thread can run at a time? It clearly does work. Or is there some distinction I’m not getting here?

Many posters/respondents also keep mentioning that threading is limited because it does not make use of multiple cores. But I would say they are still useful because they do work simultaneously and thus get the combined workload done faster. I mean why would there even be a Python thread module otherwise?

Update:

Thanks for all the answers so far. The way I understand it is that multithreading will only run in parallel for some IO tasks, but can only run one at a time for CPU-bound multiple core tasks.

I’m not entirely sure what this means for me in practical terms, so I’ll just give an example of the kind of task I’d like to multithread. For instance, let’s say I want to loop through a very long list of strings and I want to do some basic string operations on each list item. If I split up the list, send each sublist to be processed by my loop/string code in a new thread, and send the results back in a queue, will these workloads run roughly at the same time? Most importantly will this theoretically speed up the time it takes to run the script?

Another example might be if I can render and save four different pictures using PIL in four different threads, and have this be faster than processing the pictures one by one after each other? I guess this speed-component is what I’m really wondering about rather than what the correct terminology is.

I also know about the multiprocessing module but my main interest right now is for small-to-medium task loads (10-30 secs) and so I think multithreading will be more appropriate because subprocesses can be slow to initiate.


回答 0

GIL不会阻止线程化。GIL所做的全部工作就是确保一次只有一个线程在执行Python代码。控制仍然在线程之间切换。

GIL当时阻止的事情是利用多个CPU内核或单独的CPU并行运行线程。

这仅适用于Python代码。C扩展可以并且确实会发布GIL,以允许C代码的多个线程和一个Python线程跨多个内核运行。这扩展到由内核控制的I / O,例如select()对套接字读写的调用,使Python在多线程多核设置中合理有效地处理网络事件。

然后,许多服务器部署将执行多个Python进程,以使OS处理进程之间的调度,以最大程度地利用CPU内核。如果适合您的用例,您还可以使用该multiprocessing来处理来自一个代码库和父进程的多个进程的并行处理。

注意,GIL仅适用于CPython实现。Jython和IronPython使用不同的线程实现(分别是本机Java VM和.NET公共运行时线程)。

直接解决更新问题:任何尝试使用纯Python代码从并行执行中提高速度的任务都不会看到加速,因为线程化的Python代码一次只能锁定一个线程。但是,如果混用C扩展名和I / O(例如PIL或numpy操作),则任何C代码都可以与一个活动的Python线程并行运行。

Python线程非常适合创建响应式GUI或处理多个简短的Web请求,其中I / O比Python代码更多地成为瓶颈。它不适用于并行化计算量大的Python代码,不适合执行multiprocessing此类任务的模块或委托给专用的外部库。

The GIL does not prevent threading. All the GIL does is make sure only one thread is executing Python code at a time; control still switches between threads.

What the GIL prevents then, is making use of more than one CPU core or separate CPUs to run threads in parallel.

This only applies to Python code. C extensions can and do release the GIL to allow multiple threads of C code and one Python thread to run across multiple cores. This extends to I/O controlled by the kernel, such as select() calls for socket reads and writes, making Python handle network events reasonably efficiently in a multi-threaded multi-core setup.

What many server deployments then do, is run more than one Python process, to let the OS handle the scheduling between processes to utilize your CPU cores to the max. You can also use the multiprocessing library to handle parallel processing across multiple processes from one codebase and parent process, if that suits your use cases.

Note that the GIL is only applicable to the CPython implementation; Jython and IronPython use a different threading implementation (the native Java VM and .NET common runtime threads respectively).

To address your update directly: Any task that tries to get a speed boost from parallel execution, using pure Python code, will not see a speed-up as threaded Python code is locked to one thread executing at a time. If you mix in C extensions and I/O, however (such as PIL or numpy operations) and any C code can run in parallel with one active Python thread.

Python threading is great for creating a responsive GUI, or for handling multiple short web requests where I/O is the bottleneck more than the Python code. It is not suitable for parallelizing computationally intensive Python code, stick to the multiprocessing module for such tasks or delegate to a dedicated external library.


回答 1

是。:)

您具有低级线程模块和高级线程模块。但是您只需要使用多核计算机,多处理模块就是您的理想之选。

文档引用:

在CPython中,由于具有全局解释器锁,因此只有一个线程可以一次执行Python代码(即使某些面向性能的库可能克服了此限制)。如果希望您的应用程序更好地利用多核计算机的计算资源,建议您使用多处理。但是,如果您要同时运行多个I / O绑定任务,则线程化仍然是合适的模型。

Yes. :)

You have the low level thread module and the higher level threading module. But it you simply want to use multicore machines, the multiprocessing module is the way to go.

Quote from the docs:

In CPython, due to the Global Interpreter Lock, only one thread can execute Python code at once (even though certain performance-oriented libraries might overcome this limitation). If you want your application to make better use of the computational resources of multi-core machines, you are advised to use multiprocessing. However, threading is still an appropriate model if you want to run multiple I/O-bound tasks simultaneously.


回答 2

Python允许线程化,唯一的问题是GIL将确保一次只执行一个线程(无并行性)。

因此,基本上,如果您想对代码进行多线程处理以加快计算速度,那么一次只能执行一个线程就不会加快代码的处理速度,但是例如,如果您使用它与数据库进行交互,它就会加快速度。

Threading is Allowed in Python, the only problem is that the GIL will make sure that just one thread is executed at a time (no parallelism).

So basically if you want to multi-thread the code to speed up calculation it won’t speed it up as just one thread is executed at a time, but if you use it to interact with a database for example it will.