Python多重处理模块的.join()方法到底在做什么?

问题:Python多重处理模块的.join()方法到底在做什么?

了解Python 多重处理(来自PMOTW文章),并且希望澄清一下join()方法的。

2008年的旧教程中,它指出,没有p.join()以下代码中的调用,“子进程将处于空闲状态而不会终止,成为必须手动杀死的僵尸”。

from multiprocessing import Process

def say_hello(name='world'):
    print "Hello, %s" % name

p = Process(target=say_hello)
p.start()
p.join()

我添加的打印输出PID,以及一个time.sleep测试,并就我所知道的,在自己的进程终止:

from multiprocessing import Process
import sys
import time

def say_hello(name='world'):
    print "Hello, %s" % name
    print 'Starting:', p.name, p.pid
    sys.stdout.flush()
    print 'Exiting :', p.name, p.pid
    sys.stdout.flush()
    time.sleep(20)

p = Process(target=say_hello)
p.start()
# no p.join()

20秒内:

936 ttys000    0:00.05 /Library/Frameworks/Python.framework/Versions/2.7/Reso
938 ttys000    0:00.00 /Library/Frameworks/Python.framework/Versions/2.7/Reso
947 ttys001    0:00.13 -bash

20秒后:

947 ttys001    0:00.13 -bash

行为与p.join()在文件末尾添加回的行为相同。本周Python模块为模块提供了非常易读的解释 ; “要等到进程完成工作并退出后,请使用join()方法。”,但看来至少OS X仍在这样做。

我也想知道该方法的名称。该.join()方法在此处串联吗?它是在连接过程的结尾吗?还是只是与Python的本地.join()方法共享一个名称?

Learning about Python Multiprocessing (from a PMOTW article) and would love some clarification on what exactly the join() method is doing.

In an old tutorial from 2008 it states that without the p.join() call in the code below, “the child process will sit idle and not terminate, becoming a zombie you must manually kill”.

from multiprocessing import Process

def say_hello(name='world'):
    print "Hello, %s" % name

p = Process(target=say_hello)
p.start()
p.join()

I added a printout of the PID as well as a time.sleep to test and as far as I can tell, the process terminates on its own:

from multiprocessing import Process
import sys
import time

def say_hello(name='world'):
    print "Hello, %s" % name
    print 'Starting:', p.name, p.pid
    sys.stdout.flush()
    print 'Exiting :', p.name, p.pid
    sys.stdout.flush()
    time.sleep(20)

p = Process(target=say_hello)
p.start()
# no p.join()

within 20 seconds:

936 ttys000    0:00.05 /Library/Frameworks/Python.framework/Versions/2.7/Reso
938 ttys000    0:00.00 /Library/Frameworks/Python.framework/Versions/2.7/Reso
947 ttys001    0:00.13 -bash

after 20 seconds:

947 ttys001    0:00.13 -bash

Behavior is the same with p.join() added back at end of the file. Python Module of the Week offers a very readable explanation of the module; “To wait until a process has completed its work and exited, use the join() method.”, but it seems like at least OS X was doing that anyway.

Am also wondering about the name of the method. Is the .join() method concatenating anything here? Is it concatenating a process with it’s end? Or does it just share a name with Python’s native .join() method?


回答 0

join()方法与threading或一起使用时multiprocessing不相关str.join()-它实际上并没有将任何内容串联在一起。相反,它仅表示“等待此[线程/进程]完成”。join之所以使用该名称,是因为该multiprocessing模块的API看起来类似于该threading模块的API,并且该threading模块join用于其Thread对象。join在许多编程语言中,使用该术语表示“等待线程完成”是很常见的,因此Python也采用了它。

现在,您看到有和没有调用都延迟20秒的原因join()是因为默认情况下,当主进程准备退出时,它将隐式调用join()所有正在运行的multiprocessing.Process实例。在multiprocessing文档中并未对此进行明确说明,但在“ 编程指南”部分中进行了提及:

还请记住,非守护进程将自动加入。

您可以通过设置覆盖此行为daemon上的标志ProcessTrue之前,要启动的过程:

p = Process(target=say_hello)
p.daemon = True
p.start()
# Both parent and child will exit here, since the main process has completed.

如果这样做,则子进程将在主进程完成后立即终止

守护程序

进程的守护程序标志,一个布尔值。必须在调用start()之前进行设置。

初始值是从创建过程继承的。

进程退出时,它将尝试终止其所有守护程序子进程。

The join() method, when used with threading or multiprocessing, is not related to str.join() – it’s not actually concatenating anything together. Rather, it just means “wait for this [thread/process] to complete”. The name join is used because the multiprocessing module’s API is meant to look as similar to the threading module’s API, and the threading module uses join for its Thread object. Using the term join to mean “wait for a thread to complete” is common across many programming languages, so Python just adopted it as well.

Now, the reason you see the 20 second delay both with and without the call to join() is because by default, when the main process is ready to exit, it will implicitly call join() on all running multiprocessing.Process instances. This isn’t as clearly stated in the multiprocessing docs as it should be, but it is mentioned in the Programming Guidelines section:

Remember also that non-daemonic processes will be automatically be joined.

You can override this behavior by setting the daemon flag on the Process to True prior to starting the process:

p = Process(target=say_hello)
p.daemon = True
p.start()
# Both parent and child will exit here, since the main process has completed.

If you do that, the child process will be terminated as soon as the main process completes:

daemon

The process’s daemon flag, a Boolean value. This must be set before start() is called.

The initial value is inherited from the creating process.

When a process exits, it attempts to terminate all of its daemonic child processes.


回答 1

没有 join(),主进程可以在子进程之前完成。我不确定在什么情况下会导致僵尸。

主要目的 join()是确保子流程在主流程执行任何依赖于子流程的工作之前完成。

的词源join()是与的相反fork,后者是Unix系列操作系统中用于创建子进程的常用术语。单个过程“分叉”成多个,然后“连接”成一个。

Without the join(), the main process can complete before the child process does. I’m not sure under what circumstances that leads to zombieism.

The main purpose of join() is to ensure that a child process has completed before the main process does anything that depends on the work of the child process.

The etymology of join() is that it’s the opposite of fork, which is the common term in Unix-family operating systems for creating child processes. A single process “forks” into several, then “joins” back into one.


回答 2

我不会详细解释join它的作用,但是这里是它的词源和直觉,这应该有助于您更轻松地记住它的含义。

这个想法是执行“ 分叉 ”到多个进程中,其中一个是主进程,其余是工人(或“奴隶”)。工作人员完成后,他们“加入”主服务器,以便可以恢复串行执行。

join方法使主进程等待工作人员加入。该方法最好称为“等待”,因为这是它在主服务器中引起的实际行为(这就是POSIX中所称的内容,尽管POSIX线程也称其为“ join”)。连接仅是由于线程正确协作而产生的,不是主服务器执行的操作

自1963年以来,名称“ fork”和“ join”已在多处理中使用此含义。

I’m not going to explain in detail what join does, but here’s the etymology and the intuition behind it, which should help you remember its meaning more easily.

The idea is that execution “forks” into multiple processes of which one is the master, the rest workers (or “slaves”). When the workers are done, they “join” the master so that serial execution may be resumed.

The join method causes the master process to wait for a worker to join it. The method might better have been called “wait”, since that’s the actual behavior it causes in the master (and that’s what it’s called in POSIX, although POSIX threads call it “join” as well). The joining only occurs as an effect of the threads cooperating properly, it’s not something the master does.

The names “fork” and “join” have been used with this meaning in multiprocessing since 1963.


回答 3

join()用于等待辅助进程退出。必须先调用close()terminate()使用join()

就像@罗素提到的 的那样,join就像fork的相反对象(子流程)。

要运行加入,您必须先运行close(),这将阻止所有其他任务提交到池中,并在所有任务完成后退出。或者,terminate()通过立即停止所有工作进程退出运行。

"the child process will sit idle and not terminate, becoming a zombie you must manually kill" 当主(父)进程退出但子进程仍在运行并且完成后,它没有父进程可以将其退出状态返回时,这是可能的。

join() is used to wait for the worker processes to exit. One must call close() or terminate() before using join().

Like @Russell mentioned join is like the opposite of fork (which Spawns sub-processes).

For join to run you have to run close() which will prevent any more tasks from being submitted to the pool and exit once all tasks complete. Alternatively, running terminate() will just exit by stopping all worker processes immediately.

"the child process will sit idle and not terminate, becoming a zombie you must manually kill" this is possible when the main (parent) process exits but the child process is still running and once completed it has no parent process to return its exit status to.


回答 4

join()调用可确保在完成所有多处理过程之前不会调用代码的后续行。

例如,如果不使用join(),则restart_program()甚至在进程完成之前,也会调用以下代码,这类似于异步,不是我们想要的(您可以尝试):

num_processes = 5

for i in range(num_processes):
    p = multiprocessing.Process(target=calculate_stuff, args=(i,))
    p.start()
    processes.append(p)
for p in processes:
    p.join() # call to ensure subsequent line (e.g. restart_program) 
             # is not called until all processes finish

restart_program()

The join() call ensures that subsequent lines of your code are not called before all the multiprocessing processes are completed.

For example, without the join(), the following code will call restart_program() even before the processes finish, which is similar to asynchronous and is not what we want (you can try):

num_processes = 5

for i in range(num_processes):
    p = multiprocessing.Process(target=calculate_stuff, args=(i,))
    p.start()
    processes.append(p)
for p in processes:
    p.join() # call to ensure subsequent line (e.g. restart_program) 
             # is not called until all processes finish

restart_program()

回答 5

要等待进程完成其工作并退出,请使用join()方法。

注意终止进程后,必须对join()进程进行处理,以便让后台机制有时间更新对象的状态以反映终止。

这是一个很好的例子,帮助我理解了:这里

我个人注意到的一件事是,我的主要过程暂停了,直到孩子使用join()方法完成了过程,这首先挫败了我的使用multiprocessing.Process()意图。

To wait until a process has completed its work and exited, use the join() method.

and

Note It is important to join() the process after terminating it in order to give the background machinery time to update the status of the object to reflect the termination.

This is a good example helped me understand it: here

One thing I noticed personally was my main process paused until the child had finished its process using the join() method which defeated the point of me using multiprocessing.Process() in the first place.