标签归档:linux

如何使Python脚本像Linux中的服务或守护程序一样运行

问题:如何使Python脚本像Linux中的服务或守护程序一样运行

我已经编写了一个Python脚本,该脚本检查特定的电子邮件地址并将新的电子邮件传递给外部程序。如何获得此脚本以执行24/7,例如在Linux中将其转换为守护程序或服务。我是否还需要一个永无休止的循环,还是只需要多次重新执行代码就可以完成?

I have written a Python script that checks a certain e-mail address and passes new e-mails to an external program. How can I get this script to execute 24/7, such as turning it into daemon or service in Linux. Would I also need a loop that never ends in the program, or can it be done by just having the code re executed multiple times?


回答 0

您在这里有两个选择。

  1. 进行适当的cron作业来调用您的脚本。Cron是GNU / Linux守护程序的通用名称,该守护程序会根据您设置的时间表定期启动脚本。您将脚本添加到crontab中,或将其符号链接放置到特殊目录中,守护程序将在后台启动该脚本。您可以在Wikipedia上阅读更多内容。有各种不同的cron守护程序,但是您的GNU / Linux系统应该已经安装了它。

  2. 对您的脚本使用某种python方法(例如,一个库)可以使其自身守护进程。是的,这将需要一个简单的事件循环(您的事件可能是计时器触发的,可能由睡眠功能提供)。

我不建议您选择2.,因为实际上您将重复cron功能。Linux系统范例是让多个简单的工具交互并解决您的问题。除非有其他原因(除了定期触发)之外,您还应创建守护程序,否则请选择其他方法。

另外,如果将daemonize与循环一起使用,并且发生崩溃,此后没有人会检查邮件(如Ivan Nevostruev对此答案的评论中指出的)。如果将脚本添加为cron作业,它将再次触发。

You have two options here.

  1. Make a proper cron job that calls your script. Cron is a common name for a GNU/Linux daemon that periodically launches scripts according to a schedule you set. You add your script into a crontab or place a symlink to it into a special directory and the daemon handles the job of launching it in the background. You can read more at Wikipedia. There is a variety of different cron daemons, but your GNU/Linux system should have it already installed.

  2. Use some kind of python approach (a library, for example) for your script to be able to daemonize itself. Yes, it will require a simple event loop (where your events are timer triggering, possibly, provided by sleep function).

I wouldn’t recommend you to choose 2., because you would be, in fact, repeating cron functionality. The Linux system paradigm is to let multiple simple tools interact and solve your problems. Unless there are additional reasons why you should make a daemon (in addition to trigger periodically), choose the other approach.

Also, if you use daemonize with a loop and a crash happens, no one will check the mail after that (as pointed out by Ivan Nevostruev in comments to this answer). While if the script is added as a cron job, it will just trigger again.


回答 1

这是一个不错的类,它是从这里获取的

#!/usr/bin/env python

import sys, os, time, atexit
from signal import SIGTERM

class Daemon:
        """
        A generic daemon class.

        Usage: subclass the Daemon class and override the run() method
        """
        def __init__(self, pidfile, stdin='/dev/null', stdout='/dev/null', stderr='/dev/null'):
                self.stdin = stdin
                self.stdout = stdout
                self.stderr = stderr
                self.pidfile = pidfile

        def daemonize(self):
                """
                do the UNIX double-fork magic, see Stevens' "Advanced
                Programming in the UNIX Environment" for details (ISBN 0201563177)
                http://www.erlenstar.demon.co.uk/unix/faq_2.html#SEC16
                """
                try:
                        pid = os.fork()
                        if pid > 0:
                                # exit first parent
                                sys.exit(0)
                except OSError, e:
                        sys.stderr.write("fork #1 failed: %d (%s)\n" % (e.errno, e.strerror))
                        sys.exit(1)

                # decouple from parent environment
                os.chdir("/")
                os.setsid()
                os.umask(0)

                # do second fork
                try:
                        pid = os.fork()
                        if pid > 0:
                                # exit from second parent
                                sys.exit(0)
                except OSError, e:
                        sys.stderr.write("fork #2 failed: %d (%s)\n" % (e.errno, e.strerror))
                        sys.exit(1)

                # redirect standard file descriptors
                sys.stdout.flush()
                sys.stderr.flush()
                si = file(self.stdin, 'r')
                so = file(self.stdout, 'a+')
                se = file(self.stderr, 'a+', 0)
                os.dup2(si.fileno(), sys.stdin.fileno())
                os.dup2(so.fileno(), sys.stdout.fileno())
                os.dup2(se.fileno(), sys.stderr.fileno())

                # write pidfile
                atexit.register(self.delpid)
                pid = str(os.getpid())
                file(self.pidfile,'w+').write("%s\n" % pid)

        def delpid(self):
                os.remove(self.pidfile)

        def start(self):
                """
                Start the daemon
                """
                # Check for a pidfile to see if the daemon already runs
                try:
                        pf = file(self.pidfile,'r')
                        pid = int(pf.read().strip())
                        pf.close()
                except IOError:
                        pid = None

                if pid:
                        message = "pidfile %s already exist. Daemon already running?\n"
                        sys.stderr.write(message % self.pidfile)
                        sys.exit(1)

                # Start the daemon
                self.daemonize()
                self.run()

        def stop(self):
                """
                Stop the daemon
                """
                # Get the pid from the pidfile
                try:
                        pf = file(self.pidfile,'r')
                        pid = int(pf.read().strip())
                        pf.close()
                except IOError:
                        pid = None

                if not pid:
                        message = "pidfile %s does not exist. Daemon not running?\n"
                        sys.stderr.write(message % self.pidfile)
                        return # not an error in a restart

                # Try killing the daemon process       
                try:
                        while 1:
                                os.kill(pid, SIGTERM)
                                time.sleep(0.1)
                except OSError, err:
                        err = str(err)
                        if err.find("No such process") > 0:
                                if os.path.exists(self.pidfile):
                                        os.remove(self.pidfile)
                        else:
                                print str(err)
                                sys.exit(1)

        def restart(self):
                """
                Restart the daemon
                """
                self.stop()
                self.start()

        def run(self):
                """
                You should override this method when you subclass Daemon. It will be called after the process has been
                daemonized by start() or restart().
                """

Here’s a nice class that is taken from here:

#!/usr/bin/env python

import sys, os, time, atexit
from signal import SIGTERM

class Daemon:
        """
        A generic daemon class.

        Usage: subclass the Daemon class and override the run() method
        """
        def __init__(self, pidfile, stdin='/dev/null', stdout='/dev/null', stderr='/dev/null'):
                self.stdin = stdin
                self.stdout = stdout
                self.stderr = stderr
                self.pidfile = pidfile

        def daemonize(self):
                """
                do the UNIX double-fork magic, see Stevens' "Advanced
                Programming in the UNIX Environment" for details (ISBN 0201563177)
                http://www.erlenstar.demon.co.uk/unix/faq_2.html#SEC16
                """
                try:
                        pid = os.fork()
                        if pid > 0:
                                # exit first parent
                                sys.exit(0)
                except OSError, e:
                        sys.stderr.write("fork #1 failed: %d (%s)\n" % (e.errno, e.strerror))
                        sys.exit(1)

                # decouple from parent environment
                os.chdir("/")
                os.setsid()
                os.umask(0)

                # do second fork
                try:
                        pid = os.fork()
                        if pid > 0:
                                # exit from second parent
                                sys.exit(0)
                except OSError, e:
                        sys.stderr.write("fork #2 failed: %d (%s)\n" % (e.errno, e.strerror))
                        sys.exit(1)

                # redirect standard file descriptors
                sys.stdout.flush()
                sys.stderr.flush()
                si = file(self.stdin, 'r')
                so = file(self.stdout, 'a+')
                se = file(self.stderr, 'a+', 0)
                os.dup2(si.fileno(), sys.stdin.fileno())
                os.dup2(so.fileno(), sys.stdout.fileno())
                os.dup2(se.fileno(), sys.stderr.fileno())

                # write pidfile
                atexit.register(self.delpid)
                pid = str(os.getpid())
                file(self.pidfile,'w+').write("%s\n" % pid)

        def delpid(self):
                os.remove(self.pidfile)

        def start(self):
                """
                Start the daemon
                """
                # Check for a pidfile to see if the daemon already runs
                try:
                        pf = file(self.pidfile,'r')
                        pid = int(pf.read().strip())
                        pf.close()
                except IOError:
                        pid = None

                if pid:
                        message = "pidfile %s already exist. Daemon already running?\n"
                        sys.stderr.write(message % self.pidfile)
                        sys.exit(1)

                # Start the daemon
                self.daemonize()
                self.run()

        def stop(self):
                """
                Stop the daemon
                """
                # Get the pid from the pidfile
                try:
                        pf = file(self.pidfile,'r')
                        pid = int(pf.read().strip())
                        pf.close()
                except IOError:
                        pid = None

                if not pid:
                        message = "pidfile %s does not exist. Daemon not running?\n"
                        sys.stderr.write(message % self.pidfile)
                        return # not an error in a restart

                # Try killing the daemon process       
                try:
                        while 1:
                                os.kill(pid, SIGTERM)
                                time.sleep(0.1)
                except OSError, err:
                        err = str(err)
                        if err.find("No such process") > 0:
                                if os.path.exists(self.pidfile):
                                        os.remove(self.pidfile)
                        else:
                                print str(err)
                                sys.exit(1)

        def restart(self):
                """
                Restart the daemon
                """
                self.stop()
                self.start()

        def run(self):
                """
                You should override this method when you subclass Daemon. It will be called after the process has been
                daemonized by start() or restart().
                """

回答 2

您应该使用python-daemon库,它可以处理所有事情。

来自PyPI:库,用于实现行为良好的Unix守护进程。

You should use the python-daemon library, it takes care of everything.

From PyPI: Library to implement a well-behaved Unix daemon process.


回答 3

您可以使用fork()将脚本与tty分离,并使其继续运行,如下所示:

import os, sys
fpid = os.fork()
if fpid!=0:
  # Running as daemon now. PID is fpid
  sys.exit(0)

当然,您还需要实现一个无限循环,例如

while 1:
  do_your_check()
  sleep(5)

希望这可以帮助您开始。

You can use fork() to detach your script from the tty and have it continue to run, like so:

import os, sys
fpid = os.fork()
if fpid!=0:
  # Running as daemon now. PID is fpid
  sys.exit(0)

Of course you also need to implement an endless loop, like

while 1:
  do_your_check()
  sleep(5)

Hope this get’s you started.


回答 4

您还可以使用Shell脚本使python脚本作为服务运行。首先创建一个shell脚本来像这样运行python脚本(脚本名任意名称)

#!/bin/sh
script='/home/.. full path to script'
/usr/bin/python $script &

现在在/etc/init.d/scriptname中创建一个文件

#! /bin/sh

PATH=/bin:/usr/bin:/sbin:/usr/sbin
DAEMON=/home/.. path to shell script scriptname created to run python script
PIDFILE=/var/run/scriptname.pid

test -x $DAEMON || exit 0

. /lib/lsb/init-functions

case "$1" in
  start)
     log_daemon_msg "Starting feedparser"
     start_daemon -p $PIDFILE $DAEMON
     log_end_msg $?
   ;;
  stop)
     log_daemon_msg "Stopping feedparser"
     killproc -p $PIDFILE $DAEMON
     PID=`ps x |grep feed | head -1 | awk '{print $1}'`
     kill -9 $PID       
     log_end_msg $?
   ;;
  force-reload|restart)
     $0 stop
     $0 start
   ;;
  status)
     status_of_proc -p $PIDFILE $DAEMON atd && exit 0 || exit $?
   ;;
 *)
   echo "Usage: /etc/init.d/atd {start|stop|restart|force-reload|status}"
   exit 1
  ;;
esac

exit 0

现在,您可以使用命令/etc/init.d/scriptname start或stop启动和停止python脚本。

You can also make the python script run as a service using a shell script. First create a shell script to run the python script like this (scriptname arbitary name)

#!/bin/sh
script='/home/.. full path to script'
/usr/bin/python $script &

now make a file in /etc/init.d/scriptname

#! /bin/sh

PATH=/bin:/usr/bin:/sbin:/usr/sbin
DAEMON=/home/.. path to shell script scriptname created to run python script
PIDFILE=/var/run/scriptname.pid

test -x $DAEMON || exit 0

. /lib/lsb/init-functions

case "$1" in
  start)
     log_daemon_msg "Starting feedparser"
     start_daemon -p $PIDFILE $DAEMON
     log_end_msg $?
   ;;
  stop)
     log_daemon_msg "Stopping feedparser"
     killproc -p $PIDFILE $DAEMON
     PID=`ps x |grep feed | head -1 | awk '{print $1}'`
     kill -9 $PID       
     log_end_msg $?
   ;;
  force-reload|restart)
     $0 stop
     $0 start
   ;;
  status)
     status_of_proc -p $PIDFILE $DAEMON atd && exit 0 || exit $?
   ;;
 *)
   echo "Usage: /etc/init.d/atd {start|stop|restart|force-reload|status}"
   exit 1
  ;;
esac

exit 0

Now you can start and stop your python script using the command /etc/init.d/scriptname start or stop.


回答 5

一个简单且受支持的版本Daemonize

从Python软件包索引(PyPI)安装它:

$ pip install daemonize

然后像这样使用:

...
import os, sys
from daemonize import Daemonize
...
def main()
      # your code here

if __name__ == '__main__':
        myname=os.path.basename(sys.argv[0])
        pidfile='/tmp/%s' % myname       # any name
        daemon = Daemonize(app=myname,pid=pidfile, action=main)
        daemon.start()

A simple and supported version is Daemonize.

Install it from Python Package Index (PyPI):

$ pip install daemonize

and then use like:

...
import os, sys
from daemonize import Daemonize
...
def main()
      # your code here

if __name__ == '__main__':
        myname=os.path.basename(sys.argv[0])
        pidfile='/tmp/%s' % myname       # any name
        daemon = Daemonize(app=myname,pid=pidfile, action=main)
        daemon.start()

回答 6

cron显然,在许多方面都是不错的选择。但是,它不会按照您在OP中的请求创建服务或守护程序。 cron只是周期性地运行作业(意味着作业开始和停止),并且不超过一次/分钟。出现问题cron-例如,如果您的脚本的先前实例在下次cron计划表出现并启动新实例时仍在运行,可以吗? cron不处理依赖关系;时间表说的话,它只是试图开始工作。

如果发现确实需要守护程序的情况(一个永不停止运行的进程),请看一下supervisord。它提供了一种简单的方法来包装普通的,非守护进程的脚本或程序,并使其像守护进程一样运行。这比创建本地Python守护程序更好。

cron is clearly a great choice for many purposes. However it doesn’t create a service or daemon as you requested in the OP. cron just runs jobs periodically (meaning the job starts and stops), and no more often than once / minute. There are issues with cron — for example, if a prior instance of your script is still running the next time the cron schedule comes around and launches a new instance, is that OK? cron doesn’t handle dependencies; it just tries to start a job when the schedule says to.

If you find a situation where you truly need a daemon (a process that never stops running), take a look at supervisord. It provides a simple way to wrapper a normal, non-daemonized script or program and make it operate like a daemon. This is a much better way than creating a native Python daemon.


回答 7

$nohup在Linux上使用命令怎么样?

我使用它在Bluehost服务器上运行命令。

如果我错了,请指教。

how about using $nohup command on linux?

I use it for running my commands on my Bluehost server.

Please advice if I am wrong.


回答 8

如果您正在使用终端(ssh或其他东西),并且想要从终端注销后保持长时间运行的脚本,则可以尝试以下操作:

screen

apt-get install screen

在内部创建一个虚拟终端(即abc): screen -dmS abc

现在我们连接到abc: screen -r abc

因此,现在我们可以运行python脚本了: python keep_sending_mails.py

从现在开始,您可以直接关闭终端,但是python脚本将继续运行而不是被关闭

由于此keep_sending_mails.pyPID是虚拟屏幕的子进程,而不是终端(ssh)

如果要返回以检查脚本的运行状态,可以screen -r abc再次使用

If you are using terminal(ssh or something) and you want to keep a long-time script working after you log out from the terminal, you can try this:

screen

apt-get install screen

create a virtual terminal inside( namely abc): screen -dmS abc

now we connect to abc: screen -r abc

So, now we can run python script: python keep_sending_mails.py

from now on, you can directly close your terminal, however, the python script will keep running rather than being shut down

Since this keep_sending_mails.py‘s PID is a child process of the virtual screen rather than the terminal(ssh)

If you want to go back check your script running status, you can use screen -r abc again


回答 9

首先,阅读邮件别名。邮件别名将在邮件系统内执行此操作,而您无需四处寻找守护程序或服务或任何类似的内容。

您可以编写一个简单的脚本,该脚本将在每次将邮件发送到特定邮箱时由sendmail执行。

参见http://www.feep.net/sendmail/tutorial/intro/aliases.html

如果您确实想编写不必要的复杂服务器,则可以执行此操作。

nohup python myscript.py &

这就是全部。您的脚本只是循环而进入休眠状态。

import time
def do_the_work():
    # one round of polling -- checking email, whatever.
while True:
    time.sleep( 600 ) # 10 min.
    try:
        do_the_work()
    except:
        pass

First, read up on mail aliases. A mail alias will do this inside the mail system without you having to fool around with daemons or services or anything of the sort.

You can write a simple script that will be executed by sendmail each time a mail message is sent to a specific mailbox.

See http://www.feep.net/sendmail/tutorial/intro/aliases.html

If you really want to write a needlessly complex server, you can do this.

nohup python myscript.py &

That’s all it takes. Your script simply loops and sleeps.

import time
def do_the_work():
    # one round of polling -- checking email, whatever.
while True:
    time.sleep( 600 ) # 10 min.
    try:
        do_the_work()
    except:
        pass

回答 10

我会推荐这种解决方案。您需要继承和重写method run

import sys
import os
from signal import SIGTERM
from abc import ABCMeta, abstractmethod



class Daemon(object):
    __metaclass__ = ABCMeta


    def __init__(self, pidfile):
        self._pidfile = pidfile


    @abstractmethod
    def run(self):
        pass


    def _daemonize(self):
        # decouple threads
        pid = os.fork()

        # stop first thread
        if pid > 0:
            sys.exit(0)

        # write pid into a pidfile
        with open(self._pidfile, 'w') as f:
            print >> f, os.getpid()


    def start(self):
        # if daemon is started throw an error
        if os.path.exists(self._pidfile):
            raise Exception("Daemon is already started")

        # create and switch to daemon thread
        self._daemonize()

        # run the body of the daemon
        self.run()


    def stop(self):
        # check the pidfile existing
        if os.path.exists(self._pidfile):
            # read pid from the file
            with open(self._pidfile, 'r') as f:
                pid = int(f.read().strip())

            # remove the pidfile
            os.remove(self._pidfile)

            # kill daemon
            os.kill(pid, SIGTERM)

        else:
            raise Exception("Daemon is not started")


    def restart(self):
        self.stop()
        self.start()

I would recommend this solution. You need to inherit and override method run.

import sys
import os
from signal import SIGTERM
from abc import ABCMeta, abstractmethod



class Daemon(object):
    __metaclass__ = ABCMeta


    def __init__(self, pidfile):
        self._pidfile = pidfile


    @abstractmethod
    def run(self):
        pass


    def _daemonize(self):
        # decouple threads
        pid = os.fork()

        # stop first thread
        if pid > 0:
            sys.exit(0)

        # write pid into a pidfile
        with open(self._pidfile, 'w') as f:
            print >> f, os.getpid()


    def start(self):
        # if daemon is started throw an error
        if os.path.exists(self._pidfile):
            raise Exception("Daemon is already started")

        # create and switch to daemon thread
        self._daemonize()

        # run the body of the daemon
        self.run()


    def stop(self):
        # check the pidfile existing
        if os.path.exists(self._pidfile):
            # read pid from the file
            with open(self._pidfile, 'r') as f:
                pid = int(f.read().strip())

            # remove the pidfile
            os.remove(self._pidfile)

            # kill daemon
            os.kill(pid, SIGTERM)

        else:
            raise Exception("Daemon is not started")


    def restart(self):
        self.stop()
        self.start()

回答 11

创建一些像服务一样运行的东西,您可以使用以下东西:

您必须做的第一件事是安装Cement框架:Cement框架是一个CLI框架,您可以在其上部署应用程序。

应用程序的命令行界面:

interface.py

 from cement.core.foundation import CementApp
 from cement.core.controller import CementBaseController, expose
 from YourApp import yourApp

 class Meta:
    label = 'base'
    description = "your application description"
    arguments = [
        (['-r' , '--run'],
          dict(action='store_true', help='Run your application')),
        (['-v', '--version'],
          dict(action='version', version="Your app version")),
        ]
        (['-s', '--stop'],
          dict(action='store_true', help="Stop your application")),
        ]

    @expose(hide=True)
    def default(self):
        if self.app.pargs.run:
            #Start to running the your app from there !
            YourApp.yourApp()
        if self.app.pargs.stop:
            #Stop your application
            YourApp.yourApp.stop()

 class App(CementApp):
       class Meta:
       label = 'Uptime'
       base_controller = 'base'
       handlers = [MyBaseController]

 with App() as app:
       app.run()

YourApp.py类:

 import threading

 class yourApp:
     def __init__:
        self.loger = log_exception.exception_loger()
        thread = threading.Thread(target=self.start, args=())
        thread.daemon = True
        thread.start()

     def start(self):
        #Do every thing you want
        pass
     def stop(self):
        #Do some things to stop your application

请记住,您的应用必须在线程上运行才能成为守护进程

要运行该应用程序,只需在命令行中执行此操作

python interface.py-帮助

to creating some thing that is running like service you can use this thing :

The first thing that you must do is installing the Cement framework: Cement frame work is a CLI frame work that you can deploy your application on it.

command line interface of the app :

interface.py

 from cement.core.foundation import CementApp
 from cement.core.controller import CementBaseController, expose
 from YourApp import yourApp

 class Meta:
    label = 'base'
    description = "your application description"
    arguments = [
        (['-r' , '--run'],
          dict(action='store_true', help='Run your application')),
        (['-v', '--version'],
          dict(action='version', version="Your app version")),
        ]
        (['-s', '--stop'],
          dict(action='store_true', help="Stop your application")),
        ]

    @expose(hide=True)
    def default(self):
        if self.app.pargs.run:
            #Start to running the your app from there !
            YourApp.yourApp()
        if self.app.pargs.stop:
            #Stop your application
            YourApp.yourApp.stop()

 class App(CementApp):
       class Meta:
       label = 'Uptime'
       base_controller = 'base'
       handlers = [MyBaseController]

 with App() as app:
       app.run()

YourApp.py class:

 import threading

 class yourApp:
     def __init__:
        self.loger = log_exception.exception_loger()
        thread = threading.Thread(target=self.start, args=())
        thread.daemon = True
        thread.start()

     def start(self):
        #Do every thing you want
        pass
     def stop(self):
        #Do some things to stop your application

Keep in mind that your app must run on a thread to be daemon

To run the app just do this in command line

python interface.py –help


回答 12

使用系统提供的任何服务管理器-例如在Ubuntu下使用upstart。这将为您处理所有详细信息,例如启动时启动,崩溃时重启等。

Use whatever service manager your system offers – for example under Ubuntu use upstart. This will handle all the details for you such as start on boot, restart on crash, etc.


回答 13

假设您真的希望循环将24/7作为后台服务运行

对于不涉及使用库注入代码的解决方案,您可以简单地创建一个服务模板,因为您使用的是Linux:

将该文件放置在守护程序服务文件夹中(通常为/etc/systemd/system/),然后使用以下systemctl命令进行安装(可能需要sudo特权):

systemctl enable <service file name without extension>

systemctl daemon-reload

systemctl start <service file name without extension>

然后可以使用以下命令检查服务是否正在运行:

systemctl | grep running

Assuming that you would really want your loop to run 24/7 as a background service

For a solution that doesn’t involve injecting your code with libraries, you can simply create a service template, since you are using linux:

[Unit]
Description = <Your service description here>
After = network.target # Assuming you want to start after network interfaces are made available
 
[Service]
Type = simple
ExecStart = python <Path of the script you want to run>
User = # User to run the script as
Group = # Group to run the script as
Restart = on-failure # Restart when there are errors
SyslogIdentifier = <Name of logs for the service>
RestartSec = 5
TimeoutStartSec = infinity
 
[Install]
WantedBy = multi-user.target # Make it accessible to other users

Place that file in your daemon service folder (usually /etc/systemd/system/), in a *.service file, and install it using the following systemctl commands (will likely require sudo privileges):

systemctl enable <service file name without .service extension>

systemctl daemon-reload

systemctl start <service file name without .service extension>

You can then check that your service is running by using the command:

systemctl | grep running

如何防止Google Colab断开连接?

问题:如何防止Google Colab断开连接?

问:是否可以通过编程方式防止Google Colab在超时时断开连接?

下面介绍导致笔记本计算机自动断开连接的情况:

Google Colab笔记本的空闲超时为90分钟,绝对超时为12小时。这意味着,如果用户在超过90分钟的时间内未与其Google Colab笔记本互动,则其实例将自动终止。另外,Colab实例的最大生存期为12小时。

自然,我们希望自动将最大值从实例中挤出,而不必不断地手动与之交互。在这里,我将假定常见的系统要求:

  • Ubuntu 18 LTS / Windows 10 / Mac操作系统
  • 对于基于Linux的系统,请使用流行的DE,例如Gnome 3或Unity
  • Firefox或Chromium浏览器

我要在这里指出,这种行为并未违反 Google Colab的使用条款,尽管根据其常见问题解答不鼓励这样做(简而言之:从道德上讲,如果您真的不需要它,则用尽所有GPU是不可行的))。


我当前的解决方案非常愚蠢:

  • 首先,我关闭屏幕保护程序,因此我的屏幕始终保持打开状态。
  • 我有一个Arduino开发板,所以我只是将它变成了一个橡胶鸭子USB,并使其在我睡觉时模拟原始用户交互(只是因为我手边有其他用例)。

有更好的方法吗?

Q: Is there any way to programmatically prevent Google Colab from disconnecting on a timeout?

The following describes the conditions causing a notebook to automatically disconnect:

Google Colab notebooks have an idle timeout of 90 minutes and absolute timeout of 12 hours. This means, if user does not interact with his Google Colab notebook for more than 90 minutes, its instance is automatically terminated. Also, maximum lifetime of a Colab instance is 12 hours.

Naturally, we want to automatically squeeze the maximum out of the instance, without having to manually interact with it constantly. Here I will assume commonly seen system requirements:

  • Ubuntu 18 LTS / Windows 10 / Mac Operating systems
  • In case of Linux-based systems, using popular DEs like Gnome 3 or Unity
  • Firefox or Chromium browsers

I should point out here that such behavior does not violate Google Colab’s Terms of Use, although it is not encouraged according to their FAQ (in short: morally it is not okay to use up all of the GPUs if you don’t really need it).


My current solution is very dumb:

  • First, I turn the screensaver off, so my sreen is always on.
  • I have an Arduino board, so I just turned it into a rubber ducky usb and make it emulate primitive user interaction while I sleep (just because I have it at hand for other use-cases).

Are there better ways?


回答 0

编辑: 显然,该解决方案非常简单,并且不需要任何JavaScript。只需在底部创建具有以下行的新单元格:

while True:pass

现在将单元格保持在运行顺序中,以便无限循环不会停止,从而使会话保持活动状态。

旧方法: 设置一个JavaScript间隔,每60秒点击一次connect按钮。使用Ctrl + Shift + I打开开发人员设置(在您的Web浏览器中),然后单击控制台选项卡,然后在控制台提示符下键入此设置。(对于Mac,请按Option + Command + I)

function ConnectButton(){
    console.log("Connect pushed"); 
    document.querySelector("#top-toolbar > colab-connect-button").shadowRoot.querySelector("#connect").click() 
}
setInterval(ConnectButton,60000);

Edit: Apparently the solution is very easy, and doesn’t need any JavaScript. Just create a new cell at the bottom having the following line:

while True:pass

now keep the cell in the run sequence so that the infinite loop won’t stop and thus keep your session alive.

Old method: Set a javascript interval to click on the connect button every 60 seconds. Open developer-settings (in your web-browser) with Ctrl+Shift+I then click on console tab and type this on the console prompt. (for mac press Option+Command+I)

function ConnectButton(){
    console.log("Connect pushed"); 
    document.querySelector("#top-toolbar > colab-connect-button").shadowRoot.querySelector("#connect").click() 
}
setInterval(ConnectButton,60000);

回答 1

由于现在将连接按钮的ID更改为“ colab-connect-button”,因此可以使用以下代码来继续单击该按钮。

function ClickConnect(){
    console.log("Clicked on connect button"); 
    document.querySelector("colab-connect-button").click()
}
setInterval(ClickConnect,60000)

如果仍然无法解决问题,请按照以下步骤操作:

  1. 右键单击连接按钮(位于colab的右上方)
  2. 点击检查
  3. 获取按钮的HTML ID并替换为以下代码
function ClickConnect(){
    console.log("Clicked on connect button"); 
    document.querySelector("Put ID here").click() // Change id here
}
setInterval(ClickConnect,60000)

Since the id of the connect button is now changed to “colab-connect-button”, the following code can be used to keep clicking on the button.

function ClickConnect(){
    console.log("Clicked on connect button"); 
    document.querySelector("colab-connect-button").click()
}
setInterval(ClickConnect,60000)

If still, this doesn’t work, then follow the steps given below:

  1. Right-click on the connect button (on the top-right side of the colab)
  2. Click on inspect
  3. Get the HTML id of the button and substitute in the following code
function ClickConnect(){
    console.log("Clicked on connect button"); 
    document.querySelector("Put ID here").click() // Change id here
}
setInterval(ClickConnect,60000)

回答 2

嗯,这对我有用-

在控制台中运行以下代码,它将阻止您断开连接。Ctrl + Shift + i打开检查器视图。然后进入控制台。

function ClickConnect(){
    console.log("Working"); 
    document.querySelector("colab-toolbar-button#connect").click() 
}
setInterval(ClickConnect,60000)

如何防止Google Colab断开连接

Well this is working for me –

run the following code in the console and it will prevent you from disconnecting. Ctrl+ Shift + i to open inspector view . Then go to console.

function ClickConnect(){
    console.log("Working"); 
    document.querySelector("colab-toolbar-button#connect").click() 
}
setInterval(ClickConnect,60000)

How to prevent google colab from disconnecting


回答 3

对我而言,以下示例:

  • document.querySelector("#connect").click() 要么
  • document.querySelector("colab-toolbar-button#connect").click() 要么
  • document.querySelector("colab-connect-button").click()

抛出错误。

我必须使它们适应以下条件:

版本1:

function ClickConnect(){
  console.log("Connnect Clicked - Start"); 
  document.querySelector("#top-toolbar > colab-connect-button").shadowRoot.querySelector("#connect").click();
  console.log("Connnect Clicked - End"); 
};
setInterval(ClickConnect, 60000)

版本2: 如果您希望能够停止该功能,请使用以下新代码:

var startClickConnect = function startClickConnect(){
    var clickConnect = function clickConnect(){
        console.log("Connnect Clicked - Start");
        document.querySelector("#top-toolbar > colab-connect-button").shadowRoot.querySelector("#connect").click();
        console.log("Connnect Clicked - End"); 
    };

    var intervalId = setInterval(clickConnect, 60000);

    var stopClickConnectHandler = function stopClickConnect() {
        console.log("Connnect Clicked Stopped - Start");
        clearInterval(intervalId);
        console.log("Connnect Clicked Stopped - End");
    };

    return stopClickConnectHandler;
};

var stopClickConnect = startClickConnect();

为了停止,请调用:

stopClickConnect();

For me the following examples:

  • document.querySelector("#connect").click() or
  • document.querySelector("colab-toolbar-button#connect").click() or
  • document.querySelector("colab-connect-button").click()

were throwing errors.

I had to adapt them to the following:

Version 1:

function ClickConnect(){
  console.log("Connnect Clicked - Start"); 
  document.querySelector("#top-toolbar > colab-connect-button").shadowRoot.querySelector("#connect").click();
  console.log("Connnect Clicked - End"); 
};
setInterval(ClickConnect, 60000)

Version 2: If you would like to be able to stop the function, here is the new code:

var startClickConnect = function startClickConnect(){
    var clickConnect = function clickConnect(){
        console.log("Connnect Clicked - Start");
        document.querySelector("#top-toolbar > colab-connect-button").shadowRoot.querySelector("#connect").click();
        console.log("Connnect Clicked - End"); 
    };

    var intervalId = setInterval(clickConnect, 60000);

    var stopClickConnectHandler = function stopClickConnect() {
        console.log("Connnect Clicked Stopped - Start");
        clearInterval(intervalId);
        console.log("Connnect Clicked Stopped - End");
    };

    return stopClickConnectHandler;
};

var stopClickConnect = startClickConnect();

In order to stop, call:

stopClickConnect();

回答 4

使用Pynput在您的PC中创建python代码

from pynput.mouse import Button, Controller
import time

mouse = Controller()

while True:
    mouse.click(Button.left, 1)
    time.sleep(30)

在您的桌面上运行此代码,然后将鼠标指针悬停在任何目录的目录结构上(左侧的左侧栏-文件部分),此代码将每30秒不断单击一次目录,因此每30秒将展开和缩小一次,因此您的会话不会过期重要-您必须在PC中运行此代码

create a python code in your pc with pynput

from pynput.mouse import Button, Controller
import time

mouse = Controller()

while True:
    mouse.click(Button.left, 1)
    time.sleep(30)

Run this code in your Desktop, Then point mouse arrow over (colabs left panel – file section) directory structure on any directory this code will keep clicking on directory on every 30 seconds so it will expand and shrink every 30 seconds so your session will not get expired Important – you have to run this code in your pc


回答 5

我没有单击“连接”按钮,而是单击“评论”按钮以使会话保持活动状态。(2020年8月)

function ClickConnect(){

console.log("Working"); 
document.querySelector("#comments > span").click() 
}
setInterval(ClickConnect,5000)

Instead of clicking the connect button, i just clicking on comment button to keep my session alive. (August-2020)

function ClickConnect(){

console.log("Working"); 
document.querySelector("#comments > span").click() 
}
setInterval(ClickConnect,5000)

回答 6

我使用宏程序定期单击RAM / Disk按钮以整夜训练模型。诀窍是配置一个宏程序,以两次单击Ram / Disk Colab工具栏按钮,两次单击之间的间隔很短,这样即使运行时断开连接,它也将重新连接。(第一次单击用于关闭对话框,第二次单击用于重新连接)。但是,您仍然必须整夜打开笔记本电脑,甚至可以固定Colab标签。

I use a Macro Program to periodically click on the RAM/Disk button to train the model all night. The trick is to configure a macro program to click on the Ram/Disk Colab Toolbar Button twice with a short interval between the two clicks so that even if the Runtime gets disconnected it will reconnect back. (the first click used to close the dialog box and the second click used to RECONNECT). However, you still have to leave your laptop open all night and maybe pin the Colab tab.


回答 7

在某些脚本的帮助下,以上答案可能效果很好。对于没有脚本的烦人的断开连接,我有一个解决方案(或一种技巧),尤其是当您的程序必须从google驱动器读取数据时,例如训练深度学习网络模型时,使用脚本进行reconnect操作就没有用了,因为一旦您断开与colab的连接,该程序就死了,应该再次手动连接到Google驱动器,以使您的模型能够再次读取数据集,但是脚本不会执行此操作。
我已经测试了很多次,并且效果很好。
当您使用浏览器(我使用Chrome)在colab页面上运行程序时,请记住,一旦程序开始运行,就不要对浏览器进行任何操作,例如:切换到其他网页,打开或关闭另一个网页,以及依此类推,只需将其放置在那里,等待程序完成运行,就可以切换到pycharm等其他软件来继续编写代码,而不必切换到另一个网页。我不知道为什么打开或关闭或切换到其他页面会导致google colab页面的连接问题,但是每次我尝试打扰我的浏览器(如执行某些搜索工作)时,我与colab的连接都会很快断开。

The above answers with the help of some scripts maybe work well. I have a solution(or a kind of trick) for that annoying disconnection without scripts, especially when your program must read data from your google drive, like training a deep learning network model, where using scripts to do reconnect operation is of no use because once you disconnect with your colab, the program is just dead, you should manually connect to your google drive again to make your model able to read dataset again, but the scripts will not do that thing.
I’ve already test it many times and it works well.
When you run a program on the colab page with a browser(I use Chrome), just remember that don’t do any operation to your browser once your program starts running, like: switch to other webpages, open or close another webpage, and so on, just just leave it alone there and waiting for your program finish running, you can switch to another software, like pycharm to keep writing your codes but not switch to another webpage. I don’t know why open or close or switch to other pages will cause the connection problem of the google colab page, but each time I try to bothered my browser, like do some search job, my connection to colab will soon break down.


回答 8

尝试这个:

function ClickConnect(){
  console.log("Working"); 
  document
    .querySelector("#top-toolbar > colab-connect-button")
    .shadowRoot
    .querySelector("#connect")
    .click()
}

setInterval(ClickConnect,60000)

Try this:

function ClickConnect(){
  console.log("Working"); 
  document
    .querySelector("#top-toolbar > colab-connect-button")
    .shadowRoot
    .querySelector("#connect")
    .click()
}

setInterval(ClickConnect,60000)

回答 9

使用python硒

from selenium.webdriver.common.keys import Keys
from selenium import webdriver
import time   

driver = webdriver.Chrome('/usr/lib/chromium-browser/chromedriver')

notebook_url = ''
driver.get(notebook_url)

# run all cells
driver.find_element_by_tag_name('body').send_keys(Keys.CONTROL + Keys.F9)
time.sleep(5)

# click to stay connected
start_time = time.time()
current_time = time.time()
max_time = 11*59*60 #12hours

while (current_time - start_time) < max_time:
    webdriver.ActionChains(driver).send_keys(Keys.ESCAPE).perform()
    driver.find_element_by_xpath('//*[@id="top-toolbar"]/colab-connect-button').click()
    time.sleep(30)
    current_time = time.time()

Using python selenium

from selenium.webdriver.common.keys import Keys
from selenium import webdriver
import time   

driver = webdriver.Chrome('/usr/lib/chromium-browser/chromedriver')

notebook_url = ''
driver.get(notebook_url)

# run all cells
driver.find_element_by_tag_name('body').send_keys(Keys.CONTROL + Keys.F9)
time.sleep(5)

# click to stay connected
start_time = time.time()
current_time = time.time()
max_time = 11*59*60 #12hours

while (current_time - start_time) < max_time:
    webdriver.ActionChains(driver).send_keys(Keys.ESCAPE).perform()
    driver.find_element_by_xpath('//*[@id="top-toolbar"]/colab-connect-button').click()
    time.sleep(30)
    current_time = time.time()

回答 10

我认为JavaScript解决方案不再有效。我在笔记本中使用以下命令进行操作:

    from IPython.display import display, HTML
    js = ('<script>function ConnectButton(){ '
           'console.log("Connect pushed"); '
           'document.querySelector("#connect").click()} '
           'setInterval(ConnectButton,3000);</script>')
    display(HTML(js))

首次执行全部运行时(在启动JavaScript或Python代码之前),控制台将显示:

Connected to 
wss://colab.research.google.com/api/kernels/0e1ce105-0127-4758-90e48cf801ce01a3/channels?session_id=5d8...

但是,每次运行JavaScript时,您都会看到console.log部分,但是click部分仅给出:

Connect pushed

Uncaught TypeError: Cannot read property 'click' of null
 at ConnectButton (<anonymous>:1:92)

其他人建议将按钮名称更改为#colab-connect-button,但这会产生相同的错误。

启动运行系统后,该按钮将更改为显示RAM / DISK,并显示一个下拉列表。单击下拉列表创建一个<DIV class=goog menu...>以前未在DOM中显示的新内容,并带有2个选项“连接到托管运行时”和“连接到本地运行时”。如果控制台窗口已打开并显示元素,则在单击下拉元素时可以看到此DIV出现。只需在出现的新窗口中的两个选项之间移动鼠标焦点,即可向DOM添加其他元素,一旦鼠标释放焦点,它们便会从DOM中完全删除,甚至无需单击即可。

I don’t believe the JavaScript solutions work anymore. I was doing it from within my notebook with:

    from IPython.display import display, HTML
    js = ('<script>function ConnectButton(){ '
           'console.log("Connect pushed"); '
           'document.querySelector("#connect").click()} '
           'setInterval(ConnectButton,3000);</script>')
    display(HTML(js))

When you first do a Run all (before the JavaScript or Python code has started), the console displays:

Connected to 
wss://colab.research.google.com/api/kernels/0e1ce105-0127-4758-90e48cf801ce01a3/channels?session_id=5d8...

However, ever time the JavaScript runs, you see the console.log portion, but the click portion simply gives:

Connect pushed

Uncaught TypeError: Cannot read property 'click' of null
 at ConnectButton (<anonymous>:1:92)

Others suggested the button name has changed to #colab-connect-button, but that gives same error.

After the runtime is started, the button is changed to show RAM/DISK, and a drop down is presented. Clicking on the drop down creates a new <DIV class=goog menu...> that was not shown in the DOM previously, with 2 options “Connect to hosted runtime” and “Connect to local runtime”. If the console window is open and showing elements, you can see this DIV appear when you click the dropdown element. Simply moving the mouse focus between the two options in the new window that appears adds additional elements to the DOM, as soon as the mouse looses focus, they are removed from the DOM completely, even without clicking.


回答 11

我尝试了上面的代码,但它们对我不起作用。这是我重新连接的JS代码。

let interval = setInterval(function(){
let ok = document.getElementById('ok');
if(ok != null){
   console.log("Connect pushed");
ok.click();
}},60000)

您可以使用相同的方式(在浏览器的控制台上运行)来运行它。如果要停止脚本,可以输入clearInterval(interval)并再次运行setInterval(interval)

我希望这可以帮助你。

I tried the codes above but they did not work for me. So here is my JS code for reconnecting.

let interval = setInterval(function(){
let ok = document.getElementById('ok');
if(ok != null){
   console.log("Connect pushed");
ok.click();
}},60000)

You can use it with the same way (run it on the console of your browser) to run it. If you want to stop the script, you can enter clearInterval(interval) and want to run again setInterval(interval).

I hope this helps you.


回答 12

更新了一个。这个对我有用。

function ClickConnect(){
console.log("Working"); 
document.querySelector("paper-icon-button").click()
}
Const myjob = setInterval(ClickConnect, 60000)

如果对您不起作用,请尝试运行以下命令清除它:

clearInterval(myjob)

Updated one. it works for me.

function ClickConnect(){
console.log("Working"); 
document.querySelector("paper-icon-button").click()
}
Const myjob = setInterval(ClickConnect, 60000)

If isn’t working you for you guys try clear it by running:

clearInterval(myjob)

回答 13

这对我有用(似乎他们更改了按钮的类名或ID):

function ClickConnect(){
    console.log("Working"); 
    document.querySelector("colab-connect-button").click() 
}
setInterval(ClickConnect,60000)

This one worked for me (it seems like they changed the button classname or id) :

function ClickConnect(){
    console.log("Working"); 
    document.querySelector("colab-connect-button").click() 
}
setInterval(ClickConnect,60000)

回答 14

投票得最多的答案当然对我有用,但这会使“管理会话”窗口一次又一次地弹出。
我已经解决了这一问题,方法是使用浏览器控制台如下所示自动单击刷新按钮

function ClickRefresh(){
    console.log("Clicked on refresh button"); 
    document.querySelector("paper-icon-button").click()
}
setInterval(ClickRefresh, 60000)

随时在此要点上为此贡献更多代码片段https://gist.github.com/Subangkar/fd1ef276fd40dc374a7c80acc247613e

The most voted answer certainly works for me but it makes the Manage session window popping up again and again.
I’ve solved that by auto clicking the refresh button using browser console like below

function ClickRefresh(){
    console.log("Clicked on refresh button"); 
    document.querySelector("paper-icon-button").click()
}
setInterval(ClickRefresh, 60000)

Feel free to contribute more snippets for this at this gist https://gist.github.com/Subangkar/fd1ef276fd40dc374a7c80acc247613e


回答 15

也许以前的许多解决方案都不再起作用。例如,下面的代码继续在Colab中创建新的代码单元,但仍在工作。无疑,创建一堆代码单元是一个不便之处。如果在运行几个小时后创建了太多的代码单元,而没有足够的RAM,则浏览器可能会冻结。

反复创建代码单元-

function ClickConnect(){
console.log("Working"); 
document.querySelector("colab-toolbar-button").click() 
}setInterval(ClickConnect,60000)

但是我发现下面的代码正在运行,它不会引起任何问题。在Colab笔记本选项卡中,Ctrl + Shift + i同时单击该键,然后将以下代码粘贴到控制台中。120000个间隔就足够了。

function ClickConnect(){
console.log("Working"); 
document.querySelector("colab-toolbar-button#connect").click() 
}setInterval(ClickConnect,120000)

我已在2020年11月在firefox中测试了此代码。它也将在chrome上工作。

Perhaps many of the previous solutions are no longer working. For example, this bellow code continues to create new code cells in Colab, working though. Undoubtedly, creating a bunch of code cells is an inconvenience. If too many code cells are created in some hours of running and there is no enough RAM, the browser may freeze.

This repetedly creates code cells—

function ClickConnect(){
console.log("Working"); 
document.querySelector("colab-toolbar-button").click() 
}setInterval(ClickConnect,60000)

But I found the code below is working, it doesn’t cause any problems. In the Colab notebook tab, click on the Ctrl + Shift + i key simultaneously and paste the below code in the console. 120000 intervals are enough.

function ClickConnect(){
console.log("Working"); 
document.querySelector("colab-toolbar-button#connect").click() 
}setInterval(ClickConnect,120000)

I have tested this code in firefox, in November 2020. It will work on chrome too.


回答 16

我建议使用JQuery(似乎Co-lab默认包含JQuery)。

function ClickConnect(){
  console.log("Working");
  $("colab-toolbar-button").click();
}
setInterval(ClickConnect,60000);

I would recommend using JQuery (It seems that Co-lab includes JQuery by default).

function ClickConnect(){
  console.log("Working");
  $("colab-toolbar-button").click();
}
setInterval(ClickConnect,60000);

回答 17

这些JavaScript函数存在问题:

function ClickConnect(){
    console.log("Clicked on connect button"); 
    document.querySelector("colab-connect-button").click()
}
setInterval(ClickConnect,60000)

他们实际在单击按钮之前在控制台上打印“ Clicked on connect button”。从该线程的不同答案中可以看出,自Google Colab启动以来,connect按钮的ID已经更改了两次。而且将来也可能会更改。因此,如果您打算从该线程中复制旧答案,则可能会说“单击了连接按钮”,但实际上可能不会这样做。当然,如果单击不起作用,它将在控制台上显示一个错误,但是如果您可能不会意外看到该怎么办?因此,您最好这样做:

function ClickConnect(){
    document.querySelector("colab-connect-button").click()
    console.log("Clicked on connect button"); 
}
setInterval(ClickConnect,60000)

您肯定会看到它是否真正起作用。

I have a problem with these javascript functions:

function ClickConnect(){
    console.log("Clicked on connect button"); 
    document.querySelector("colab-connect-button").click()
}
setInterval(ClickConnect,60000)

They print the “Clicked on connect button” on the console before the button is actually clicked. As you can see from different answers in this thread, the id of the connect button has changed a couple of times since Google Colab was launched. And it could be changed in the future as well. So if you’re going to copy an old answer from this thread it may say “Clicked on connect button” but it may actually not do that. Of course if the clicking won’t work it will print an error on the console but what if you may not accidentally see it? So you better do this:

function ClickConnect(){
    document.querySelector("colab-connect-button").click()
    console.log("Clicked on connect button"); 
}
setInterval(ClickConnect,60000)

And you’ll definitely see if it truly works or not.


回答 18

function ClickConnect()
{
    console.log("Working...."); 
    document.querySelector("paper-button#comments").click()
}
setInterval(ClickConnect,600)

这对我有用,但明智地使用

快乐学习:)

function ClickConnect()
{
    console.log("Working...."); 
    document.querySelector("paper-button#comments").click()
}
setInterval(ClickConnect,600)

this worked for me but use wisely

happy learning :)


回答 19

以下最新解决方案适用于我:

function ClickConnect(){
  colab.config
  console.log("Connnect Clicked - Start"); 
  document.querySelector("#top-toolbar > colab-connect-button").shadowRoot.querySelector("#connect").click();
  console.log("Connnect Clicked - End");
};
setInterval(ClickConnect, 60000)

the following LATEST solution works for me:

function ClickConnect(){
  colab.config
  console.log("Connnect Clicked - Start"); 
  document.querySelector("#top-toolbar > colab-connect-button").shadowRoot.querySelector("#connect").click();
  console.log("Connnect Clicked - End");
};
setInterval(ClickConnect, 60000)

回答 20

下面的javascript对我有用。学分@ artur.k.space

function ColabReconnect() {
    var dialog = document.querySelector("colab-dialog.yes-no-dialog");
    var dialogTitle = dialog && dialog.querySelector("div.content-area>h2");
    if (dialogTitle && dialogTitle.innerText == "Runtime disconnected") {
        dialog.querySelector("paper-button#ok").click();
        console.log("Reconnecting...");
    } else {
        console.log("ColabReconnect is in service.");
    }
}
timerId = setInterval(ColabReconnect, 60000);

在Colab笔记本中,同时单击Ctrl + Shift +i键。将脚本复制并粘贴到提示行中。然后Enter在关闭编辑器之前点击。

这样,该功能将每60秒检查一次,以查看是否显示了屏幕连接对话框,如果显示,则该功能将ok自动为您单击该按钮。

The javascript below works for me. Credits to @artur.k.space.

function ColabReconnect() {
    var dialog = document.querySelector("colab-dialog.yes-no-dialog");
    var dialogTitle = dialog && dialog.querySelector("div.content-area>h2");
    if (dialogTitle && dialogTitle.innerText == "Runtime disconnected") {
        dialog.querySelector("paper-button#ok").click();
        console.log("Reconnecting...");
    } else {
        console.log("ColabReconnect is in service.");
    }
}
timerId = setInterval(ColabReconnect, 60000);

In the Colab notebook, click on Ctrl + Shift + the i key simultaneously. Copy and paste the script into the prompt line. Then hit Enter before closing the editor.

By doing so, the function will check every 60 seconds to see if the onscreen connection dialog is shown, and if it is, the function would then click the ok button automatically for you.


回答 21

好吧,我不是python家伙,也不知道这个’Colab’的实际用途是什么,我将其用作构建系统。我以前在其中设置了ssh转发,然后将这段代码放到运行中,是的。

import getpass
authtoken = getpass.getpass()

Well I am not a python guy nor I know what is the actual use of this ‘Colab’, I use it as a build system lol. And I used to setup ssh forwarding in it then put this code and just leave it running and yeah it works.

import getpass
authtoken = getpass.getpass()

回答 22

此代码在文件资源管理器窗格中单击“刷新文件夹”。

function ClickRefresh(){
  console.log("Working"); 
  document.querySelector("[icon='colab:folder-refresh']").click()
}
const myjob = setInterval(ClickRefresh, 60000)

This code keep clicking “Refresh folder” in the file explorer pane.

function ClickRefresh(){
  console.log("Working"); 
  document.querySelector("[icon='colab:folder-refresh']").click()
}
const myjob = setInterval(ClickRefresh, 60000)

回答 23

GNU Colab使您可以在Colaboratory实例之上运行标准的持久桌面环境。

实际上,它包含一种不让机器死掉的机制。

这是一个视频演示

GNU Colab lets you run a standard persistent desktop environment on top of a Colaboratory instance.

Indeed it contains a mechanism to not let machines die of idling.

Here’s a video demonstration.


回答 24

您也可以使用Python按下箭头键。我也在以下代码中添加了一些随机性。

from pyautogui import press, typewrite, hotkey
import time
from random import shuffle

array = ["left", "right", "up", "down"]

while True:
    shuffle(array)
    time.sleep(10)
    press(array[0])
    press(array[1])
    press(array[2])
    press(array[3])

You can also use Python to press the arrow keys. I added a little bit of randomness in the following code as well.

from pyautogui import press, typewrite, hotkey
import time
from random import shuffle

array = ["left", "right", "up", "down"]

while True:
    shuffle(array)
    time.sleep(10)
    press(array[0])
    press(array[1])
    press(array[2])
    press(array[3])

回答 25

只需在要运行的单元格之后运行以下代码,以防止数据丢失。

!python

同样要退出此模式,请写

exit()

Just run the code below after the cell you want to run to save from data loss.

!python

Also to exit from this mode, write

exit()

回答 26

我一直在寻找解决方案,直到找到一个Python3,该Python3总是在同一位置来回移动鼠标并单击,但这足以使Colab误以为我在笔记本电脑上很活跃并且没有断开连接。

import numpy as np
import time
import mouse
import threading

def move_mouse():
    while True:
        random_row = np.random.random_sample()*100
        random_col = np.random.random_sample()*10
        random_time = np.random.random_sample()*np.random.random_sample() * 100
        mouse.wheel(1000)
        mouse.wheel(-1000)
        mouse.move(random_row, random_col, absolute=False, duration=0.2)
        mouse.move(-random_row, -random_col, absolute=False, duration = 0.2)
        mouse.LEFT
        time.sleep(random_time)


x = threading.Thread(target=move_mouse)
x.start()

您需要安装所需的软件包:sudo -H pip3 install <package_name> 您只需要使用(在本地计算机中)运行它即可(sudo因为它可以控制鼠标)并且它应该可以工作,从而使您能够充分利用Colab的12h会话。

积分: 对于使用Colab(Pro)的用户:防止会话由于不活动而断开连接

I was looking for a solution until I found a Python3 that randomly moves the mouse back and forth and clicks, always on the same place, but that’s enough to fool Colab into thinking I’m active on the notebook and not disconnect.

import numpy as np
import time
import mouse
import threading

def move_mouse():
    while True:
        random_row = np.random.random_sample()*100
        random_col = np.random.random_sample()*10
        random_time = np.random.random_sample()*np.random.random_sample() * 100
        mouse.wheel(1000)
        mouse.wheel(-1000)
        mouse.move(random_row, random_col, absolute=False, duration=0.2)
        mouse.move(-random_row, -random_col, absolute=False, duration = 0.2)
        mouse.LEFT
        time.sleep(random_time)


x = threading.Thread(target=move_mouse)
x.start()

You need to install the needed packages: sudo -H pip3 install <package_name> You just need to run it (in your local machine) with sudo (as it takes control of the mouse) and it should work, allowing you to take full advantage of Colab’s 12h sessions.

Credits: For those using Colab (Pro): Preventing Session from disconnecting due to inactivity


回答 27

function ClickConnect(){
    console.log("Clicked on connect button"); 
    document.querySelector("connect").click() // Change id here
}
setInterval(ClickConnect,60000)

试试上面对我有用的代码:)

function ClickConnect(){
    console.log("Clicked on connect button"); 
    document.querySelector("connect").click() // Change id here
}
setInterval(ClickConnect,60000)

Try above code it worked for me:)


在Alpine Linux上安装Pillow时,没有这样的文件或目录“ limits.h”

问题:在Alpine Linux上安装Pillow时,没有这样的文件或目录“ limits.h”

我在Raspberry Pi 2上运行alpine-linux。我正在尝试通过以下命令安装Pillow:

pip install pillow

这是命令的输出:

Installing collected packages: pillow
Running setup.py install for pillow
    Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-gNq0WA/pillow/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-nDKwei-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-armv7l-2.7
    creating build/lib.linux-armv7l-2.7/PIL
    copying PIL/XVThumbImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/XpmImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/XbmImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/WmfImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/WebPImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/WalImageFile.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/TiffTags.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/TiffImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/TgaImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/TarIO.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/SunImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/SpiderImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/SgiImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/PyAccess.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/PSDraw.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/PsdImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/PpmImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/PngImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/PixarImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/PdfImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/PcxImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/PcfFontFile.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/PcdImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/PalmImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/PaletteFile.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/OleFileIO.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/MspImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/MpoImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/MpegImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/MicImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/McIdasImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/JpegPresets.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/JpegImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/Jpeg2KImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/IptcImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImtImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageWin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageTransform.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageTk.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageStat.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageShow.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageSequence.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageQt.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImagePath.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImagePalette.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageOps.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageMorph.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageMode.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageMath.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageGrab.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageFont.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageFilter.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageFileIO.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageFile.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageEnhance.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageDraw2.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageDraw.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageColor.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageCms.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageChops.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/Image.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/IcoImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/IcnsImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/Hdf5StubImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/GribStubImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/GimpPaletteFile.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/GimpGradientFile.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/GifImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/GdImageFile.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/GbrImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/FpxImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/FontFile.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/FliImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/FitsStubImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ExifTags.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/EpsImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/DcxImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/CurImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ContainerIO.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/BufrStubImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/BmpImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/BdfFontFile.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/_util.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/_binary.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/__init__.py -> build/lib.linux-armv7l-2.7/PIL
    running egg_info
    writing Pillow.egg-info/PKG-INFO
    writing top-level names to Pillow.egg-info/top_level.txt
    writing dependency_links to Pillow.egg-info/dependency_links.txt
    warning: manifest_maker: standard file '-c' not found

    reading manifest file 'Pillow.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    warning: no files found matching 'LICENSE' under directory 'docs'
    writing manifest file 'Pillow.egg-info/SOURCES.txt'
    copying PIL/OleFileIO-README.md -> build/lib.linux-armv7l-2.7/PIL
    running build_ext
    building 'PIL._imaging' extension
    creating build/temp.linux-armv7l-2.7/libImaging
    gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -DHAVE_LIBJPEG -I/tmp/pip-build-gNq0WA/pillow/libImaging -I/usr/include -I/usr/include/python2.7 -c _imaging.c -o build/temp.linux-armv7l-2.7/_imaging.o
    gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -DHAVE_LIBJPEG -I/tmp/pip-build-gNq0WA/pillow/libImaging -I/usr/include -I/usr/include/python2.7 -c outline.c -o build/temp.linux-armv7l-2.7/outline.o
    gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -DHAVE_LIBJPEG -I/tmp/pip-build-gNq0WA/pillow/libImaging -I/usr/include -I/usr/include/python2.7 -c libImaging/Bands.c -o build/temp.linux-armv7l-2.7/libImaging/Bands.o
    gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -DHAVE_LIBJPEG -I/tmp/pip-build-gNq0WA/pillow/libImaging -I/usr/include -I/usr/include/python2.7 -c libImaging/ConvertYCbCr.c -o build/temp.linux-armv7l-2.7/libImaging/ConvertYCbCr.o
    In file included from _imaging.c:76:0:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.
    In file included from outline.c:20:0:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.
    In file included from libImaging/ImPlatform.h:10:0,
                    from libImaging/Imaging.h:14,
                    from libImaging/ConvertYCbCr.c:15:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.
    In file included from libImaging/ImPlatform.h:10:0,
                    from libImaging/Imaging.h:14,
                    from libImaging/Bands.c:19:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.
    gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -DHAVE_LIBJPEG -I/tmp/pip-build-gNq0WA/pillow/libImaging -I/usr/include -I/usr/include/python2.7 -c libImaging/Draw.c -o build/temp.linux-armv7l-2.7/libImaging/Draw.o
    gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -DHAVE_LIBJPEG -I/tmp/pip-build-gNq0WA/pillow/libImaging -I/usr/include -I/usr/include/python2.7 -c libImaging/Filter.c -o build/temp.linux-armv7l-2.7/libImaging/Filter.o
    gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -DHAVE_LIBJPEG -I/tmp/pip-build-gNq0WA/pillow/libImaging -I/usr/include -I/usr/include/python2.7 -c libImaging/GifEncode.c -o build/temp.linux-armv7l-2.7/libImaging/GifEncode.o
    gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -DHAVE_LIBJPEG -I/tmp/pip-build-gNq0WA/pillow/libImaging -I/usr/include -I/usr/include/python2.7 -c libImaging/LzwDecode.c -o build/temp.linux-armv7l-2.7/libImaging/LzwDecode.o
    In file included from libImaging/ImPlatform.h:10:0,
                    from libImaging/Imaging.h:14,
                    from libImaging/Draw.c:35:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.
    In file included from libImaging/ImPlatform.h:10:0,
                    from libImaging/Imaging.h:14,
                    from libImaging/Filter.c:27:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.
    In file included from libImaging/ImPlatform.h:10:0,
                    from libImaging/Imaging.h:14,
                    from libImaging/GifEncode.c:20:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.
    In file included from libImaging/ImPlatform.h:10:0,
                    from libImaging/Imaging.h:14,
                    from libImaging/LzwDecode.c:31:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.
    gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -DHAVE_LIBJPEG -I/tmp/pip-build-gNq0WA/pillow/libImaging -I/usr/include -I/usr/include/python2.7 -c libImaging/Offset.c -o build/temp.linux-armv7l-2.7/libImaging/Offset.o
    gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -DHAVE_LIBJPEG -I/tmp/pip-build-gNq0WA/pillow/libImaging -I/usr/include -I/usr/include/python2.7 -c libImaging/Quant.c -o build/temp.linux-armv7l-2.7/libImaging/Quant.o
    gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -DHAVE_LIBJPEG -I/tmp/pip-build-gNq0WA/pillow/libImaging -I/usr/include -I/usr/include/python2.7 -c libImaging/PcxDecode.c -o build/temp.linux-armv7l-2.7/libImaging/PcxDecode.o
    gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -DHAVE_LIBJPEG -I/tmp/pip-build-gNq0WA/pillow/libImaging -I/usr/include -I/usr/include/python2.7 -c libImaging/RawEncode.c -o build/temp.linux-armv7l-2.7/libImaging/RawEncode.o
    In file included from libImaging/ImPlatform.h:10:0,
                    from libImaging/Imaging.h:14,
                    from libImaging/Offset.c:18:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.
    In file included from libImaging/ImPlatform.h:10:0,
                    from libImaging/Imaging.h:14,
                    from libImaging/Quant.c:21:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.
    In file included from libImaging/ImPlatform.h:10:0,
                    from libImaging/Imaging.h:14,
                    from libImaging/PcxDecode.c:17:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.
    In file included from libImaging/ImPlatform.h:10:0,
                    from libImaging/Imaging.h:14,
                    from libImaging/RawEncode.c:21:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.
    gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -DHAVE_LIBJPEG -I/tmp/pip-build-gNq0WA/pillow/libImaging -I/usr/include -I/usr/include/python2.7 -c libImaging/UnpackYCC.c -o build/temp.linux-armv7l-2.7/libImaging/UnpackYCC.o
    gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -DHAVE_LIBJPEG -I/tmp/pip-build-gNq0WA/pillow/libImaging -I/usr/include -I/usr/include/python2.7 -c libImaging/ZipEncode.c -o build/temp.linux-armv7l-2.7/libImaging/ZipEncode.o
    gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -DHAVE_LIBJPEG -I/tmp/pip-build-gNq0WA/pillow/libImaging -I/usr/include -I/usr/include/python2.7 -c libImaging/BoxBlur.c -o build/temp.linux-armv7l-2.7/libImaging/BoxBlur.o
    In file included from libImaging/ImPlatform.h:10:0,
                    from libImaging/Imaging.h:14,
                    from libImaging/UnpackYCC.c:17:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.
    In file included from libImaging/ImPlatform.h:10:0,
                    from libImaging/Imaging.h:14,
                    from libImaging/ZipEncode.c:18:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.
    In file included from libImaging/BoxBlur.c:1:0:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.
    Building using 4 processes
    gcc -shared -Wl,--as-needed build/temp.linux-armv7l-2.7/_imaging.o build/temp.linux-armv7l-2.7/decode.o build/temp.linux-armv7l-2.7/encode.o build/temp.linux-armv7l-2.7/map.o build/temp.linux-armv7l-2.7/display.o build/temp.linux-armv7l-2.7/outline.o build/temp.linux-armv7l-2.7/path.o build/temp.linux-armv7l-2.7/libImaging/Access.o build/temp.linux-armv7l-2.7/libImaging/AlphaComposite.o build/temp.linux-armv7l-2.7/libImaging/Resample.o build/temp.linux-armv7l-2.7/libImaging/Bands.o build/temp.linux-armv7l-2.7/libImaging/BitDecode.o build/temp.linux-armv7l-2.7/libImaging/Blend.o build/temp.linux-armv7l-2.7/libImaging/Chops.o build/temp.linux-armv7l-2.7/libImaging/Convert.o build/temp.linux-armv7l-2.7/libImaging/ConvertYCbCr.o build/temp.linux-armv7l-2.7/libImaging/Copy.o build/temp.linux-armv7l-2.7/libImaging/Crc32.o build/temp.linux-armv7l-2.7/libImaging/Crop.o build/temp.linux-armv7l-2.7/libImaging/Dib.o build/temp.linux-armv7l-2.7/libImaging/Draw.o build/temp.linux-armv7l-2.7/libImaging/Effects.o build/temp.linux-armv7l-2.7/libImaging/EpsEncode.o build/temp.linux-armv7l-2.7/libImaging/File.o build/temp.linux-armv7l-2.7/libImaging/Fill.o build/temp.linux-armv7l-2.7/libImaging/Filter.o build/temp.linux-armv7l-2.7/libImaging/FliDecode.o build/temp.linux-armv7l-2.7/libImaging/Geometry.o build/temp.linux-armv7l-2.7/libImaging/GetBBox.o build/temp.linux-armv7l-2.7/libImaging/GifDecode.o build/temp.linux-armv7l-2.7/libImaging/GifEncode.o build/temp.linux-armv7l-2.7/libImaging/HexDecode.o build/temp.linux-armv7l-2.7/libImaging/Histo.o build/temp.linux-armv7l-2.7/libImaging/JpegDecode.o build/temp.linux-armv7l-2.7/libImaging/JpegEncode.o build/temp.linux-armv7l-2.7/libImaging/LzwDecode.o build/temp.linux-armv7l-2.7/libImaging/Matrix.o build/temp.linux-armv7l-2.7/libImaging/ModeFilter.o build/temp.linux-armv7l-2.7/libImaging/MspDecode.o build/temp.linux-armv7l-2.7/libImaging/Negative.o build/temp.linux-armv7l-2.7/libImaging/Offset.o build/temp.linux-armv7l-2.7/libImaging/Pack.o build/temp.linux-armv7l-2.7/libImaging/PackDecode.o build/temp.linux-armv7l-2.7/libImaging/Palette.o build/temp.linux-armv7l-2.7/libImaging/Paste.o build/temp.linux-armv7l-2.7/libImaging/Quant.o build/temp.linux-armv7l-2.7/libImaging/QuantOctree.o build/temp.linux-armv7l-2.7/libImaging/QuantHash.o build/temp.linux-armv7l-2.7/libImaging/QuantHeap.o build/temp.linux-armv7l-2.7/libImaging/PcdDecode.o build/temp.linux-armv7l-2.7/libImaging/PcxDecode.o build/temp.linux-armv7l-2.7/libImaging/PcxEncode.o build/temp.linux-armv7l-2.7/libImaging/Point.o build/temp.linux-armv7l-2.7/libImaging/RankFilter.o build/temp.linux-armv7l-2.7/libImaging/RawDecode.o build/temp.linux-armv7l-2.7/libImaging/RawEncode.o build/temp.linux-armv7l-2.7/libImaging/Storage.o build/temp.linux-armv7l-2.7/libImaging/SunRleDecode.o build/temp.linux-armv7l-2.7/libImaging/TgaRleDecode.o build/temp.linux-armv7l-2.7/libImaging/Unpack.o build/temp.linux-armv7l-2.7/libImaging/UnpackYCC.o build/temp.linux-armv7l-2.7/libImaging/UnsharpMask.o build/temp.linux-armv7l-2.7/libImaging/XbmDecode.o build/temp.linux-armv7l-2.7/libImaging/XbmEncode.o build/temp.linux-armv7l-2.7/libImaging/ZipDecode.o build/temp.linux-armv7l-2.7/libImaging/ZipEncode.o build/temp.linux-armv7l-2.7/libImaging/TiffDecode.o build/temp.linux-armv7l-2.7/libImaging/Incremental.o build/temp.linux-armv7l-2.7/libImaging/Jpeg2KDecode.o build/temp.linux-armv7l-2.7/libImaging/Jpeg2KEncode.o build/temp.linux-armv7l-2.7/libImaging/BoxBlur.o -L/usr/lib -L/usr/local/lib -L/usr/lib -ljpeg -lpython2.7 -o build/lib.linux-armv7l-2.7/PIL/_imaging.so
    gcc: error: build/temp.linux-armv7l-2.7/_imaging.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/decode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/encode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/map.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/display.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/outline.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/path.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Access.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/AlphaComposite.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Resample.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Bands.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/BitDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Blend.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Chops.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Convert.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/ConvertYCbCr.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Copy.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Crc32.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Crop.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Dib.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Draw.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Effects.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/EpsEncode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/File.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Fill.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Filter.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/FliDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Geometry.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/GetBBox.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/GifDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/GifEncode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/HexDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Histo.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/JpegDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/JpegEncode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/LzwDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Matrix.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/ModeFilter.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/MspDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Negative.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Offset.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Pack.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/PackDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Palette.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Paste.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Quant.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/QuantOctree.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/QuantHash.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/QuantHeap.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/PcdDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/PcxDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/PcxEncode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Point.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/RankFilter.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/RawDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/RawEncode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Storage.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/SunRleDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/TgaRleDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Unpack.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/UnpackYCC.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/UnsharpMask.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/XbmDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/XbmEncode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/ZipDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/ZipEncode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/TiffDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Incremental.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Jpeg2KDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Jpeg2KEncode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/BoxBlur.o: No such file or directory
    error: command 'gcc' failed with exit status 1

    ----------------------------------------
Command "/usr/bin/python -c "import setup tools, tokenize;__file__='/tmp/pip-build-gNq0WA/pillow/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-nDKwei-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-gNq0WA/pillow

我认为这可能是相关的部分:

In file included from libImaging/BoxBlur.c:1:0:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.

我的研究表明,头文件可能与此有关。我已经安装了这些:

apk add py-configobj libusb py-pip python-dev gcc linux-headers
pip install --upgrade pip
pip install -U setuptools
pip install Cheetah
pip install pyusb

I’m running alpine-linux on a Raspberry Pi 2. I’m trying to install Pillow via this command:

pip install pillow

This is the output from the command:

Installing collected packages: pillow
Running setup.py install for pillow
    Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-build-gNq0WA/pillow/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-nDKwei-record/install-record.txt --single-version-externally-managed --compile:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-armv7l-2.7
    creating build/lib.linux-armv7l-2.7/PIL
    copying PIL/XVThumbImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/XpmImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/XbmImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/WmfImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/WebPImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/WalImageFile.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/TiffTags.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/TiffImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/TgaImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/TarIO.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/SunImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/SpiderImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/SgiImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/PyAccess.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/PSDraw.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/PsdImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/PpmImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/PngImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/PixarImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/PdfImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/PcxImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/PcfFontFile.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/PcdImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/PalmImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/PaletteFile.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/OleFileIO.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/MspImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/MpoImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/MpegImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/MicImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/McIdasImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/JpegPresets.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/JpegImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/Jpeg2KImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/IptcImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImtImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageWin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageTransform.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageTk.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageStat.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageShow.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageSequence.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageQt.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImagePath.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImagePalette.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageOps.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageMorph.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageMode.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageMath.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageGrab.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageFont.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageFilter.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageFileIO.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageFile.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageEnhance.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageDraw2.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageDraw.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageColor.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageCms.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ImageChops.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/Image.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/IcoImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/IcnsImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/Hdf5StubImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/GribStubImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/GimpPaletteFile.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/GimpGradientFile.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/GifImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/GdImageFile.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/GbrImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/FpxImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/FontFile.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/FliImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/FitsStubImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ExifTags.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/EpsImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/DcxImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/CurImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/ContainerIO.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/BufrStubImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/BmpImagePlugin.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/BdfFontFile.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/_util.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/_binary.py -> build/lib.linux-armv7l-2.7/PIL
    copying PIL/__init__.py -> build/lib.linux-armv7l-2.7/PIL
    running egg_info
    writing Pillow.egg-info/PKG-INFO
    writing top-level names to Pillow.egg-info/top_level.txt
    writing dependency_links to Pillow.egg-info/dependency_links.txt
    warning: manifest_maker: standard file '-c' not found

    reading manifest file 'Pillow.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    warning: no files found matching 'LICENSE' under directory 'docs'
    writing manifest file 'Pillow.egg-info/SOURCES.txt'
    copying PIL/OleFileIO-README.md -> build/lib.linux-armv7l-2.7/PIL
    running build_ext
    building 'PIL._imaging' extension
    creating build/temp.linux-armv7l-2.7/libImaging
    gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -DHAVE_LIBJPEG -I/tmp/pip-build-gNq0WA/pillow/libImaging -I/usr/include -I/usr/include/python2.7 -c _imaging.c -o build/temp.linux-armv7l-2.7/_imaging.o
    gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -DHAVE_LIBJPEG -I/tmp/pip-build-gNq0WA/pillow/libImaging -I/usr/include -I/usr/include/python2.7 -c outline.c -o build/temp.linux-armv7l-2.7/outline.o
    gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -DHAVE_LIBJPEG -I/tmp/pip-build-gNq0WA/pillow/libImaging -I/usr/include -I/usr/include/python2.7 -c libImaging/Bands.c -o build/temp.linux-armv7l-2.7/libImaging/Bands.o
    gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -DHAVE_LIBJPEG -I/tmp/pip-build-gNq0WA/pillow/libImaging -I/usr/include -I/usr/include/python2.7 -c libImaging/ConvertYCbCr.c -o build/temp.linux-armv7l-2.7/libImaging/ConvertYCbCr.o
    In file included from _imaging.c:76:0:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.
    In file included from outline.c:20:0:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.
    In file included from libImaging/ImPlatform.h:10:0,
                    from libImaging/Imaging.h:14,
                    from libImaging/ConvertYCbCr.c:15:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.
    In file included from libImaging/ImPlatform.h:10:0,
                    from libImaging/Imaging.h:14,
                    from libImaging/Bands.c:19:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.
    gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -DHAVE_LIBJPEG -I/tmp/pip-build-gNq0WA/pillow/libImaging -I/usr/include -I/usr/include/python2.7 -c libImaging/Draw.c -o build/temp.linux-armv7l-2.7/libImaging/Draw.o
    gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -DHAVE_LIBJPEG -I/tmp/pip-build-gNq0WA/pillow/libImaging -I/usr/include -I/usr/include/python2.7 -c libImaging/Filter.c -o build/temp.linux-armv7l-2.7/libImaging/Filter.o
    gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -DHAVE_LIBJPEG -I/tmp/pip-build-gNq0WA/pillow/libImaging -I/usr/include -I/usr/include/python2.7 -c libImaging/GifEncode.c -o build/temp.linux-armv7l-2.7/libImaging/GifEncode.o
    gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -DHAVE_LIBJPEG -I/tmp/pip-build-gNq0WA/pillow/libImaging -I/usr/include -I/usr/include/python2.7 -c libImaging/LzwDecode.c -o build/temp.linux-armv7l-2.7/libImaging/LzwDecode.o
    In file included from libImaging/ImPlatform.h:10:0,
                    from libImaging/Imaging.h:14,
                    from libImaging/Draw.c:35:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.
    In file included from libImaging/ImPlatform.h:10:0,
                    from libImaging/Imaging.h:14,
                    from libImaging/Filter.c:27:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.
    In file included from libImaging/ImPlatform.h:10:0,
                    from libImaging/Imaging.h:14,
                    from libImaging/GifEncode.c:20:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.
    In file included from libImaging/ImPlatform.h:10:0,
                    from libImaging/Imaging.h:14,
                    from libImaging/LzwDecode.c:31:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.
    gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -DHAVE_LIBJPEG -I/tmp/pip-build-gNq0WA/pillow/libImaging -I/usr/include -I/usr/include/python2.7 -c libImaging/Offset.c -o build/temp.linux-armv7l-2.7/libImaging/Offset.o
    gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -DHAVE_LIBJPEG -I/tmp/pip-build-gNq0WA/pillow/libImaging -I/usr/include -I/usr/include/python2.7 -c libImaging/Quant.c -o build/temp.linux-armv7l-2.7/libImaging/Quant.o
    gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -DHAVE_LIBJPEG -I/tmp/pip-build-gNq0WA/pillow/libImaging -I/usr/include -I/usr/include/python2.7 -c libImaging/PcxDecode.c -o build/temp.linux-armv7l-2.7/libImaging/PcxDecode.o
    gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -DHAVE_LIBJPEG -I/tmp/pip-build-gNq0WA/pillow/libImaging -I/usr/include -I/usr/include/python2.7 -c libImaging/RawEncode.c -o build/temp.linux-armv7l-2.7/libImaging/RawEncode.o
    In file included from libImaging/ImPlatform.h:10:0,
                    from libImaging/Imaging.h:14,
                    from libImaging/Offset.c:18:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.
    In file included from libImaging/ImPlatform.h:10:0,
                    from libImaging/Imaging.h:14,
                    from libImaging/Quant.c:21:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.
    In file included from libImaging/ImPlatform.h:10:0,
                    from libImaging/Imaging.h:14,
                    from libImaging/PcxDecode.c:17:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.
    In file included from libImaging/ImPlatform.h:10:0,
                    from libImaging/Imaging.h:14,
                    from libImaging/RawEncode.c:21:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.
    gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -DHAVE_LIBJPEG -I/tmp/pip-build-gNq0WA/pillow/libImaging -I/usr/include -I/usr/include/python2.7 -c libImaging/UnpackYCC.c -o build/temp.linux-armv7l-2.7/libImaging/UnpackYCC.o
    gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -DHAVE_LIBJPEG -I/tmp/pip-build-gNq0WA/pillow/libImaging -I/usr/include -I/usr/include/python2.7 -c libImaging/ZipEncode.c -o build/temp.linux-armv7l-2.7/libImaging/ZipEncode.o
    gcc -fno-strict-aliasing -Os -fomit-frame-pointer -DNDEBUG -Os -fomit-frame-pointer -fPIC -DHAVE_LIBJPEG -I/tmp/pip-build-gNq0WA/pillow/libImaging -I/usr/include -I/usr/include/python2.7 -c libImaging/BoxBlur.c -o build/temp.linux-armv7l-2.7/libImaging/BoxBlur.o
    In file included from libImaging/ImPlatform.h:10:0,
                    from libImaging/Imaging.h:14,
                    from libImaging/UnpackYCC.c:17:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.
    In file included from libImaging/ImPlatform.h:10:0,
                    from libImaging/Imaging.h:14,
                    from libImaging/ZipEncode.c:18:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.
    In file included from libImaging/BoxBlur.c:1:0:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.
    Building using 4 processes
    gcc -shared -Wl,--as-needed build/temp.linux-armv7l-2.7/_imaging.o build/temp.linux-armv7l-2.7/decode.o build/temp.linux-armv7l-2.7/encode.o build/temp.linux-armv7l-2.7/map.o build/temp.linux-armv7l-2.7/display.o build/temp.linux-armv7l-2.7/outline.o build/temp.linux-armv7l-2.7/path.o build/temp.linux-armv7l-2.7/libImaging/Access.o build/temp.linux-armv7l-2.7/libImaging/AlphaComposite.o build/temp.linux-armv7l-2.7/libImaging/Resample.o build/temp.linux-armv7l-2.7/libImaging/Bands.o build/temp.linux-armv7l-2.7/libImaging/BitDecode.o build/temp.linux-armv7l-2.7/libImaging/Blend.o build/temp.linux-armv7l-2.7/libImaging/Chops.o build/temp.linux-armv7l-2.7/libImaging/Convert.o build/temp.linux-armv7l-2.7/libImaging/ConvertYCbCr.o build/temp.linux-armv7l-2.7/libImaging/Copy.o build/temp.linux-armv7l-2.7/libImaging/Crc32.o build/temp.linux-armv7l-2.7/libImaging/Crop.o build/temp.linux-armv7l-2.7/libImaging/Dib.o build/temp.linux-armv7l-2.7/libImaging/Draw.o build/temp.linux-armv7l-2.7/libImaging/Effects.o build/temp.linux-armv7l-2.7/libImaging/EpsEncode.o build/temp.linux-armv7l-2.7/libImaging/File.o build/temp.linux-armv7l-2.7/libImaging/Fill.o build/temp.linux-armv7l-2.7/libImaging/Filter.o build/temp.linux-armv7l-2.7/libImaging/FliDecode.o build/temp.linux-armv7l-2.7/libImaging/Geometry.o build/temp.linux-armv7l-2.7/libImaging/GetBBox.o build/temp.linux-armv7l-2.7/libImaging/GifDecode.o build/temp.linux-armv7l-2.7/libImaging/GifEncode.o build/temp.linux-armv7l-2.7/libImaging/HexDecode.o build/temp.linux-armv7l-2.7/libImaging/Histo.o build/temp.linux-armv7l-2.7/libImaging/JpegDecode.o build/temp.linux-armv7l-2.7/libImaging/JpegEncode.o build/temp.linux-armv7l-2.7/libImaging/LzwDecode.o build/temp.linux-armv7l-2.7/libImaging/Matrix.o build/temp.linux-armv7l-2.7/libImaging/ModeFilter.o build/temp.linux-armv7l-2.7/libImaging/MspDecode.o build/temp.linux-armv7l-2.7/libImaging/Negative.o build/temp.linux-armv7l-2.7/libImaging/Offset.o build/temp.linux-armv7l-2.7/libImaging/Pack.o build/temp.linux-armv7l-2.7/libImaging/PackDecode.o build/temp.linux-armv7l-2.7/libImaging/Palette.o build/temp.linux-armv7l-2.7/libImaging/Paste.o build/temp.linux-armv7l-2.7/libImaging/Quant.o build/temp.linux-armv7l-2.7/libImaging/QuantOctree.o build/temp.linux-armv7l-2.7/libImaging/QuantHash.o build/temp.linux-armv7l-2.7/libImaging/QuantHeap.o build/temp.linux-armv7l-2.7/libImaging/PcdDecode.o build/temp.linux-armv7l-2.7/libImaging/PcxDecode.o build/temp.linux-armv7l-2.7/libImaging/PcxEncode.o build/temp.linux-armv7l-2.7/libImaging/Point.o build/temp.linux-armv7l-2.7/libImaging/RankFilter.o build/temp.linux-armv7l-2.7/libImaging/RawDecode.o build/temp.linux-armv7l-2.7/libImaging/RawEncode.o build/temp.linux-armv7l-2.7/libImaging/Storage.o build/temp.linux-armv7l-2.7/libImaging/SunRleDecode.o build/temp.linux-armv7l-2.7/libImaging/TgaRleDecode.o build/temp.linux-armv7l-2.7/libImaging/Unpack.o build/temp.linux-armv7l-2.7/libImaging/UnpackYCC.o build/temp.linux-armv7l-2.7/libImaging/UnsharpMask.o build/temp.linux-armv7l-2.7/libImaging/XbmDecode.o build/temp.linux-armv7l-2.7/libImaging/XbmEncode.o build/temp.linux-armv7l-2.7/libImaging/ZipDecode.o build/temp.linux-armv7l-2.7/libImaging/ZipEncode.o build/temp.linux-armv7l-2.7/libImaging/TiffDecode.o build/temp.linux-armv7l-2.7/libImaging/Incremental.o build/temp.linux-armv7l-2.7/libImaging/Jpeg2KDecode.o build/temp.linux-armv7l-2.7/libImaging/Jpeg2KEncode.o build/temp.linux-armv7l-2.7/libImaging/BoxBlur.o -L/usr/lib -L/usr/local/lib -L/usr/lib -ljpeg -lpython2.7 -o build/lib.linux-armv7l-2.7/PIL/_imaging.so
    gcc: error: build/temp.linux-armv7l-2.7/_imaging.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/decode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/encode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/map.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/display.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/outline.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/path.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Access.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/AlphaComposite.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Resample.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Bands.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/BitDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Blend.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Chops.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Convert.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/ConvertYCbCr.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Copy.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Crc32.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Crop.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Dib.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Draw.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Effects.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/EpsEncode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/File.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Fill.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Filter.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/FliDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Geometry.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/GetBBox.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/GifDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/GifEncode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/HexDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Histo.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/JpegDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/JpegEncode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/LzwDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Matrix.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/ModeFilter.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/MspDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Negative.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Offset.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Pack.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/PackDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Palette.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Paste.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Quant.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/QuantOctree.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/QuantHash.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/QuantHeap.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/PcdDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/PcxDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/PcxEncode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Point.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/RankFilter.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/RawDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/RawEncode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Storage.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/SunRleDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/TgaRleDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Unpack.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/UnpackYCC.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/UnsharpMask.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/XbmDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/XbmEncode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/ZipDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/ZipEncode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/TiffDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Incremental.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Jpeg2KDecode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/Jpeg2KEncode.o: No such file or directory
    gcc: error: build/temp.linux-armv7l-2.7/libImaging/BoxBlur.o: No such file or directory
    error: command 'gcc' failed with exit status 1

    ----------------------------------------
Command "/usr/bin/python -c "import setup tools, tokenize;__file__='/tmp/pip-build-gNq0WA/pillow/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-nDKwei-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-gNq0WA/pillow

I think this is probably the relevant section:

In file included from libImaging/BoxBlur.c:1:0:
    /usr/include/python2.7/Python.h:19:20: fatal error: limits.h: No such file or directory
    #include <limits.h>
                        ^
    compilation terminated.

My research shows it’s probably something with the header files. I have installed these:

apk add py-configobj libusb py-pip python-dev gcc linux-headers
pip install --upgrade pip
pip install -U setuptools
pip install Cheetah
pip install pyusb

回答 0

高山Linux使用musl libc。您可能需要安装musl-dev

Alpine Linux uses musl libc. You probably need to install musl-dev.


回答 1

@zakaria答案是正确的,但是如果您偶然发现

fatal error: linux/limits.h: No such file or directory

那么你需要的包linux-headers(注意前缀linux之前limits.h

apk add linux-headers

@zakaria answer is correct, but if you stumble upon

fatal error: linux/limits.h: No such file or directory

then you need the package linux-headers (notice the prefix linux before limits.h

apk add linux-headers

回答 2

limits.h位于libc-dev

apk add libc-dev

limits.h is located in libc-dev:

apk add libc-dev

回答 3

我在docker pyhton:3.6-alpine image,Alpine linux> = 3.3中安装python库正则表达式时遇到了非常相似的问题。

pip install regex

我必须添加gcc和musl-dev软件包

apk --no-cache add gcc musl-dev

I had very similar problem with installing python library regex in docker pyhton:3.6-alpine image, Alpine linux >= 3.3.

pip install regex

I had to add gcc and musl-dev packages

apk --no-cache add gcc musl-dev

回答 4

我发现有些python软件包无法通过pip install安装,但是如果您安装了相关的alpine linux软件包,它们可以工作。例如,pip install uwsgi无法抱怨limits.h,但是apk add uwsgi-python可以正常工作。建议尝试apk添加py-pillow而不是pip安装枕头。

I’ve found some python packages fail to install via pip install but work if you install the associated alpine linux package. For example pip install uwsgi fails complaining about limits.h, but apk add uwsgi-python works fine. Suggest trying apk add py-pillow instead of pip install pillow.


为什么在导入numpy之后多处理仅使用单个内核?

问题:为什么在导入numpy之后多处理仅使用单个内核?

我不确定这是否更多的是操作系统问题,但是我想在这里问一下,以防有人对Python有所了解。

我一直在尝试使用并行化CPU繁重的for循环joblib,但是我发现不是将每个工作进程分配给不同的内核,而是最终将所有工作进程分配给相同的内核,并且没有性能提升。

这是一个非常简单的例子…

from joblib import Parallel,delayed
import numpy as np

def testfunc(data):
    # some very boneheaded CPU work
    for nn in xrange(1000):
        for ii in data[0,:]:
            for jj in data[1,:]:
                ii*jj

def run(niter=10):
    data = (np.random.randn(2,100) for ii in xrange(niter))
    pool = Parallel(n_jobs=-1,verbose=1,pre_dispatch='all')
    results = pool(delayed(testfunc)(dd) for dd in data)

if __name__ == '__main__':
    run()

htop这是该脚本运行时看到的内容:

我在具有4核的笔记本电脑上运行Ubuntu 12.10(3.5.0-26)。显然joblib.Parallel是为不同的工作人员生成了单独的进程,但是有什么方法可以使这些进程在不同的内核上执行?

I am not sure whether this counts more as an OS issue, but I thought I would ask here in case anyone has some insight from the Python end of things.

I’ve been trying to parallelise a CPU-heavy for loop using joblib, but I find that instead of each worker process being assigned to a different core, I end up with all of them being assigned to the same core and no performance gain.

Here’s a very trivial example…

from joblib import Parallel,delayed
import numpy as np

def testfunc(data):
    # some very boneheaded CPU work
    for nn in xrange(1000):
        for ii in data[0,:]:
            for jj in data[1,:]:
                ii*jj

def run(niter=10):
    data = (np.random.randn(2,100) for ii in xrange(niter))
    pool = Parallel(n_jobs=-1,verbose=1,pre_dispatch='all')
    results = pool(delayed(testfunc)(dd) for dd in data)

if __name__ == '__main__':
    run()

…and here’s what I see in htop while this script is running:

I’m running Ubuntu 12.10 (3.5.0-26) on a laptop with 4 cores. Clearly joblib.Parallel is spawning separate processes for the different workers, but is there any way that I can make these processes execute on different cores?


回答 0

经过更多的谷歌搜索后,我在这里找到了答案。

事实证明,某些Python模块(numpyscipytablespandasskimage对进口核心相关性……)的混乱。据我所知,这个问题似乎是由它们链接到多线程OpenBLAS库引起的。

解决方法是使用

os.system("taskset -p 0xff %d" % os.getpid())

在导入模块之后粘贴了这一行,我的示例现在可以在所有内核上运行:

到目前为止,我的经验是,这似乎对numpy机器的性能没有任何负面影响,尽管这可能是特定于机器和任务的。

更新:

还有两种方法可以禁用OpenBLAS本身的CPU关联性重置行为。在运行时,您可以使用环境变量OPENBLAS_MAIN_FREE(或GOTOBLAS_MAIN_FREE),例如

OPENBLAS_MAIN_FREE=1 python myscript.py

或者,如果您要从源代码编译OpenBLAS,则可以在构建时通过编辑Makefile.rule使其包含该行来永久禁用它

NO_AFFINITY=1

After some more googling I found the answer here.

It turns out that certain Python modules (numpy, scipy, tables, pandas, skimage…) mess with core affinity on import. As far as I can tell, this problem seems to be specifically caused by them linking against multithreaded OpenBLAS libraries.

A workaround is to reset the task affinity using

os.system("taskset -p 0xff %d" % os.getpid())

With this line pasted in after the module imports, my example now runs on all cores:

My experience so far has been that this doesn’t seem to have any negative effect on numpy‘s performance, although this is probably machine- and task-specific .

Update:

There are also two ways to disable the CPU affinity-resetting behaviour of OpenBLAS itself. At run-time you can use the environment variable OPENBLAS_MAIN_FREE (or GOTOBLAS_MAIN_FREE), for example

OPENBLAS_MAIN_FREE=1 python myscript.py

Or alternatively, if you’re compiling OpenBLAS from source you can permanently disable it at build-time by editing the Makefile.rule to contain the line

NO_AFFINITY=1

回答 1

Python 3现在公开了直接设置亲和力的方法

>>> import os
>>> os.sched_getaffinity(0)
{0, 1, 2, 3}
>>> os.sched_setaffinity(0, {1, 3})
>>> os.sched_getaffinity(0)
{1, 3}
>>> x = {i for i in range(10)}
>>> x
{0, 1, 2, 3, 4, 5, 6, 7, 8, 9}
>>> os.sched_setaffinity(0, x)
>>> os.sched_getaffinity(0)
{0, 1, 2, 3}

Python 3 now exposes the methods to directly set the affinity

>>> import os
>>> os.sched_getaffinity(0)
{0, 1, 2, 3}
>>> os.sched_setaffinity(0, {1, 3})
>>> os.sched_getaffinity(0)
{1, 3}
>>> x = {i for i in range(10)}
>>> x
{0, 1, 2, 3, 4, 5, 6, 7, 8, 9}
>>> os.sched_setaffinity(0, x)
>>> os.sched_getaffinity(0)
{0, 1, 2, 3}

回答 2

这似乎是Ubuntu上Python的常见问题,并不特定于joblib

我建议尝试使用CPU相似性(taskset)。


Linux上的两个版本的python。如何使2.7成为默认值

问题:Linux上的两个版本的python。如何使2.7成为默认值

我的linuxbox上有两个版本的python:

$python
Python 2.6.6 (r266:84292, Jul 10 2013, 22:48:45) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 


$ /usr/local/bin/python2.7
Python 2.7.3 (default, Oct  8 2013, 15:53:09) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 

$ which python
/usr/bin/python
$ ls -al /usr/bin/python
-rwxr-xr-x. 2 root root 4864 Jul 10 22:49 /usr/bin/python

如何将2.7设置为默认版本,以便在键入python时将其置于2.7版本?

I’ve got two versions of python on my linuxbox:

$python
Python 2.6.6 (r266:84292, Jul 10 2013, 22:48:45) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 


$ /usr/local/bin/python2.7
Python 2.7.3 (default, Oct  8 2013, 15:53:09) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 

$ which python
/usr/bin/python
$ ls -al /usr/bin/python
-rwxr-xr-x. 2 root root 4864 Jul 10 22:49 /usr/bin/python

How can I make 2.7 be the default version so when I type python it puts me in 2.7?


回答 0

您可能实际上不想更改默认的Python。

您的发行版在中安装了标准系统Python /usr/bin,并且可能具有依赖于此脚本的脚本,并由选择#! /usr/bin/env python通常,您可以在2.7中运行Python 2.6脚本,但是您要冒险吗?

最重要的是,闲逛/usr/bin可能会破坏您的包管理器管理包的能力。并且更改目录中的顺序PATH将影响除Python以外的许多其他因素。(实际上,在/usr/local/bin之前更常见/usr/bin,这可能是您真正想要的-但是如果您有其他选择,则可能有充分的理由。)

但是,您无需更改默认的Python即可在键入时使系统运行2.7 python


首先,您可以设置一个shell别名:

alias python=/usr/local/bin/python2.7

在提示符下键入该命令,或者~/.bashrc如果您想使更改持久化,则将其放入您的计算机,现在键入该命令时python将运行您选择的2.7,但是当系统上的某个程序尝试使用/usr/bin/env python标准2.6 运行该脚本时。


或者,只需在2.7(或针对不同项目的单独venv)中创建一个虚拟环境,然后在venv中进行工作。

You probably don’t actually want to change your default Python.

Your distro installed a standard system Python in /usr/bin, and may have scripts that depend on this being present, and selected by #! /usr/bin/env python. You can usually get away with running Python 2.6 scripts in 2.7, but do you want to risk it?

On top of that, monkeying with /usr/bin can break your package manager’s ability to manage packages. And changing the order of directories in your PATH will affect a lot of other things besides Python. (In fact, it’s more common to have /usr/local/bin ahead of /usr/bin, and it may be what you actually want—but if you have it the other way around, presumably there’s a good reason for that.)

But you don’t need to change your default Python to get the system to run 2.7 when you type python.


First, you can set up a shell alias:

alias python=/usr/local/bin/python2.7

Type that at a prompt, or put it in your ~/.bashrc if you want the change to be persistent, and now when you type python it runs your chosen 2.7, but when some program on your system tries to run a script with /usr/bin/env python it runs the standard 2.6.


Alternatively, just create a virtual environment out of your 2.7 (or separate venvs for different projects), and do your work inside the venv.


回答 1

添加/usr/local/bin到您的PATH环境变量中,在列表中早于/usr/bin

通常,这是在您外壳的rc文件中完成的,例如,对于bash,您可以将其放入.bashrc

export PATH="/usr/local/bin:$PATH"

这将导致你的shell先寻找一个python/usr/local/bin,它会与一个之前/usr/bin

(当然,这意味着您还需要/usr/local/bin/python指向python2.7-如果尚未指向,则需要对其进行符号链接。)

Add /usr/local/bin to your PATH environment variable, earlier in the list than /usr/bin.

Generally this is done in your shell’s rc file, e.g. for bash, you’d put this in .bashrc:

export PATH="/usr/local/bin:$PATH"

This will cause your shell to look first for a python in /usr/local/bin, before it goes with the one in /usr/bin.

(Of course, this means you also need to have /usr/local/bin/python point to python2.7 – if it doesn’t already, you’ll need to symlink it.)


回答 2

通过以下方式验证python的当前版本:

$ python --version

然后检查python是指向哪个文件的符号链接。

  $ ll /usr/bin/python

输出示例:

 lrwxrwxrwx 1 root root 9 Jun 16  2014 /usr/bin/python -> python2.7*

检查其他可用的python版本:

$ ls /usr/bin/python*

输出示例:

/usr/bin/python     /usr/bin/python2.7-config  /usr/bin/python3.4         /usr/bin/python3.4m-config  /usr/bin/python3.6m         /usr/bin/python3m
/usr/bin/python2    /usr/bin/python2-config    /usr/bin/python3.4-config  /usr/bin/python3.6          /usr/bin/python3.6m-config  /usr/bin/python3m-config
/usr/bin/python2.7  /usr/bin/python3           /usr/bin/python3.4m        /usr/bin/python3.6-config   /usr/bin/python3-config     /usr/bin/python-config

如果要将python的当前版本更改为3.6版本,请编辑文件〜/ .bashrc:

vim ~/.bashrc

在文件末尾添加以下行并保存:

alias python=/usr/local/bin/python3.6

为python 3.6安装pip

$ sudo apt-get install python3.6 python3.6-dev
$ sudo curl https://bootstrap.pypa.io/ez_setup.py -o - | sudo python3.6
$ sudo easy_install pip

成功后,检查pip的当前版本:

$ pip3 -V

输出示例:

pip 1.5.4 from /usr/lib/python3/dist-packages (python 3.6)

Verify current version of python by:

$ python --version

then check python is symbolic link to which file.

  $ ll /usr/bin/python

Output Ex:

 lrwxrwxrwx 1 root root 9 Jun 16  2014 /usr/bin/python -> python2.7*

Check other available versions of python:

$ ls /usr/bin/python*

Output Ex:

/usr/bin/python     /usr/bin/python2.7-config  /usr/bin/python3.4         /usr/bin/python3.4m-config  /usr/bin/python3.6m         /usr/bin/python3m
/usr/bin/python2    /usr/bin/python2-config    /usr/bin/python3.4-config  /usr/bin/python3.6          /usr/bin/python3.6m-config  /usr/bin/python3m-config
/usr/bin/python2.7  /usr/bin/python3           /usr/bin/python3.4m        /usr/bin/python3.6-config   /usr/bin/python3-config     /usr/bin/python-config

If want to change current version of python to 3.6 version edit file ~/.bashrc:

vim ~/.bashrc

add below line in the end of file and save:

alias python=/usr/local/bin/python3.6

To install pip for python 3.6

$ sudo apt-get install python3.6 python3.6-dev
$ sudo curl https://bootstrap.pypa.io/ez_setup.py -o - | sudo python3.6
$ sudo easy_install pip

On Success, check current version of pip:

$ pip3 -V

Output Ex:

pip 1.5.4 from /usr/lib/python3/dist-packages (python 3.6)

回答 3

输入命令

which python

//output:
/usr/bin/python

cd /usr/bin
ls -l

在这里你可以看到类似的东西

lrwxrwxrwx 1 root   root            9 Mar  7 17:04  python -> python2.7

您的默认python2.7被软链接到文本’python’

所以删除softlink python

sudo rm -r python

然后重试上面的命令

ls -l

您可以看到该软链接已删除

-rwxr-xr-x 1 root   root      3670448 Nov 12 20:01  python2.7

然后为python3.6创建一个新的软链接

ln -s /usr/bin/python3.6 python

然后python在终端中尝试命令

//output:
Python 3.6.7 (default, Oct 22 2018, 11:32:17) 
[GCC 8.2.0] on linux

类型helpcopyrightcreditslicense了解更多信息。

Enter the command

which python

//output:
/usr/bin/python

cd /usr/bin
ls -l

Here you can see something like this

lrwxrwxrwx 1 root   root            9 Mar  7 17:04  python -> python2.7

your default python2.7 is soft linked to the text ‘python’

So remove the softlink python

sudo rm -r python

then retry the above command

ls -l

you can see the softlink is removed

-rwxr-xr-x 1 root   root      3670448 Nov 12 20:01  python2.7

Then create a new softlink for python3.6

ln -s /usr/bin/python3.6 python

Then try the command python in terminal

//output:
Python 3.6.7 (default, Oct 22 2018, 11:32:17) 
[GCC 8.2.0] on linux

Type help, copyright, credits or license for more information.


回答 4

所有操作系统都带有python的默认版本,并且位于/ usr / bin中。操作系统随附的所有脚本(例如yum)都指向/ usr / bin中驻留的该版本的python。当您想安装新版本的python时,您不想破坏可能不适用于新版本python的现有脚本。

正确的方法是将python安装为替代版本。

e.g.
wget http://www.python.org/ftp/python/2.7.3/Python-2.7.3.tar.bz2 
tar xf Python-2.7.3.tar.bz2
cd Python-2.7.3
./configure --prefix=/usr/local/
make && make altinstall

现在,通过执行此操作,现有的脚本(如yum)仍可与/ usr / bin / python一起使用。而您的默认python版本将是/ usr / local / bin中安装的版本。即当您键入python时,您将获得2.7.3

这是因为。$ PATH变量在usr / bin之前具有/ usr / local / bin。

/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin

如果python2.7仍然无法作为默认python版本生效,则需要执行此操作

export PATH="/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin"

All OS comes with a default version of python and it resides in /usr/bin. All scripts that come with the OS (e.g. yum) point this version of python residing in /usr/bin. When you want to install a new version of python you do not want to break the existing scripts which may not work with new version of python.

The right way of doing this is to install the python as an alternate version.

e.g.
wget http://www.python.org/ftp/python/2.7.3/Python-2.7.3.tar.bz2 
tar xf Python-2.7.3.tar.bz2
cd Python-2.7.3
./configure --prefix=/usr/local/
make && make altinstall

Now by doing this the existing scripts like yum still work with /usr/bin/python. and your default python version would be the one installed in /usr/local/bin. i.e. when you type python you would get 2.7.3

This happens because. $PATH variable has /usr/local/bin before usr/bin.

/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin

If python2.7 still does not take effect as the default python version you would need to do

export PATH="/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin"

回答 5

我想您手动安装了2.7版本,而2.6是从软件包中获得的?

简单的答案是:卸载python软件包。

比较复杂的是:不要在/ usr / local中手动安装。用2.7版本构建软件包,然后升级。

包处理取决于您使用的发行版。

I guess you have installed the 2.7 version manually, while 2.6 comes from a package?

The simple answer is: uninstall python package.

The more complex one is: do not install manually in /usr/local. Build a package with 2.7 version and then upgrade.

Package handling depends on what distribution you use.


无法使用Ctrl-C终止Python脚本

问题:无法使用Ctrl-C终止Python脚本

我正在使用以下脚本测试Python线程:

import threading

class FirstThread (threading.Thread):
    def run (self):
        while True:
            print 'first'

class SecondThread (threading.Thread):
    def run (self):
        while True:
            print 'second'

FirstThread().start()
SecondThread().start()

它在Kubuntu 11.10上的Python 2.7中运行。Ctrl+ C不会杀死它。我还尝试为系统信号添加处理程序,但这没有帮助:

import signal 
import sys
def signal_handler(signal, frame):
    sys.exit(0)
signal.signal(signal.SIGINT, signal_handler)

为了终止进程,我使用Ctrl+ 将程序发送到后台后通过PID将其终止Z,这不会被忽略。为什么Ctrl+ C被如此持久地忽略?我该如何解决?

I am testing Python threading with the following script:

import threading

class FirstThread (threading.Thread):
    def run (self):
        while True:
            print 'first'

class SecondThread (threading.Thread):
    def run (self):
        while True:
            print 'second'

FirstThread().start()
SecondThread().start()

This is running in Python 2.7 on Kubuntu 11.10. Ctrl+C will not kill it. I also tried adding a handler for system signals, but that did not help:

import signal 
import sys
def signal_handler(signal, frame):
    sys.exit(0)
signal.signal(signal.SIGINT, signal_handler)

To kill the process I am killing it by PID after sending the program to the background with Ctrl+Z, which isn’t being ignored. Why is Ctrl+C being ignored so persistently? How can I resolve this?


回答 0

Ctrl+ C终止主线程,但是因为您的线程不在守护程序模式下,所以它们继续运行,并且使进程保持活动状态。我们可以使它们成为守护进程:

f = FirstThread()
f.daemon = True
f.start()
s = SecondThread()
s.daemon = True
s.start()

但是,还有另一个问题-一旦主线程启动了线程,就别无他法了。因此它退出了,线程立即被销毁。因此,让主线程保持活动状态:

import time
while True:
    time.sleep(1)

现在它将保留打印“ first”和“ second”,直到您按Ctrl+ 为止C

编辑:正如评论者所指出的,守护进程线程可能没有机会清理临时文件之类的东西。如果需要,请抓住KeyboardInterrupt主线程并协调其清理和关闭。但是在很多情况下,让守护线程突然死掉可能已经足够了。

Ctrl+C terminates the main thread, but because your threads aren’t in daemon mode, they keep running, and that keeps the process alive. We can make them daemons:

f = FirstThread()
f.daemon = True
f.start()
s = SecondThread()
s.daemon = True
s.start()

But then there’s another problem – once the main thread has started your threads, there’s nothing else for it to do. So it exits, and the threads are destroyed instantly. So let’s keep the main thread alive:

import time
while True:
    time.sleep(1)

Now it will keep print ‘first’ and ‘second’ until you hit Ctrl+C.

Edit: as commenters have pointed out, the daemon threads may not get a chance to clean up things like temporary files. If you need that, then catch the KeyboardInterrupt on the main thread and have it co-ordinate cleanup and shutdown. But in many cases, letting daemon threads die suddenly is probably good enough.


回答 1

KeyboardInterrupt和信号仅在进程(即主线程)中可见…看看Ctrl-c即KeyboardInterrupt杀死python中的线程

KeyboardInterrupt and signals are only seen by the process (ie the main thread)… Have a look at Ctrl-c i.e. KeyboardInterrupt to kill threads in python


回答 2

我认为最好在您希望线程死亡时在线程上调用join()。我对您的代码采取了一些自由以结束循环(您也可以在其中添加所需的任何清理需求)。每次通过时都要检查变量die是否为真,当它为True时,程序将退出。

import threading
import time

class MyThread (threading.Thread):
    die = False
    def __init__(self, name):
        threading.Thread.__init__(self)
        self.name = name

    def run (self):
        while not self.die:
            time.sleep(1)
            print (self.name)

    def join(self):
        self.die = True
        super().join()

if __name__ == '__main__':
    f = MyThread('first')
    f.start()
    s = MyThread('second')
    s.start()
    try:
        while True:
            time.sleep(2)
    except KeyboardInterrupt:
        f.join()
        s.join()

I think it’s best to call join() on your threads when you expect them to die. I’ve taken some liberty with your code to make the loops end (you can add whatever cleanup needs are required to there as well). The variable die is checked for truth on each pass and when it’s True then the program exits.

import threading
import time

class MyThread (threading.Thread):
    die = False
    def __init__(self, name):
        threading.Thread.__init__(self)
        self.name = name

    def run (self):
        while not self.die:
            time.sleep(1)
            print (self.name)

    def join(self):
        self.die = True
        super().join()

if __name__ == '__main__':
    f = MyThread('first')
    f.start()
    s = MyThread('second')
    s.start()
    try:
        while True:
            time.sleep(2)
    except KeyboardInterrupt:
        f.join()
        s.join()

回答 3

@Thomas K的答案的改进版本:

  • is_any_thread_alive()根据此要旨定义助手功能,该功能可以main()自动终止。

示例代码:

import threading

def job1():
    ...

def job2():
    ...

def is_any_thread_alive(threads):
    return True in [t.is_alive() for t in threads]

if __name__ == "__main__":
    ...
    t1 = threading.Thread(target=job1,daemon=True)
    t2 = threading.Thread(target=job2,daemon=True)
    t1.start()
    t2.start()

    while is_any_thread_alive([t1,t2]):
        time.sleep(0)

An improved version of @Thomas K’s answer:

  • Defining an assistant function is_any_thread_alive() according to this gist, which can terminates the main() automatically.

Example codes:

import threading

def job1():
    ...

def job2():
    ...

def is_any_thread_alive(threads):
    return True in [t.is_alive() for t in threads]

if __name__ == "__main__":
    ...
    t1 = threading.Thread(target=job1,daemon=True)
    t2 = threading.Thread(target=job2,daemon=True)
    t1.start()
    t2.start()

    while is_any_thread_alive([t1,t2]):
        time.sleep(0)

在Python脚本中,如何设置PYTHONPATH?

问题:在Python脚本中,如何设置PYTHONPATH?

我知道如何在/ etc / profile和环境变量中进行设置。

但是,如果我想在脚本中进行设置怎么办?是导入os,sys吗?我该怎么做?

I know how to set it in my /etc/profile and in my environment variables.

But what if I want to set it during a script? Is it import os, sys? How do I do it?


回答 0

您没有设置PYTHONPATH,而是向中添加条目sys.path。这是应该在其中搜索Python软件包的目录列表,因此您只需将目录追加到该列表即可。

sys.path.append('/path/to/whatever')

实际上,sys.path是通过分割PYTHONPATH路径分隔符:上的值来初始化的(在类似Linux的系统上,;在Windows上)。

您也可以使用来添加目录site.addsitedir,该方法还将考虑.pth您传递的目录内存在的文件。(对于您在中指定的目录,情况并非如此PYTHONPATH。)

You don’t set PYTHONPATH, you add entries to sys.path. It’s a list of directories that should be searched for Python packages, so you can just append your directories to that list.

sys.path.append('/path/to/whatever')

In fact, sys.path is initialized by splitting the value of PYTHONPATH on the path separator character (: on Linux-like systems, ; on Windows).

You can also add directories using site.addsitedir, and that method will also take into account .pth files existing within the directories you pass. (That would not be the case with directories you specify in PYTHONPATH.)


回答 1

您可以通过os.environ以下方式获取和设置环境变量:

import os
user_home = os.environ["HOME"]

os.environ["PYTHONPATH"] = "..."

但是,由于您的解释器已经在运行,因此不会起作用。你最好用

import sys
sys.path.append("...")

这是您PYTHONPATH将在解释程序启动时转换为的数组。

You can get and set environment variables via os.environ:

import os
user_home = os.environ["HOME"]

os.environ["PYTHONPATH"] = "..."

But since your interpreter is already running, this will have no effect. You’re better off using

import sys
sys.path.append("...")

which is the array that your PYTHONPATH will be transformed into on interpreter startup.


回答 2

如果您sys.path.append('dir/to/path')不加检查就放了它,则可以在中生成一个长列表sys.path。为此,我建议这样做:

import sys
import os # if you want this directory

try:
    sys.path.index('/dir/path') # Or os.getcwd() for this directory
except ValueError:
    sys.path.append('/dir/path') # Or os.getcwd() for this directory

If you put sys.path.append('dir/to/path') without check it is already added, you could generate a long list in sys.path. For that, I recommend this:

import sys
import os # if you want this directory

try:
    sys.path.index('/dir/path') # Or os.getcwd() for this directory
except ValueError:
    sys.path.append('/dir/path') # Or os.getcwd() for this directory

回答 3

PYTHONPATH结尾于sys.path,您可以在运行时进行修改。

import sys
sys.path += ["whatever"]

PYTHONPATH ends up in sys.path, which you can modify at runtime.

import sys
sys.path += ["whatever"]

回答 4

您可以通过设置PYTHONPATHos.environ['PATHPYTHON']=/some/path然后需要调用os.system('python')以重新启动python shell,以使新添加的路径生效。

you can set PYTHONPATH, by os.environ['PATHPYTHON']=/some/path, then you need to call os.system('python') to restart the python shell to make the newly added path effective.


回答 5

我的Linux也可以:

import sys
sys.path.extend(["/path/to/dotpy/file/"])

I linux this works too:

import sys
sys.path.extend(["/path/to/dotpy/file/"])

从PHP运行Python脚本

问题:从PHP运行Python脚本

我正在尝试使用以下命令从PHP运行Python脚本:

exec('/usr/bin/python2.7 /srv/http/assets/py/switch.py arg1 arg2');

但是,PHP根本不会产生任何输出。错误报告设置为E_ALL,并且display_errors打开。

这是我尝试过的:

  • 我使用python2/usr/bin/python2python2.7不是/usr/bin/python2.7
  • 我还使用了相对路径而不是绝对路径,它也没有改变任何东西。
  • 我试着使用的命令execshell_execsystem

但是,如果我跑步

if (exec('echo TEST') == 'TEST')
{
    echo 'exec works!';
}

shutdown now什么也没做,却可以正常工作。

PHP有权访问和执行文件。

编辑:感谢亚历杭德罗,我能够解决此问题。如果您遇到相同的问题,请不要忘记您的Web服务器可能/希望不是以root用户身份运行。尝试以您的Web服务器用户或具有类似权限的用户身份登录,然后尝试自己运行命令。

I’m trying to run a Python script from PHP using the following command:

exec('/usr/bin/python2.7 /srv/http/assets/py/switch.py arg1 arg2');

However, PHP simply doesn’t produce any output. Error reporting is set to E_ALL and display_errors is on.

Here’s what I’ve tried:

  • I used python2, /usr/bin/python2 and python2.7 instead of /usr/bin/python2.7
  • I also used a relative path instead of an absolute path which didn’t change anything either.
  • I tried using the commands exec, shell_exec, system.

However, if I run

if (exec('echo TEST') == 'TEST')
{
    echo 'exec works!';
}

it works perfectly fine while shutdown now doesn’t do anything.

PHP has the permissions to access and execute the file.

EDIT: Thanks to Alejandro, I was able to fix the problem. If you have the same problem, don’t forget that your webserver probably/hopefully doesn’t run as root. Try logging in as your webserver’s user or a user with similar permissions and try to run the commands yourself.


回答 0

在Ubuntu Server 10.04上测试。希望它对Arch Linux也有帮助。

在PHP中使用shell_exec函数

通过shell执行命令并以字符串形式返回完整的输出。

它从执行的命令返回输出,如果发生错误或命令不产生任何输出,则返回NULL。

<?php 

$command = escapeshellcmd('/usr/custom/test.py');
$output = shell_exec($command);
echo $output;

?>

在Python文件中test.py,在第一行中验证以下文本:(请参见shebang解释)

#!/usr/bin/env python

此外,Python文件必须具有正确的特权(如果PHP脚本在浏览器或curl中运行,则对用户www-data / apache的执行)和/或必须是“可执行的”。此外,所有进入.py文件的命令都必须具有正确的特权:

采取从PHP手册

对于那些试图在unix类型的平台上使用shell_exec并且似乎无法使其正常工作的人,请快速提醒一下。PHP以系统上的Web用户身份执行(对于Apache,通常为www),因此您需要确保Web用户对您在shell_exec命令中尝试使用的任何文件或目录具有权限。否则,它似乎什么也没做。

为了使在UNIX型平台上的可执行文件

chmod +x myscript.py

Tested on Ubuntu Server 10.04. I hope it helps you also on Arch Linux.

In PHP use shell_exec function:

Execute command via shell and return the complete output as a string.

It returns the output from the executed command or NULL if an error occurred or the command produces no output.

<?php 

$command = escapeshellcmd('/usr/custom/test.py');
$output = shell_exec($command);
echo $output;

?>

In Python file test.py, verify this text in first line: (see shebang explain):

#!/usr/bin/env python

Also Python file must have correct privileges (execution for user www-data / apache if PHP script runs in browser or curl) and/or must be “executable”. Also all commands into .py file must have correct privileges:

Taken from php manual:

Just a quick reminder for those trying to use shell_exec on a unix-type platform and can’t seem to get it to work. PHP executes as the web user on the system (generally www for Apache), so you need to make sure that the web user has rights to whatever files or directories that you are trying to use in the shell_exec command. Other wise, it won’t appear to be doing anything.

To make executable a file on unix-type platforms:

chmod +x myscript.py

回答 1

我建议passthru直接使用和处理输出缓冲区:

ob_start();
passthru('/usr/bin/python2.7 /srv/http/assets/py/switch.py arg1 arg2');
$output = ob_get_clean(); 

I recommend using passthru and handling the output buffer directly:

ob_start();
passthru('/usr/bin/python2.7 /srv/http/assets/py/switch.py arg1 arg2');
$output = ob_get_clean(); 

回答 2

如果您想知道命令的返回状态并获取整个stdout输出,则可以实际使用exec

$command = 'ls';
exec($command, $out, $status);

$out是所有行的数组。$status是退货状态。对于调试非常有用。

如果您还想查看stderr输出,则可以使用proc_open或直接将其添加2>&1到中$command。后者通常足以使事情正常运行,并更快地实现。

If you want to know the return status of the command and get the entire stdout output you can actually use exec:

$command = 'ls';
exec($command, $out, $status);

$out is an array of all lines. $status is the return status. Very useful for debugging.

If you also want to see the stderr output you can either play with proc_open or simply add 2>&1 to your $command. The latter is often sufficient to get things working and way faster to “implement”.


回答 3

亚历杭德罗(Alejandro)钉上了钉子,为异常(Ubuntu或Debian)添加了说明-我没有代表要添加到答案本身:

sudoers文件: sudo visudo

添加了异常: www-data ALL=(ALL) NOPASSWD: ALL

Alejandro nailed it, adding clarification to the exception (Ubuntu or Debian) – I don’t have the rep to add to the answer itself:

sudoers file: sudo visudo

exception added: www-data ALL=(ALL) NOPASSWD: ALL


回答 4

根据情况明确使用哪个命令

exec() -执行外部程序

system() -执行外部程序并显示输出

passthru() -执行外部程序并显示原始输出

资料来源:http : //php.net/manual/en/function.exec.php

To clarify which command to use based on the situation

exec() – Execute an external program

system() – Execute an external program and display the output

passthru() – Execute an external program and display raw output

Source: http://php.net/manual/en/function.exec.php


回答 5

就我而言,我需要在www名为的目录中创建一个新文件夹scripts。在其中scripts添加了一个名为的新文件test.py

然后sudo chown www-data:root scripts,我使用和sudo chown www-data:root test.py

然后我转到新scripts目录并使用sudo chmod +x test.py

我的test.py文件看起来像这样。请注意不同的Python版本:

#!/usr/bin/env python3.5
print("Hello World!")

从PHP,我现在这样做:

$message = exec("/var/www/scripts/test.py 2>&1");
print_r($message);

您应该看到:Hello World!

In my case I needed to create a new folder in the www directory called scripts. Within scripts I added a new file called test.py.

I then used sudo chown www-data:root scripts and sudo chown www-data:root test.py.

Then I went to the new scripts directory and used sudo chmod +x test.py.

My test.py file it looks like this. Note the different Python version:

#!/usr/bin/env python3.5
print("Hello World!")

From php I now do this:

$message = exec("/var/www/scripts/test.py 2>&1");
print_r($message);

And you should see: Hello World!


回答 6

上述方法似乎很复杂。使用我的方法作为参考。

我有两个文件:

  • 运行.php

  • mkdir.py

在这里,我创建了一个包含GO按钮的HTML页面。每当您按下此按钮时,都会在您提到的路径中的目录中创建一个新文件夹。

运行.php

<html>
 <body>
  <head>
   <title>
     run
   </title>
  </head>

   <form method="post">

    <input type="submit" value="GO" name="GO">
   </form>
 </body>
</html>

<?php
	if(isset($_POST['GO']))
	{
		shell_exec("python /var/www/html/lab/mkdir.py");
		echo"success";
	}
?>

mkdir.py

#!/usr/bin/env python    
import os    
os.makedirs("thisfolder");

The above methods seem to be complex. Use my method as a reference.

I have these two files:

  • run.php

  • mkdir.py

Here, I’ve created an HTML page which contains a GO button. Whenever you press this button a new folder will be created in directory whose path you have mentioned.

run.php

<html>
 <body>
  <head>
   <title>
     run
   </title>
  </head>

   <form method="post">

    <input type="submit" value="GO" name="GO">
   </form>
 </body>
</html>

<?php
	if(isset($_POST['GO']))
	{
		shell_exec("python /var/www/html/lab/mkdir.py");
		echo"success";
	}
?>

mkdir.py

#!/usr/bin/env python    
import os    
os.makedirs("thisfolder");

回答 7

这很琐碎,但只是想帮助已经遵循亚历杭德罗建议但遇到此错误的任何人:

sh:blabla.py:找不到命令

如果有人遇到该错误,那么Alejandro需要对php文件进行一些更改:

$command = escapeshellcmd('python blabla.py');

This is so trivial, but just wanted to help anyone who already followed along Alejandro’s suggestion but encountered this error:

sh: blabla.py: command not found

If anyone encountered that error, then a little change needs to be made to the php file by Alejandro:

$command = escapeshellcmd('python blabla.py');

Python子进程。Popen“ OSError:[Errno 12]无法分配内存”

问题:Python子进程。Popen“ OSError:[Errno 12]无法分配内存”

注意:此问题最初是在此处提出的但赏金时间已过,即使实际上未找到可接受的答案。我正在重新询问这个问题,包括原始问题中提供的所有详细信息。

python脚本使用sched模块每60秒运行一组类函数:

# sc is a sched.scheduler instance
sc.enter(60, 1, self.doChecks, (sc, False))

该脚本使用此处的代码作为守护进程运行。

在doChecks中调用的许多类方法使用子过程模块来调用系统函数,以获取系统统计信息:

ps = subprocess.Popen(['ps', 'aux'], stdout=subprocess.PIPE).communicate()[0]

在整个脚本崩溃并出现以下错误之前,它可以正常运行一段时间:

File "/home/admin/sd-agent/checks.py", line 436, in getProcesses
File "/usr/lib/python2.4/subprocess.py", line 533, in __init__
File "/usr/lib/python2.4/subprocess.py", line 835, in _get_handles
OSError: [Errno 12] Cannot allocate memory

脚本崩溃后,服务器上的free -m输出为:

$ free -m
                  total       used       free     shared     buffers    cached
Mem:                894        345        549          0          0          0
-/+ buffers/cache:  345        549
Swap:                 0          0          0

服务器正在运行CentOS 5.3。我无法在自己的CentOS盒子上或任何其他报告相同问题的用户上进行复制。

我已经尝试了许多方法来调试此问题,如原始问题中所建议:

  1. 在Popen调用之前和之后记录free -m的输出。内存使用没有显着变化,即,脚本运行时内存不会逐渐消耗完。

  2. 我在Popen调用中添加了close_fds = True,但这没有什么不同-脚本仍然因相同的错误而崩溃。建议在这里这里

  3. 我检查了这所建议双方RLIMIT_DATA和RLIMIT_AS显示(-1,-1)的rlimits 这里

  4. 一篇文章建议没有交换空间可能是原因,但是交换实际上是按需提供的(根据Web主机),这在这里也被认为是虚假的原因。

  5. 进程已关闭,因为这是使用.communicate()的行为,该行为由Python源代码和此处的注释支持。

可以在GitHub上的第442行定义的getProcesses函数中找到整个检查。此操作由从第520行开始的doChecks()调用。

崩溃前,该脚本使用strace运行,输出如下:

recv(4, "Total Accesses: 516662\nTotal kBy"..., 234, 0) = 234
gettimeofday({1250893252, 887805}, NULL) = 0
write(3, "2009-08-21 17:20:52,887 - checks"..., 91) = 91
gettimeofday({1250893252, 888362}, NULL) = 0
write(3, "2009-08-21 17:20:52,888 - checks"..., 74) = 74
gettimeofday({1250893252, 888897}, NULL) = 0
write(3, "2009-08-21 17:20:52,888 - checks"..., 67) = 67
gettimeofday({1250893252, 889184}, NULL) = 0
write(3, "2009-08-21 17:20:52,889 - checks"..., 81) = 81
close(4)                                = 0
gettimeofday({1250893252, 889591}, NULL) = 0
write(3, "2009-08-21 17:20:52,889 - checks"..., 63) = 63
pipe([4, 5])                            = 0
pipe([6, 7])                            = 0
fcntl64(7, F_GETFD)                     = 0
fcntl64(7, F_SETFD, FD_CLOEXEC)         = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb7f12708) = -1 ENOMEM (Cannot allocate memory)
write(2, "Traceback (most recent call last"..., 35) = 35
open("/usr/bin/sd-agent/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/bin/sd-agent/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python24.zip/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/plat-linux2/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python2.4/lib-tk/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/lib-dynload/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/site-packages/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
write(2, "  File \"/usr/bin/sd-agent/agent."..., 52) = 52
open("/home/admin/sd-agent/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/bin/sd-agent/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python24.zip/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/plat-linux2/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python2.4/lib-tk/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/lib-dynload/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/site-packages/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
write(2, "  File \"/home/admin/sd-agent/dae"..., 60) = 60
open("/usr/bin/sd-agent/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/bin/sd-agent/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python24.zip/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/plat-linux2/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python2.4/lib-tk/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/lib-dynload/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/site-packages/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
write(2, "  File \"/usr/bin/sd-agent/agent."..., 54) = 54
open("/usr/lib/python2.4/sched.py", O_RDONLY|O_LARGEFILE) = 8
write(2, "  File \"/usr/lib/python2.4/sched"..., 55) = 55
fstat64(8, {st_mode=S_IFREG|0644, st_size=4054, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7d28000
read(8, "\"\"\"A generally useful event sche"..., 4096) = 4054
write(2, "    ", 4)                     = 4
write(2, "void = action(*argument)\n", 25) = 25
close(8)                                = 0
munmap(0xb7d28000, 4096)                = 0
open("/usr/bin/sd-agent/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/bin/sd-agent/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python24.zip/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/plat-linux2/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python2.4/lib-tk/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/lib-dynload/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/site-packages/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
write(2, "  File \"/usr/bin/sd-agent/checks"..., 60) = 60
open("/usr/bin/sd-agent/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/bin/sd-agent/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python24.zip/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/plat-linux2/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python2.4/lib-tk/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/lib-dynload/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/site-packages/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
write(2, "  File \"/usr/bin/sd-agent/checks"..., 64) = 64
open("/usr/lib/python2.4/subprocess.py", O_RDONLY|O_LARGEFILE) = 8
write(2, "  File \"/usr/lib/python2.4/subpr"..., 65) = 65
fstat64(8, {st_mode=S_IFREG|0644, st_size=39931, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7d28000
read(8, "# subprocess - Subprocesses with"..., 4096) = 4096
read(8, "lso, the newlines attribute of t"..., 4096) = 4096
read(8, "code < 0:\n        print >>sys.st"..., 4096) = 4096
read(8, "alse does not exist on 2.2.0\ntry"..., 4096) = 4096
read(8, " p2cread\n        # c2pread    <-"..., 4096) = 4096
write(2, "    ", 4)                     = 4
write(2, "errread, errwrite)\n", 19)    = 19
close(8)                                = 0
munmap(0xb7d28000, 4096)                = 0
open("/usr/lib/python2.4/subprocess.py", O_RDONLY|O_LARGEFILE) = 8
write(2, "  File \"/usr/lib/python2.4/subpr"..., 71) = 71
fstat64(8, {st_mode=S_IFREG|0644, st_size=39931, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7d28000
read(8, "# subprocess - Subprocesses with"..., 4096) = 4096
read(8, "lso, the newlines attribute of t"..., 4096) = 4096
read(8, "code < 0:\n        print >>sys.st"..., 4096) = 4096
read(8, "alse does not exist on 2.2.0\ntry"..., 4096) = 4096
read(8, " p2cread\n        # c2pread    <-"..., 4096) = 4096
read(8, "table(self, handle):\n           "..., 4096) = 4096
read(8, "rrno using _sys_errlist (or siml"..., 4096) = 4096
read(8, " p2cwrite = None, None\n         "..., 4096) = 4096
write(2, "    ", 4)                     = 4
write(2, "self.pid = os.fork()\n", 21)  = 21
close(8)                                = 0
munmap(0xb7d28000, 4096)                = 0
write(2, "OSError", 7)                  = 7
write(2, ": ", 2)                       = 2
write(2, "[Errno 12] Cannot allocate memor"..., 33) = 33
write(2, "\n", 1)                       = 1
unlink("/var/run/sd-agent.pid")         = 0
close(3)                                = 0
munmap(0xb7e0d000, 4096)                = 0
rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x589978}, {0xb89a60, [], SA_RESTORER, 0x589978}, 8) = 0
brk(0xa022000)                          = 0xa022000
exit_group(1)                           = ?

Note: This question was originally asked here but the bounty time expired even though an acceptable answer was not actually found. I am re-asking this question including all details provided in the original question.

A python script is running a set of class functions every 60 seconds using the sched module:

# sc is a sched.scheduler instance
sc.enter(60, 1, self.doChecks, (sc, False))

The script is running as a daemonised process using the code here.

A number of class methods that are called as part of doChecks use the subprocess module to call system functions in order to get system statistics:

ps = subprocess.Popen(['ps', 'aux'], stdout=subprocess.PIPE).communicate()[0]

This runs fine for a period of time before the entire script crashing with the following error:

File "/home/admin/sd-agent/checks.py", line 436, in getProcesses
File "/usr/lib/python2.4/subprocess.py", line 533, in __init__
File "/usr/lib/python2.4/subprocess.py", line 835, in _get_handles
OSError: [Errno 12] Cannot allocate memory

The output of free -m on the server once the script has crashed is:

$ free -m
                  total       used       free     shared     buffers    cached
Mem:                894        345        549          0          0          0
-/+ buffers/cache:  345        549
Swap:                 0          0          0

The server is running CentOS 5.3. I am unable to reproduce on my own CentOS boxes nor with any other user reporting the same problem.

I have tried a number of things to debug this as suggested in the original question:

  1. Logging the output of free -m before and after the Popen call. There is no significant change in memory usage i.e. memory is not gradually being used up as the script runs.

  2. I added close_fds=True to the Popen call but this made no difference – the script still crashed with the same error. Suggested here and here.

  3. I checked the rlimits which showed (-1, -1) on both RLIMIT_DATA and RLIMIT_AS as suggested here.

  4. An article suggested the having no swap space might be the cause but swap is actually available on demand (according to the web host) and this was also suggested as a bogus cause here.

  5. The processes are being closed because that is the behaviour of using .communicate() as backed up by the Python source code and comments here.

The entire checks can be found at on GitHub here with the getProcesses function defined from line 442. This is called by doChecks() starting at line 520.

The script was run with strace with the following output before the crash:

recv(4, "Total Accesses: 516662\nTotal kBy"..., 234, 0) = 234
gettimeofday({1250893252, 887805}, NULL) = 0
write(3, "2009-08-21 17:20:52,887 - checks"..., 91) = 91
gettimeofday({1250893252, 888362}, NULL) = 0
write(3, "2009-08-21 17:20:52,888 - checks"..., 74) = 74
gettimeofday({1250893252, 888897}, NULL) = 0
write(3, "2009-08-21 17:20:52,888 - checks"..., 67) = 67
gettimeofday({1250893252, 889184}, NULL) = 0
write(3, "2009-08-21 17:20:52,889 - checks"..., 81) = 81
close(4)                                = 0
gettimeofday({1250893252, 889591}, NULL) = 0
write(3, "2009-08-21 17:20:52,889 - checks"..., 63) = 63
pipe([4, 5])                            = 0
pipe([6, 7])                            = 0
fcntl64(7, F_GETFD)                     = 0
fcntl64(7, F_SETFD, FD_CLOEXEC)         = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb7f12708) = -1 ENOMEM (Cannot allocate memory)
write(2, "Traceback (most recent call last"..., 35) = 35
open("/usr/bin/sd-agent/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/bin/sd-agent/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python24.zip/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/plat-linux2/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python2.4/lib-tk/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/lib-dynload/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/site-packages/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
write(2, "  File \"/usr/bin/sd-agent/agent."..., 52) = 52
open("/home/admin/sd-agent/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/bin/sd-agent/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python24.zip/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/plat-linux2/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python2.4/lib-tk/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/lib-dynload/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/site-packages/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
write(2, "  File \"/home/admin/sd-agent/dae"..., 60) = 60
open("/usr/bin/sd-agent/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/bin/sd-agent/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python24.zip/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/plat-linux2/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python2.4/lib-tk/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/lib-dynload/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/site-packages/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
write(2, "  File \"/usr/bin/sd-agent/agent."..., 54) = 54
open("/usr/lib/python2.4/sched.py", O_RDONLY|O_LARGEFILE) = 8
write(2, "  File \"/usr/lib/python2.4/sched"..., 55) = 55
fstat64(8, {st_mode=S_IFREG|0644, st_size=4054, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7d28000
read(8, "\"\"\"A generally useful event sche"..., 4096) = 4054
write(2, "    ", 4)                     = 4
write(2, "void = action(*argument)\n", 25) = 25
close(8)                                = 0
munmap(0xb7d28000, 4096)                = 0
open("/usr/bin/sd-agent/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/bin/sd-agent/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python24.zip/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/plat-linux2/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python2.4/lib-tk/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/lib-dynload/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/site-packages/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
write(2, "  File \"/usr/bin/sd-agent/checks"..., 60) = 60
open("/usr/bin/sd-agent/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/bin/sd-agent/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python24.zip/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/plat-linux2/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory)
open("/usr/lib/python2.4/lib-tk/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/lib-dynload/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
open("/usr/lib/python2.4/site-packages/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory)
write(2, "  File \"/usr/bin/sd-agent/checks"..., 64) = 64
open("/usr/lib/python2.4/subprocess.py", O_RDONLY|O_LARGEFILE) = 8
write(2, "  File \"/usr/lib/python2.4/subpr"..., 65) = 65
fstat64(8, {st_mode=S_IFREG|0644, st_size=39931, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7d28000
read(8, "# subprocess - Subprocesses with"..., 4096) = 4096
read(8, "lso, the newlines attribute of t"..., 4096) = 4096
read(8, "code < 0:\n        print >>sys.st"..., 4096) = 4096
read(8, "alse does not exist on 2.2.0\ntry"..., 4096) = 4096
read(8, " p2cread\n        # c2pread    <-"..., 4096) = 4096
write(2, "    ", 4)                     = 4
write(2, "errread, errwrite)\n", 19)    = 19
close(8)                                = 0
munmap(0xb7d28000, 4096)                = 0
open("/usr/lib/python2.4/subprocess.py", O_RDONLY|O_LARGEFILE) = 8
write(2, "  File \"/usr/lib/python2.4/subpr"..., 71) = 71
fstat64(8, {st_mode=S_IFREG|0644, st_size=39931, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7d28000
read(8, "# subprocess - Subprocesses with"..., 4096) = 4096
read(8, "lso, the newlines attribute of t"..., 4096) = 4096
read(8, "code < 0:\n        print >>sys.st"..., 4096) = 4096
read(8, "alse does not exist on 2.2.0\ntry"..., 4096) = 4096
read(8, " p2cread\n        # c2pread    <-"..., 4096) = 4096
read(8, "table(self, handle):\n           "..., 4096) = 4096
read(8, "rrno using _sys_errlist (or siml"..., 4096) = 4096
read(8, " p2cwrite = None, None\n         "..., 4096) = 4096
write(2, "    ", 4)                     = 4
write(2, "self.pid = os.fork()\n", 21)  = 21
close(8)                                = 0
munmap(0xb7d28000, 4096)                = 0
write(2, "OSError", 7)                  = 7
write(2, ": ", 2)                       = 2
write(2, "[Errno 12] Cannot allocate memor"..., 33) = 33
write(2, "\n", 1)                       = 1
unlink("/var/run/sd-agent.pid")         = 0
close(3)                                = 0
munmap(0xb7e0d000, 4096)                = 0
rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x589978}, {0xb89a60, [], SA_RESTORER, 0x589978}, 8) = 0
brk(0xa022000)                          = 0xa022000
exit_group(1)                           = ?

回答 0

作为一般规则(即香草内核),fork/ clone有故障ENOMEM 发生的具体原因的任何一个诚实的神了内存不足的条件dup_mmdup_task_structalloc_pidmpol_dupmm_init等呱呱叫),或者是因为security_vm_enough_memory_mm你失望实施过载策略

首先,在尝试进行分叉时,检查未能分叉的进程的vmsize,然后将其与过量使用策略相关的可用内存(物理和交换)量进行比较(插入数字)。

在您的特定情况下,请注意,Virtuozzo 在过量使用执法方面还有其他检查。而且,我不确定您容器内部交换和过量使用配置真正拥有多少控制权(以影响执行结果)。

现在,为了真正前进,我想告诉您,您还有两个选择

  • 切换到更大的实例,或者
  • 投入一些编码工作来更有效地控制脚本的内存占用量

注意,如果事实证明不是您自己,而是与您运行amock在同一台服务器上的其他实例并置在另一个实例中,那么编码工作可能就一事无成。

在内存方面,我们已经知道subprocess.Popen使用fork/ clone 在幕后,这意味着每次调用它时,您都在请求与Python已经耗尽的内存一样多的内存,即增加数百MB,以便exec很小的10kB可执行文件,例如freeps。如果出现不利的过量使用政策,您很快就会看到ENOMEM

替代方法fork没有此父页面表等。复制问题为vforkposix_spawn。但是,如果您不想subprocess.Popenvfork/ 重写大块的代码posix_spawn,请考虑suprocess.Popen在脚本开始时(Python的内存占用最小的情况下)仅使用一次,以生成一个shell脚本,然后再运行free/ ps/ sleep以及其他与脚本并行循环;轮询脚本的输出或同步读取它,如果您还有其他要异步处理的内容,则可能从一个单独的线程中读取它-使用Python处理数据,但将分叉交给下级进程处理。

无论其,在您的特定情况下,你可以跳过调用psfree干脆; 无论您选择自己亲自还是通过现有的库和/或程序包访问这些信息,都可以直接在Python中直接从中procfs使用。如果和是你正在运行的唯一的实用工具,那么你就可以弄死完全psfreesubprocess.Popen

最后,无论您做什么subprocess.Popen,如果脚本泄漏内存,您最终还是会碰壁。密切注意它,并检查是否有内存泄漏

As a general rule (i.e. in vanilla kernels), fork/clone failures with ENOMEM occur specifically because of either an honest to God out-of-memory condition (dup_mm, dup_task_struct, alloc_pid, mpol_dup, mm_init etc. croak), or because security_vm_enough_memory_mm failed you while enforcing the overcommit policy.

Start by checking the vmsize of the process that failed to fork, at the time of the fork attempt, and then compare to the amount of free memory (physical and swap) as it relates to the overcommit policy (plug the numbers in.)

In your particular case, note that Virtuozzo has additional checks in overcommit enforcement. Moreover, I’m not sure how much control you truly have, from within your container, over swap and overcommit configuration (in order to influence the outcome of the enforcement.)

Now, in order to actually move forward I’d say you’re left with two options:

  • switch to a larger instance, or
  • put some coding effort into more effectively controlling your script’s memory footprint

NOTE that the coding effort may be all for naught if it turns out that it’s not you, but some other guy collocated in a different instance on the same server as you running amock.

Memory-wise, we already know that subprocess.Popen uses fork/clone under the hood, meaning that every time you call it you’re requesting once more as much memory as Python is already eating up, i.e. in the hundreds of additional MB, all in order to then exec a puny 10kB executable such as free or ps. In the case of an unfavourable overcommit policy, you’ll soon see ENOMEM.

Alternatives to fork that do not have this parent page tables etc. copy problem are vfork and posix_spawn. But if you do not feel like rewriting chunks of subprocess.Popen in terms of vfork/posix_spawn, consider using suprocess.Popen only once, at the beginning of your script (when Python’s memory footprint is minimal), to spawn a shell script that then runs free/ps/sleep and whatever else in a loop parallel to your script; poll the script’s output or read it synchronously, possibly from a separate thread if you have other stuff to take care of asynchronously — do your data crunching in Python but leave the forking to the subordinate process.

HOWEVER, in your particular case you can skip invoking ps and free altogether; that information is readily available to you in Python directly from procfs, whether you choose to access it yourself or via existing libraries and/or packages. If ps and free were the only utilities you were running, then you can do away with subprocess.Popen completely.

Finally, whatever you do as far as subprocess.Popen is concerned, if your script leaks memory you will still hit the wall eventually. Keep an eye on it, and check for memory leaks.


回答 1

free -m我看来,从输出看,您实际上没有可用的交换内存。我不确定在Linux中交换是否总是可以按需自动进行,但是我遇到了同样的问题,这里的答案都没有真正帮助我。但是,添加一些交换内存可以解决我的问题,因为这可能会帮助其他面临相同问题的人,所以我发布了有关如何添加1GB交换的答案(在Ubuntu 12.04上,但对于其他发行版也应类似地工作)。

您可以首先检查是否启用了任何交换内存。

$sudo swapon -s

如果为空,则表示您没有启用任何交换。要添加1GB交换空间:

$sudo dd if=/dev/zero of=/swapfile bs=1024 count=1024k
$sudo mkswap /swapfile
$sudo swapon /swapfile

将以下行添加到中,fstab以使交换永久生效。

$sudo vim /etc/fstab

     /swapfile       none    swap    sw      0       0 

来源和更多信息可以在这里找到。

Looking at the output of free -m it seems to me that you actually do not have swap memory available. I am not sure if in Linux the swap always will be available automatically on demand, but I was having the same problem and none of the answers here really helped me. Adding some swap memory however, fixed the problem in my case so since this might help other people facing the same problem, I post my answer on how to add a 1GB swap (on Ubuntu 12.04 but it should work similarly for other distributions.)

You can first check if there is any swap memory enabled.

$sudo swapon -s

if it is empty, it means you don’t have any swap enabled. To add a 1GB swap:

$sudo dd if=/dev/zero of=/swapfile bs=1024 count=1024k
$sudo mkswap /swapfile
$sudo swapon /swapfile

Add the following line to the fstab to make the swap permanent.

$sudo vim /etc/fstab

     /swapfile       none    swap    sw      0       0 

Source and more information can be found here.


回答 2

swap可能不是以前建议的红色鲱鱼。之前的python进程有多大ENOMEM

在内核2.6下,/proc/sys/vm/swappiness控制内核将如何积极地进行交换,并overcommit*归档内核可以眨眨一下头来分配多少内存以及如何精确分配内存。就像您的Facebook关系状态一样,这很复杂

…但是交换实际上是按需提供的(根据Web主机)…

但不是根据free(1)命令的输出,该命令的输出不显示服务器实例识别的交换空间。现在,您的Web主机肯定比我对这个主题了解更多,但是我使用的虚拟RHEL / CentOS系统报告了可用于来宾OS的交换。

改编Red Hat KB第15252条

只要匿名内存和系统V共享内存的总和少于RAM的3/4,红帽企业Linux 5系统就可以很好地运行,根本没有交换空间。….内存小于或等于4GB的系统 [建议]至少具有2GB的交换空间。

将您的/proc/sys/vm设置与普通的CentOS 5.3安装进行比较。添加交换文件。棘轮下来swappiness,看看你是否再活下去。

swap may not be the red herring previously suggested. How big is the python process in question just before the ENOMEM?

Under kernel 2.6, /proc/sys/vm/swappiness controls how aggressively the kernel will turn to swap, and overcommit* files how much and how precisely the kernel may apportion memory with a wink and a nod. Like your facebook relationship status, it’s complicated.

…but swap is actually available on demand (according to the web host)…

but not according to the output of your free(1) command, which shows no swap space recognized by your server instance. Now, your web host may certainly know much more than I about this topic, but virtual RHEL/CentOS systems I’ve used have reported swap available to the guest OS.

Adapting Red Hat KB Article 15252:

A Red Hat Enterprise Linux 5 system will run just fine with no swap space at all as long as the sum of anonymous memory and system V shared memory is less than about 3/4 the amount of RAM. …. Systems with 4GB of ram or less [are recommended to have] a minimum of 2GB of swap space.

Compare your /proc/sys/vm settings to a plain CentOS 5.3 installation. Add a swap file. Ratchet down swappiness and see if you live any longer.


回答 3

为了轻松解决,您可以

echo 1 > /proc/sys/vm/overcommit_memory

如果您确定系统有足够的内存。参见Linux以上的提交启发式方法

For an easy fix, you could

echo 1 > /proc/sys/vm/overcommit_memory

if your’re sure that your system has enough memory. See Linux over commit heuristic.


回答 4

我仍然怀疑您的客户/用户已加载了某些内核模块或驱动程序,从而干扰了clone()系统调用(可能是一些晦涩的安全性增强功能,例如LIDS,但更为晦涩吗?),或者是以某种方式填充了某些内核数据结构对fork()/来说是必需的clone()(进程表,页面表,文件描述符表等)。

这是fork(2)手册页的相关部分:

错误
       EAGAIN fork()无法分配足够的内存来复制父级的页表并为该任务分配任务结构
              儿童。

       EAGAIN无法创建新进程,因为遇到了调用者的RLIMIT_NPROC资源限制。至
              超过此限制,该进程必须具有CAP_SYS_ADMIN或CAP_SYS_RESOURCE能力。

       由于内存紧张,ENOMEM fork()无法分配必要的内核结构。

我建议让用户在引导到普通的通用内核之后,并仅加载最少的一组模块和驱动程序(运行应用程序/脚本的最低要求)之后尝试一下。从那里开始,假设它在该配置下有效,则他们可以在显示该问题的配置和配置之间执行二进制搜索。这是标准的系统管理员故障排除101。

您中的相关行strace是:

clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb7f12708) = -1 ENOMEM (Cannot allocate memory)

…我知道其他人已经讨论过交换和内存的可用性(我建议您至少设置一个小的交换分区,即使该分区位于RAM磁盘上,也具有讽刺意味的是…与可用零交换的那些交换(exceptions处理路径)相比,甚至很少的可用交换也被广泛地执行。

但是,我怀疑这仍然是鲱鱼。

free报告高速缓存和缓冲区正在使用的0(零)内存的事实非常令人不安。我怀疑free输出…甚至可能是您的应用程序问题,是由某种专有内核模块引起的,该模块以某种方式干扰了内存分配。

根据fork()/ clone()的手册页,如果您的调用会导致资源限制冲突(RLIMIT_NPROC),则fork()系统调用应返回EAGAIN …但是,它没有说是否要返回EAGAIN违反其他RLIMIT *。无论如何,如果目标/主机具有某种奇怪的Vormetric或其他安全设置(或者即使您的进程在某种奇怪的SELinux策略下运行),也可能导致此-ENOMEM故障。

不太可能是正常的Linux / UNIX问题。您正在那里进行一些非标准的操作。

I continue to suspect that your customer/user has some kernel module or driver loaded which is interfering with the clone() system call (perhaps some obscure security enhancement, something like LIDS but more obscure?) or is somehow filling up some of the kernel data structures that are necessary for fork()/clone() to operate (process table, page tables, file descriptor tables, etc).

Here’s the relevant portion of the fork(2) man page:

ERRORS
       EAGAIN fork() cannot allocate sufficient memory to copy the parent's page tables and allocate a task  structure  for  the
              child.

       EAGAIN It  was not possible to create a new process because the caller's RLIMIT_NPROC resource limit was encountered.  To
              exceed this limit, the process must have either the CAP_SYS_ADMIN or the CAP_SYS_RESOURCE capability.

       ENOMEM fork() failed to allocate the necessary kernel structures because memory is tight.

I suggest having the user try this after booting into a stock, generic kernel and with only a minimal set of modules and drivers loaded (minimum necessary to run your application/script). From there, assuming it works in that configuration, they can perform a binary search between that and the configuration which exhibits the issue. This is standard sysadmin troubleshooting 101.

The relevant line in your strace is:

clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb7f12708) = -1 ENOMEM (Cannot allocate memory)

… I know others have talked about swap and memory availability (and I would recommend that you set up at least a small swap partition, ironically even if it’s on a RAM disk … the code paths through the Linux kernel when it has even a tiny bit of swap available have been exercised far more extensively than those (exception handling paths) in which there is zero swap available.

However I suspect that this is still a red herring.

The fact that free is reporting 0 (ZERO) memory in use by the cache and buffers is very disturbing. I suspect that the free output … and possibly your application issue here, are caused by some proprietary kernel module which is interfering with the memory allocation in some way.

According to the man pages for fork()/clone() the fork() system call should return EAGAIN if your call would cause a resource limit violation (RLIMIT_NPROC) … however, it doesn’t say if EAGAIN is to be returned by other RLIMIT* violations. In any event if your target/host has some sort of weird Vormetric or other security settings (or even if your process is running under some weird SELinux policy) then it might be causing this -ENOMEM failure.

It’s pretty unlikely to be a normal run-of-the-mill Linux/UNIX issue. You’ve got something non-standard going on there.


回答 5

您是否尝试过使用:

(status,output) = commands.getstatusoutput("ps aux")

我认为这为我解决了完全相同的问题。但是后来我的过程最终被杀死,而不是没有产生,这更糟。

经过一些测试,我发现这仅在旧版本的python上发生:它在2.6.5中发生,而在2.7.2中不发生

我的搜索将我引到了这里python-close_fds-issue,但是取消设置closed_fds并不能解决问题。仍然值得一读。

我发现python通过仅关注它就泄漏了文件描述符:

watch "ls /proc/$PYTHONPID/fd | wc -l"

像您一样,我确实想捕获命令的输出,并且确实要避免OOM错误……但是看来,唯一的方法是让人们使用没有太多错误的Python版本。不理想…

Have you tried using:

(status,output) = commands.getstatusoutput("ps aux")

I thought this had fixed the exact same problem for me. But then my process ended up getting killed instead of failing to spawn, which is even worse..

After some testing I found that this only occurred on older versions of python: it happens with 2.6.5 but not with 2.7.2

My search had led me here python-close_fds-issue, but unsetting closed_fds had not solved the issue. It is still well worth a read.

I found that python was leaking file descriptors by just keeping an eye on it:

watch "ls /proc/$PYTHONPID/fd | wc -l"

Like you, I do want to capture the command’s output, and I do want to avoid OOM errors… but it looks like the only way is for people to use a less buggy version of Python. Not ideal…


回答 6

munmap(0xb7d28000,4096)= 0
写(2,“ OSError”,7)= 7

我看过草率的代码,看起来像这样:

serrno = errno;
some_Syscall(...)
if (serrno != errno)
/* sound alarm: CATROSTOPHIC ERROR !!! */

您应该检查这是否是python代码中正在发生的事情。Errno仅在进行中的系统调用失败时才有效。

编辑添加:

您没有说这个过程可以持续多久。可能的内存使用者

  • 分叉过程
  • 未使用的数据结构
  • 共享库
  • 内存映射文件

munmap(0xb7d28000, 4096) = 0
write(2, “OSError”, 7) = 7

I’ve seen sloppy code that looks like this:

serrno = errno;
some_Syscall(...)
if (serrno != errno)
/* sound alarm: CATROSTOPHIC ERROR !!! */

You should check to see if this is what is happening in the python code. Errno is only valid if the proceeding system call failed.

Edited to add:

You don’t say how long this process lives. Possible consumers of memory

  • forked processes
  • unused data structures
  • shared libraries
  • memory mapped files

回答 7

也许你可以简单地

$ sudo bash -c "echo vm.overcommit_memory=1 >> /etc/sysctl.conf"
$ sudo sysctl -p

它适用于我的情况。

参考:https : //github.com/openai/gym/issues/110#issuecomment-220672405

Maybe you can simply

$ sudo bash -c "echo vm.overcommit_memory=1 >> /etc/sysctl.conf"
$ sudo sysctl -p

It works for my case.

Reference: https://github.com/openai/gym/issues/110#issuecomment-220672405


获取MAC地址

问题:获取MAC地址

我需要一种跨平台的方法来在运行时确定计算机的MAC地址。对于Windows,可以使用“ wmi”模块,在Linux下,我能找到的唯一方法是运行ifconfig并在其输出中运行正则表达式。我不喜欢使用只能在一个OS上运行的程序包,而且更不用说容易出错的语法解析另一个程序的输出了。

有谁知道跨平台方法(Windows和Linux)方法来获取MAC地址?如果没有,还有谁比我上面列出的方法更优雅?

I need a cross platform method of determining the MAC address of a computer at run time. For windows the ‘wmi’ module can be used and the only method under Linux I could find was to run ifconfig and run a regex across its output. I don’t like using a package that only works on one OS, and parsing the output of another program doesn’t seem very elegant not to mention error prone.

Does anyone know a cross platform method (windows and linux) method to get the MAC address? If not, does anyone know any more elegant methods then those I listed above?


回答 0

Python 2.5包含一个uuid实现(至少在一个版本中),该实现需要mac地址。您可以轻松地将mac查找功能导入您自己的代码中:

from uuid import getnode as get_mac
mac = get_mac()

返回值是作为48位整数的mac地址。

Python 2.5 includes an uuid implementation which (in at least one version) needs the mac address. You can import the mac finding function into your own code easily:

from uuid import getnode as get_mac
mac = get_mac()

The return value is the mac address as 48 bit integer.


回答 1

在Linux下针对此问题的纯python解决方案,用于获取特定本地接口的MAC,最初由vishnubob作为注释发布,并在本activestate食谱中由Ben Mackey进行了改进

#!/usr/bin/python

import fcntl, socket, struct

def getHwAddr(ifname):
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    info = fcntl.ioctl(s.fileno(), 0x8927,  struct.pack('256s', ifname[:15]))
    return ':'.join(['%02x' % ord(char) for char in info[18:24]])

print getHwAddr('eth0')

这是Python 3兼容的代码:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import fcntl
import socket
import struct


def getHwAddr(ifname):
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    info = fcntl.ioctl(s.fileno(), 0x8927,  struct.pack('256s', bytes(ifname, 'utf-8')[:15]))
    return ':'.join('%02x' % b for b in info[18:24])


def main():
    print(getHwAddr('enp0s8'))


if __name__ == "__main__":
    main()

The pure python solution for this problem under Linux to get the MAC for a specific local interface, originally posted as a comment by vishnubob and improved by on Ben Mackey in this activestate recipe

#!/usr/bin/python

import fcntl, socket, struct

def getHwAddr(ifname):
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    info = fcntl.ioctl(s.fileno(), 0x8927,  struct.pack('256s', ifname[:15]))
    return ':'.join(['%02x' % ord(char) for char in info[18:24]])

print getHwAddr('eth0')

This is the Python 3 compatible code:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

import fcntl
import socket
import struct


def getHwAddr(ifname):
    s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    info = fcntl.ioctl(s.fileno(), 0x8927,  struct.pack('256s', bytes(ifname, 'utf-8')[:15]))
    return ':'.join('%02x' % b for b in info[18:24])


def main():
    print(getHwAddr('enp0s8'))


if __name__ == "__main__":
    main()

回答 2

netifaces是一个很好的模块,可用于获取mac地址(和其他地址)。它是跨平台的,比使用套接字或uuid更有意义。

>>> import netifaces
>>> netifaces.interfaces()
['lo', 'eth0', 'tun2']

>>> netifaces.ifaddresses('eth0')[netifaces.AF_LINK]
[{'addr': '08:00:27:50:f2:51', 'broadcast': 'ff:ff:ff:ff:ff:ff'}]

netifaces is a good module to use for getting the mac address (and other addresses). It’s crossplatform and makes a bit more sense than using socket or uuid.

>>> import netifaces
>>> netifaces.interfaces()
['lo', 'eth0', 'tun2']

>>> netifaces.ifaddresses('eth0')[netifaces.AF_LINK]
[{'addr': '08:00:27:50:f2:51', 'broadcast': 'ff:ff:ff:ff:ff:ff'}]


回答 3

您应该注意的另一件事是,uuid.getnode()可以通过返回随机的48位数字来伪造MAC地址,这可能不是您所期望的。另外,也没有明显的迹象表明MAC地址已被伪造,但是您可以通过调用getnode()两次并查看结果是否变化来检测到它。如果两个调用都返回了相同的值,则说明您具有MAC地址,否则得到的是伪造的地址。

>>> print uuid.getnode.__doc__
Get the hardware address as a 48-bit positive integer.

    The first time this runs, it may launch a separate program, which could
    be quite slow.  If all attempts to obtain the hardware address fail, we
    choose a random 48-bit number with its eighth bit set to 1 as recommended
    in RFC 4122.

One other thing that you should note is that uuid.getnode() can fake the MAC addr by returning a random 48-bit number which may not be what you are expecting. Also, there’s no explicit indication that the MAC address has been faked, but you could detect it by calling getnode() twice and seeing if the result varies. If the same value is returned by both calls, you have the MAC address, otherwise you are getting a faked address.

>>> print uuid.getnode.__doc__
Get the hardware address as a 48-bit positive integer.

    The first time this runs, it may launch a separate program, which could
    be quite slow.  If all attempts to obtain the hardware address fail, we
    choose a random 48-bit number with its eighth bit set to 1 as recommended
    in RFC 4122.

回答 4

有时我们有多个网络接口。

找出特定接口的mac地址的一种简单方法是:

def getmac(interface):

  try:
    mac = open('/sys/class/net/'+interface+'/address').readline()
  except:
    mac = "00:00:00:00:00:00"

  return mac[0:17]

调用方法很简单

myMAC = getmac("wlan0")

Sometimes we have more than one net interface.

A simple method to find out the mac address of a specific interface, is:

def getmac(interface):

  try:
    mac = open('/sys/class/net/'+interface+'/address').readline()
  except:
    mac = "00:00:00:00:00:00"

  return mac[0:17]

to call the method is simple

myMAC = getmac("wlan0")

回答 5

从这里使用我的答案:https : //stackoverflow.com/a/18031868/2362361

重要的是要知道您想要MAC到哪个iface,因为可能存在很多(蓝牙,几个nic等)。

当您知道需要使用MAC的iface的IP netifaces(使用PyPI中提供)时,就可以完成此工作:

import netifaces as nif
def mac_for_ip(ip):
    'Returns a list of MACs for interfaces that have given IP, returns None if not found'
    for i in nif.interfaces():
        addrs = nif.ifaddresses(i)
        try:
            if_mac = addrs[nif.AF_LINK][0]['addr']
            if_ip = addrs[nif.AF_INET][0]['addr']
        except IndexError, KeyError: #ignore ifaces that dont have MAC or IP
            if_mac = if_ip = None
        if if_ip == ip:
            return if_mac
    return None

测试:

>>> mac_for_ip('169.254.90.191')
'2c:41:38:0a:94:8b'

Using my answer from here: https://stackoverflow.com/a/18031868/2362361

It would be important to know to which iface you want the MAC for since many can exist (bluetooth, several nics, etc.).

This does the job when you know the IP of the iface you need the MAC for, using netifaces (available in PyPI):

import netifaces as nif
def mac_for_ip(ip):
    'Returns a list of MACs for interfaces that have given IP, returns None if not found'
    for i in nif.interfaces():
        addrs = nif.ifaddresses(i)
        try:
            if_mac = addrs[nif.AF_LINK][0]['addr']
            if_ip = addrs[nif.AF_INET][0]['addr']
        except IndexError, KeyError: #ignore ifaces that dont have MAC or IP
            if_mac = if_ip = None
        if if_ip == ip:
            return if_mac
    return None

Testing:

>>> mac_for_ip('169.254.90.191')
'2c:41:38:0a:94:8b'

回答 6

您可以使用跨平台的psutil进行此操作:

import psutil
nics = psutil.net_if_addrs()
print [j.address for j in nics[i] for i in nics if i!="lo" and j.family==17]

You can do this with psutil which is cross-platform:

import psutil
nics = psutil.net_if_addrs()
print [j.address for j in nics[i] for i in nics if i!="lo" and j.family==17]

回答 7

请注意,您可以使用条件导入在python中构建自己的跨平台库。例如

import platform
if platform.system() == 'Linux':
  import LinuxMac
  mac_address = LinuxMac.get_mac_address()
elif platform.system() == 'Windows':
  # etc

这将允许您使用os.system调用或特定于平台的库。

Note that you can build your own cross-platform library in python using conditional imports. e.g.

import platform
if platform.system() == 'Linux':
  import LinuxMac
  mac_address = LinuxMac.get_mac_address()
elif platform.system() == 'Windows':
  # etc

This will allow you to use os.system calls or platform-specific libraries.


回答 8

如果您不介意依赖,则跨平台的getmac软件包将对此有效。它适用于Python 2.7+和3.4+。它将尝试许多不同的方法,直到获得地址或返回None。

from getmac import get_mac_address
eth_mac = get_mac_address(interface="eth0")
win_mac = get_mac_address(interface="Ethernet 3")
ip_mac = get_mac_address(ip="192.168.0.1")
ip6_mac = get_mac_address(ip6="::1")
host_mac = get_mac_address(hostname="localhost")
updated_mac = get_mac_address(ip="10.0.0.1", network_request=True)

免责声明:我是软件包的作者。

更新(2019年1月14日):该软件包现在仅支持Python 2.7+和3.4+。如果您需要使用旧版本的Python(2.5、2.6、3.2、3.3),则仍可以使用该软件包的旧版本。

The cross-platform getmac package will work for this, if you don’t mind taking on a dependency. It works with Python 2.7+ and 3.4+. It will try many different methods until either getting a address or returning None.

from getmac import get_mac_address
eth_mac = get_mac_address(interface="eth0")
win_mac = get_mac_address(interface="Ethernet 3")
ip_mac = get_mac_address(ip="192.168.0.1")
ip6_mac = get_mac_address(ip6="::1")
host_mac = get_mac_address(hostname="localhost")
updated_mac = get_mac_address(ip="10.0.0.1", network_request=True)

Disclaimer: I am the author of the package.

Update (Jan 14 2019): the package now only supports Python 2.7+ and 3.4+. You can still use an older version of the package if you need to work with an older Python (2.5, 2.6, 3.2, 3.3).


回答 9

要获取eth0接口MAC地址,

import psutil

nics = psutil.net_if_addrs()['eth0']

for interface in nics:
   if interface.family == 17:
      print(interface.address)

To get the eth0 interface MAC address,

import psutil

nics = psutil.net_if_addrs()['eth0']

for interface in nics:
   if interface.family == 17:
      print(interface.address)

回答 10

我不知道统一的方式,但是以下内容可能对您有用:

http://www.codeguru.com/Cpp/IN/network/networkinformation/article.php/c5451

在这种情况下,我要做的就是将它们包装成一个函数,并根据操作系统运行适当的命令,根据需要进行解析,并仅返回所需格式的MAC地址。当然,除了您只需要执行一次,并且与主代码相比,它看起来更干净以外,其他所有内容都一样。

I dont know of a unified way, but heres something that you might find useful:

http://www.codeguru.com/Cpp/I-N/network/networkinformation/article.php/c5451

What I would do in this case would be to wrap these up into a function, and based on the OS it would run the proper command, parse as required and return only the MAC address formatted as you want. Its ofcourse all the same, except that you only have to do it once, and it looks cleaner from the main code.


回答 11

对于Linux,让我介绍一个shell脚本,该脚本将显示mac地址并允许对其进行更改(MAC嗅探)。

 ifconfig eth0 | grep HWaddr |cut -dH -f2|cut -d\  -f2
 00:26:6c:df:c3:95

减少参数(我不是专家)可以尝试:

ifconfig etho | grep HWaddr
eth0      Link encap:Ethernet  HWaddr 00:26:6c:df:c3:95  

要更改MAC,我们可以这样做:

ifconfig eth0 down
ifconfig eth0 hw ether 00:80:48:BA:d1:30
ifconfig eth0 up

会将mac地址更改为00:80:48:BA:d1:30(临时,重新启动后将还原为实际地址)。

For Linux let me introduce a shell script that will show the mac address and allows to change it (MAC sniffing).

 ifconfig eth0 | grep HWaddr |cut -dH -f2|cut -d\  -f2
 00:26:6c:df:c3:95

Cut arguements may dffer (I am not an expert) try:

ifconfig etho | grep HWaddr
eth0      Link encap:Ethernet  HWaddr 00:26:6c:df:c3:95  

To change MAC we may do:

ifconfig eth0 down
ifconfig eth0 hw ether 00:80:48:BA:d1:30
ifconfig eth0 up

will change mac address to 00:80:48:BA:d1:30 (temporarily, will restore to actual one upon reboot).


回答 12

或者,

import uuid
mac_id=(':'.join(['{:02x}'.format((uuid.getnode() >> ele) & 0xff)

Alternatively,

import uuid
mac_id=(':'.join(['{:02x}'.format((uuid.getnode() >> ele) & 0xff)

回答 13

For Linux you can retrieve the MAC address using a SIOCGIFHWADDR ioctl.

struct ifreq    ifr;
uint8_t         macaddr[6];

if ((s = socket(AF_INET, SOCK_DGRAM, IPPROTO_IP)) < 0)
    return -1;

strcpy(ifr.ifr_name, "eth0");

if (ioctl(s, SIOCGIFHWADDR, (void *)&ifr) == 0) {
    if (ifr.ifr_hwaddr.sa_family == ARPHRD_ETHER) {
        memcpy(macaddr, ifr.ifr_hwaddr.sa_data, 6);
        return 0;
... etc ...

You’ve tagged the question “python”. I don’t know of an existing Python module to get this information. You could use ctypes to call the ioctl directly.

For Linux you can retrieve the MAC address using a SIOCGIFHWADDR ioctl.

struct ifreq    ifr;
uint8_t         macaddr[6];

if ((s = socket(AF_INET, SOCK_DGRAM, IPPROTO_IP)) < 0)
    return -1;

strcpy(ifr.ifr_name, "eth0");

if (ioctl(s, SIOCGIFHWADDR, (void *)&ifr) == 0) {
    if (ifr.ifr_hwaddr.sa_family == ARPHRD_ETHER) {
        memcpy(macaddr, ifr.ifr_hwaddr.sa_data, 6);
        return 0;
... etc ...

You’ve tagged the question “python”. I don’t know of an existing Python module to get this information. You could use ctypes to call the ioctl directly.