问题:将stdout重定向到Python中的文件?

如何在Python中将stdout重定向到任意文件?

当从ssh会话中启动运行了很长时间的Python脚本(例如Web应用程序)并进行背景调整,并且ssh会话关闭时,该应用程序将在尝试写入stdout时引发IOError并失败。我需要找到一种方法来使应用程序和模块输出到文件而不是stdout,以防止由于IOError而导致失败。当前,我使用nohup将输出重定向到文件,并且可以完成工作,但是我想知道是否有一种出于好奇而无需使用nohup的方法。

我已经尝试过了sys.stdout = open('somefile', 'w'),但是这似乎并不能阻止某些外部模块仍然输出到终端(或者sys.stdout = ...线路根本没有触发)。我知道它应该可以通过我测试过的简单脚本来工作,但是我还没有时间在Web应用程序上进行测试。

How do I redirect stdout to an arbitrary file in Python?

When a long-running Python script (e.g, web application) is started from within the ssh session and backgounded, and the ssh session is closed, the application will raise IOError and fail the moment it tries to write to stdout. I needed to find a way to make the application and modules output to a file rather than stdout to prevent failure due to IOError. Currently, I employ nohup to redirect output to a file, and that gets the job done, but I was wondering if there was a way to do it without using nohup, out of curiosity.

I have already tried sys.stdout = open('somefile', 'w'), but this does not seem to prevent some external modules from still outputting to terminal (or maybe the sys.stdout = ... line did not fire at all). I know it should work from simpler scripts I’ve tested on, but I also didn’t have time yet to test on a web application yet.


回答 0

如果要在Python脚本中进行重定向,则设置sys.stdout为文件对象可以解决问题:

import sys
sys.stdout = open('file', 'w')
print('test')

一种更常见的方法是在执行时使用外壳重定向(与Windows和Linux相同):

$ python foo.py > file

If you want to do the redirection within the Python script, setting sys.stdout to a file object does the trick:

import sys
sys.stdout = open('file', 'w')
print('test')

A far more common method is to use shell redirection when executing (same on Windows and Linux):

$ python foo.py > file

回答 1

Python 3.4中有contextlib.redirect_stdout()功能

from contextlib import redirect_stdout

with open('help.txt', 'w') as f:
    with redirect_stdout(f):
        print('it now prints to `help.text`')

它类似于:

import sys
from contextlib import contextmanager

@contextmanager
def redirect_stdout(new_target):
    old_target, sys.stdout = sys.stdout, new_target # replace sys.stdout
    try:
        yield new_target # run some code with the replaced stdout
    finally:
        sys.stdout = old_target # restore to the previous value

可以在早期的Python版本中使用。后一版本不可重用。如果需要,可以将其制成一个。

它不会在文件描述符级别重定向标准输出,例如:

import os
from contextlib import redirect_stdout

stdout_fd = sys.stdout.fileno()
with open('output.txt', 'w') as f, redirect_stdout(f):
    print('redirected to a file')
    os.write(stdout_fd, b'not redirected')
    os.system('echo this also is not redirected')

b'not redirected'并且'echo this also is not redirected'不会重定向到该output.txt文件。

要在文件描述符级别重定向,os.dup2()可以使用:

import os
import sys
from contextlib import contextmanager

def fileno(file_or_fd):
    fd = getattr(file_or_fd, 'fileno', lambda: file_or_fd)()
    if not isinstance(fd, int):
        raise ValueError("Expected a file (`.fileno()`) or a file descriptor")
    return fd

@contextmanager
def stdout_redirected(to=os.devnull, stdout=None):
    if stdout is None:
       stdout = sys.stdout

    stdout_fd = fileno(stdout)
    # copy stdout_fd before it is overwritten
    #NOTE: `copied` is inheritable on Windows when duplicating a standard stream
    with os.fdopen(os.dup(stdout_fd), 'wb') as copied: 
        stdout.flush()  # flush library buffers that dup2 knows nothing about
        try:
            os.dup2(fileno(to), stdout_fd)  # $ exec >&to
        except ValueError:  # filename
            with open(to, 'wb') as to_file:
                os.dup2(to_file.fileno(), stdout_fd)  # $ exec > to
        try:
            yield stdout # allow code to be run with the redirected stdout
        finally:
            # restore stdout to its previous value
            #NOTE: dup2 makes stdout_fd inheritable unconditionally
            stdout.flush()
            os.dup2(copied.fileno(), stdout_fd)  # $ exec >&copied

如果stdout_redirected()使用代替,则现在可以使用相同的示例redirect_stdout()

import os
import sys

stdout_fd = sys.stdout.fileno()
with open('output.txt', 'w') as f, stdout_redirected(f):
    print('redirected to a file')
    os.write(stdout_fd, b'it is redirected now\n')
    os.system('echo this is also redirected')
print('this is goes back to stdout')

output.txt只要stdout_redirected()上下文管理器处于活动状态,以前打印在stdout上的输出现在将保留。

注意:stdout.flush()不会在直接在其上实现I / O的Python 3上刷新C stdio缓冲区read() / write()系统调用。要刷新所有打开的C stdio输出流,libc.fflush(None)如果某些C扩展使用基于stdio的I / O ,则可以显式调用:

try:
    import ctypes
    from ctypes.util import find_library
except ImportError:
    libc = None
else:
    try:
        libc = ctypes.cdll.msvcrt # Windows
    except OSError:
        libc = ctypes.cdll.LoadLibrary(find_library('c'))

def flush(stream):
    try:
        libc.fflush(None)
        stream.flush()
    except (AttributeError, ValueError, IOError):
        pass # unsupported

您可以使用stdout参数来重定向其他流,而不仅仅是sys.stdout合并sys.stderrsys.stdout

def merged_stderr_stdout():  # $ exec 2>&1
    return stdout_redirected(to=sys.stdout, stdout=sys.stderr)

例:

from __future__ import print_function
import sys

with merged_stderr_stdout():
     print('this is printed on stdout')
     print('this is also printed on stdout', file=sys.stderr)

注意:stdout_redirected()混合使用缓冲的I / O(sys.stdout通常)和未缓冲的I / O(直接对文件描述符进行操作)。当心,可能会有缓冲 问题

要回答,请进行编辑:您可以用来守护脚本并使用logging模块(如@ erikb85建议)代替print语句,而仅将stdout重定向到您nohup现在运行的长期运行的Python脚本。

There is contextlib.redirect_stdout() function in Python 3.4:

from contextlib import redirect_stdout

with open('help.txt', 'w') as f:
    with redirect_stdout(f):
        print('it now prints to `help.text`')

It is similar to:

import sys
from contextlib import contextmanager

@contextmanager
def redirect_stdout(new_target):
    old_target, sys.stdout = sys.stdout, new_target # replace sys.stdout
    try:
        yield new_target # run some code with the replaced stdout
    finally:
        sys.stdout = old_target # restore to the previous value

that can be used on earlier Python versions. The latter version is not reusable. It can be made one if desired.

It doesn’t redirect the stdout at the file descriptors level e.g.:

import os
from contextlib import redirect_stdout

stdout_fd = sys.stdout.fileno()
with open('output.txt', 'w') as f, redirect_stdout(f):
    print('redirected to a file')
    os.write(stdout_fd, b'not redirected')
    os.system('echo this also is not redirected')

b'not redirected' and 'echo this also is not redirected' are not redirected to the output.txt file.

To redirect at the file descriptor level, os.dup2() could be used:

import os
import sys
from contextlib import contextmanager

def fileno(file_or_fd):
    fd = getattr(file_or_fd, 'fileno', lambda: file_or_fd)()
    if not isinstance(fd, int):
        raise ValueError("Expected a file (`.fileno()`) or a file descriptor")
    return fd

@contextmanager
def stdout_redirected(to=os.devnull, stdout=None):
    if stdout is None:
       stdout = sys.stdout

    stdout_fd = fileno(stdout)
    # copy stdout_fd before it is overwritten
    #NOTE: `copied` is inheritable on Windows when duplicating a standard stream
    with os.fdopen(os.dup(stdout_fd), 'wb') as copied: 
        stdout.flush()  # flush library buffers that dup2 knows nothing about
        try:
            os.dup2(fileno(to), stdout_fd)  # $ exec >&to
        except ValueError:  # filename
            with open(to, 'wb') as to_file:
                os.dup2(to_file.fileno(), stdout_fd)  # $ exec > to
        try:
            yield stdout # allow code to be run with the redirected stdout
        finally:
            # restore stdout to its previous value
            #NOTE: dup2 makes stdout_fd inheritable unconditionally
            stdout.flush()
            os.dup2(copied.fileno(), stdout_fd)  # $ exec >&copied

The same example works now if stdout_redirected() is used instead of redirect_stdout():

import os
import sys

stdout_fd = sys.stdout.fileno()
with open('output.txt', 'w') as f, stdout_redirected(f):
    print('redirected to a file')
    os.write(stdout_fd, b'it is redirected now\n')
    os.system('echo this is also redirected')
print('this is goes back to stdout')

The output that previously was printed on stdout now goes to output.txt as long as stdout_redirected() context manager is active.

Note: stdout.flush() does not flush C stdio buffers on Python 3 where I/O is implemented directly on read()/write() system calls. To flush all open C stdio output streams, you could call libc.fflush(None) explicitly if some C extension uses stdio-based I/O:

try:
    import ctypes
    from ctypes.util import find_library
except ImportError:
    libc = None
else:
    try:
        libc = ctypes.cdll.msvcrt # Windows
    except OSError:
        libc = ctypes.cdll.LoadLibrary(find_library('c'))

def flush(stream):
    try:
        libc.fflush(None)
        stream.flush()
    except (AttributeError, ValueError, IOError):
        pass # unsupported

You could use stdout parameter to redirect other streams, not only sys.stdout e.g., to merge sys.stderr and sys.stdout:

def merged_stderr_stdout():  # $ exec 2>&1
    return stdout_redirected(to=sys.stdout, stdout=sys.stderr)

Example:

from __future__ import print_function
import sys

with merged_stderr_stdout():
     print('this is printed on stdout')
     print('this is also printed on stdout', file=sys.stderr)

Note: stdout_redirected() mixes buffered I/O (sys.stdout usually) and unbuffered I/O (operations on file descriptors directly). Beware, there could be buffering issues.

To answer, your edit: you could use to daemonize your script and use logging module (as @erikb85 suggested) instead of print statements and merely redirecting stdout for your long-running Python script that you run using nohup now.


回答 2

你可以尝试得更好

import sys

class Logger(object):
    def __init__(self, filename="Default.log"):
        self.terminal = sys.stdout
        self.log = open(filename, "a")

    def write(self, message):
        self.terminal.write(message)
        self.log.write(message)

sys.stdout = Logger("yourlogfilename.txt")
print "Hello world !" # this is should be saved in yourlogfilename.txt

you can try this too much better

import sys

class Logger(object):
    def __init__(self, filename="Default.log"):
        self.terminal = sys.stdout
        self.log = open(filename, "a")

    def write(self, message):
        self.terminal.write(message)
        self.log.write(message)

sys.stdout = Logger("yourlogfilename.txt")
print "Hello world !" # this is should be saved in yourlogfilename.txt

回答 3

其他答案未涵盖您希望分叉的进程共享新标准输出的情况。

要做到这一点:

from os import open, close, dup, O_WRONLY

old = dup(1)
close(1)
open("file", O_WRONLY) # should open on 1

..... do stuff and then restore

close(1)
dup(old) # should dup to 1
close(old) # get rid of left overs

The other answers didn’t cover the case where you want forked processes to share your new stdout.

To do that:

from os import open, close, dup, O_WRONLY

old = dup(1)
close(1)
open("file", O_WRONLY) # should open on 1

..... do stuff and then restore

close(1)
dup(old) # should dup to 1
close(old) # get rid of left overs

回答 4

引用自PEP 343-“ with”语句(添加的导入语句):

暂时重定向标准输出:

import sys
from contextlib import contextmanager
@contextmanager
def stdout_redirected(new_stdout):
    save_stdout = sys.stdout
    sys.stdout = new_stdout
    try:
        yield None
    finally:
        sys.stdout = save_stdout

用法如下:

with open(filename, "w") as f:
    with stdout_redirected(f):
        print "Hello world"

当然,这不是线程安全的,但是也没有手动进行相同的舞蹈。在单线程程序中(例如在脚本中),这是一种流行的处理方式。

Quoted from PEP 343 — The “with” Statement (added import statement):

Redirect stdout temporarily:

import sys
from contextlib import contextmanager
@contextmanager
def stdout_redirected(new_stdout):
    save_stdout = sys.stdout
    sys.stdout = new_stdout
    try:
        yield None
    finally:
        sys.stdout = save_stdout

Used as follows:

with open(filename, "w") as f:
    with stdout_redirected(f):
        print "Hello world"

This isn’t thread-safe, of course, but neither is doing this same dance manually. In single-threaded programs (for example in scripts) it is a popular way of doing things.


回答 5

import sys
sys.stdout = open('stdout.txt', 'w')
import sys
sys.stdout = open('stdout.txt', 'w')

回答 6

这是Yuda Prawira答案的一种变化:

  • 实现flush()和所有文件属性
  • 写为上下文管理器
  • stderr也捕获

import contextlib, sys

@contextlib.contextmanager
def log_print(file):
    # capture all outputs to a log file while still printing it
    class Logger:
        def __init__(self, file):
            self.terminal = sys.stdout
            self.log = file

        def write(self, message):
            self.terminal.write(message)
            self.log.write(message)

        def __getattr__(self, attr):
            return getattr(self.terminal, attr)

    logger = Logger(file)

    _stdout = sys.stdout
    _stderr = sys.stderr
    sys.stdout = logger
    sys.stderr = logger
    try:
        yield logger.log
    finally:
        sys.stdout = _stdout
        sys.stderr = _stderr


with log_print(open('mylogfile.log', 'w')):
    print('hello world')
    print('hello world on stderr', file=sys.stderr)

# you can capture the output to a string with:
# with log_print(io.StringIO()) as log:
#   ....
#   print('[captured output]', log.getvalue())

Here is a variation of Yuda Prawira answer:

  • implement flush() and all the file attributes
  • write it as a contextmanager
  • capture stderr also

.

import contextlib, sys

@contextlib.contextmanager
def log_print(file):
    # capture all outputs to a log file while still printing it
    class Logger:
        def __init__(self, file):
            self.terminal = sys.stdout
            self.log = file

        def write(self, message):
            self.terminal.write(message)
            self.log.write(message)

        def __getattr__(self, attr):
            return getattr(self.terminal, attr)

    logger = Logger(file)

    _stdout = sys.stdout
    _stderr = sys.stderr
    sys.stdout = logger
    sys.stderr = logger
    try:
        yield logger.log
    finally:
        sys.stdout = _stdout
        sys.stderr = _stderr


with log_print(open('mylogfile.log', 'w')):
    print('hello world')
    print('hello world on stderr', file=sys.stderr)

# you can capture the output to a string with:
# with log_print(io.StringIO()) as log:
#   ....
#   print('[captured output]', log.getvalue())

回答 7

基于以下答案:https : //stackoverflow.com/a/5916874/1060344,这是我弄清楚在我的一个项目中使用的另一种方法。对于要替换sys.stderr或替换的内容sys.stdout,必须确保替换符合file接口要求,尤其是在执行此操作时,因为在其他不受您控制的库中使用了stderr / stdout。该库可能正在使用文件对象的其他方法。

看看这种方式,我仍然可以让所有事情继续进行stderr / stdout(或与此有关的任何文件),并使用Python的日志记录工具将消息发送到日志文件中(但您实际上可以执行任何操作):

class FileToLogInterface(file):
    '''
    Interface to make sure that everytime anything is written to stderr, it is
    also forwarded to a file.
    '''

    def __init__(self, *args, **kwargs):
        if 'cfg' not in kwargs:
            raise TypeError('argument cfg is required.')
        else:
            if not isinstance(kwargs['cfg'], config.Config):
                raise TypeError(
                    'argument cfg should be a valid '
                    'PostSegmentation configuration object i.e. '
                    'postsegmentation.config.Config')
        self._cfg = kwargs['cfg']
        kwargs.pop('cfg')

        self._logger = logging.getlogger('access_log')

        super(FileToLogInterface, self).__init__(*args, **kwargs)

    def write(self, msg):
        super(FileToLogInterface, self).write(msg)
        self._logger.info(msg)

Based on this answer: https://stackoverflow.com/a/5916874/1060344, here is another way I figured out which I use in one of my projects. For whatever you replace sys.stderr or sys.stdout with, you have to make sure that the replacement complies with file interface, especially if this is something you are doing because stderr/stdout are used in some other library that is not under your control. That library may be using other methods of file object.

Check out this way where I still let everything go do stderr/stdout (or any file for that matter) and also send the message to a log file using Python’s logging facility (but you can really do anything with this):

class FileToLogInterface(file):
    '''
    Interface to make sure that everytime anything is written to stderr, it is
    also forwarded to a file.
    '''

    def __init__(self, *args, **kwargs):
        if 'cfg' not in kwargs:
            raise TypeError('argument cfg is required.')
        else:
            if not isinstance(kwargs['cfg'], config.Config):
                raise TypeError(
                    'argument cfg should be a valid '
                    'PostSegmentation configuration object i.e. '
                    'postsegmentation.config.Config')
        self._cfg = kwargs['cfg']
        kwargs.pop('cfg')

        self._logger = logging.getlogger('access_log')

        super(FileToLogInterface, self).__init__(*args, **kwargs)

    def write(self, msg):
        super(FileToLogInterface, self).write(msg)
        self._logger.info(msg)

回答 8

您需要一个终端多路复用器,例如tmuxGNU屏幕

令我惊讶的是,Ryan Amos对原始问题的一小段评论是唯一提及的解决方案远胜于所有其他提供的解决方案,无论python技巧有多聪明,他们收到了多少票。除了Ryan的评论,tmux是GNU屏幕的不错选择。

但是原理是一样的:如果您发现自己想在退出时让终端机继续运行,可以去咖啡馆吃三明治,然后去洗手间,回家(等),然后再连接到从任何地方或任何计算机终端会话,就好像你从来没有离开,终端多路复用器答案。将它们视为用于终端会话的VNC或远程桌面。其他任何方法都可以解决。另外,当老板和/或合伙人进来时,您无意间将ctrl-w / cmd-w终端窗口(而不是带有晦涩内容的浏览器窗口)作为ctrl-w / cmd-w,您将不会失去最后18小时的处理价值!

You need a terminal multiplexer like either tmux or GNU screen

I’m surprised that a small comment by Ryan Amos’ to the original question is the only mention of a solution far preferable to all the others on offer, no matter how clever the python trickery may be and how many upvotes they’ve received. Further to Ryan’s comment, tmux is a nice alternative to GNU screen.

But the principle is the same: if you ever find yourself wanting to leave a terminal job running while you log-out, head to the cafe for a sandwich, pop to the bathroom, go home (etc) and then later, reconnect to your terminal session from anywhere or any computer as though you’d never been away, terminal multiplexers are the answer. Think of them as VNC or remote desktop for terminal sessions. Anything else is a workaround. As a bonus, when the boss and/or partner comes in and you inadvertently ctrl-w / cmd-w your terminal window instead of your browser window with its dodgy content, you won’t have lost the last 18 hours-worth of processing!


回答 9

用其他语言(例如C)编写的程序必须做特别的魔术(称为双叉)才能与终端分离(并防止僵尸进程)。因此,我认为最好的解决方案是模拟它们。

重新执行程序的好处是,您可以在命令行上选择重定向,例如 /usr/bin/python mycoolscript.py 2>&1 1>/dev/null

有关更多信息,请参见此帖子:创建守护程序时执行双叉的原因是什么?

Programs written in other languages (e.g. C) have to do special magic (called double-forking) expressly to detach from the terminal (and to prevent zombie processes). So, I think the best solution is to emulate them.

A plus of re-executing your program is, you can choose redirections on the command-line, e.g. /usr/bin/python mycoolscript.py 2>&1 1>/dev/null

See this post for more info: What is the reason for performing a double fork when creating a daemon?


声明:本站所有文章,如无特殊说明或标注,均为本站原创发布。任何个人或组织,在未征得本站同意时,禁止复制、盗用、采集、发布本站内容到任何网站、书籍等各类媒体平台。如若本站内容侵犯了原著者的合法权益,可联系我们进行处理。