是否可以“破解” Python的打印功能?

问题:是否可以“破解” Python的打印功能?

注意:此问题仅供参考。我很想知道这样做有多深入到Python内部。

不久之前,在某个问题的内部开始了一个讨论,该问题是关于传递给print语句的字符串是否可以在调用to之后/期间进行修改print。例如,考虑以下功能:

def print_something():
    print('This cat was scared.')

现在,当print运行时,到终端的输出应显示:

This dog was scared.

请注意,单词“ cat”已被单词“ dog”代替。某处某种方式能够修改那些内部缓冲区以更改打印的内容。假设这样做是在未经原始代码作者明确许可的情况下进行的(因此,被黑客/劫持)。

这个评论从智者@abarnert,尤其让我思考:

有两种方法可以做到这一点,但是它们都很丑陋,绝不应该这样做。最丑陋的方法是code用一个带有不同co_consts 列表的对象替换 函数内部的对象。接下来可能是进入C API以访问str的内部缓冲区。[…]

因此,看起来这实际上是可能的。

这是我解决此问题的幼稚方法:

>>> import inspect
>>> exec(inspect.getsource(print_something).replace('cat', 'dog'))
>>> print_something()
This dog was scared.

当然,这exec很糟糕,但这并不能真正回答问题,因为 print调用when / after之后,它实际上并未进行任何修改。

正如@abarnert解释的那样,它将如何进行?

Note: This question is for informational purposes only. I am interested to see how deep into Python’s internals it is possible to go with this.

Not very long ago, a discussion began inside a certain question regarding whether the strings passed to print statements could be modified after/during the call to print has been made. For example, consider the function:

def print_something():
    print('This cat was scared.')

Now, when print is run, then the output to the terminal should display:

This dog was scared.

Notice the word “cat” has been replaced by the word “dog”. Something somewhere somehow was able to modify those internal buffers to change what was printed. Assume this is done without the original code author’s explicit permission (hence, hacking/hijacking).

This comment from the wise @abarnert, in particular, got me thinking:

There are a couple of ways to do that, but they’re all very ugly, and should never be done. The least ugly way is to probably replace the code object inside the function with one with a different co_consts list. Next is probably reaching into the C API to access the str’s internal buffer. […]

So, it looks like this is actually possible.

Here’s my naive way of approaching this problem:

>>> import inspect
>>> exec(inspect.getsource(print_something).replace('cat', 'dog'))
>>> print_something()
This dog was scared.

Of course, exec is bad, but that doesn’t really answer the question, because it does not actually modify anything during when/after print is called.

How would it be done as @abarnert has explained it?


回答 0

首先,实际上没有那么多hacky方式。我们要做的就是更改print打印内容,对吗?

_print = print
def print(*args, **kw):
    args = (arg.replace('cat', 'dog') if isinstance(arg, str) else arg
            for arg in args)
    _print(*args, **kw)

或者,类似地,您可以选择Monkey补丁sys.stdout而不是print


同样,这个exec … getsource …想法也没有错。好吧,这当然有很多问题,但是比这里的要少…


但是,如果您确实想修改函数对象的代码常量,则可以这样做。

如果您真的想真正使用代码对象,则应该使用bytecode(完成时)或byteplay(直到那时,或者对于较旧的Python版本)之类的库,而不是手动执行。即使对于这种琐碎的事情,CodeType初始化器还是很痛苦的。如果您确实需要做一些固定的工作lnotab,那么只有疯子才会手动进行。

另外,不用说,并非所有的Python实现都使用CPython风格的代码对象。这段代码可以在CPython 3.7中使用,并且可能所有版本都可以回溯到至少2.2,但需要进行一些细微的更改(不是代码黑客的东西,而是生成器表达式之类的东西),但是不适用于任何版本的IronPython。

import types

def print_function():
    print ("This cat was scared.")

def main():
    # A function object is a wrapper around a code object, with
    # a bit of extra stuff like default values and closure cells.
    # See inspect module docs for more details.
    co = print_function.__code__
    # A code object is a wrapper around a string of bytecode, with a
    # whole bunch of extra stuff, including a list of constants used
    # by that bytecode. Again see inspect module docs. Anyway, inside
    # the bytecode for string (which you can read by typing
    # dis.dis(string) in your REPL), there's going to be an
    # instruction like LOAD_CONST 1 to load the string literal onto
    # the stack to pass to the print function, and that works by just
    # reading co.co_consts[1]. So, that's what we want to change.
    consts = tuple(c.replace("cat", "dog") if isinstance(c, str) else c
                   for c in co.co_consts)
    # Unfortunately, code objects are immutable, so we have to create
    # a new one, copying over everything except for co_consts, which
    # we'll replace. And the initializer has a zillion parameters.
    # Try help(types.CodeType) at the REPL to see the whole list.
    co = types.CodeType(
        co.co_argcount, co.co_kwonlyargcount, co.co_nlocals,
        co.co_stacksize, co.co_flags, co.co_code,
        consts, co.co_names, co.co_varnames, co.co_filename,
        co.co_name, co.co_firstlineno, co.co_lnotab,
        co.co_freevars, co.co_cellvars)
    print_function.__code__ = co
    print_function()

main()

入侵代码对象可能会出什么问题?大多数情况下,segfaults RuntimeError会耗尽整个堆栈,更正常RuntimeError的segfault 会被处理,或者垃圾值可能只会引发a TypeErrorAttributeError当您尝试使用它们时。例如,尝试创建一个代码对象,该对象只带有一个RETURN_VALUE在堆栈上没有任何内容的字节码b'S\0'(3.6以上的字节码,b'S'之前),或者一个空元组(表示字节码中是否co_consts有a LOAD_CONST 0,或者varnames减1,因此最高的字节LOAD_FAST实际上加载了一个freevar) / cellvar单元格。为了获得一些真正的乐趣,如果您lnotab弄错了太多,那么只有在调试器中运行代码时,您的代码才会出现段错误。

使用bytecodebyteplay不会保护您免受所有这些问题的影响,但是它们确实具有一些基本的健全性检查,并且好的助手可以让您执行诸如插入代码块之类的事情,并使其担心更新所有偏移量和标签,以便您能够弄错了,等等。(此外,它们使您不必键入该可笑的6行构造函数,也不必调试由此产生的愚蠢的错字。)


现在进入第二。

我提到代码对象是不可变的。当然,const是一个元组,因此我们不能直接更改它。const元组中的东西是一个字符串,我们也不能直接更改它。这就是为什么我必须构建一个新的字符串来构建一个新的元组来构建一个新的代码对象的原因。

但是,如果您可以直接更改字符串怎么办?

好吧,在足够深入的内容下,所有内容都只是指向某些C数据的指针,对吗?如果您使用的是CPython,则有一个C API可以访问对象,并且可以使用ctypes它从Python本身内部访问该API,这是一个很糟糕的想法,他们将它们放在pythonapistdlib的ctypes模块中。:)您需要了解的最重要的技巧id(x)是实际指向x内存的指针(作为int)。

不幸的是,用于字符串的C API不能让我们安全地获取已经冻结的字符串的内部存储。因此,请放心,我们只需要阅读头文件并自己找到该存储即可。

如果您使用的是CPython 3.4-3.7(旧版本有所不同,谁知道未来),那么将使用紧凑ASCII格式存储由纯ASCII组成的模块中的字符串文字。提早结束,并且ASCII字节的缓冲区立即在内存中。如果您在字符串或某些非文字字符串中输入非ASCII字符,这将中断(可能在段错误中),但是您可以阅读其他4种方式来访问不同类型字符串的缓冲区。

为了使事情变得简单一些,我在superhackyinternalsGitHub上使用了该项目。(这是有意不可pip安装的,因为您真的不应该使用它,除非尝试在本地构建解释器等。)

import ctypes
import internals # https://github.com/abarnert/superhackyinternals/blob/master/internals.py

def print_function():
    print ("This cat was scared.")

def main():
    for c in print_function.__code__.co_consts:
        if isinstance(c, str):
            idx = c.find('cat')
            if idx != -1:
                # Too much to explain here; just guess and learn to
                # love the segfaults...
                p = internals.PyUnicodeObject.from_address(id(c))
                assert p.compact and p.ascii
                addr = id(c) + internals.PyUnicodeObject.utf8_length.offset
                buf = (ctypes.c_int8 * 3).from_address(addr + idx)
                buf[:3] = b'dog'

    print_function()

main()

如果您想玩这些东西,int则比起隐藏起来要简单得多str。而且,通过更改2to 的值来猜测可以破坏什么,容易得多1,对吗?实际上,忘记想象,让我们开始吧(superhackyinternals再次使用类型):

>>> n = 2
>>> pn = PyLongObject.from_address(id(n))
>>> pn.ob_digit[0]
2
>>> pn.ob_digit[0] = 1
>>> 2
1
>>> n * 3
3
>>> i = 10
>>> while i < 40:
...     i *= 2
...     print(i)
10
10
10

…假设代码框具有无限长的滚动条。

我在IPython中尝试过同样的事情,并且第一次尝试2在提示符下进行评估,它陷入了某种不间断的无限循环。大概2是在REPL循环中将数字用于某物,而股票解释器不是吗?

First, there’s actually a much less hacky way. All we want to do is change what print prints, right?

_print = print
def print(*args, **kw):
    args = (arg.replace('cat', 'dog') if isinstance(arg, str) else arg
            for arg in args)
    _print(*args, **kw)

Or, similarly, you can monkeypatch sys.stdout instead of print.


Also, nothing wrong with the exec … getsource … idea. Well, of course there’s plenty wrong with it, but less than what follows here…


But if you do want to modify the function object’s code constants, we can do that.

If you really want to play around with code objects for real, you should use a library like bytecode (when it’s finished) or byteplay (until then, or for older Python versions) instead of doing it manually. Even for something this trivial, the CodeType initializer is a pain; if you actually need to do stuff like fixing up lnotab, only a lunatic would do that manually.

Also, it goes without saying that not all Python implementations use CPython-style code objects. This code will work in CPython 3.7, and probably all versions back to at least 2.2 with a few minor changes (and not the code-hacking stuff, but things like generator expressions), but it won’t work with any version of IronPython.

import types

def print_function():
    print ("This cat was scared.")

def main():
    # A function object is a wrapper around a code object, with
    # a bit of extra stuff like default values and closure cells.
    # See inspect module docs for more details.
    co = print_function.__code__
    # A code object is a wrapper around a string of bytecode, with a
    # whole bunch of extra stuff, including a list of constants used
    # by that bytecode. Again see inspect module docs. Anyway, inside
    # the bytecode for string (which you can read by typing
    # dis.dis(string) in your REPL), there's going to be an
    # instruction like LOAD_CONST 1 to load the string literal onto
    # the stack to pass to the print function, and that works by just
    # reading co.co_consts[1]. So, that's what we want to change.
    consts = tuple(c.replace("cat", "dog") if isinstance(c, str) else c
                   for c in co.co_consts)
    # Unfortunately, code objects are immutable, so we have to create
    # a new one, copying over everything except for co_consts, which
    # we'll replace. And the initializer has a zillion parameters.
    # Try help(types.CodeType) at the REPL to see the whole list.
    co = types.CodeType(
        co.co_argcount, co.co_kwonlyargcount, co.co_nlocals,
        co.co_stacksize, co.co_flags, co.co_code,
        consts, co.co_names, co.co_varnames, co.co_filename,
        co.co_name, co.co_firstlineno, co.co_lnotab,
        co.co_freevars, co.co_cellvars)
    print_function.__code__ = co
    print_function()

main()

What could go wrong with hacking up code objects? Mostly just segfaults, RuntimeErrors that eat up the whole stack, more normal RuntimeErrors that can be handled, or garbage values that will probably just raise a TypeError or AttributeError when you try to use them. For examples, try creating a code object with just a RETURN_VALUE with nothing on the stack (bytecode b'S\0' for 3.6+, b'S' before), or with an empty tuple for co_consts when there’s a LOAD_CONST 0 in the bytecode, or with varnames decremented by 1 so the highest LOAD_FAST actually loads a freevar/cellvar cell. For some real fun, if you get the lnotab wrong enough, your code will only segfault when run in the debugger.

Using bytecode or byteplay won’t protect you from all of those problems, but they do have some basic sanity checks, and nice helpers that let you do things like insert a chunk of code and let it worry about updating all offsets and labels so you can’t get it wrong, and so on. (Plus, they keep you from having to type in that ridiculous 6-line constructor, and having to debug the silly typos that come from doing so.)


Now on to #2.

I mentioned that code objects are immutable. And of course the consts are a tuple, so we can’t change that directly. And the thing in the const tuple is a string, which we also can’t change directly. That’s why I had to build a new string to build a new tuple to build a new code object.

But what if you could change a string directly?

Well, deep enough under the covers, everything is just a pointer to some C data, right? If you’re using CPython, there’s a C API to access the objects, and you can use ctypes to access that API from within Python itself, which is such a terrible idea that they put a pythonapi right there in the stdlib’s ctypes module. :) The most important trick you need to know is that id(x) is the actual pointer to x in memory (as an int).

Unfortunately, the C API for strings won’t let us safely get at the internal storage of an already-frozen string. So screw safely, let’s just read the header files and find that storage ourselves.

If you’re using CPython 3.4 – 3.7 (it’s different for older versions, and who knows for the future), a string literal from a module that’s made of pure ASCII is going to be stored using the compact ASCII format, which means the struct ends early and the buffer of ASCII bytes follows immediately in memory. This will break (as in probably segfault) if you put a non-ASCII character in the string, or certain kinds of non-literal strings, but you can read up on the other 4 ways to access the buffer for different kinds of strings.

To make things slightly easier, I’m using the superhackyinternals project off my GitHub. (It’s intentionally not pip-installable because you really shouldn’t be using this except to experiment with your local build of the interpreter and the like.)

import ctypes
import internals # https://github.com/abarnert/superhackyinternals/blob/master/internals.py

def print_function():
    print ("This cat was scared.")

def main():
    for c in print_function.__code__.co_consts:
        if isinstance(c, str):
            idx = c.find('cat')
            if idx != -1:
                # Too much to explain here; just guess and learn to
                # love the segfaults...
                p = internals.PyUnicodeObject.from_address(id(c))
                assert p.compact and p.ascii
                addr = id(c) + internals.PyUnicodeObject.utf8_length.offset
                buf = (ctypes.c_int8 * 3).from_address(addr + idx)
                buf[:3] = b'dog'

    print_function()

main()

If you want to play with this stuff, int is a whole lot simpler under the covers than str. And it’s a lot easier to guess what you can break by changing the value of 2 to 1, right? Actually, forget imagining, let’s just do it (using the types from superhackyinternals again):

>>> n = 2
>>> pn = PyLongObject.from_address(id(n))
>>> pn.ob_digit[0]
2
>>> pn.ob_digit[0] = 1
>>> 2
1
>>> n * 3
3
>>> i = 10
>>> while i < 40:
...     i *= 2
...     print(i)
10
10
10

… pretend that code box has an infinite-length scrollbar.

I tried the same thing in IPython, and the first time I tried to evaluate 2 at the prompt, it went into some kind of uninterruptable infinite loop. Presumably it’s using the number 2 for something in its REPL loop, while the stock interpreter isn’t?


回答 1

Monkey补丁 print

print是一个内置函数,因此它将使用模块(或Python 2)中print定义的函数。因此,无论何时要修改或更改内置函数的行为,都可以在该模块中简单地重新分配名称。builtins__builtin__

此过程称为monkey-patching

# Store the real print function in another variable otherwise
# it will be inaccessible after being modified.
_print = print  

# Actual implementation of the new print
def custom_print(*args, **options):
    _print('custom print called')
    _print(*args, **options)

# Change the print function globally
import builtins
builtins.print = custom_print

之后,即使是在外部模块中,每个print调用也都将通过。custom_printprint

但是,您实际上并不想打印其他文本,而是要更改打印的文本。一种解决方法是将其替换为要打印的字符串:

_print = print  

def custom_print(*args, **options):
    # Get the desired seperator or the default whitspace
    sep = options.pop('sep', ' ')
    # Create the final string
    printed_string = sep.join(args)
    # Modify the final string
    printed_string = printed_string.replace('cat', 'dog')
    # Call the default print function
    _print(printed_string, **options)

import builtins
builtins.print = custom_print

实际上,如果您运行:

>>> def print_something():
...     print('This cat was scared.')
>>> print_something()
This dog was scared.

或者,如果您将其写入文件:

test_file.py

def print_something():
    print('This cat was scared.')

print_something()

并导入:

>>> import test_file
This dog was scared.
>>> test_file.print_something()
This dog was scared.

因此,它确实按预期工作。

但是,如果您只想临时打印Monkey补丁,可以将其包装在上下文管理器中:

import builtins

class ChangePrint(object):
    def __init__(self):
        self.old_print = print

    def __enter__(self):
        def custom_print(*args, **options):
            # Get the desired seperator or the default whitspace
            sep = options.pop('sep', ' ')
            # Create the final string
            printed_string = sep.join(args)
            # Modify the final string
            printed_string = printed_string.replace('cat', 'dog')
            # Call the default print function
            self.old_print(printed_string, **options)

        builtins.print = custom_print

    def __exit__(self, *args, **kwargs):
        builtins.print = self.old_print

因此,当您运行时,它取决于上下文,显示的内容是:

>>> with ChangePrint() as x:
...     test_file.print_something()
... 
This dog was scared.
>>> test_file.print_something()
This cat was scared.

这样便可以print通过Monkey补丁“破解” 。

修改目标,而不是 print

如果您看一下签名,print则会注意到默认情况下有一个file参数sys.stdout。请注意,这是一个动态默认参数(每次调用时都会真正查找),而不像Python中的普通默认参数。因此,如果您进行更改,则实际上将打印到其他目标会更加方便,因为Python还提供了一个功能(从Python 3.4开始,但是为早期的Python版本创建等效功能很容易)。sys.stdoutprintsys.stdout printredirect_stdout

缺点是它不适用于print不打印到的语句,sys.stdout并且创建自己的语句stdout并不是很简单。

import io
import sys

class CustomStdout(object):
    def __init__(self, *args, **kwargs):
        self.current_stdout = sys.stdout

    def write(self, string):
        self.current_stdout.write(string.replace('cat', 'dog'))

但是,这也可以:

>>> import contextlib
>>> with contextlib.redirect_stdout(CustomStdout()):
...     test_file.print_something()
... 
This dog was scared.
>>> test_file.print_something()
This cat was scared.

摘要

@abarnet已经提到了其中一些观点,但是我想更详细地探讨这些选项。特别是如何跨模块(使用builtins/ __builtin__)修改它,以及如何仅临时更改(使用contextmanagers)。

Monkey-patch print

print is a builtin function so it will use the print function defined in the builtins module (or __builtin__ in Python 2). So whenever you want to modify or change the behavior of a builtin function you can simply reassign the name in that module.

This process is called monkey-patching.

# Store the real print function in another variable otherwise
# it will be inaccessible after being modified.
_print = print  

# Actual implementation of the new print
def custom_print(*args, **options):
    _print('custom print called')
    _print(*args, **options)

# Change the print function globally
import builtins
builtins.print = custom_print

After that every print call will go through custom_print, even if the print is in an external module.

However you don’t really want to print additional text, you want to change the text that is printed. One way to go about that is to replace it in the string that would be printed:

_print = print  

def custom_print(*args, **options):
    # Get the desired seperator or the default whitspace
    sep = options.pop('sep', ' ')
    # Create the final string
    printed_string = sep.join(args)
    # Modify the final string
    printed_string = printed_string.replace('cat', 'dog')
    # Call the default print function
    _print(printed_string, **options)

import builtins
builtins.print = custom_print

And indeed if you run:

>>> def print_something():
...     print('This cat was scared.')
>>> print_something()
This dog was scared.

Or if you write that to a file:

test_file.py

def print_something():
    print('This cat was scared.')

print_something()

and import it:

>>> import test_file
This dog was scared.
>>> test_file.print_something()
This dog was scared.

So it really works as intended.

However, in case you only temporarily want to monkey-patch print you could wrap this in a context-manager:

import builtins

class ChangePrint(object):
    def __init__(self):
        self.old_print = print

    def __enter__(self):
        def custom_print(*args, **options):
            # Get the desired seperator or the default whitspace
            sep = options.pop('sep', ' ')
            # Create the final string
            printed_string = sep.join(args)
            # Modify the final string
            printed_string = printed_string.replace('cat', 'dog')
            # Call the default print function
            self.old_print(printed_string, **options)

        builtins.print = custom_print

    def __exit__(self, *args, **kwargs):
        builtins.print = self.old_print

So when you run that it depends on the context what is printed:

>>> with ChangePrint() as x:
...     test_file.print_something()
... 
This dog was scared.
>>> test_file.print_something()
This cat was scared.

So that’s how you could “hack” print by monkey-patching.

Modify the target instead of the print

If you look at the signature of print you’ll notice a file argument which is sys.stdout by default. Note that this is a dynamic default argument (it really looks up sys.stdout every time you call print) and not like normal default arguments in Python. So if you change sys.stdout print will actually print to the different target even more convenient that Python also provides a redirect_stdout function (from Python 3.4 on, but it’s easy to create an equivalent function for earlier Python versions).

The downside is that it won’t work for print statements that don’t print to sys.stdout and that creating your own stdout isn’t really straightforward.

import io
import sys

class CustomStdout(object):
    def __init__(self, *args, **kwargs):
        self.current_stdout = sys.stdout

    def write(self, string):
        self.current_stdout.write(string.replace('cat', 'dog'))

However this also works:

>>> import contextlib
>>> with contextlib.redirect_stdout(CustomStdout()):
...     test_file.print_something()
... 
This dog was scared.
>>> test_file.print_something()
This cat was scared.

Summary

Some of these points have already be mentioned by @abarnet but I wanted to explore these options in more detail. Especially how to modify it across modules (using builtins/__builtin__) and how to make that change only temporary (using contextmanagers).


回答 2

print函数捕获所有输出然后对其进行处理的一种简单方法是将输出流更改为其他内容,例如文件。

我将使用PHP命名约定(ob_startob_get_contents,…)

from functools import partial
output_buffer = None
print_orig = print
def ob_start(fname="print.txt"):
    global print
    global output_buffer
    print = partial(print_orig, file=output_buffer)
    output_buffer = open(fname, 'w')
def ob_end():
    global output_buffer
    close(output_buffer)
    print = print_orig
def ob_get_contents(fname="print.txt"):
    return open(fname, 'r').read()

用法:

print ("Hi John")
ob_start()
print ("Hi John")
ob_end()
print (ob_get_contents().replace("Hi", "Bye"))

将打印

嗨约翰再见约翰

A simple way to capture all output from a print function and then process it, is to change the output stream to something else, e.g. a file.

I’ll use a PHP naming conventions (ob_start, ob_get_contents,…)

from functools import partial
output_buffer = None
print_orig = print
def ob_start(fname="print.txt"):
    global print
    global output_buffer
    print = partial(print_orig, file=output_buffer)
    output_buffer = open(fname, 'w')
def ob_end():
    global output_buffer
    close(output_buffer)
    print = print_orig
def ob_get_contents(fname="print.txt"):
    return open(fname, 'r').read()

Usage:

print ("Hi John")
ob_start()
print ("Hi John")
ob_end()
print (ob_get_contents().replace("Hi", "Bye"))

Would print

Hi John Bye John


回答 3

让我们将其与帧自省结合起来!

import sys

_print = print

def print(*args, **kw):
    frame = sys._getframe(1)
    _print(frame.f_code.co_name)
    _print(*args, **kw)

def greetly(name, greeting = "Hi")
    print(f"{greeting}, {name}!")

class Greeter:
    def __init__(self, greeting = "Hi"):
        self.greeting = greeting
    def greet(self, name):
        print(f"{self.greeting}, {name}!")

您会发现此技巧在调用函数或方法的每个问候语前都有序。这对于日志记录或调试可能非常有用;特别是因为它可以让您“劫持”第三方代码中的打印语句。

Let’s combine this with frame introspection!

import sys

_print = print

def print(*args, **kw):
    frame = sys._getframe(1)
    _print(frame.f_code.co_name)
    _print(*args, **kw)

def greetly(name, greeting = "Hi")
    print(f"{greeting}, {name}!")

class Greeter:
    def __init__(self, greeting = "Hi"):
        self.greeting = greeting
    def greet(self, name):
        print(f"{self.greeting}, {name}!")

You’ll find this trick prefaces every greeting with the calling function or method. This might be very useful for logging or debugging; especially as it lets you “hijack” print statements in third party code.