在Python中处理字符串中的转义序列

问题:在Python中处理字符串中的转义序列

有时,当我从文件或用户那里得到输入时,我会得到一个带有转义序列的字符串。我想以与Python处理字符串文字中的转义序列相同的方式来处理转义序列

例如,假设myString定义为:

>>> myString = "spam\\neggs"
>>> print(myString)
spam\neggs

我想要一个process执行此操作的函数(我称之为):

>>> print(process(myString))
spam
eggs

该函数可以处理Python中的所有转义序列(在上面的链接的表格中列出),这一点很重要。

Python是否具有执行此操作的功能?

Sometimes when I get input from a file or the user, I get a string with escape sequences in it. I would like to process the escape sequences in the same way that Python processes escape sequences in string literals.

For example, let’s say myString is defined as:

>>> myString = "spam\\neggs"
>>> print(myString)
spam\neggs

I want a function (I’ll call it process) that does this:

>>> print(process(myString))
spam
eggs

It’s important that the function can process all of the escape sequences in Python (listed in a table in the link above).

Does Python have a function to do this?


回答 0

正确的做法是使用“字符串转义”代码对字符串进行解码。

>>> myString = "spam\\neggs"
>>> decoded_string = bytes(myString, "utf-8").decode("unicode_escape") # python3 
>>> decoded_string = myString.decode('string_escape') # python2
>>> print(decoded_string)
spam
eggs

不要使用AST或eval。使用字符串编解码器更加安全。

The correct thing to do is use the ‘string-escape’ code to decode the string.

>>> myString = "spam\\neggs"
>>> decoded_string = bytes(myString, "utf-8").decode("unicode_escape") # python3 
>>> decoded_string = myString.decode('string_escape') # python2
>>> print(decoded_string)
spam
eggs

Don’t use the AST or eval. Using the string codecs is much safer.


回答 1

unicode_escape 总的来说不起作用

事实证明,string_escapeor unicode_escape解决方案通常无法正常工作-尤其是在存在实际Unicode的情况下,它不能正常工作。

如果您可以确定每个非ASCII字符都会被转义(并且请记住,前128个字符以外的任何字符都是非ASCII),unicode_escape将为您做正确的事。但是,如果您的字符串中已经有任何文字上的非ASCII字符,则会出错。

unicode_escape从根本上来说是设计用来将字节转换为Unicode文本。但是在许多地方(例如Python源代码),源数据已经是Unicode文本。

唯一可以正常工作的方法是首先将文本编码为字节。UTF-8是所有文本的明智编码,因此应该可以使用,对吧?

以下示例是Python 3中的示例,因此字符串文字更清晰,但在Python 2和3上,存在相同的问题,但表现形式略有不同。

>>> s = 'naïve \\t test'
>>> print(s.encode('utf-8').decode('unicode_escape'))
naïve   test

好吧,那是错误的。

建议使用编解码器将文本解码为文本的新方法是codecs.decode直接调用。有帮助吗?

>>> import codecs
>>> print(codecs.decode(s, 'unicode_escape'))
naïve   test

一点也不。(此外,以上是Python 2上的UnicodeError。)

unicode_escape编解码器,尽管它的名字,原来假设所有非ASCII字节拉丁-1(ISO-8859-1)编码。因此,您必须这样做:

>>> print(s.encode('latin-1').decode('unicode_escape'))
naïve    test

但这太可怕了。这将您限制为256个Latin-1字符,就好像根本没有发明Unicode一样!

>>> print('Ernő \\t Rubik'.encode('latin-1').decode('unicode_escape'))
UnicodeEncodeError: 'latin-1' codec can't encode character '\u0151'
in position 3: ordinal not in range(256)

添加正则表达式以解决问题

(令人惊讶的是,我们现在没有两个问题。)

我们需要做的只是将unicode_escape解码器应用于我们确定为ASCII文本的内容。特别是,我们可以确保仅将其应用于有效的Python转义序列,这些序列必须保证为ASCII文本。

计划是,我们将使用正则表达式查找转义序列,并使用函数作为参数以re.sub将其替换为未转义的值。

import re
import codecs

ESCAPE_SEQUENCE_RE = re.compile(r'''
    ( \\U........      # 8-digit hex escapes
    | \\u....          # 4-digit hex escapes
    | \\x..            # 2-digit hex escapes
    | \\[0-7]{1,3}     # Octal escapes
    | \\N\{[^}]+\}     # Unicode characters by name
    | \\[\\'"abfnrtv]  # Single-character escapes
    )''', re.UNICODE | re.VERBOSE)

def decode_escapes(s):
    def decode_match(match):
        return codecs.decode(match.group(0), 'unicode-escape')

    return ESCAPE_SEQUENCE_RE.sub(decode_match, s)

然后:

>>> print(decode_escapes('Ernő \\t Rubik'))
Ernő     Rubik

unicode_escape doesn’t work in general

It turns out that the string_escape or unicode_escape solution does not work in general — particularly, it doesn’t work in the presence of actual Unicode.

If you can be sure that every non-ASCII character will be escaped (and remember, anything beyond the first 128 characters is non-ASCII), unicode_escape will do the right thing for you. But if there are any literal non-ASCII characters already in your string, things will go wrong.

unicode_escape is fundamentally designed to convert bytes into Unicode text. But in many places — for example, Python source code — the source data is already Unicode text.

The only way this can work correctly is if you encode the text into bytes first. UTF-8 is the sensible encoding for all text, so that should work, right?

The following examples are in Python 3, so that the string literals are cleaner, but the same problem exists with slightly different manifestations on both Python 2 and 3.

>>> s = 'naïve \\t test'
>>> print(s.encode('utf-8').decode('unicode_escape'))
naïve   test

Well, that’s wrong.

The new recommended way to use codecs that decode text into text is to call codecs.decode directly. Does that help?

>>> import codecs
>>> print(codecs.decode(s, 'unicode_escape'))
naïve   test

Not at all. (Also, the above is a UnicodeError on Python 2.)

The unicode_escape codec, despite its name, turns out to assume that all non-ASCII bytes are in the Latin-1 (ISO-8859-1) encoding. So you would have to do it like this:

>>> print(s.encode('latin-1').decode('unicode_escape'))
naïve    test

But that’s terrible. This limits you to the 256 Latin-1 characters, as if Unicode had never been invented at all!

>>> print('Ernő \\t Rubik'.encode('latin-1').decode('unicode_escape'))
UnicodeEncodeError: 'latin-1' codec can't encode character '\u0151'
in position 3: ordinal not in range(256)

Adding a regular expression to solve the problem

(Surprisingly, we do not now have two problems.)

What we need to do is only apply the unicode_escape decoder to things that we are certain to be ASCII text. In particular, we can make sure only to apply it to valid Python escape sequences, which are guaranteed to be ASCII text.

The plan is, we’ll find escape sequences using a regular expression, and use a function as the argument to re.sub to replace them with their unescaped value.

import re
import codecs

ESCAPE_SEQUENCE_RE = re.compile(r'''
    ( \\U........      # 8-digit hex escapes
    | \\u....          # 4-digit hex escapes
    | \\x..            # 2-digit hex escapes
    | \\[0-7]{1,3}     # Octal escapes
    | \\N\{[^}]+\}     # Unicode characters by name
    | \\[\\'"abfnrtv]  # Single-character escapes
    )''', re.UNICODE | re.VERBOSE)

def decode_escapes(s):
    def decode_match(match):
        return codecs.decode(match.group(0), 'unicode-escape')

    return ESCAPE_SEQUENCE_RE.sub(decode_match, s)

And with that:

>>> print(decode_escapes('Ernő \\t Rubik'))
Ernő     Rubik

回答 2

python 3的实际正确答案:

>>> import codecs
>>> myString = "spam\\neggs"
>>> print(codecs.escape_decode(bytes(myString, "utf-8"))[0].decode("utf-8"))
spam
eggs
>>> myString = "naïve \\t test"
>>> print(codecs.escape_decode(bytes(myString, "utf-8"))[0].decode("utf-8"))
naïve    test

有关的详细信息codecs.escape_decode

  • codecs.escape_decode 是一个逐字节解码器
  • codecs.escape_decode解码ascii转义序列,例如:b"\\n"-> b"\n"b"\\xce"-> b"\xce"
  • codecs.escape_decode 不需要或不需要了解字节对象的编码,但是转义字节的编码应与对象其余部分的编码匹配。

背景:

The actually correct and convenient answer for python 3:

>>> import codecs
>>> myString = "spam\\neggs"
>>> print(codecs.escape_decode(bytes(myString, "utf-8"))[0].decode("utf-8"))
spam
eggs
>>> myString = "naïve \\t test"
>>> print(codecs.escape_decode(bytes(myString, "utf-8"))[0].decode("utf-8"))
naïve    test

Details regarding codecs.escape_decode:

  • codecs.escape_decode is a bytes-to-bytes decoder
  • codecs.escape_decode decodes ascii escape sequences, such as: b"\\n" -> b"\n", b"\\xce" -> b"\xce".
  • codecs.escape_decode does not care or need to know about the byte object’s encoding, but the encoding of the escaped bytes should match the encoding of the rest of the object.

Background:

  • @rspeer is correct: unicode_escape is the incorrect solution for python3. This is because unicode_escape decodes escaped bytes, then decodes bytes to unicode string, but receives no information regarding which codec to use for the second operation.
  • @Jerub is correct: avoid the AST or eval.
  • I first discovered codecs.escape_decode from this answer to “how do I .decode(‘string-escape’) in Python3?”. As that answer states, that function is currently not documented for python 3.

回答 3

ast.literal_eval函数将关闭,但是它将期望该字符串先被正确引用。

当然反斜杠Python的解释依赖于字符串的方式引用(""VS r""VS u"",三引号等),所以你可能想包装在合适的报价的用户输入和传递给literal_eval。将其包装在引号中还可以防止literal_eval返回数字,元组,字典等。

如果用户键入您打算在字符串周围使用的引号引起来,事情可能仍然会变得棘手。

The ast.literal_eval function comes close, but it will expect the string to be properly quoted first.

Of course Python’s interpretation of backslash escapes depends on how the string is quoted ("" vs r"" vs u"", triple quotes, etc) so you may want to wrap the user input in suitable quotes and pass to literal_eval. Wrapping it in quotes will also prevent literal_eval from returning a number, tuple, dictionary, etc.

Things still might get tricky if the user types unquoted quotes of the type you intend to wrap around the string.


回答 4

这是一个不好的方法,但是当我尝试解释在字符串参数中传递的转义八进制时,它对我有用。

input_string = eval('b"' + sys.argv[1] + '"')

值得一提的是,eval和ast.literal_eval之间存在区别(eval更加不安全)。请参阅使用python的eval()与ast.literal_eval()吗?

This is a bad way of doing it, but it worked for me when trying to interpret escaped octals passed in a string argument.

input_string = eval('b"' + sys.argv[1] + '"')

It’s worth mentioning that there is a difference between eval and ast.literal_eval (eval being way more unsafe). See Using python’s eval() vs. ast.literal_eval()?


回答 5

下面的代码应该适用于\ n,要求将其显示在字符串上。

import string

our_str = 'The String is \\n, \\n and \\n!'
new_str = string.replace(our_str, '/\\n', '/\n', 1)
print(new_str)

Below code should work for \n is required to be displayed on the string.

import string

our_str = 'The String is \\n, \\n and \\n!'
new_str = string.replace(our_str, '/\\n', '/\n', 1)
print(new_str)