Sometimes when I get input from a file or the user, I get a string with escape sequences in it. I would like to process the escape sequences in the same way that Python processes escape sequences in string literals.

For example, let’s say myString is defined as:

>>> myString = "spam\\neggs"
>>> print(myString)
spam\neggs

I want a function (I’ll call it process) that does this:

>>> print(process(myString))
spam
eggs

It’s important that the function can process all of the escape sequences in Python (listed in a table in the link above).

Does Python have a function to do this?

The correct thing to do is use the ‘string-escape’ code to decode the string.

>>> myString = "spam\\neggs"
>>> decoded_string = bytes(myString, "utf-8").decode("unicode_escape") # python3 
>>> decoded_string = myString.decode('string_escape') # python2
>>> print(decoded_string)
spam
eggs

Don’t use the AST or eval. Using the string codecs is much safer.

unicode_escape doesn’t work in general

It turns out that the string_escape or unicode_escape solution does not work in general — particularly, it doesn’t work in the presence of actual Unicode.

If you can be sure that every non-ASCII character will be escaped (and remember, anything beyond the first 128 characters is non-ASCII), unicode_escape will do the right thing for you. But if there are any literal non-ASCII characters already in your string, things will go wrong.

unicode_escape is fundamentally designed to convert bytes into Unicode text. But in many places — for example, Python source code — the source data is already Unicode text.

The only way this can work correctly is if you encode the text into bytes first. UTF-8 is the sensible encoding for all text, so that should work, right?

The following examples are in Python 3, so that the string literals are cleaner, but the same problem exists with slightly different manifestations on both Python 2 and 3.

>>> s = 'naïve \\t test'
>>> print(s.encode('utf-8').decode('unicode_escape'))
naïve   test

Well, that’s wrong.

The new recommended way to use codecs that decode text into text is to call codecs.decode directly. Does that help?

>>> import codecs
>>> print(codecs.decode(s, 'unicode_escape'))
naïve   test

Not at all. (Also, the above is a UnicodeError on Python 2.)

The unicode_escape codec, despite its name, turns out to assume that all non-ASCII bytes are in the Latin-1 (ISO-8859-1) encoding. So you would have to do it like this:

>>> print(s.encode('latin-1').decode('unicode_escape'))
naïve    test

But that’s terrible. This limits you to the 256 Latin-1 characters, as if Unicode had never been invented at all!

>>> print('Ernő \\t Rubik'.encode('latin-1').decode('unicode_escape'))
UnicodeEncodeError: 'latin-1' codec can't encode character '\u0151'
in position 3: ordinal not in range(256)

Adding a regular expression to solve the problem

(Surprisingly, we do not now have two problems.)

What we need to do is only apply the unicode_escape decoder to things that we are certain to be ASCII text. In particular, we can make sure only to apply it to valid Python escape sequences, which are guaranteed to be ASCII text.

The plan is, we’ll find escape sequences using a regular expression, and use a function as the argument to re.sub to replace them with their unescaped value.

import re
import codecs

ESCAPE_SEQUENCE_RE = re.compile(r'''
    ( \\U........      # 8-digit hex escapes
    | \\u....          # 4-digit hex escapes
    | \\x..            # 2-digit hex escapes
    | \\[0-7]{1,3}     # Octal escapes
    | \\N\{[^}]+\}     # Unicode characters by name
    | \\[\\'"abfnrtv]  # Single-character escapes
    )''', re.UNICODE | re.VERBOSE)

def decode_escapes(s):
    def decode_match(match):
        return codecs.decode(match.group(0), 'unicode-escape')

    return ESCAPE_SEQUENCE_RE.sub(decode_match, s)

And with that:

>>> print(decode_escapes('Ernő \\t Rubik'))
Ernő     Rubik

The actually correct and convenient answer for python 3:

>>> import codecs
>>> myString = "spam\\neggs"
>>> print(codecs.escape_decode(bytes(myString, "utf-8"))[0].decode("utf-8"))
spam
eggs
>>> myString = "naïve \\t test"
>>> print(codecs.escape_decode(bytes(myString, "utf-8"))[0].decode("utf-8"))
naïve    test

Details regarding codecs.escape_decode:

• codecs.escape_decode is a bytes-to-bytes decoder
• codecs.escape_decode decodes ascii escape sequences, such as: b"\\n" -> b"\n", b"\\xce" -> b"\xce".
• codecs.escape_decode does not care or need to know about the byte object’s encoding, but the encoding of the escaped bytes should match the encoding of the rest of the object.

Background:

• @rspeer is correct: unicode_escape is the incorrect solution for python3. This is because unicode_escape decodes escaped bytes, then decodes bytes to unicode string, but receives no information regarding which codec to use for the second operation.
• @Jerub is correct: avoid the AST or eval.
• I first discovered codecs.escape_decode from this answer to “how do I .decode(‘string-escape’) in Python3?”. As that answer states, that function is currently not documented for python 3.

The ast.literal_eval function comes close, but it will expect the string to be properly quoted first.

Of course Python’s interpretation of backslash escapes depends on how the string is quoted ("" vs r"" vs u"", triple quotes, etc) so you may want to wrap the user input in suitable quotes and pass to literal_eval. Wrapping it in quotes will also prevent literal_eval from returning a number, tuple, dictionary, etc.

Things still might get tricky if the user types unquoted quotes of the type you intend to wrap around the string.

This is a bad way of doing it, but it worked for me when trying to interpret escaped octals passed in a string argument.

input_string = eval('b"' + sys.argv[1] + '"')

It’s worth mentioning that there is a difference between eval and ast.literal_eval (eval being way more unsafe). See Using python’s eval() vs. ast.literal_eval()?

Below code should work for \n is required to be displayed on the string.

import string

our_str = 'The String is \\n, \\n and \\n!'
new_str = string.replace(our_str, '/\\n', '/\n', 1)
print(new_str)

Is there a way to declare a string variable in python such that everything inside of it is automatically escaped, or has its literal character value?

I’m not asking how to escape the quotes with slashes, that’s obvious. What I’m asking for is a general purpose way for making everything in a string literal so that I don’t have to manually go through and escape everything for very large strings. Anyone know of a solution? Thanks!

Raw string literals:

>>> r'abc\dev\t'
'abc\\dev\\t'

If you’re dealing with very large strings, specifically multiline strings, be aware of the triple-quote syntax:

a = r"""This is a multiline string
with more than one line
in the source code."""

There is no such thing. It looks like you want something like “here documents” in Perl and the shells, but Python doesn’t have that.

Using raw strings or multiline strings only means that there are fewer things to worry about. If you use a raw string then you still have to work around a terminal “\” and with any string solution you’ll have to worry about the closing “, ‘, ”’ or “”” if it is included in your data.

That is, there’s no way to have the string

 '   ''' """  " \

properly stored in any Python string literal without internal escaping of some sort.

You will find Python’s string literal documentation here:

http://docs.python.org/tutorial/introduction.html#strings

and here:

http://docs.python.org/reference/lexical_analysis.html#literals

The simplest example would be using the ‘r’ prefix:

ss = r'Hello\nWorld'
print(ss)
Hello\nWorld

(Assuming you are not required to input the string from directly within Python code)

to get around the Issue Andrew Dalke pointed out, simply type the literal string into a text file and then use this;

input_ = '/directory_of_text_file/your_text_file.txt' 
input_open   = open(input_,'r+')
input_string = input_open.read()

print input_string

This will print the literal text of whatever is in the text file, even if it is;

 '   ''' """  “ \

Not fun or optimal, but can be useful, especially if you have 3 pages of code that would’ve needed character escaping.

if string is a variable, use the .repr method on it:

>>> s = '\tgherkin\n'

>>> s
'\tgherkin\n'

>>> print(s)
    gherkin

>>> print(s.__repr__())
'\tgherkin\n'

Suppose I have a string which is a backslash-escaped version of another string. Is there an easy way, in Python, to unescape the string? I could, for example, do:

>>> escaped_str = '"Hello,\\nworld!"'
>>> raw_str = eval(escaped_str)
>>> print raw_str
Hello,
world!
>>> 

However that involves passing a (possibly untrusted) string to eval() which is a security risk. Is there a function in the standard lib which takes a string and produces a string with no security implications?

>>> print '"Hello,\\nworld!"'.decode('string_escape')
"Hello,
world!"

You can use ast.literal_eval which is safe:

Safely evaluate an expression node or a string containing a Python expression. The string or node provided may only consist of the following Python literal structures: strings, numbers, tuples, lists, dicts, booleans, and None. (END)

Like this:

>>> import ast
>>> escaped_str = '"Hello,\\nworld!"'
>>> print ast.literal_eval(escaped_str)
Hello,
world!

All given answers will break on general Unicode strings. The following works for Python3 in all cases, as far as I can tell:

from codecs import encode, decode
sample = u'mon€y\\nröcks'
result = decode(encode(sample, 'latin-1', 'backslashreplace'), 'unicode-escape')
print(result)

As outlined in the comments, you can also use the literal_eval method from the ast module like so:

import ast
sample = u'mon€y\\nröcks'
print(ast.literal_eval(F'"{sample}"'))

Or like this when your string really contains a string literal (including the quotes):

import ast
sample = u'"mon€y\\nröcks"'
print(ast.literal_eval(sample))

However, if you are uncertain whether the input string uses double or single quotes as delimiters, or when you cannot assume it to be properly escaped at all, then literal_eval may raise a SyntaxError while the encode/decode method will still work.

In python 3, str objects don’t have a decode method and you have to use a bytes object. ChristopheD’s answer covers python 2.

# create a `bytes` object from a `str`
my_str = "Hello,\\nworld"
# (pick an encoding suitable for your str, e.g. 'latin1')
my_bytes = my_str.encode("utf-8")

# or directly
my_bytes = b"Hello,\\nworld"

print(my_bytes.decode("unicode_escape"))
# "Hello,
# world"