Python 实用宝典

Question 1

Python recognizes the following as instruction which defines file’s encoding:

# -*- coding: utf-8 -*-

I definitely saw this kind of instructions before (-*- var: value -*-). Where does it come from? What is the full specification, e.g. can the value include spaces, special symbols, newlines, even -*- itself?

My program will be writing plain text files and I’d like to include some metadata in them using this format.

Question 2

This way of specifying the encoding of a Python file comes from PEP 0263 – Defining Python Source Code Encodings.

It is also recognized by GNU Emacs (see Python Language Reference, 2.1.4 Encoding declarations), though I don’t know if it was the first program to use that syntax.

Question 3

# -*- coding: utf-8 -*- is a Python 2 thing. In Python 3+, the default encoding of source files is already UTF-8 and that line is useless.

See: Should I use encoding declaration in Python 3?

pyupgrade is a tool you can run on your code to remove those comments and other no-longer-useful leftovers from Python 2, like having all your classes inherit from object.

Question 4

This is so called file local variables, that are understood by Emacs and set correspondingly. See corresponding section in Emacs manual – you can define them either in header or in footer of file

Question 5

In PyCharm, I’d leave it out. It turns off the UTF-8 indicator at the bottom with a warning that the encoding is hard-coded. Don’t think you need the PyCharm comment mentioned above.

Question 6

I’m still learning python and I have a doubt:

In python 2.6.x I usually declare encoding in the file header like this (as in PEP 0263)

# -*- coding: utf-8 -*-

After that, my strings are written as usual:

a = "A normal string without declared Unicode"

But everytime I see a python project code, the encoding is not declared at the header. Instead, it is declared at every string like this:

a = u"A string with declared Unicode"

What’s the difference? What’s the purpose of this? I know Python 2.6.x sets ASCII encoding by default, but it can be overriden by the header declaration, so what’s the point of per string declaration?

Addendum: Seems that I’ve mixed up file encoding with string encoding. Thanks for explaining it :)

Question 7

Those are two different things, as others have mentioned.

When you specify # -*- coding: utf-8 -*-, you’re telling Python the source file you’ve saved is utf-8. The default for Python 2 is ASCII (for Python 3 it’s utf-8). This just affects how the interpreter reads the characters in the file.

In general, it’s probably not the best idea to embed high unicode characters into your file no matter what the encoding is; you can use string unicode escapes, which work in either encoding.

When you declare a string with a u in front, like u'This is a string', it tells the Python compiler that the string is Unicode, not bytes. This is handled mostly transparently by the interpreter; the most obvious difference is that you can now embed unicode characters in the string (that is, u'\u2665' is now legal). You can use from __future__ import unicode_literals to make it the default.

This only applies to Python 2; in Python 3 the default is Unicode, and you need to specify a b in front (like b'These are bytes', to declare a sequence of bytes).

Question 8

As others have said, # coding: specifies the encoding the source file is saved in. Here are some examples to illustrate this:

A file saved on disk as cp437 (my console encoding), but no encoding declared

b = 'über'
u = u'über'
print b,repr(b)
print u,repr(u)

Output:

  File "C:\ex.py", line 1
SyntaxError: Non-ASCII character '\x81' in file C:\ex.py on line 1, but no
encoding declared; see http://www.python.org/peps/pep-0263.html for details

Output of file with # coding: cp437 added:

über '\x81ber'
über u'\xfcber'

At first, Python didn’t know the encoding and complained about the non-ASCII character. Once it knew the encoding, the byte string got the bytes that were actually on disk. For the Unicode string, Python read \x81, knew that in cp437 that was a ü, and decoded it into the Unicode codepoint for ü which is U+00FC. When the byte string was printed, Python sent the hex value 81 to the console directly. When the Unicode string was printed, Python correctly detected my console encoding as cp437 and translated Unicode ü to the cp437 value for ü.

Here’s what happens with a file declared and saved in UTF-8:

├╝ber '\xc3\xbcber'
über u'\xfcber'

In UTF-8, ü is encoded as the hex bytes C3 BC, so the byte string contains those bytes, but the Unicode string is identical to the first example. Python read the two bytes and decoded it correctly. Python printed the byte string incorrectly, because it sent the two UTF-8 bytes representing ü directly to my cp437 console.

Here the file is declared cp437, but saved in UTF-8:

├╝ber '\xc3\xbcber'
├╝ber u'\u251c\u255dber'

The byte string still got the bytes on disk (UTF-8 hex bytes C3 BC), but interpreted them as two cp437 characters instead of a single UTF-8-encoded character. Those two characters where translated to Unicode code points, and everything prints incorrectly.

Question 9

That doesn’t set the format of the string; it sets the format of the file. Even with that header, "hello" is a byte string, not a Unicode string. To make it Unicode, you’re going to have to use u"hello" everywhere. The header is just a hint of what format to use when reading the .py file.

Question 10

The header definition is to define the encoding of the code itself, not the resulting strings at runtime.

putting a non-ascii character like ۲ in the python script without the utf-8 header definition will throw a warning

Question 11

I made the following module called unicoder to be able to do the transformation on variables:

import sys
import os

def ustr(string):

    string = 'u"%s"'%string

    with open('_unicoder.py', 'w') as script:

        script.write('# -*- coding: utf-8 -*-\n')
        script.write('_ustr = %s'%string)

    import _unicoder
    value = _unicoder._ustr

    del _unicoder
    del sys.modules['_unicoder']

    os.system('del _unicoder.py')
    os.system('del _unicoder.pyc')

    return value

Then in your program you could do the following:

# -*- coding: utf-8 -*-

from unicoder import ustr

txt = 'Hello, Unicode World'
txt = ustr(txt)

print type(txt) # <type 'unicode'>

Question 12

Python 3 uses UTF-8 encoding for source-code files by default. Should I still use the encoding declaration at the beginning of every source file? Like # -*- coding: utf-8 -*-

Question 13

Because the default is UTF-8, you only need to use that declaration when you deviate from the default, or if you rely on other tools (like your IDE or text editor) to make use of that information.

In other words, as far as Python is concerned, only when you want to use an encoding that differs do you have to use that declaration.

Other tools, such as your editor, can support similar syntax, which is why the PEP 263 specification allows for considerable flexibility in the syntax (it must be a comment, the text coding must be there, followed by either a : or = character and optional whitespace, followed by a recognised codec).

Note that it only applies to how Python reads the source code. It doesn’t apply to executing that code, so not to how printing, opening files, or any other I/O operations translate between bytes and Unicode. For more details on Python, Unicode, and encodings, I strongly urge you to read the Python Unicode HOWTO, or the very thorough Pragmatic Unicode talk by Ned Batchelder.

Question 14

No, if:

entire project use only the UTF-8, which is a default.
and you’re sure your IDE tool doesn’t need that encoding declaration in each file.

Yes, if

your project relies on different encoding
or relies on many encodings.

For multi-encodings projects:

If some files are encoded in the non-utf-8, then even for these encoded in UTF-8 you should add encoding declaration too, because the golden rule is Explicit is better than implicit.

Reference:

PyCharm doesn’t need that declaration:

configuring encoding for specific file in pycharm

vim doesn’t need that declaration, but:

# vim: set fileencoding=<encoding name> :

Question 15

In a text file, there is a string “I don’t like this”.

However, when I read it into a string, it becomes “I don\xe2\x80\x98t like this”. I understand that \u2018 is the unicode representation of “‘”. I use

f1 = open (file1, "r")
text = f1.read()

command to do the reading.

Now, is it possible to read the string in such a way that when it is read into the string, it is “I don’t like this”, instead of “I don\xe2\x80\x98t like this like this”?

Second edit: I have seen some people use mapping to solve this problem, but really, is there no built-in conversion that does this kind of ANSI to unicode ( and vice versa) conversion?

Question 16

Ref: http://docs.python.org/howto/unicode

Reading Unicode from a file is therefore simple:

import codecs
with codecs.open('unicode.rst', encoding='utf-8') as f:
    for line in f:
        print repr(line)

It’s also possible to open files in update mode, allowing both reading and writing:

with codecs.open('test', encoding='utf-8', mode='w+') as f:
    f.write(u'\u4500 blah blah blah\n')
    f.seek(0)
    print repr(f.readline()[:1])

EDIT: I’m assuming that your intended goal is just to be able to read the file properly into a string in Python. If you’re trying to convert to an ASCII string from Unicode, then there’s really no direct way to do so, since the Unicode characters won’t necessarily exist in ASCII.

If you’re trying to convert to an ASCII string, try one of the following:

Replace the specific unicode chars with ASCII equivalents, if you are only looking to handle a few special cases such as this particular example
Use the unicodedata module’s normalize() and the string.encode() method to convert as best you can to the next closest ASCII equivalent (Ref https://web.archive.org/web/20090228203858/http://techxplorer.com/2006/07/18/converting-unicode-to-ascii-using-python):
```
>>> teststr
u'I don\xe2\x80\x98t like this'
>>> unicodedata.normalize('NFKD', teststr).encode('ascii', 'ignore')
'I donat like this'
```

Question 17

There are a few points to consider.

A \u2018 character may appear only as a fragment of representation of a unicode string in Python, e.g. if you write:

>>> text = u'‘'
>>> print repr(text)
u'\u2018'

Now if you simply want to print the unicode string prettily, just use unicode’s encode method:

>>> text = u'I don\u2018t like this'
>>> print text.encode('utf-8')
I don‘t like this

To make sure that every line from any file would be read as unicode, you’d better use the codecs.open function instead of just open, which allows you to specify file’s encoding:

>>> import codecs
>>> f1 = codecs.open(file1, "r", "utf-8")
>>> text = f1.read()
>>> print type(text)
<type 'unicode'>
>>> print text.encode('utf-8')
I don‘t like this

Question 18

But it really is “I don\u2018t like this” and not “I don’t like this”. The character u’\u2018′ is a completely different character than “‘” (and, visually, should correspond more to ‘`’).

If you’re trying to convert encoded unicode into plain ASCII, you could perhaps keep a mapping of unicode punctuation that you would like to translate into ASCII.

punctuation = {
  u'\u2018': "'",
  u'\u2019': "'",
}
for src, dest in punctuation.iteritems():
  text = text.replace(src, dest)

There are an awful lot of punctuation characters in unicode, however, but I suppose you can count on only a few of them actually being used by whatever application is creating the documents you’re reading.

Question 19

It is also possible to read an encoded text file using the python 3 read method:

f = open (file.txt, 'r', encoding='utf-8')
text = f.read()
f.close()

With this variation, there is no need to import any additional libraries

Question 20

Leaving aside the fact that your text file is broken (U+2018 is a left quotation mark, not an apostrophe): iconv can be used to transliterate unicode characters to ascii.

You’ll have to google for “iconvcodec”, since the module seems not to be supported anymore and I can’t find a canonical home page for it.

>>> import iconvcodec
>>> from locale import setlocale, LC_ALL
>>> setlocale(LC_ALL, '')
>>> u'\u2018'.encode('ascii//translit')
"'"

Alternatively you can use the iconv command line utility to clean up your file:

$ xxd foo
0000000: e280 980a                                ....
$ iconv -t 'ascii//translit' foo | xxd
0000000: 270a                                     '.

Question 21

There is a possibility that somehow you have a non-unicode string with unicode escape characters, e.g.:

>>> print repr(text)
'I don\\u2018t like this'

This actually happened to me once before. You can use a unicode_escape codec to decode the string to unicode and then encode it to any format you want:

>>> uni = text.decode('unicode_escape')
>>> print type(uni)
<type 'unicode'>
>>> print uni.encode('utf-8')
I don‘t like this

Question 22

This is Pythons way do show you unicode encoded strings. But i think you should be able to print the string on the screen or write it into a new file without any problems.

>>> test = u"I don\u2018t like this"
>>> test
u'I don\u2018t like this'
>>> print test
I don‘t like this

Question 23

Actually, U+2018 is the Unicode representation of the special character ‘ . If you want, you can convert instances of that character to U+0027 with this code:

text = text.replace (u"\u2018", "'")

In addition, what are you using to write the file? f1.read() should return a string that looks like this:

'I don\xe2\x80\x98t like this'

If it’s returning this string, the file is being written incorrectly:

'I don\u2018t like this'

Question 24

I have seen few py scripts which use this at the top of the script. In what cases one should use it?

import sys
reload(sys)
sys.setdefaultencoding("utf-8")

Question 25

As per the documentation: This allows you to switch from the default ASCII to other encodings such as UTF-8, which the Python runtime will use whenever it has to decode a string buffer to unicode.

This function is only available at Python start-up time, when Python scans the environment. It has to be called in a system-wide module, sitecustomize.py, After this module has been evaluated, the setdefaultencoding() function is removed from the sys module.

The only way to actually use it is with a reload hack that brings the attribute back.

Also, the use of sys.setdefaultencoding() has always been discouraged, and it has become a no-op in py3k. The encoding of py3k is hard-wired to “utf-8” and changing it raises an error.

I suggest some pointers for reading:

Question 26

tl;dr

The answer is NEVER! _{(unless you really know what you’re doing)}

9/10 times the solution can be resolved with a proper understanding of encoding/decoding.

1/10 people have an incorrectly defined locale or environment and need to set:

PYTHONIOENCODING="UTF-8"

in their environment to fix console printing problems.

What does it do?

~~sys.setdefaultencoding("utf-8")~~ (struck through to avoid re-use) changes the default encoding/decoding used whenever Python 2.x needs to convert a Unicode() to a str() (and vice-versa) and the encoding is not given. I.e:

str(u"\u20AC")
unicode("€")
"{}".format(u"\u20AC")

In Python 2.x, the default encoding is set to ASCII and the above examples will fail with:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)

(My console is configured as UTF-8, so "€" = '\xe2\x82\xac', hence exception on \xe2)

or

UnicodeEncodeError: 'ascii' codec can't encode character u'\u20ac' in position 0: ordinal not in range(128)

~~sys.setdefaultencoding("utf-8")~~ will allow these to work for me, but won’t necessarily work for people who don’t use UTF-8. The default of ASCII ensures that assumptions of encoding are not baked into code

Console

~~sys.setdefaultencoding("utf-8")~~ also has a side effect of appearing to fix sys.stdout.encoding, used when printing characters to the console. Python uses the user’s locale (Linux/OS X/Un*x) or codepage (Windows) to set this. Occasionally, a user’s locale is broken and just requires PYTHONIOENCODING to fix the console encoding.

Example:

$ export LANG=en_GB.gibberish
$ python
>>> import sys
>>> sys.stdout.encoding
'ANSI_X3.4-1968'
>>> print u"\u20AC"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\u20ac' in position 0: ordinal not in range(128)
>>> exit()

$ PYTHONIOENCODING=UTF-8 python
>>> import sys
>>> sys.stdout.encoding
'UTF-8'
>>> print u"\u20AC"
€

What’s so bad with sys.setdefaultencoding(“utf-8”)?

People have been developing against Python 2.x for 16 years on the understanding that the default encoding is ASCII. UnicodeError exception handling methods have been written to handle string to Unicode conversions on strings that are found to contain non-ASCII.

From https://anonbadger.wordpress.com/2015/06/16/why-sys-setdefaultencoding-will-break-code/

def welcome_message(byte_string):
    try:
        return u"%s runs your business" % byte_string
    except UnicodeError:
        return u"%s runs your business" % unicode(byte_string,
            encoding=detect_encoding(byte_string))

print(welcome_message(u"Angstrom (Å®)".encode("latin-1"))

Previous to setting defaultencoding this code would be unable to decode the “Å” in the ascii encoding and then would enter the exception handler to guess the encoding and properly turn it into unicode. Printing: Angstrom (Å®) runs your business. Once you’ve set the defaultencoding to utf-8 the code will find that the byte_string can be interpreted as utf-8 and so it will mangle the data and return this instead: Angstrom (Ů) runs your business.

Changing what should be a constant will have dramatic effects on modules you depend upon. It’s better to just fix the data coming in and out of your code.

Example problem

While the setting of defaultencoding to UTF-8 isn’t the root cause in the following example, it shows how problems are masked and how, when the input encoding changes, the code breaks in an unobvious way: UnicodeDecodeError: ‘utf8’ codec can’t decode byte 0x80 in position 3131: invalid start byte

Question 27

#!/usr/bin/env python
#-*- coding: utf-8 -*-
u = u'moçambique'
print u.encode("utf-8")
print u

chmod +x test.py
./test.py
moçambique
moçambique

./test.py > output.txt
Traceback (most recent call last):
  File "./test.py", line 5, in <module>
    print u
UnicodeEncodeError: 'ascii' codec can't encode character 
u'\xe7' in position 2: ordinal not in range(128)

on shell works , sending to sdtout not , so that is one workaround, to write to stdout .

I made other approach, which is not run if sys.stdout.encoding is not define, or in others words , need export PYTHONIOENCODING=UTF-8 first to write to stdout.

import sys
if (sys.stdout.encoding is None):            
    print >> sys.stderr, "please set python env PYTHONIOENCODING=UTF-8, example: export PYTHONIOENCODING=UTF-8, when write to stdout." 
    exit(1)

so, using same example:

export PYTHONIOENCODING=UTF-8
./test.py > output.txt

will work

Question 28

The first danger lies in reload(sys).

When you reload a module, you actually get two copies of the module in your runtime. The old module is a Python object like everything else, and stays alive as long as there are references to it. So, half of the objects will be pointing to the old module, and half to the new one. When you make some change, you will never see it coming when some random object doesn’t see the change:
```
(This is IPython shell)

In [1]: import sys

In [2]: sys.stdout
Out[2]: <colorama.ansitowin32.StreamWrapper at 0x3a2aac8>

In [3]: reload(sys)
<module 'sys' (built-in)>

In [4]: sys.stdout
Out[4]: <open file '<stdout>', mode 'w' at 0x00000000022E20C0>

In [11]: import IPython.terminal

In [14]: IPython.terminal.interactiveshell.sys.stdout
Out[14]: <colorama.ansitowin32.StreamWrapper at 0x3a9aac8>
```
Now, sys.setdefaultencoding() proper

All that it affects is implicit conversion str<->unicode. Now, utf-8 is the sanest encoding on the planet (backward-compatible with ASCII and all), the conversion now “just works”, what could possibly go wrong?

Well, anything. And that is the danger.
- There may be some code that relies on the UnicodeError being thrown for non-ASCII input, or does the transcoding with an error handler, which now produces an unexpected result. And since all code is tested with the default setting, you’re strictly on “unsupported” territory here, and no-one gives you guarantees about how their code will behave.
- The transcoding may produce unexpected or unusable results if not everything on the system uses UTF-8 because Python 2 actually has multiple independent “default string encodings”. (Remember, a program must work for the customer, on the customer’s equipment.)
  - Again, the worst thing is you will never know that because the conversion is implicit — you don’t really know when and where it happens. (Python Zen, koan 2 ahoy!) You will never know why (and if) your code works on one system and breaks on another. (Or better yet, works in IDE and breaks in console.)

Question 29

PEP 263 defines how to declare Python source code encoding.

Normally, the first 2 lines of a Python file should start with:

#!/usr/bin/python
# -*- coding: <encoding name> -*-

But I have seen a lot of files starting with:

#!/usr/bin/python
# -*- encoding: <encoding name> -*-

=> encoding instead of coding.

So what is the correct way of declaring the file encoding?

Is encoding permitted because the regex used is lazy? Or is it just another form of declaring the file encoding?

I’m asking this question because the PEP does not talk about encoding, it just talks about coding.

Question 30

Check the docs here:

“If a comment in the first or second line of the Python script matches the regular expression coding[=:]\s*([-\w.]+), this comment is processed as an encoding declaration”

“The recommended forms of this expression are

# -*- coding: <encoding-name> -*-

which is recognized also by GNU Emacs, and

# vim:fileencoding=<encoding-name>

which is recognized by Bram Moolenaar’s VIM.”

So, you can put pretty much anything before the “coding” part, but stick to “coding” (with no prefix) if you want to be 100% python-docs-recommendation-compatible.

More specifically, you need to use whatever is recognized by Python and the specific editing software you use (if it needs/accepts anything at all). E.g. the coding form is recognized (out of the box) by GNU Emacs but not Vim (yes, without a universal agreement, it’s essentially a turf war).

Question 31

PEP 263:

the first or second line must match the regular expression “coding[:=]\s*([-\w.]+)”

So, “encoding: UTF-8” matches.

PEP provides some examples:

#!/usr/bin/python
# vim: set fileencoding=<encoding name> :

# This Python file uses the following encoding: utf-8
import os, sys

Question 32

Just copy paste below statement on the top of your program.It will solve character encoding problems

#!/usr/bin/env python
# -*- coding: utf-8 -*-

Question 33

As of today — June 2018

PEP 263 itself mentions the regex it follows:

To define a source code encoding, a magic comment must be placed into the source files either as first or second line in the file, such as:
# coding=<encoding name>
or (using formats recognized by popular editors):
#!/usr/bin/python
# -*- coding: <encoding name> -*-
or:
#!/usr/bin/python
# vim: set fileencoding=<encoding name> : 
More precisely, the first or second line must match the following regular expression:
^[ \t\f]*#.*?coding[:=][ \t]*([-_.a-zA-Z0-9]+)

So, as already summed up by other answers, it’ll match coding with any prefix, but if you’d like to be as PEP-compliant as it gets (even though, as far as I can tell, using encoding instead of coding does not violate PEP 263 in any way) — stick with ‘plain’ coding, with no prefixes.

Question 34

If I’m not mistaken, the original proposal for source file encodings was to use a regular expression for the first couple of lines, which would allow both.

I think the regex was something along the lines of coding: followed by something.

I found this: http://www.python.org/dev/peps/pep-0263/ Which is the original proposal, but I can’t seem to find the final spec stating exactly what they did.

I’ve certainly used encoding: to great effect, so obviously that works.

Try changing to something completely different, like duhcoding: ... to see if that works just as well.

Question 35

I suspect it is similar to Ruby – either method is okay.

This is largely because different text editors use different methods (ie, these two) of marking encoding.

With Ruby, as long as the first, or second if there is a shebang line contains a string that matches:

coding: encoding-name

and ignoring any whitespace and other fluff on those lines. (It can often be a = instead of :, too).

Question 36

I am trying to send a simple dictionary to a json file from python, but I keep getting the “TypeError: 1425 is not JSON serializable” message.

import json
alerts = {'upper':[1425],'lower':[576],'level':[2],'datetime':['2012-08-08 15:30']}
afile = open('test.json','w')
afile.write(json.dumps(alerts,encoding='UTF-8'))
afile.close()

If I add the default argument, then it writes, but the integer values are written to the json file as strings, which is undesirable.

afile.write(json.dumps(alerts,encoding='UTF-8',default=str))

Question 37

I found my problem. The issue was that my integers were actually type numpy.int64.

Question 38

It seems like there may be a issue to dump numpy.int64 into json string in Python 3 and the python team already have a conversation about it. More details can be found here.

There is a workaround provided by Serhiy Storchaka. It works very well so I paste it here:

def convert(o):
    if isinstance(o, numpy.int64): return int(o)  
    raise TypeError

json.dumps({'value': numpy.int64(42)}, default=convert)

Question 39

This solved the problem for me:

def serialize(self):
    return {
        my_int: int(self.my_int), 
        my_float: float(self.my_float)
    }

Question 40

Just convert numbers from `int64` (from numpy) to `int`.

For example, if variable x is a int64:

int(x)

If is array of int64:

map(int, x)

Question 41

as @JAC pointed out in the comments of the highest rated answer, the generic solution (for all numpy types) can be found in the thread Converting numpy dtypes to native python types.

Nevertheless, I´ll add my version of the solution below, as my in my case I needed a generic solution that combines these answers and with the answers of the other thread. This should work with almost all numpy types.

def convert(o):
    if isinstance(o, np.generic): return o.item()  
    raise TypeError

json.dumps({'value': numpy.int64(42)}, default=convert)

Question 42

This might be the late response, but recently i got the same error. After lot of surfing this solution helped me.

alerts = {'upper':[1425],'lower':[576],'level':[2],'datetime':['2012-08-08 15:30']}
def myconverter(obj):
        if isinstance(obj, np.integer):
            return int(obj)
        elif isinstance(obj, np.floating):
            return float(obj)
        elif isinstance(obj, np.ndarray):
            return obj.tolist()
        elif isinstance(obj, datetime.datetime):
            return obj.__str__()

Call myconverter in json.dumps() like below. json.dumps(alerts, default=myconverter).

Question 43

Alternatively you can convert your object into a dataframe first:

df = pd.DataFrame(obj)

and then save this dataframe in a json file:

df.to_json(path_or_buf='df.json')

Hope this helps

Question 44

You have Numpy Data Type, Just change to normal int() or float() data type. it will work fine.

Question 45

Same problem. List contained numbers of type numpy.int64 which throws a TypeError. Quick workaround for me was to

mylist = eval(str(mylist_of_integers))
json.dumps({'mylist': mylist})

which converts list to str() and eval() function evaluates the String like a Python expression and returns the result as a list of integers in my case.

Question 46

Use the below code to resolve the issue.

import json
from numpyencoder import NumpyEncoder
alerts = {'upper':[1425],'lower':[576],'level':[2],'datetime':['2012-08-08 
15:30']}
afile = open('test.json','w')
afile.write(json.dumps(alerts,encoding='UTF-8',cls=NumpyEncoder))
afile.close()

Question 47

I found that in 123, \d matches 1 and 3 but not 2. I was wondering if \d matches a digit satisfying what kind of requirement? I am talking about Python style regex.

Regular expression plugin in Gedit is using Python style regex. I created a text file with its content being

Only 1 and 3 are matched by the regex \d; 2 is not.

Generally for a sequence of digit numbers without other characters in between, only the odd order digits are matches, and the even order digits are not. For example in 12345, the matches are 1, 3 and 5.

Question 48

[0-9] is not always equivalent to \d. In python3, [0-9] matches only 0123456789 characters, while \d matches [0-9] and other digit characters, for example Eastern Arabic numerals ٠١٢٣٤٥٦٧٨٩.

Question 49

\d matches any single digit in most regex grammar styles, including python. Regex Reference

Question 50

In Python-style regex, \d matches any individual digit. If you’re seeing something that doesn’t seem to do that, please provide the full regex you’re using, as opposed to just describing that one particular symbol.

>>> import re
>>> re.match(r'\d', '3')
<_sre.SRE_Match object at 0x02155B80>
>>> re.match(r'\d', '2')
<_sre.SRE_Match object at 0x02155BB8>
>>> re.match(r'\d', '1')
<_sre.SRE_Match object at 0x02155B80>

Question 51

\\d{3} matches any sequence of three digits in Java.

Question 52

This is just a guess, but I think your editor actually matches every single digit — 1 2 3 — but only odd matches are highlighted, to distinguish it from the case when the whole 123 string is matched.

Most regex consoles highlight contiguous matches with different colors, but due to the plugin settings, terminal limitations or for some other reason, only every other group might be highlighted in your case.

Question 53

Info regarding .NET / C#:

Decimal digit character: \d \d matches any decimal digit. It is equivalent to the \p{Nd} regular expression pattern, which includes the standard decimal digits 0-9 as well as the decimal digits of a number of other character sets.

If ECMAScript-compliant behavior is specified, \d is equivalent to [0-9]. For information on ECMAScript regular expressions, see the “ECMAScript Matching Behavior” section in Regular Expression Options.

Info: https://docs.microsoft.com/en-us/dotnet/standard/base-types/character-classes-in-regular-expressions#decimal-digit-character-d

Question 54

I have many “can’t encode” and “can’t decode” problems with Python when I run my applications from the console. But in the Eclipse PyDev IDE, the default character encoding is set to UTF-8, and I’m fine.

I searched around for setting the default encoding, and people say that Python deletes the sys.setdefaultencoding function on startup, and we can not use it.

So what’s the best solution for it?

Question 55

Here is a simpler method (hack) that gives you back the setdefaultencoding() function that was deleted from sys:

import sys
# sys.setdefaultencoding() does not exist, here!
reload(sys)  # Reload does the trick!
sys.setdefaultencoding('UTF8')

(Note for Python 3.4+: reload() is in the importlib library.)

This is not a safe thing to do, though: this is obviously a hack, since sys.setdefaultencoding() is purposely removed from sys when Python starts. Reenabling it and changing the default encoding can break code that relies on ASCII being the default (this code can be third-party, which would generally make fixing it impossible or dangerous).

Question 56

If you get this error when you try to pipe/redirect output of your script

UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-5: ordinal not in range(128)

Just export PYTHONIOENCODING in console and then run your code.

export PYTHONIOENCODING=utf8

Question 57

A) To control sys.getdefaultencoding() output:

python -c 'import sys; print(sys.getdefaultencoding())'

ascii

Then

echo "import sys; sys.setdefaultencoding('utf-16-be')" > sitecustomize.py

and

PYTHONPATH=".:$PYTHONPATH" python -c 'import sys; print(sys.getdefaultencoding())'

utf-16-be

You could put your sitecustomize.py higher in your PYTHONPATH.

Also you might like to try reload(sys).setdefaultencoding by @EOL

B) To control stdin.encoding and stdout.encoding you want to set PYTHONIOENCODING:

python -c 'import sys; print(sys.stdin.encoding, sys.stdout.encoding)'

ascii ascii

Then

PYTHONIOENCODING="utf-16-be" python -c 'import sys; 
print(sys.stdin.encoding, sys.stdout.encoding)'

utf-16-be utf-16-be

Finally: you can use A) or B) or both!

Question 58

Starting with PyDev 3.4.1, the default encoding is not being changed anymore. See this ticket for details.

For earlier versions a solution is to make sure PyDev does not run with UTF-8 as the default encoding. Under Eclipse, run dialog settings (“run configurations”, if I remember correctly); you can choose the default encoding on the common tab. Change it to US-ASCII if you want to have these errors ‘early’ (in other words: in your PyDev environment). Also see an original blog post for this workaround.

Question 59

Regarding python2 (and python2 only), some of the former answers rely on using the following hack:

import sys
reload(sys)  # Reload is a hack
sys.setdefaultencoding('UTF8')

It is discouraged to use it (check this or this)

In my case, it come with a side-effect: I’m using ipython notebooks, and once I run the code the ´print´ function no longer works. I guess there would be solution to it, but still I think using the hack should not be the correct option.

After trying many options, the one that worked for me was using the same code in the sitecustomize.py, where that piece of code is meant to be. After evaluating that module, the setdefaultencoding function is removed from sys.

So the solution is to append to file /usr/lib/python2.7/sitecustomize.py the code:

import sys
sys.setdefaultencoding('UTF8')

When I use virtualenvwrapper the file I edit is ~/.virtualenvs/venv-name/lib/python2.7/sitecustomize.py.

And when I use with python notebooks and conda, it is ~/anaconda2/lib/python2.7/sitecustomize.py

Question 60

There is an insightful blog post about it.

See https://anonbadger.wordpress.com/2015/06/16/why-sys-setdefaultencoding-will-break-code/.

I paraphrase its content below.

In python 2 which was not as strongly typed regarding the encoding of strings you could perform operations on differently encoded strings, and succeed. E.g. the following would return True.

u'Toshio' == 'Toshio'

That would hold for every (normal, unprefixed) string that was encoded in sys.getdefaultencoding(), which defaulted to ascii, but not others.

The default encoding was meant to be changed system-wide in site.py, but not somewhere else. The hacks (also presented here) to set it in user modules were just that: hacks, not the solution.

Python 3 did changed the system encoding to default to utf-8 (when LC_CTYPE is unicode-aware), but the fundamental problem was solved with the requirement to explicitly encode “byte”strings whenever they are used with unicode strings.

Question 61

First: reload(sys) and setting some random default encoding just regarding the need of an output terminal stream is bad practice. reload often changes things in sys which have been put in place depending on the environment – e.g. sys.stdin/stdout streams, sys.excepthook, etc.

Solving the encode problem on stdout

The best solution I know for solving the encode problem of print‘ing unicode strings and beyond-ascii str‘s (e.g. from literals) on sys.stdout is: to take care of a sys.stdout (file-like object) which is capable and optionally tolerant regarding the needs:

When sys.stdout.encoding is None for some reason, or non-existing, or erroneously false or “less” than what the stdout terminal or stream really is capable of, then try to provide a correct .encoding attribute. At last by replacing sys.stdout & sys.stderr by a translating file-like object.
When the terminal / stream still cannot encode all occurring unicode chars, and when you don’t want to break print‘s just because of that, you can introduce an encode-with-replace behavior in the translating file-like object.

Here an example:

#!/usr/bin/env python
# encoding: utf-8
import sys

class SmartStdout:
    def __init__(self, encoding=None, org_stdout=None):
        if org_stdout is None:
            org_stdout = getattr(sys.stdout, 'org_stdout', sys.stdout)
        self.org_stdout = org_stdout
        self.encoding = encoding or \
                        getattr(org_stdout, 'encoding', None) or 'utf-8'
    def write(self, s):
        self.org_stdout.write(s.encode(self.encoding, 'backslashreplace'))
    def __getattr__(self, name):
        return getattr(self.org_stdout, name)

if __name__ == '__main__':
    if sys.stdout.isatty():
        sys.stdout = sys.stderr = SmartStdout()

    us = u'aouäöüфżß²'
    print us
    sys.stdout.flush()

Using beyond-ascii plain string literals in Python 2 / 2 + 3 code

The only good reason to change the global default encoding (to UTF-8 only) I think is regarding an application source code decision – and not because of I/O stream encodings issues: For writing beyond-ascii string literals into code without being forced to always use u'string' style unicode escaping. This can be done rather consistently (despite what anonbadger‘s article says) by taking care of a Python 2 or Python 2 + 3 source code basis which uses ascii or UTF-8 plain string literals consistently – as far as those strings potentially undergo silent unicode conversion and move between modules or potentially go to stdout. For that, prefer “# encoding: utf-8” or ascii (no declaration). Change or drop libraries which still rely in a very dumb way fatally on ascii default encoding errors beyond chr #127 (which is rare today).

And do like this at application start (and/or via sitecustomize.py) in addition to the SmartStdout scheme above – without using reload(sys):

...
def set_defaultencoding_globally(encoding='utf-8'):
    assert sys.getdefaultencoding() in ('ascii', 'mbcs', encoding)
    import imp
    _sys_org = imp.load_dynamic('_sys_org', 'sys')
    _sys_org.setdefaultencoding(encoding)

if __name__ == '__main__':
    sys.stdout = sys.stderr = SmartStdout()
    set_defaultencoding_globally('utf-8') 
    s = 'aouäöüфżß²'
    print s

This way string literals and most operations (except character iteration) work comfortable without thinking about unicode conversion as if there would be Python3 only. File I/O of course always need special care regarding encodings – as it is in Python3.

Note: plains strings then are implicitely converted from utf-8 to unicode in SmartStdout before being converted to the output stream enconding.

Question 62

Here is the approach I used to produce code that was compatible with both python2 and python3 and always produced utf8 output. I found this answer elsewhere, but I can’t remember the source.

This approach works by replacing sys.stdout with something that isn’t quite file-like (but still only using things in the standard library). This may well cause problems for your underlying libraries, but in the simple case where you have good control over how sys.stdout out is used through your framework this can be a reasonable approach.

sys.stdout = io.open(sys.stdout.fileno(), 'w', encoding='utf8')

Question 63

This fixed the issue for me.

import os
os.environ["PYTHONIOENCODING"] = "utf-8"

Question 64

This is a quick hack for anyone who is (1) On a Windows platform (2) running Python 2.7 and (3) annoyed because a nice piece of software (i.e., not written by you so not immediately a candidate for encode/decode printing maneuvers) won’t display the “pretty unicode characters” in the IDLE environment (Pythonwin prints unicode fine), For example, the neat First Order Logic symbols that Stephan Boyer uses in the output from his pedagogic prover at First Order Logic Prover.

I didn’t like the idea of forcing a sys reload and I couldn’t get the system to cooperate with setting environment variables like PYTHONIOENCODING (tried direct Windows environment variable and also dropping that in a sitecustomize.py in site-packages as a one liner =’utf-8′).

So, if you are willing to hack your way to success, go to your IDLE directory, typically: “C:\Python27\Lib\idlelib” Locate the file IOBinding.py. Make a copy of that file and store it somewhere else so you can revert to original behavior when you choose. Open the file in the idlelib with an editor (e.g., IDLE). Go to this code area:

# Encoding for file names
filesystemencoding = sys.getfilesystemencoding()

encoding = "ascii"
if sys.platform == 'win32':
    # On Windows, we could use "mbcs". However, to give the user
    # a portable encoding name, we need to find the code page 
    try:
        # --> 6/5/17 hack to force IDLE to display utf-8 rather than cp1252
        # --> encoding = locale.getdefaultlocale()[1]
        encoding = 'utf-8'
        codecs.lookup(encoding)
    except LookupError:
        pass

In other words, comment out the original code line following the ‘try‘ that was making the encoding variable equal to locale.getdefaultlocale (because that will give you cp1252 which you don’t want) and instead brute force it to ‘utf-8’ (by adding the line ‘encoding = ‘utf-8‘ as shown).

I believe this only affects IDLE display to stdout and not the encoding used for file names etc. (that is obtained in the filesystemencoding prior). If you have a problem with any other code you run in IDLE later, just replace the IOBinding.py file with the original unmodified file.

问题：这是从哪里来的：-*-编码：utf-8-*-

回答 0

回答 1

回答 2

回答 3

问题：为什么要在python中通过字符串声明unicode？

回答 0

回答 1

回答 2

回答 3

回答 4

问题：我应该在Python 3中使用编码声明吗？

回答 0

回答 1

不，如果：

是的，如果

参考：

No, if:

Yes, if

Reference:

问题：用Python从文件中读取字符

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

回答 7

问题：为什么我们不应该在py脚本中使用sys.setdefaultencoding（“ utf-8”）？

回答 0

回答 1

tl; dr

它有什么作用？

安慰

sys.setdefaultencoding（“ utf-8”）有什么不好？

示例问题

tl;dr

What does it do?

Console

What’s so bad with sys.setdefaultencoding(“utf-8”)?

Example problem

回答 2

回答 3

问题：定义Python源代码编码的正确方法

回答 0

回答 1

回答 2

回答 3

截至今天-2018年6月

As of today — June 2018

回答 4

回答 5

问题：在Python中序列化JSON时，“ TypeError ：（整数）不可JSON序列化”？

回答 0

回答 1

回答 2

回答 3

只需将数字从int64（从numpy）转换为int。

Just convert numbers from int64 (from numpy) to int.

回答 4

回答 5

回答 6

回答 7

回答 8

回答 9

问题：正则表达式中的“ \ d”表示数字吗？

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

问题：更改Python的默认编码？

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

问题：这是从哪里来的：--编码：utf-8--

只需将数字从`int64`（从numpy）转换为`int`。

Just convert numbers from `int64` (from numpy) to `int`.