Python 实用宝典

Question 1

What does sys.stdout.flush() do?

Question 2

Python’s standard out is buffered (meaning that it collects some of the data “written” to standard out before it writes it to the terminal). Calling sys.stdout.flush() forces it to “flush” the buffer, meaning that it will write everything in the buffer to the terminal, even if normally it would wait before doing so.

Here’s some good information about (un)buffered I/O and why it’s useful:
http://en.wikipedia.org/wiki/Data_buffer
Buffered vs unbuffered IO

Question 3

Consider the following simple Python script:

import time
import sys

for i in range(5):
    print(i),
    #sys.stdout.flush()
    time.sleep(1)

This is designed to print one number every second for five seconds, but if you run it as it is now (depending on your default system buffering) you may not see any output until the script completes, and then all at once you will see 0 1 2 3 4 printed to the screen.

This is because the output is being buffered, and unless you flush sys.stdout after each print you won’t see the output immediately. Remove the comment from the sys.stdout.flush() line to see the difference.

Question 4

As per my understanding, When ever we execute print statements output will be written to buffer. And we will see the output on screen when buffer get flushed(cleared). By default buffer will be flushed when program exits. BUT WE CAN ALSO FLUSH THE BUFFER MANUALLY by using “sys.stdout.flush()” statement in the program. In the below code buffer will be flushed when value of i reaches 5.

You can understand by executing the below code.

chiru@online:~$ cat flush.py

import time
import sys

for i in range(10):
    print i
    if i == 5:
        print "Flushing buffer"
        sys.stdout.flush()
    time.sleep(1)

for i in range(10):
    print i,
    if i == 5:
        print "Flushing buffer"
        sys.stdout.flush()

chiru@online:~$ python flush.py 
0 1 2 3 4 5 Flushing buffer
6 7 8 9 0 1 2 3 4 5 Flushing buffer
6 7 8 9

Question 5

import sys
for x in range(10000):
    print "HAPPY >> %s <<\r" % str(x),
    sys.stdout.flush()

Question 6

As per my understanding sys.stdout.flush() pushes out all the data that has been buffered to that point to a file object. While using stdout, data is stored in buffer memory (for some time or until the memory gets filled) before it gets written to terminal. Using flush() forces to empty the buffer and write to terminal even before buffer has empty space.

Question 7

Where is Python’s sys.path initialized from?

UPD: Python is adding some paths before refering to PYTHONPATH:

    >>> import sys
    >>> from pprint import pprint as p
    >>> p(sys.path)
    ['',
     'C:\\Python25\\lib\\site-packages\\setuptools-0.6c9-py2.5.egg',
     'C:\\Python25\\lib\\site-packages\\orbited-0.7.8-py2.5.egg',
     'C:\\Python25\\lib\\site-packages\\morbid-0.8.6.1-py2.5.egg',
     'C:\\Python25\\lib\\site-packages\\demjson-1.4-py2.5.egg',
     'C:\\Python25\\lib\\site-packages\\stomper-0.2.2-py2.5.egg',
     'C:\\Python25\\lib\\site-packages\\uuid-1.30-py2.5.egg',
     'C:\\Python25\\lib\\site-packages\\stompservice-0.1.0-py2.5.egg',
     'C:\\Python25\\lib\\site-packages\\cherrypy-3.0.1-py2.5.egg',
     'C:\\Python25\\lib\\site-packages\\pyorbited-0.2.2-py2.5.egg',
     'C:\\Python25\\lib\\site-packages\\flup-1.0.1-py2.5.egg',
     'C:\\Python25\\lib\\site-packages\\wsgilog-0.1-py2.5.egg',
     'c:\\testdir',
     'C:\\Windows\\system32\\python25.zip',
     'C:\\Python25\\DLLs',
     'C:\\Python25\\lib',
     'C:\\Python25\\lib\\plat-win',
     'C:\\Python25\\lib\\lib-tk',
     'C:\\Python25',
     'C:\\Python25\\lib\\site-packages',
     'C:\\Python25\\lib\\site-packages\\PIL',
     'C:\\Python25\\lib\\site-packages\\win32',
     'C:\\Python25\\lib\\site-packages\\win32\\lib',
     'C:\\Python25\\lib\\site-packages\\Pythonwin']

My PYTHONPATH is:

    PYTHONPATH=c:\testdir

I wonder where those paths before PYTHONPATH’s ones come from?

Question 8

“Initialized from the environment variable PYTHONPATH, plus an installation-dependent default”

— http://docs.python.org/library/sys.html#sys.path

Question 9

Python really tries hard to intelligently set sys.path. How it is set can get really complicated. The following guide is a watered-down, somewhat-incomplete, somewhat-wrong, but hopefully-useful guide for the rank-and-file python programmer of what happens when python figures out what to use as the initial values of sys.path, sys.executable, sys.exec_prefix, and sys.prefix on a normal python installation.

First, python does its level best to figure out its actual physical location on the filesystem based on what the operating system tells it. If the OS just says “python” is running, it finds itself in $PATH. It resolves any symbolic links. Once it has done this, the path of the executable that it finds is used as the value for sys.executable, no ifs, ands, or buts.

Next, it determines the initial values for sys.exec_prefix and sys.prefix.

If there is a file called pyvenv.cfg in the same directory as sys.executable or one directory up, python looks at it. Different OSes do different things with this file.

One of the values in this config file that python looks for is the configuration option home = <DIRECTORY>. Python will use this directory instead of the directory containing sys.executable when it dynamically sets the initial value of sys.prefix later. If the applocal = true setting appears in the pyvenv.cfg file on Windows, but not the home = <DIRECTORY> setting, then sys.prefix will be set to the directory containing sys.executable.

Next, the PYTHONHOME environment variable is examined. On Linux and Mac, sys.prefix and sys.exec_prefix are set to the PYTHONHOME environment variable, if it exists, superseding any home = <DIRECTORY> setting in pyvenv.cfg. On Windows, sys.prefix and sys.exec_prefix is set to the PYTHONHOME environment variable, if it exists, unless a home = <DIRECTORY> setting is present in pyvenv.cfg, which is used instead.

Otherwise, these sys.prefix and sys.exec_prefix are found by walking backwards from the location of sys.executable, or the home directory given by pyvenv.cfg if any.

If the file lib/python<version>/dyn-load is found in that directory or any of its parent directories, that directory is set to be to be sys.exec_prefix on Linux or Mac. If the file lib/python<version>/os.py is is found in the directory or any of its subdirectories, that directory is set to be sys.prefix on Linux, Mac, and Windows, with sys.exec_prefix set to the same value as sys.prefix on Windows. This entire step is skipped on Windows if applocal = true is set. Either the directory of sys.executable is used or, if home is set in pyvenv.cfg, that is used instead for the initial value of sys.prefix.

If it can’t find these “landmark” files or sys.prefix hasn’t been found yet, then python sets sys.prefix to a “fallback” value. Linux and Mac, for example, use pre-compiled defaults as the values of sys.prefix and sys.exec_prefix. Windows waits until sys.path is fully figured out to set a fallback value for sys.prefix.

Then, (what you’ve all been waiting for,) python determines the initial values that are to be contained in sys.path.

The directory of the script which python is executing is added to sys.path. On Windows, this is always the empty string, which tells python to use the full path where the script is located instead.
The contents of PYTHONPATH environment variable, if set, is added to sys.path, unless you’re on Windows and applocal is set to true in pyvenv.cfg.
The zip file path, which is <prefix>/lib/python35.zip on Linux/Mac and os.path.join(os.dirname(sys.executable), "python.zip") on Windows, is added to sys.path.
If on Windows and no applocal = true was set in pyvenv.cfg, then the contents of the subkeys of the registry key HK_CURRENT_USER\Software\Python\PythonCore\<DLLVersion>\PythonPath\ are added, if any.
If on Windows and no applocal = true was set in pyvenv.cfg, and sys.prefix could not be found, then the core contents of the of the registry key HK_CURRENT_USER\Software\Python\PythonCore\<DLLVersion>\PythonPath\ is added, if it exists;
If on Windows and no applocal = true was set in pyvenv.cfg, then the contents of the subkeys of the registry key HK_LOCAL_MACHINE\Software\Python\PythonCore\<DLLVersion>\PythonPath\ are added, if any.
If on Windows and no applocal = true was set in pyvenv.cfg, and sys.prefix could not be found, then the core contents of the of the registry key HK_CURRENT_USER\Software\Python\PythonCore\<DLLVersion>\PythonPath\ is added, if it exists;
If on Windows, and PYTHONPATH was not set, the prefix was not found, and no registry keys were present, then the relative compile-time value of PYTHONPATH is added; otherwise, this step is ignored.
Paths in the compile-time macro PYTHONPATH are added relative to the dynamically-found sys.prefix.
On Mac and Linux, the value of sys.exec_prefix is added. On Windows, the directory which was used (or would have been used) to search dynamically for sys.prefix is added.

At this stage on Windows, if no prefix was found, then python will try to determine it by searching all the directories in sys.path for the landmark files, as it tried to do with the directory of sys.executable previously, until it finds something. If it doesn’t, sys.prefix is left blank.

Finally, after all this, Python loads the site module, which adds stuff yet further to sys.path:

It starts by constructing up to four directories from a head and a tail part. For the head part, it uses sys.prefix and sys.exec_prefix; empty heads are skipped. For the tail part, it uses the empty string and then lib/site-packages (on Windows) or lib/pythonX.Y/site-packages and then lib/site-python (on Unix and Macintosh). For each of the distinct head-tail combinations, it sees if it refers to an existing directory, and if so, adds it to sys.path and also inspects the newly added path for configuration files.

Question 10

I have seen few py scripts which use this at the top of the script. In what cases one should use it?

import sys
reload(sys)
sys.setdefaultencoding("utf-8")

Question 11

As per the documentation: This allows you to switch from the default ASCII to other encodings such as UTF-8, which the Python runtime will use whenever it has to decode a string buffer to unicode.

This function is only available at Python start-up time, when Python scans the environment. It has to be called in a system-wide module, sitecustomize.py, After this module has been evaluated, the setdefaultencoding() function is removed from the sys module.

The only way to actually use it is with a reload hack that brings the attribute back.

Also, the use of sys.setdefaultencoding() has always been discouraged, and it has become a no-op in py3k. The encoding of py3k is hard-wired to “utf-8” and changing it raises an error.

I suggest some pointers for reading:

Question 12

tl;dr

The answer is NEVER! _{(unless you really know what you’re doing)}

9/10 times the solution can be resolved with a proper understanding of encoding/decoding.

1/10 people have an incorrectly defined locale or environment and need to set:

PYTHONIOENCODING="UTF-8"

in their environment to fix console printing problems.

What does it do?

~~sys.setdefaultencoding("utf-8")~~ (struck through to avoid re-use) changes the default encoding/decoding used whenever Python 2.x needs to convert a Unicode() to a str() (and vice-versa) and the encoding is not given. I.e:

str(u"\u20AC")
unicode("€")
"{}".format(u"\u20AC")

In Python 2.x, the default encoding is set to ASCII and the above examples will fail with:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)

(My console is configured as UTF-8, so "€" = '\xe2\x82\xac', hence exception on \xe2)

or

UnicodeEncodeError: 'ascii' codec can't encode character u'\u20ac' in position 0: ordinal not in range(128)

~~sys.setdefaultencoding("utf-8")~~ will allow these to work for me, but won’t necessarily work for people who don’t use UTF-8. The default of ASCII ensures that assumptions of encoding are not baked into code

Console

~~sys.setdefaultencoding("utf-8")~~ also has a side effect of appearing to fix sys.stdout.encoding, used when printing characters to the console. Python uses the user’s locale (Linux/OS X/Un*x) or codepage (Windows) to set this. Occasionally, a user’s locale is broken and just requires PYTHONIOENCODING to fix the console encoding.

Example:

$ export LANG=en_GB.gibberish
$ python
>>> import sys
>>> sys.stdout.encoding
'ANSI_X3.4-1968'
>>> print u"\u20AC"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\u20ac' in position 0: ordinal not in range(128)
>>> exit()

$ PYTHONIOENCODING=UTF-8 python
>>> import sys
>>> sys.stdout.encoding
'UTF-8'
>>> print u"\u20AC"
€

What’s so bad with sys.setdefaultencoding(“utf-8”)?

People have been developing against Python 2.x for 16 years on the understanding that the default encoding is ASCII. UnicodeError exception handling methods have been written to handle string to Unicode conversions on strings that are found to contain non-ASCII.

From https://anonbadger.wordpress.com/2015/06/16/why-sys-setdefaultencoding-will-break-code/

def welcome_message(byte_string):
    try:
        return u"%s runs your business" % byte_string
    except UnicodeError:
        return u"%s runs your business" % unicode(byte_string,
            encoding=detect_encoding(byte_string))

print(welcome_message(u"Angstrom (Å®)".encode("latin-1"))

Previous to setting defaultencoding this code would be unable to decode the “Å” in the ascii encoding and then would enter the exception handler to guess the encoding and properly turn it into unicode. Printing: Angstrom (Å®) runs your business. Once you’ve set the defaultencoding to utf-8 the code will find that the byte_string can be interpreted as utf-8 and so it will mangle the data and return this instead: Angstrom (Ů) runs your business.

Changing what should be a constant will have dramatic effects on modules you depend upon. It’s better to just fix the data coming in and out of your code.

Example problem

While the setting of defaultencoding to UTF-8 isn’t the root cause in the following example, it shows how problems are masked and how, when the input encoding changes, the code breaks in an unobvious way: UnicodeDecodeError: ‘utf8’ codec can’t decode byte 0x80 in position 3131: invalid start byte

Question 13

#!/usr/bin/env python
#-*- coding: utf-8 -*-
u = u'moçambique'
print u.encode("utf-8")
print u

chmod +x test.py
./test.py
moçambique
moçambique

./test.py > output.txt
Traceback (most recent call last):
  File "./test.py", line 5, in <module>
    print u
UnicodeEncodeError: 'ascii' codec can't encode character 
u'\xe7' in position 2: ordinal not in range(128)

on shell works , sending to sdtout not , so that is one workaround, to write to stdout .

I made other approach, which is not run if sys.stdout.encoding is not define, or in others words , need export PYTHONIOENCODING=UTF-8 first to write to stdout.

import sys
if (sys.stdout.encoding is None):            
    print >> sys.stderr, "please set python env PYTHONIOENCODING=UTF-8, example: export PYTHONIOENCODING=UTF-8, when write to stdout." 
    exit(1)

so, using same example:

export PYTHONIOENCODING=UTF-8
./test.py > output.txt

will work

Question 14

The first danger lies in reload(sys).

When you reload a module, you actually get two copies of the module in your runtime. The old module is a Python object like everything else, and stays alive as long as there are references to it. So, half of the objects will be pointing to the old module, and half to the new one. When you make some change, you will never see it coming when some random object doesn’t see the change:
```
(This is IPython shell)

In [1]: import sys

In [2]: sys.stdout
Out[2]: <colorama.ansitowin32.StreamWrapper at 0x3a2aac8>

In [3]: reload(sys)
<module 'sys' (built-in)>

In [4]: sys.stdout
Out[4]: <open file '<stdout>', mode 'w' at 0x00000000022E20C0>

In [11]: import IPython.terminal

In [14]: IPython.terminal.interactiveshell.sys.stdout
Out[14]: <colorama.ansitowin32.StreamWrapper at 0x3a9aac8>
```
Now, sys.setdefaultencoding() proper

All that it affects is implicit conversion str<->unicode. Now, utf-8 is the sanest encoding on the planet (backward-compatible with ASCII and all), the conversion now “just works”, what could possibly go wrong?

Well, anything. And that is the danger.
- There may be some code that relies on the UnicodeError being thrown for non-ASCII input, or does the transcoding with an error handler, which now produces an unexpected result. And since all code is tested with the default setting, you’re strictly on “unsupported” territory here, and no-one gives you guarantees about how their code will behave.
- The transcoding may produce unexpected or unusable results if not everything on the system uses UTF-8 because Python 2 actually has multiple independent “default string encodings”. (Remember, a program must work for the customer, on the customer’s equipment.)
  - Again, the worst thing is you will never know that because the conversion is implicit — you don’t really know when and where it happens. (Python Zen, koan 2 ahoy!) You will never know why (and if) your code works on one system and breaks on another. (Or better yet, works in IDE and breaks in console.)

Question 15

I’m using python to analyse some large files and I’m running into memory issues, so I’ve been using sys.getsizeof() to try and keep track of the usage, but it’s behaviour with numpy arrays is bizarre. Here’s an example involving a map of albedos that I’m having to open:

>>> import numpy as np
>>> import struct
>>> from sys import getsizeof
>>> f = open('Albedo_map.assoc', 'rb')
>>> getsizeof(f)
144
>>> albedo = struct.unpack('%df' % (7200*3600), f.read(7200*3600*4))
>>> getsizeof(albedo)
207360056
>>> albedo = np.array(albedo).reshape(3600,7200)
>>> getsizeof(albedo)
80

Well the data’s still there, but the size of the object, a 3600×7200 pixel map, has gone from ~200 Mb to 80 bytes. I’d like to hope that my memory issues are over and just convert everything to numpy arrays, but I feel that this behaviour, if true, would in some way violate some law of information theory or thermodynamics, or something, so I’m inclined to believe that getsizeof() doesn’t work with numpy arrays. Any ideas?

Question 16

You can use array.nbytes for numpy arrays, for example:

>>> import numpy as np
>>> from sys import getsizeof
>>> a = [0] * 1024
>>> b = np.array(a)
>>> getsizeof(a)
8264
>>> b.nbytes
8192

Question 17

The field nbytes will give you the size in bytes of all the elements of the array in a numpy.array:

size_in_bytes = my_numpy_array.nbytes

Notice that this does not measures “non-element attributes of the array object” so the actual size in bytes can be a few bytes larger than this.

Question 18

In python notebooks I often want to filter out ‘dangling’ numpy.ndarray‘s, in particular the ones that are stored in _1, _2, etc that were never really meant to stay alive.

I use this code to get a listing of all of them and their size.

Not sure if locals() or globals() is better here.

import sys
import numpy
from humanize import naturalsize

for size, name in sorted(
    (value.nbytes, name)
    for name, value in locals().items()
    if isinstance(value, numpy.ndarray)):
  print("{:>30}: {:>8}".format(name, naturalsize(size)))

Python 实用宝典

标签归档：sys

sys.stdout.flush（）方法的用法

问题：sys.stdout.flush（）方法的用法

回答 0

回答 1

回答 2

回答 3

回答 4

Python的sys.path是从哪里初始化的？

问题：Python的sys.path是从哪里初始化的？

回答 0

回答 1

为什么我们不应该在py脚本中使用sys.setdefaultencoding（“ utf-8”）？

问题：为什么我们不应该在py脚本中使用sys.setdefaultencoding（“ utf-8”）？

回答 0

回答 1

tl; dr

它有什么作用？

安慰

sys.setdefaultencoding（“ utf-8”）有什么不好？

示例问题

tl;dr

What does it do?

Console

What’s so bad with sys.setdefaultencoding(“utf-8”)?

Example problem

回答 2

回答 3

numpy数组的Python内存使用情况

问题：numpy数组的Python内存使用情况

回答 0

回答 1

回答 2

有趣好用的Python教程