Python 实用宝典

Question 1

I am using Ubuntu and have installed Python 2.7.5 and 3.4.0. In Python 2.7.5 I am able to successfully assign a variable x = Value('i', 2), but not in 3.4.0. I am getting:

Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "/usr/local/lib/python3.4/multiprocessing/context.py", line 132, in Value
      from .sharedctypes import Value
   File "/usr/local/lib/python3.4/multiprocessing/sharedctypes.py", line 10, in <
module>
   import ctypes
   File "/usr/local/lib/python3.4/ctypes/__init__.py", line 7, in <module>
      from _ctypes import Union, Structure, Array
ImportError: No module named '_ctypes'

I just updated to 3.3.2 through installing the source of 3.4.0. It installed in /usr/local/lib/python3.4.

Did I update to Python 3.4 correctly?

One thing I noticed that Python 3.4 is installed in usr/local/lib, while Python 3.3.2 is still installed in usr/lib, so it was not overwritten.

Question 2

Installing libffi-dev and re-installing python3.7 fixed the problem for me.

to cleanly build py 3.7 libffi-dev is required or else later stuff will fail

If using RHEL/Fedora:

yum install libffi-devel

or

sudo dnf install libffi-devel

If using Debian/Ubuntu:

sudo apt-get install libffi-dev

Question 3

On a fresh Debian image, cloning https://github.com/python/cpython and running:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get dist-upgrade
sudo apt-get install build-essential python-dev python-setuptools python-pip python-smbus
sudo apt-get install libncursesw5-dev libgdbm-dev libc6-dev
sudo apt-get install zlib1g-dev libsqlite3-dev tk-dev
sudo apt-get install libssl-dev openssl
sudo apt-get install libffi-dev

Now execute the configure file cloned above:

./configure
make # alternatively `make -j 4` will utilize 4 threads
sudo make altinstall

Got 3.7 installed and working for me.

SLIGHT UPDATE

Looks like I said I would update this answer with some more explanation and two years later I don’t have much to add.

this SO post explains why certain libraries like python-dev might be necessary.
this SO post explains why one might use the altinstall as opposed to install argument in the make command.

Aside from that I guess the choice would be to either read through the cpython codebase looking for #include directives that need to be met, but what I usually do is keep trying to install the package and just keep reading through the output installing the required packages until it succeeds.

Reminds me of the story of the Engineer, the Manager and the Programmer whose car rolls down a hill.

Question 4

If you use pyenv and get error “No module named ‘_ctypes'” (like i am) on Debian/Raspbian/Ubuntu you need to run this commands:

sudo apt-get install libffi-dev
pyenv uninstall 3.7.6
pyenv install 3.7.6

Put your version of python instead of 3.7.6

Question 5

Detailed steps to install Python 3.7 in CentOS or any redhat linux machine:

Download Python from https://www.python.org/ftp/python/3.7.0/Python-3.7.0.tar.xz
Extract the content in new folder
Open Terminal in the same directory
Run below code step by step :

sudo yum -y install gcc gcc-c++ 
sudo yum -y install zlib zlib-devel
sudo yum -y install libffi-devel 
./configure
make
make install

Question 6

Thought I’d add the Centos installs:

sudo yum -y install gcc gcc-c++ 
sudo yum -y install zlib zlib-devel
sudo yum -y install libffi-devel

Check python version:

python3 -V

Create virtualenv:

virtualenv -p python3 venv

Question 7

I run into this error when I tried to install Python 3.7.3 in Ubuntu 18.04 with next command: $ pyenv install 3.7.3. Installation succeeded after running $ sudo apt-get update && sudo apt-get install libffi-dev (as suggested here). The issue was solved there.

Question 8

None of the solution worked. You have to recompile your python again; once all the required packages were completely installed.

Follow this:

Install required packages
Run ./configure --enable-optimizations

https://gist.github.com/jerblack/798718c1910ccdd4ede92481229043be

Question 9

Refer to this thread or this thread, for customized installation of libffi, it is difficult for Python3.7 to find the library location of libffi. An alternative method is to set the CONFIGURE_LDFLAGS variable in the Makefile, for example CONFIGURE_LDFLAGS="-L/path/to/libffi-3.2.1/lib64".

Question 10

My solution: Installing libffi-dev with apt-get didn’t help. But this helped: Installing libffi from source and then installing Python 3.8 from source.

My configuration: Ubuntu 16.04 LTS Python 3.8.2

Step by step:

I got the error message “ModuleNotFoundError: No module named ‘_ctypes'” when starting the debugger from Visual Studio Code, and when running python3 -c "import sklearn; sklearn.show_versions()".

download libffi v3.3 from https://github.com/libffi/libffi/releases
install libtool: sudo apt-get install libtool The file README.md from libffi mentions that autoconf and automake are also necessary. They were already installed on my system.
configure libffi without docs:

./configure --disable-docs

make check

sudo make install

download python 3.8 from https://www.python.org/downloads/
./configure
make
make test
make install

After that my python installation could find _ctypes.

Question 11

This solved the same error for me on Debian:

sudo apt-get install libffi-dev

and compile again

Reference: issue31652

Question 12

If you don’t mind using Miniconda, the necessary external libraries and _ctypes are installed by default. It does take more space and may require using a moderately older version of Python (e.g. 3.7.6 instead of 3.8.2 as of this writing).

Question 13

You have to load the missing php3 (Python3) modules from the package manager. If you have Ubuntu I recommend the Synaptic Package Manager:

sudo apt-get install synaptic

There you can simply search for the missing modules. search for ctypes and install all the packages. Then go to your Python dir and do

./configure
make install.

This should solve your problem.

Question 14

If you are doing something nobody here will listen you about because “you’re doing it the wrong way”, but you have to do it “the wrong way” for reasons too asinine (for instance, in my case it quickly degrades into foul words about somebody on devops team overweight mother), you need to first:

Get libffi and install it into your user install area the usual way.

git clone https://github.com/libffi/libffi.git
cd libffi
./configure --prefix=path/to/your/install/root
make
make install

Then go back to your Python 3 source and find this part of the code in setup.py at the top level of the python source directory

        ffi_inc = [sysconfig.get_config_var("LIBFFI_INCLUDEDIR")]
        if not ffi_inc or ffi_inc[0] == '':
            ffi_inc = find_file('ffi.h', [], inc_dirs)
        if ffi_inc is not None:
            ffi_h = ffi_inc[0] + '/ffi.h'
            if not os.path.exists(ffi_h):
                ffi_inc = None
                print('Header file {} does not exist'.format(ffi_h))
        ffi_lib = None
        if ffi_inc is not None:
            for lib_name in ('ffi', 'ffi_pic'):
                if (self.compiler.find_library_file(lib_dirs, lib_name)):
                    ffi_lib = lib_name
                    break

        ffi_lib="ffi"  # --- AND INSERT THIS LINE HERE THAT DOES NOT APPEAR ---
        if ffi_inc and ffi_lib:
            ext.include_dirs.extend(ffi_inc)
            ext.libraries.append(ffi_lib)
            self.use_system_libffi = True

and add the line I have marked above with the comment. Why it is necessary, and why there is no way to get configure to respect ‘–without-system-ffi` on Linux platforms, perhaps I will find out why that is “unsupported” in the next couple of hours, but everything has worked ever since. Otherwise, best of luck… YMMV.

WHAT IT DOES: just overrides the logic there and causes the compiler linking command to add “-lffi” which is all that it really needs. If you have the library user-installed, it is probably detecting the headers fine as long as your PKG_CONFIG_PATH includes path/to/your/install/root/lib/pkgconfig.

Question 15

I want to do parallel http request tasks in asyncio, but I find that python-requests would block the event loop of asyncio. I’ve found aiohttp but it couldn’t provide the service of http request using a http proxy.

So I want to know if there’s a way to do asynchronous http requests with the help of asyncio.

Question 16

To use requests (or any other blocking libraries) with asyncio, you can use BaseEventLoop.run_in_executor to run a function in another thread and yield from it to get the result. For example:

import asyncio
import requests

@asyncio.coroutine
def main():
    loop = asyncio.get_event_loop()
    future1 = loop.run_in_executor(None, requests.get, 'http://www.google.com')
    future2 = loop.run_in_executor(None, requests.get, 'http://www.google.co.uk')
    response1 = yield from future1
    response2 = yield from future2
    print(response1.text)
    print(response2.text)

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

This will get both responses in parallel.

With python 3.5 you can use the new await/async syntax:

import asyncio
import requests

async def main():
    loop = asyncio.get_event_loop()
    future1 = loop.run_in_executor(None, requests.get, 'http://www.google.com')
    future2 = loop.run_in_executor(None, requests.get, 'http://www.google.co.uk')
    response1 = await future1
    response2 = await future2
    print(response1.text)
    print(response2.text)

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

See PEP0492 for more.

Question 17

aiohttp can be used with HTTP proxy already:

import asyncio
import aiohttp


@asyncio.coroutine
def do_request():
    proxy_url = 'http://localhost:8118'  # your proxy address
    response = yield from aiohttp.request(
        'GET', 'http://google.com',
        proxy=proxy_url,
    )
    return response

loop = asyncio.get_event_loop()
loop.run_until_complete(do_request())

Question 18

The answers above are still using the old Python 3.4 style coroutines. Here is what you would write if you got Python 3.5+.

aiohttp supports http proxy now

import aiohttp
import asyncio

async def fetch(session, url):
    async with session.get(url) as response:
        return await response.text()

async def main():
    urls = [
            'http://python.org',
            'https://google.com',
            'http://yifei.me'
        ]
    tasks = []
    async with aiohttp.ClientSession() as session:
        for url in urls:
            tasks.append(fetch(session, url))
        htmls = await asyncio.gather(*tasks)
        for html in htmls:
            print(html[:100])

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())

Question 19

Requests does not currently support asyncio and there are no plans to provide such support. It’s likely that you could implement a custom “Transport Adapter” (as discussed here) that knows how to use asyncio.

If I find myself with some time it’s something I might actually look into, but I can’t promise anything.

Question 20

There is a good case of async/await loops and threading in an article by Pimin Konstantin Kefaloukos Easy parallel HTTP requests with Python and asyncio:

To minimize the total completion time, we could increase the size of the thread pool to match the number of requests we have to make. Luckily, this is easy to do as we will see next. The code listing below is an example of how to make twenty asynchronous HTTP requests with a thread pool of twenty worker threads:

# Example 3: asynchronous requests with larger thread pool
import asyncio
import concurrent.futures
import requests

async def main():

    with concurrent.futures.ThreadPoolExecutor(max_workers=20) as executor:

        loop = asyncio.get_event_loop()
        futures = [
            loop.run_in_executor(
                executor, 
                requests.get, 
                'http://example.org/'
            )
            for i in range(20)
        ]
        for response in await asyncio.gather(*futures):
            pass


loop = asyncio.get_event_loop()
loop.run_until_complete(main())

Question 21

I’m trying to split my huge class into two; well, basically into the “main” class and a mixin with additional functions, like so:

main.py file:

import mymixin.py

class Main(object, MyMixin):
    def func1(self, xxx):
        ...

mymixin.py file:

class MyMixin(object):
    def func2(self: Main, xxx):  # <--- note the type hint
        ...

Now, while this works just fine, the type hint in MyMixin.func2 of course can’t work. I can’t import main.py, because I’d get a cyclic import and without the hint, my editor (PyCharm) can’t tell what self is.

I’m using Python 3.4, willing to move to 3.5 if a solution is available there.

Is there any way I can split my class into two files and keep all the “connections” so that my IDE still offers me auto completion & all the other goodies that come from it knowing the types?

Question 22

There isn’t a hugely elegant way to handle import cycles in general, I’m afraid. Your choices are to either redesign your code to remove the cyclic dependency, or if it isn’t feasible, do something like this:

# some_file.py

from typing import TYPE_CHECKING
if TYPE_CHECKING:
    from main import Main

class MyObject(object):
    def func2(self, some_param: 'Main'):
        ...

The TYPE_CHECKING constant is always False at runtime, so the import won’t be evaluated, but mypy (and other type-checking tools) will evaluate the contents of that block.

We also need to make the Main type annotation into a string, effectively forward declaring it since the Main symbol isn’t available at runtime.

If you are using Python 3.7+, we can at least skip having to provide an explicit string annotation by taking advantage of PEP 563:

# some_file.py

from __future__ import annotations
from typing import TYPE_CHECKING
if TYPE_CHECKING:
    from main import Main

class MyObject(object):
    # Hooray, cleaner annotations!
    def func2(self, some_param: Main):
        ...

The from __future__ import annotations import will make all type hints be strings and skip evaluating them. This can help make our code here mildly more ergonomic.

All that said, using mixins with mypy will likely require a bit more structure then you currently have. Mypy recommends an approach that’s basically what deceze is describing — to create an ABC that both your Main and MyMixin classes inherit. I wouldn’t be surprised if you ended up needing to do something similar in order to make Pycharm’s checker happy.

Question 23

For people struggling with cyclic imports when importing class only for Type checking: you will likely want to use a Forward Reference (PEP 484 – Type Hints):

When a type hint contains names that have not been defined yet, that definition may be expressed as a string literal, to be resolved later.

So instead of:

class Tree:
    def __init__(self, left: Tree, right: Tree):
        self.left = left
        self.right = right

you do:

class Tree:
    def __init__(self, left: 'Tree', right: 'Tree'):
        self.left = left
        self.right = right

Question 24

The bigger issue is that your types aren’t sane to begin with. MyMixin makes a hardcoded assumption that it will be mixed into Main, whereas it could be mixed into any number of other classes, in which case it would probably break. If your mixin is hardcoded to be mixed into one specific class, you may as well write the methods directly into that class instead of separating them out.

To properly do this with sane typing, MyMixin should be coded against an interface, or abstract class in Python parlance:

import abc


class MixinDependencyInterface(abc.ABC):
    @abc.abstractmethod
    def foo(self):
        pass


class MyMixin:
    def func2(self: MixinDependencyInterface, xxx):
        self.foo()  # ← mixin only depends on the interface


class Main(MixinDependencyInterface, MyMixin):
    def foo(self):
        print('bar')

Question 25

Turns out my original attempt was quite close to the solution as well. This is what I’m currently using:

# main.py
import mymixin.py

class Main(object, MyMixin):
    def func1(self, xxx):
        ...


# mymixin.py
if False:
    from main import Main

class MyMixin(object):
    def func2(self: 'Main', xxx):  # <--- note the type hint
        ...

Note the import within if False statement that never gets imported (but IDE knows about it anyway) and using the Main class as string because it’s not known at runtime.

Question 26

I think the perfect way should be to import all the classes and dependencies in a file (like __init__.py) and then from __init__ import * in all the other files.

In this case you are

avoiding multiple references to those files and classes and
also only have to add one line in each of the other files and
the third would be the pycharm knowing about all of the classes that you might use.

Question 27

What is the BEST way to clear out all the __pycache__ folders and .pyc/.pyo files from a python3 project. I have seen multiple users suggest the pyclean script bundled with Debian, but this does not remove the folders. I want a simple way to clean up the project before pushing the files to my DVS.

Question 28

You can do it manually with the next command:

find . | grep -E "(__pycache__|\.pyc|\.pyo$)" | xargs rm -rf

This will remove all *.pyc files and __pycache__ directories recursively in the current directory.

Question 29

I found the answer myself when I mistyped pyclean as pycclean:

    No command 'pycclean' found, did you mean:
     Command 'py3clean' from package 'python3-minimal' (main)
     Command 'pyclean' from package 'python-minimal' (main)
    pycclean: command not found

Running py3clean . cleaned it up very nicely.

Question 30

macOS & Linux

BSD’s find implementation on macOS is different from GNU find – this is compatible with both BSD and GNU find. Start with a globbing implementation, using -name and the -o for or – Put this function in your .bashrc file:

pyclean () {
    find . -type f -name '*.py[co]' -delete -o -type d -name __pycache__ -delete
}

Then cd to the directory you want to recursively clean, and type pyclean.

GNU find-only

This is a GNU find, only (i.e. Linux) solution, but I feel it’s a little nicer with the regex:

pyclean () {
    find . -regex '^.*\(__pycache__\|\.py[co]\)$' -delete
}

Any platform, using Python 3

On Windows, you probably don’t even have find. You do, however, probably have Python 3, which starting in 3.4 has the convenient pathlib module:

python3 -Bc "import pathlib; [p.unlink() for p in pathlib.Path('.').rglob('*.py[co]')]"
python3 -Bc "import pathlib; [p.rmdir() for p in pathlib.Path('.').rglob('__pycache__')]"

The -B flag tells Python not to write .pyc files. (See also the PYTHONDONTWRITEBYTECODE environment variable.)

The above abuses list comprehensions for looping, but when using python -c, style is rather a secondary concern. Alternatively we could abuse (for example) __import__:

python3 -Bc "for p in __import__('pathlib').Path('.').rglob('*.py[co]'): p.unlink()"
python3 -Bc "for p in __import__('pathlib').Path('.').rglob('__pycache__'): p.rmdir()"

Critique of an answer

The top answer used to say:

find . | grep -E "(__pycache__|\.pyc|\.pyo$)" | xargs rm -rf

This would seem to be less efficient because it uses three processes. find takes a regular expression, so we don’t need a separate invocation of grep. Similarly, it has -delete, so we don’t need a separate invocation of rm —and contrary to a comment here, it will delete non-empty directories so long as they get emptied by virtue of the regular expression match.

From the xargs man page:

find /tmp -depth -name core -type f -delete
Find files named core in or below the directory /tmp and delete them, but more efficiently than in the previous example (because we avoid the need to use fork(2) and exec(2) to launch rm and we don’t need the extra xargs process).

Question 31

Since this is a Python 3 project, you only need to delete __pycache__ directories — all .pyc/.pyo files are inside them.

find . -type d -name __pycache__ -exec rm -r {} \+

or its simpler form,

find . -type d -name __pycache__ -delete

which didn’t work for me for some reason (files were deleted but directories weren’t), so I’m including both for the sake of completeness.

Alternatively, if you’re doing this in a directory that’s under revision control, you can tell the RCS to ignore __pycache__ folders recursively. Then, at the required moment, just clean up all the ignored files. This will likely be more convenient because there’ll probably be more to clean up than just __pycache__.

Question 32

This is my alias that works both with Python 2 and Python 3 removing all .pyc .pyo files as well __pycache__ directories recursively.

alias pyclean='find . -name "*.py[co]" -o -name __pycache__ -exec rm -rf {} +'

Question 33

If you need a permanent solution for keeping Python cache files out of your project directories:

Starting with Python 3.8 you can use the environment variable PYTHONPYCACHEPREFIX to define a cache directory for Python.

From the Python docs:

If this is set, Python will write .pyc files in a mirror directory tree at this path, instead of in pycache directories within the source tree. This is equivalent to specifying the -X pycache_prefix=PATH option.

Example

If you add the following line to your ./profile in Linux:

export PYTHONPYCACHEPREFIX="$HOME/.cache/cpython/"

Python won’t create the annoying __pycache__ directories in your project directory, instead it will put all of them under ~/.cache/cpython/

Question 34

The command I’ve used:

find . -type d -name "__pycache__" -exec rm -r {} +

Explains:

First finds all __pycache__ folders in current directory.
Execute rm -r {} + to delete each folder at step above ({} signify for placeholder and + to end the command)

Edited 1:

I’m using Linux, to reuse the command I’ve added the line below to the ~/.bashrc file

alias rm-pycache='find . -type d -name  "__pycache__" -exec rm -r {} +'

Edited 2: If you’re using VS Code, you don’t need to remove __pycache__ manually. You can add the snippet below to settings.json file. After that, VS Code will hide all __pycache__ folders for you

"files.exclude": {
     "**/__pycache__": true
}

Hope it helps !!!

Question 35

From the project directory type the following:

Deleting all .pyc files

find . -path "*/*.pyc" -delete

Deleting all .pyo files:

find . -path "*/*.pyo" -delete

Finally, to delete all ‘__pycache__’, type:

find . -path "*/__pycache__" -type d -exec rm -r {} ';'

If you encounter permission denied error, add sudo at the begining of all the above command.

Question 36

Using PyCharm

To remove Python compiled files

In the Project Tool Window, right-click a project or directory, where Python compiled files should be deleted from.
On the context menu, choose Clean Python compiled files.

The .pyc files residing in the selected directory are silently deleted.

Question 37

Thanks a lot for the other answers, based on them this is what I used for my Debian package’s prerm file:

#!/bin/sh
set -e

deb_package='package-name'
python_package='package_name'

if which pyclean >/dev/null 2>&1; then
    py3clean -p $deb_package
else
    dpkg -L $deb_package | grep ${python_package}$ | while read file
    do
        find ${file} -type d -name __pycache__ -exec rm -r {} \+
    done
fi

Question 38

Why not just use rm -rf __pycache__? Run git add -A afterwards to remove them from your repository and add __pycache__/ to your .gitignore file.

Question 39

Please just go to your terminal then type:

$rm __pycache__

and it will be removed.

问题：Python3：ImportError：使用模块多处理中的值时，没有名为“ _ctypes”的模块

回答 0

回答 1

轻微更新

SLIGHT UPDATE

回答 2

回答 3

回答 4

回答 5

回答 6

回答 7

回答 8

回答 9

回答 10

回答 11

回答 12

问题：如何在asyncio中使用请求？

回答 0

回答 1

回答 2

回答 3

回答 4

问题：没有循环导入的Python类型提示

回答 0

回答 1

回答 2

回答 3

回答 4

问题：Python3项目删除__pycache__文件夹和.pyc文件

回答 0

回答 1

回答 2

macOS和Linux

GNU仅查找

任何使用Python 3的平台

批判答案

macOS & Linux

GNU find-only

Any platform, using Python 3

Critique of an answer

回答 3

回答 4

回答 5

回答 6

回答 7

回答 8

使用PyCharm

删除Python编译文件

Using PyCharm

To remove Python compiled files

回答 9

回答 10

回答 11

有趣好用的Python教程

问题：Python3项目删除pycache文件夹和.pyc文件