Python 实用宝典

Question 1

I have a directory structure similar to the following

meta_project
    project1
        __init__.py
        lib
            module.py
            __init__.py
    notebook_folder
        notebook.jpynb

When working in notebook.jpynb if I try to use a relative import to access a function function() in module.py with:

from ..project1.lib.module import function

I get the following error:

SystemError                               Traceback (most recent call last)
<ipython-input-7-6393744d93ab> in <module>()
----> 1 from ..project1.lib.module import function

SystemError: Parent module '' not loaded, cannot perform relative import

Is there any way to get this to work using relative imports?

Note, the notebook server is instantiated at the level of the meta_project directory, so it should have access to the information in those files.

Note, also, that at least as originally intended project1 wasn’t thought of as a module and therefore does not have an __init__.py file, it was just meant as a file-system directory. If the solution to the problem requires treating it as a module and including an __init__.py file (even a blank one) that is fine, but doing so is not enough to solve the problem.

I share this directory between machines and relative imports allow me to use the same code everywhere, & I often use notebooks for quick prototyping, so suggestions that involve hacking together absolute paths are unlikely to be helpful.

Edit: This is unlike Relative imports in Python 3, which talks about relative imports in Python 3 in general and – in particular – running a script from within a package directory. This has to do with working within a jupyter notebook trying to call a function in a local module in another directory which has both different general and particular aspects.

Question 2

I had almost the same example as you in this notebook where I wanted to illustrate the usage of an adjacent module’s function in a DRY manner.

My solution was to tell Python of that additional module import path by adding a snippet like this one to the notebook:

import os
import sys
module_path = os.path.abspath(os.path.join('..'))
if module_path not in sys.path:
    sys.path.append(module_path)

This allows you to import the desired function from the module hierarchy:

from project1.lib.module import function
# use the function normally
function(...)

Note that it is necessary to add empty __init__.py files to project1/ and lib/ folders if you don’t have them already.

Question 3

Came here searching for best practices in abstracting code to submodules when working in Notebooks. I’m not sure that there is a best practice. I have been proposing this.

A project hierarchy as such:

├── ipynb
│   ├── 20170609-Examine_Database_Requirements.ipynb
│   └── 20170609-Initial_Database_Connection.ipynb
└── lib
    ├── __init__.py
    └── postgres.py

And from 20170609-Initial_Database_Connection.ipynb:

    In [1]: cd ..

    In [2]: from lib.postgres import database_connection

This works because by default the Jupyter Notebook can parse the cd command. Note that this does not make use of Python Notebook magic. It simply works without prepending %bash.

Considering that 99 times out of a 100 I am working in Docker using one of the Project Jupyter Docker images, the following modification is idempotent

    In [1]: cd /home/jovyan

    In [2]: from lib.postgres import database_connection

Question 4

So far, the accepted answer has worked best for me. However, my concern has always been that there is a likely scenario where I might refactor the notebooks directory into subdirectories, requiring to change the module_path in every notebook. I decided to add a python file within each notebook directory to import the required modules.

Thus, having the following project structure:

project
|__notebooks
   |__explore
      |__ notebook1.ipynb
      |__ notebook2.ipynb
      |__ project_path.py
   |__ explain
       |__notebook1.ipynb
       |__project_path.py
|__lib
   |__ __init__.py
   |__ module.py

I added the file project_path.py in each notebook subdirectory (notebooks/explore and notebooks/explain). This file contains the code for relative imports (from @metakermit):

import sys
import os

module_path = os.path.abspath(os.path.join(os.pardir, os.pardir))
if module_path not in sys.path:
    sys.path.append(module_path)

This way, I just need to do relative imports within the project_path.py file, and not in the notebooks. The notebooks files would then just need to import project_path before importing lib. For example in 0.0-notebook.ipynb:

import project_path
import lib

The caveat here is that reversing the imports would not work. THIS DOES NOT WORK:

import lib
import project_path

Thus care must be taken during imports.

Question 5

I have just found this pretty solution:

import sys; sys.path.insert(0, '..') # add parent folder path where lib folder is
import lib.store_load # store_load is a file on my library folder

You just want some functions of that file

from lib.store_load import your_function_name

If python version >= 3.3 you do not need init.py file in the folder

Question 6

Researching this topic myself and having read the answers I recommend using the path.py library since it provides a context manager for changing the current working directory.

You then have something like

import path
if path.Path('../lib').isdir():
    with path.Path('..'):
        import lib

Although, you might just omit the isdir statement.

Here I’ll add print statements to make it easy to follow what’s happening

import path
import pandas

print(path.Path.getcwd())
print(path.Path('../lib').isdir())
if path.Path('../lib').isdir():
    with path.Path('..'):
        print(path.Path.getcwd())
        import lib
        print('Success!')
print(path.Path.getcwd())

which outputs in this example (where lib is at /home/jovyan/shared/notebooks/by-team/data-vis/demos/lib):

/home/jovyan/shared/notebooks/by-team/data-vis/demos/custom-chart
/home/jovyan/shared/notebooks/by-team/data-vis/demos
/home/jovyan/shared/notebooks/by-team/data-vis/demos/custom-chart

Since the solution uses a context manager, you are guaranteed to go back to your previous working directory, no matter what state your kernel was in before the cell and no matter what exceptions are thrown by importing your library code.

Question 7

Here’s my 2 cents:

import sys

map the path where the module file is located. In my case it was the desktop

sys.path.append(‘/Users/John/Desktop’)

Either import the whole mapping module BUT then you have to use the .notation to map the classes like mapping.Shipping()

import mapping #mapping.py is the name of my module file

shipit = mapping.Shipment() #Shipment is the name of the class I need to use in the mapping module

Or import the specific class from the mapping module

from mapping import Mapping

shipit = Shipment() #Now you don’t have to use the .notation

Question 8

I have found that python-dotenv helps solve this issue pretty effectively. Your project structure ends up changing slightly, but the code in your notebook is a bit simpler and consistent across notebooks.

For your project, do a little install.

pipenv install python-dotenv

Then, project changes to:

├── .env (this can be empty)
├── ipynb
│   ├── 20170609-Examine_Database_Requirements.ipynb
│   └── 20170609-Initial_Database_Connection.ipynb
└── lib
    ├── __init__.py
    └── postgres.py

And finally, your import changes to:

import os
import sys

from dotenv import find_dotenv


sys.path.append(os.path.dirname(find_dotenv()))

A +1 for this package is that your notebooks can be several directories deep. python-dotenv will find the closest one in a parent directory and use it. A +2 for this approach is that jupyter will load environment variables from the .env file on startup. Double whammy.

Question 9

I’ve got a python project with a configuration file in the project root. The configuration file needs to be accessed in a few different files throughout the project.

So it looks something like: <ROOT>/configuration.conf <ROOT>/A/a.py, <ROOT>/A/B/b.py (when b,a.py access the configuration file).

What’s the best / easiest way to get the path to the project root and the configuration file without depending on which file inside the project I’m in? i.e without using ../../? It’s okay to assume that we know the project root’s name.

Question 10

You can do this how Django does it: define a variable to the Project Root from a file that is in the top-level of the project. For example, if this is what your project structure looks like:

project/
    configuration.conf
    definitions.py
    main.py
    utils.py

In definitions.py you can define (this requires import os):

ROOT_DIR = os.path.dirname(os.path.abspath(__file__)) # This is your Project Root

Thus, with the Project Root known, you can create a variable that points to the location of the configuration (this can be defined anywhere, but a logical place would be to put it in a location where constants are defined – e.g. definitions.py):

CONFIG_PATH = os.path.join(ROOT_DIR, 'configuration.conf')  # requires `import os`

Then, you can easily access the constant (in any of the other files) with the import statement (e.g. in utils.py): from definitions import CONFIG_PATH.

Question 11

Other answers advice to use a file in the top-level of the project. This is not necessary if you use pathlib.Path and parent (Python 3.4 and up). Consider the following directory structure where all files except README.md and utils.py have been omitted.

project
│   README.md
|
└───src
│   │   utils.py
|   |   ...
|   ...

In utils.py we define the following function.

from pathlib import Path

def get_project_root() -> Path:
    return Path(__file__).parent.parent

In any module in the project we can now get the project root as follows.

from src.utils import get_project_root

root = get_project_root()

Benefits: Any module which calls get_project_root can be moved without changing program behavior. Only when the module utils.py is moved we have to update get_project_root and the imports (refactoring tools can be used to automate this).

Question 12

All the previous solutions seem to be overly complicated for what I think you need, and often didn’t work for me. The following one-line command does what you want:

import os
ROOT_DIR = os.path.abspath(os.curdir)

Question 13

To get the path of the “root” module, you can use:

import os
import sys
os.path.dirname(sys.modules['__main__'].__file__)

But more interestingly if you have an config “object” in your top-most module you could -read- from it like so:

app = sys.modules['__main__']
stuff = app.config.somefunc()

Question 14

A standard way to achieve this would be to use the pkg_resources module which is part of the setuptools package. setuptools is used to create an install-able python package.

You can use pkg_resources to return the contents of your desired file as a string and you can use pkg_resources to get the actual path of the desired file on your system.

Let’s say that you have a package called stackoverflow.

stackoverflow/
|-- app
|   `-- __init__.py
`-- resources
    |-- bands
    |   |-- Dream\ Theater
    |   |-- __init__.py
    |   |-- King's\ X
    |   |-- Megadeth
    |   `-- Rush
    `-- __init__.py

3 directories, 7 files

Now let’s say that you want to access the file Rush from a module app.run. Use pkg_resources.resouces_filename to get the path to Rush and pkg_resources.resource_string to get the contents of Rush; thusly:

import pkg_resources

if __name__ == "__main__":
    print pkg_resources.resource_filename('resources.bands', 'Rush')
    print pkg_resources.resource_string('resources.bands', 'Rush')

The output:

/home/sri/workspace/stackoverflow/resources/bands/Rush
Base: Geddy Lee
Vocals: Geddy Lee
Guitar: Alex Lifeson
Drums: Neil Peart

This works for all packages in your python path. So if you want to know where lxml.etree exists on your system:

import pkg_resources

if __name__ == "__main__":
    print pkg_resources.resource_filename('lxml', 'etree')

output:

/usr/lib64/python2.7/site-packages/lxml/etree

The point is that you can use this standard method to access files that are installed on your system (e.g pip install xxx or yum -y install python-xxx) and files that are within the module that you’re currently working on.

Question 15

Try:

ROOT_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))

Question 16

Below Code Returns the path until your project root

import sys
print(sys.path[1])

Question 17

I struggled with this problem too until I came to this solution. This is the cleanest solution in my opinion.

In your setup.py add “packages”

setup(
name='package_name'
version='0.0.1'
.
.
.
packages=['package_name']
.
.
.
)

In your python_script.py

import pkg_resources
import os

resource_package = pkg_resources.get_distribution(
    'package_name').location
config_path = os.path.join(resource_package,'configuration.conf')

Question 18

Just an example: I want to run runio.py from within helper1.py

Project tree example:

myproject_root
- modules_dir/helpers_dir/helper1.py
- tools_dir/runio.py

Get project root:

import os
rootdir = os.path.dirname(os.path.realpath(__file__)).rsplit(os.sep, 2)[0]

Build path to script:

runme = os.path.join(rootdir, "tools_dir", "runio.py")
execfile(runme)

Question 19

This worked for me using a standard PyCharm project with my virtual environment (venv) under the project root directory.

Code below isnt the prettiest, but consistently gets the project root. It returns the full directory path to venv from the VIRTUAL_ENV environment variable e.g. /Users/NAME/documents/PROJECT/venv

It then splits the path at the last /, giving an array with two elements. The first element will be the project path e.g. /Users/NAME/documents/PROJECT

import os

print(os.path.split(os.environ['VIRTUAL_ENV'])[0])

Question 20

I’ve recently been trying to do something similar and I have found these answers inadequate for my use cases (a distributed library that needs to detect project root). Mainly I’ve been battling different environments and platforms, and still haven’t found something perfectly universal.

Code local to project

I’ve seen this example mentioned and used in a few places, Django, etc.

import os
print(os.path.dirname(os.path.abspath(__file__)))

Simple as this is, it only works when the file that the snippet is in is actually part of the project. We do not retrieve the project directory, but instead the snippet’s directory

Similarly, the sys.modules approach breaks down when called from outside the entrypoint of the application, specifically I’ve observed a child thread cannot determine this without relation back to the ‘main‘ module. I’ve explicitly put the import inside a function to demonstrate an import from a child thread, moving it to top level of app.py would fix it.

app/
|-- config
|   `-- __init__.py
|   `-- settings.py
`-- app.py

app.py

#!/usr/bin/env python
import threading


def background_setup():
    # Explicitly importing this from the context of the child thread
    from config import settings
    print(settings.ROOT_DIR)


# Spawn a thread to background preparation tasks
t = threading.Thread(target=background_setup)
t.start()

# Do other things during initialization

t.join()

# Ready to take traffic

settings.py

import os
import sys


ROOT_DIR = None


def setup():
    global ROOT_DIR
    ROOT_DIR = os.path.dirname(sys.modules['__main__'].__file__)
    # Do something slow

Running this program produces an attribute error:

>>> import main
>>> Exception in thread Thread-1:
Traceback (most recent call last):
  File "C:\Python2714\lib\threading.py", line 801, in __bootstrap_inner
    self.run()
  File "C:\Python2714\lib\threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "main.py", line 6, in background_setup
    from config import settings
  File "config\settings.py", line 34, in <module>
    ROOT_DIR = get_root()
  File "config\settings.py", line 31, in get_root
    return os.path.dirname(sys.modules['__main__'].__file__)
AttributeError: 'module' object has no attribute '__file__'

…hence a threading-based solution

Location independent

Using the same application structure as before but modifying settings.py

import os
import sys
import inspect
import platform
import threading


ROOT_DIR = None


def setup():
    main_id = None
    for t in threading.enumerate():
        if t.name == 'MainThread':
            main_id = t.ident
            break

    if not main_id:
        raise RuntimeError("Main thread exited before execution")

    current_main_frame = sys._current_frames()[main_id]
    base_frame = inspect.getouterframes(current_main_frame)[-1]

    if platform.system() == 'Windows':
        filename = base_frame.filename
    else:
        filename = base_frame[0].f_code.co_filename

    global ROOT_DIR
    ROOT_DIR = os.path.dirname(os.path.abspath(filename))

Breaking this down: First we want to accurately find the thread ID of the main thread. In Python3.4+ the threading library has threading.main_thread() however, everybody doesn’t use 3.4+ so we search through all threads looking for the main thread save it’s ID. If the main thread has already exited, it won’t be listed in the threading.enumerate(). We raise a RuntimeError() in this case until I find a better solution.

main_id = None
for t in threading.enumerate():
    if t.name == 'MainThread':
        main_id = t.ident
        break

if not main_id:
    raise RuntimeError("Main thread exited before execution")

Next we find the very first stack frame of the main thread. Using the cPython specific function sys._current_frames() we get a dictionary of every thread’s current stack frame. Then utilizing inspect.getouterframes() we can retrieve the entire stack for the main thread and the very first frame. current_main_frame = sys._current_frames()[main_id] base_frame = inspect.getouterframes(current_main_frame)[-1] Finally, the differences between Windows and Linux implementations of inspect.getouterframes() need to be handled. Using the cleaned up filename, os.path.abspath() and os.path.dirname() clean things up.

if platform.system() == 'Windows':
    filename = base_frame.filename
else:
    filename = base_frame[0].f_code.co_filename

global ROOT_DIR
ROOT_DIR = os.path.dirname(os.path.abspath(filename))

So far I’ve tested this on Python2.7 and 3.6 on Windows as well as Python3.4 on WSL

Question 21

If you are working with anaconda-project, you can query the PROJECT_ROOT from the environment variable –> os.getenv(‘PROJECT_ROOT’). This works only if the script is executed via anaconda-project run .

If you do not want your script run by anaconda-project, you can query the absolute path of the executable binary of the Python interpreter you are using and extract the path string up to the envs directory exclusiv. For example: The python interpreter of my conda env is located at:

/home/user/project_root/envs/default/bin/python

# You can first retrieve the env variable PROJECT_DIR.
# If not set, get the python interpreter location and strip off the string till envs inclusiv...

if os.getenv('PROJECT_DIR'):
    PROJECT_DIR = os.getenv('PROJECT_DIR')
else:
    PYTHON_PATH = sys.executable
    path_rem = os.path.join('envs', 'default', 'bin', 'python')
    PROJECT_DIR = py_path.split(path_rem)[0]

This works only with conda-project with fixed project structure of a anaconda-project

Question 22

I used the ../ method to fetch the current project path.

Example: Project1 — D:\projects

src

ConfigurationFiles

Configuration.cfg

Path=”../src/ConfigurationFiles/Configuration.cfg”

Question 23

At the time of writing, none of the other solutions are very self-contained. They depend either on an environment variable or the position of the module in the package structure. The top answer with the ‘Django’ solution falls victim to the latter by requiring a relative import. It also has the disadvantage of having to modify a module at the top level.

This should be the correct approach for finding the directory path of the top-level package:

import sys
import os

root_name, _, _ = __name__.partition('.')
root_module = sys.modules[root_name]
root_dir = os.path.dirname(root_module.__file__)

config_path = os.path.join(root_dir, 'configuration.conf')

It works by taking the first component in the dotted string contained in __name__ and using it as a key in sys.modules which returns the module object of the top-level package. Its __file__ attribute contains the path we want after trimming off /__init__.py using os.path.dirname().

This solution is self-contained. It works anywhere in any module of the package, including in the top-level __init__.py file.

Question 24

I had to implement a custom solution because it’s not as simple as you might think. My solution is based on stack trace inspection (inspect.stack()) + sys.path and is working fine no matter the location of the python module in which the function is invoked nor the interpreter (I tried by running it in PyCharm, in a poetry shell and other…). This is the full implementation with comments:

def get_project_root_dir() -> str:
    """
    Returns the name of the project root directory.

    :return: Project root directory name
    """

    # stack trace history related to the call of this function
    frame_stack: [FrameInfo] = inspect.stack()

    # get info about the module that has invoked this function
    # (index=0 is always this very module, index=1 is fine as long this function is not called by some other
    # function in this module)
    frame_info: FrameInfo = frame_stack[1]

    # if there are multiple calls in the stacktrace of this very module, we have to skip those and take the first
    # one which comes from another module
    if frame_info.filename == __file__:
        for frame in frame_stack:
            if frame.filename != __file__:
                frame_info = frame
                break

    # path of the module that has invoked this function
    caller_path: str = frame_info.filename

    # absolute path of the of the module that has invoked this function
    caller_absolute_path: str = os.path.abspath(caller_path)

    # get the top most directory path which contains the invoker module
    paths: [str] = [p for p in sys.path if p in caller_absolute_path]
    paths.sort(key=lambda p: len(p))
    caller_root_path: str = paths[0]

    if not os.path.isabs(caller_path):
        # file name of the invoker module (eg: "mymodule.py")
        caller_module_name: str = Path(caller_path).name

        # this piece represents a subpath in the project directory
        # (eg. if the root folder is "myproject" and this function has ben called from myproject/foo/bar/mymodule.py
        # this will be "foo/bar")
        project_related_folders: str = caller_path.replace(os.sep + caller_module_name, '')

        # fix root path by removing the undesired subpath
        caller_root_path = caller_root_path.replace(project_related_folders, '')

    dir_name: str = Path(caller_root_path).name

    return dir_name

Question 25

There are many answers here but I couldn’t find something simple that covers all cases so allow me to suggest my solution too:

import pathlib
import os

def get_project_root():
    """
    There is no way in python to get project root. This function uses a trick.
    We know that the function that is currently running is in the project.
    We know that the root project path is in the list of PYTHONPATH
    look for any path in PYTHONPATH list that is contained in this function's path
    Lastly we filter and take the shortest path because we are looking for the root.
    :return: path to project root
    """
    apth = str(pathlib.Path().absolute())
    ppth = os.environ['PYTHONPATH'].split(':')
    matches = [x for x in ppth if x in apth]
    project_root = min(matches, key=len)
    return project_root

Question 26

Lets suppose I have a list like this:

mylist = ["a","b","c","d"]

To get the values printed along with their index I can use Python’s enumerate function like this

>>> for i,j in enumerate(mylist):
...     print i,j
...
0 a
1 b
2 c
3 d
>>>

Now, when I try to use it inside a list comprehension it gives me this error

>>> [i,j for i,j in enumerate(mylist)]
  File "<stdin>", line 1
    [i,j for i,j in enumerate(mylist)]
           ^
SyntaxError: invalid syntax

So, my question is: what is the correct way of using enumerate inside list comprehension?

Question 27

Try this:

[(i, j) for i, j in enumerate(mylist)]

You need to put i,j inside a tuple for the list comprehension to work. Alternatively, given that enumerate() already returns a tuple, you can return it directly without unpacking it first:

[pair for pair in enumerate(mylist)]

Either way, the result that gets returned is as expected:

> [(0, 'a'), (1, 'b'), (2, 'c'), (3, 'd')]

Question 28

Just to be really clear, this has nothing to do with enumerate and everything to do with list comprehension syntax.

This list comprehension returns a list of tuples:

[(i,j) for i in range(3) for j in 'abc']

this a list of dicts:

[{i:j} for i in range(3) for j in 'abc']

a list of lists:

[[i,j] for i in range(3) for j in 'abc']

a syntax error:

[i,j for i in range(3) for j in 'abc']

Which is inconsistent (IMHO) and confusing with dictionary comprehensions syntax:

>>> {i:j for i,j in enumerate('abcdef')}
{0: 'a', 1: 'b', 2: 'c', 3: 'd', 4: 'e', 5: 'f'}

And a set of tuples:

>>> {(i,j) for i,j in enumerate('abcdef')}
set([(0, 'a'), (4, 'e'), (1, 'b'), (2, 'c'), (5, 'f'), (3, 'd')])

As Óscar López stated, you can just pass the enumerate tuple directly:

>>> [t for t in enumerate('abcdef') ] 
[(0, 'a'), (1, 'b'), (2, 'c'), (3, 'd'), (4, 'e'), (5, 'f')]

Question 29

Or, if you don’t insist on using a list comprehension:

>>> mylist = ["a","b","c","d"]
>>> list(enumerate(mylist))
[(0, 'a'), (1, 'b'), (2, 'c'), (3, 'd')]

Question 30

If you’re using long lists, it appears the list comprehension’s faster, not to mention more readable.

~$ python -mtimeit -s"mylist = ['a','b','c','d']" "list(enumerate(mylist))"
1000000 loops, best of 3: 1.61 usec per loop
~$ python -mtimeit -s"mylist = ['a','b','c','d']" "[(i, j) for i, j in enumerate(mylist)]"
1000000 loops, best of 3: 0.978 usec per loop
~$ python -mtimeit -s"mylist = ['a','b','c','d']" "[t for t in enumerate(mylist)]"
1000000 loops, best of 3: 0.767 usec per loop

Question 31

Here’s a way to do it:

>>> mylist = ['a', 'b', 'c', 'd']
>>> [item for item in enumerate(mylist)]
[(0, 'a'), (1, 'b'), (2, 'c'), (3, 'd')]

Alternatively, you can do:

>>> [(i, j) for i, j in enumerate(mylist)]
[(0, 'a'), (1, 'b'), (2, 'c'), (3, 'd')]

The reason you got an error was that you were missing the () around i and j to make it a tuple.

Question 32

Be explicit about the tuples.

[(i, j) for (i, j) in enumerate(mylist)]

Question 33

All great answer guys. I know the question here is specific to enumeration but how about something like this, just another perspective

from itertools import izip, count
a = ["5", "6", "1", "2"]
tupleList = list( izip( count(), a ) )
print(tupleList)

It becomes more powerful, if one has to iterate multiple lists in parallel in terms of performance. Just a thought

a = ["5", "6", "1", "2"]
b = ["a", "b", "c", "d"]
tupleList = list( izip( count(), a, b ) )
print(tupleList)

Question 34

Virtualenv is great: it lets me keep a number of distinct Python installations so that different projects’ dependencies aren’t all thrown together into a common pile.

But if I want to install a package on Windows that’s packaged as a .exe installer, how can I direct it to install into the virtualenv? For example, I have pycuda-0.94rc.win32-py2.6.exe. When I run it, it examines the registry, and finds only one Python26 to install into, the common one that my virtualenv is based off of.

How can I direct it to install into the virtualenv?

Question 35

Yes, you can. All you need is

easy_install binary_installer_built_with_distutils.exe

Surprised? It looks like binary installers for Windows made with distutils combine .exe with .zip into one .exe file. Change extension to .zip to see it’s a valid zip file. I discovered this after reading answers to my question Where can I download binary eggs with psycopg2 for Windows?

UPDATE

As noted by Tritium21 in his answer nowadays you should use pip instead of easy_install. Pip can’t install binary packages created by distutils but it can install binary packages in the new wheel format. You can convert from old format to the new one using wheel package, which you have to install first.

Question 36

I know this is quite an old question, and predates the tools I am about to talk about, but for the sake of Google, I think it is a good idea to mention it. easy_install is the black sheep of python packaging. No one wants to admit using it with the new hotness of pip around. Also, while playing registry tricks will work best for non-standard EXE installers (someone built the installer themselves instead of using distutils, and is checking the registry for the installation path), there is now a Better Way(c) for standard EXE installers.

pip install wheel
wheel convert INSTALLER.EXE
pip install NEW_FILE_CREATED_IN_LAST_STEP.whl

The wheel format, introduced recently as of this post, is the replacement for the egg format, filling much the same role. This format is also supported by pip (a tool already installed in your virtualenv).

if for some reason pip install WHEELFILE does not work, try wheel install WHEELFILE

Question 37

I ended up adapting a script (http://effbot.org/zone/python-register.htm) to register a Python installation in the registry. I can pick the Python to be the Python in the registry, run the Windows installer, then set the registry back:

# -*- encoding: utf-8 -*-
#
# script to register Python 2.0 or later for use with win32all
# and other extensions that require Python registry settings
#
# Adapted by Ned Batchelder from a script
# written by Joakim Löw for Secret Labs AB / PythonWare
#
# source:
# http://www.pythonware.com/products/works/articles/regpy20.htm

import sys

from _winreg import *

# tweak as necessary
version = sys.version[:3]
installpath = sys.prefix

regpath = "SOFTWARE\\Python\\Pythoncore\\%s\\" % (version)
installkey = "InstallPath"
pythonkey = "PythonPath"
pythonpath = "%s;%s\\Lib\\;%s\\DLLs\\" % (
    installpath, installpath, installpath
)

def RegisterPy():
    try:
        reg = OpenKey(HKEY_LOCAL_MACHINE, regpath)
    except EnvironmentError:
        try:
            reg = CreateKey(HKEY_LOCAL_MACHINE, regpath)
        except Exception, e:
            print "*** Unable to register: %s" % e
            return

    SetValue(reg, installkey, REG_SZ, installpath)
    SetValue(reg, pythonkey, REG_SZ, pythonpath)
    CloseKey(reg)
    print "--- Python %s at %s is now registered!" % (version, installpath)

if __name__ == "__main__":
    RegisterPy()

Run this script with the Python you want to be registered, and it will be entered into the registry. Note that on Windows 7 and Vista, you’ll need Administrator privileges.

Question 38

easy_install is able to install .exe packages as long as they were built using distutils’ bdist_wininst target, which covers many popular packages. However, there are many others that aren’t (wxPython is one that I’ve struggled with)

Question 39

You can use environment’s easy_install to install PyCUDA.

dev-env-path/bin/easy_install pycuda

it will give you the same version 0.94rc.

On Windows easy_install.exe will be in Scripts directory.

Question 40

If it’s a .msi, you might be able to specify command line options using msiexec. The Python installer itself allows TARGETDIR, but I’m not sure if distutils bakes this into distribution installers.

If you’re using a .exe, I don’t think there’s a clean way. One option is to use a program like 7Zip (or winzip, etc) to directly extract the contents of the exe, then copy the relevent folders into your virtual site-packages folder. For example, if I extract “processing-0.5.2.win32-py2.5.exe”, I find a folder “PLATLIB\processing” which I copy to a virtualenv path and use without any runtime problems. (I’m not sure it’s always that simple though.)

Question 41

I’m starting to learn python and loving it. I work on a Mac mainly as well as Linux. I’m finding that on Linux (Ubuntu 9.04 mostly) when I install a python module using apt-get it works fine. I can import it with no trouble.

On the Mac, I’m used to using Macports to install all the Unixy stuff. However, I’m finding that most of the python modules I install with it are not being seen by python. I’ve spent some time playing around with PATH settings and using python_select . Nothing has really worked and at this point I’m not really understanding, instead I’m just poking around.

I get the impression that Macports isn’t universally loved for managing python modules. I’d like to start fresh using a more “accepted” (if that’s the right word) approach.

So, I was wondering, what is the method that Mac python developers use to manage their modules?

Bonus questions:

Do you use Apple’s python, or some other version? Do you compile everything from source or is there a package manger that works well (Fink?).

Question 42

The most popular way to manage python packages (if you’re not using your system package manager) is to use setuptools and easy_install. It is probably already installed on your system. Use it like this:

easy_install django

easy_install uses the Python Package Index which is an amazing resource for python developers. Have a look around to see what packages are available.

A better option is pip, which is gaining traction, as it attempts to fix a lot of the problems associated with easy_install. Pip uses the same package repository as easy_install, it just works better. Really the only time use need to use easy_install is for this command:

easy_install pip

After that, use:

pip install django

At some point you will probably want to learn a bit about virtualenv. If you do a lot of python development on projects with conflicting package requirements, virtualenv is a godsend. It will allow you to have completely different versions of various packages, and switch between them easily depending your needs.

Regarding which python to use, sticking with Apple’s python will give you the least headaches, but If you need a newer version (Leopard is 2.5.1 I believe), I would go with the macports python 2.6.

Question 43

Your question is already three years old and there are some details not covered in other answers:

Most people I know use HomeBrew or MacPorts, I prefer MacPorts because of its clean cut of what is a default Mac OS X environment and my development setup. Just move out your /opt folder and test your packages with a normal user Python environment

MacPorts is only portable within Mac, but with easy_install or pip you will learn how to setup your environment in any platform (Win/Mac/Linux/Bsd…). Furthermore it will always be more up to date and with more packages

I personally let MacPorts handle my Python modules to keep everything updated. Like any other high level package manager (ie: apt-get) it is much better for the heavy lifting of modules with lots of binary dependencies. There is no way I would build my Qt bindings (PySide) with easy_install or pip. Qt is huge and takes a lot to compile. As soon as you want a Python package that needs a library used by non Python programs, try to avoid easy_install or pip

At some point you will find that there are some packages missing within MacPorts. I do not believe that MacPorts will ever give you the whole CheeseShop. For example, recently I needed the Elixir module, but MacPorts only offers py25-elixir and py26-elixir, no py27 version. In cases like these you have:

pip-2.7 install –user elixir

( make sure you always type pip-(version) )

That will build an extra Python library in your home dir. Yes, Python will work with more than one library location: one controlled by MacPorts and a user local one for everything missing within MacPorts.

Now notice that I favor pip over easy_install. There is a good reason you should avoid setuptools and easy_install. Here is a good explanation and I try to keep away from them. One very useful feature of pip is giving you a list of all the modules (along their versions) that you installed with MacPorts, easy_install and pip itself:

pip-2.7 freeze

If you already started using easy_install, don’t worry, pip can recognize everything done already by easy_install and even upgrade the packages installed with it.

If you are a developer keep an eye on virtualenv for controlling different setups and combinations of module versions. Other answers mention it already, what is not mentioned so far is the Tox module, a tool for testing that your package installs correctly with different Python versions.

Although I usually do not have version conflicts, I like to have virtualenv to set up a clean environment and get a clear view of my packages dependencies. That way I never forget any dependencies in my setup.py

If you go for MacPorts be aware that multiple versions of the same package are not selected anymore like the old Debian style with an extra python_select package (it is still there for compatibility). Now you have the select command to choose which Python version will be used (you can even select the Apple installed ones):

$  port select python
Available versions for python:
    none
    python25-apple
    python26-apple
    python27 (active)
    python27-apple
    python32

$ port select python python32

Add tox on top of it and your programs should be really portable

Question 44

Please see Python OS X development environment. The best way is to use MacPorts. Download and install MacPorts, then install Python via MacPorts by typing the following commands in the Terminal:

sudo port install python26 python_select
sudo port select --set python python26

OR

sudo port install python30 python_select
sudo port select --set python python30

Use the first set of commands to install Python 2.6 and the second set to install Python 3.0. Then use:

sudo port install py26-packagename

OR

sudo port install py30-packagename

In the above commands, replace packagename with the name of the package, for example:

sudo port install py26-setuptools

These commands will automatically install the package (and its dependencies) for the given Python version.

For a full list of available packages for Python, type:

port list | grep py26-

OR

port list | grep py30-

Which command you use depends on which version of Python you chose to install.

Question 45

I use MacPorts to install Python and any third-party modules tracked by MacPorts into /opt/local, and I install any manually installed modules (those not in the MacPorts repository) into /usr/local, and this has never caused any problems. I think you may be confused as to the use of certain MacPorts scripts and environment variables.

MacPorts python_select is used to select the “current” version of Python, but it has nothing to do with modules. This allows you to, e.g., install both Python 2.5 and Python 2.6 using MacPorts, and switch between installs.

The $PATH environment variables does not affect what Python modules are loaded. $PYTHONPATH is what you are looking for. $PYTHONPATH should point to directories containing Python modules you want to load. In my case, my $PYTHONPATH variable contains /usr/local/lib/python26/site-packages. If you use MacPorts’ Python, it sets up the other proper directories for you, so you only need to add additional paths to $PYTHONPATH. But again, $PATH isn’t used at all when Python searches for modules you have installed.

$PATH is used to find executables, so if you install MacPorts’ Python, make sure /opt/local/bin is in your $PATH.

Question 46

There’s nothing wrong with using a MacPorts Python installation. If you are installing python modules from MacPorts but then not seeing them, that likely means you are not invoking the MacPorts python you installed to. In a terminal shell, you can use absolute paths to invoke the various Pythons that may be installed. For example:

$ /usr/bin/python2.5         # Apple-supplied 2.5 (Leopard)
$ /opt/local/bin/python2.5   # MacPorts 2.5
$ /opt/local/bin/python2.6   # MacPorts 2.6
$ /usr/local/bin/python2.6   # python.org (MacPython) 2.6
$ /usr/local/bin/python3.1   # python.org (MacPython) 3.1

To get the right python by default requires ensuring your shell $PATH is set properly to ensure that the right executable is found first. Another solution is to define shell aliases to the various pythons.

A python.org (MacPython) installation is fine, too, as others have suggested. easy_install can help but, again, because each Python instance may have its own easy_install command, make sure you are invoking the right easy_install.

Question 47

If you use Python from MacPorts, it has it’s own easy_install located at: /opt/local/bin/easy_install-2.6 (for py26, that is). It’s not the same one as simply calling easy_install directly, even if you used python_select to change your default python command.

Question 48

Have you looked into easy_install at all? It won’t synchronize your macports or anything like that, but it will automatically download the latest package and all necessary dependencies, i.e.

easy_install nose

for the nose unit testing package, or

easy_install trac

for the trac bug tracker.

There’s a bit more information on their EasyInstall page too.

Question 49

For MacPython installations, I found an effective solution to fixing the problem with setuptools (easy_install) in this blog post:

http://droidism.com/getting-running-with-django-and-macpython-26-on-leopard

One handy tip includes finding out which version of python is active in the terminal:

which python

Question 50

When you install modules with MacPorts, it does not go into Apple’s version of Python. Instead those modules are installed onto the MacPorts version of Python selected.

You can change which version of Python is used by default using a mac port called python_select. instructions here.

Also, there’s easy_install. Which will use python to install python modules.

Question 51

You may already have pip3 pre-installed, so just try it!

Question 52

Regarding which python version to use, Mac OS usually ships an old version of python. It’s a good idea to upgrade to a newer version. You can download a .dmg from http://www.python.org/download/ . If you do that, remember to update the path. You can find the exact commands here http://farmdev.com/thoughts/66/python-3-0-on-mac-os-x-alongside-2-6-2-5-etc-/

Question 53

I use easy_install with Apple’s Python, and it works like a charm.

Question 54

Directly install one of the fink packages (Django 1.6 as of 2013-Nov)

fink install django-py27
fink install django-py33

Or create yourself a virtualenv:

fink install virtualenv-py27
virtualenv django-env
source django-env/bin/activate
pip install django
deactivate # when you are done

Or use fink django plus any other pip installed packages in a virtualenv

fink install django-py27
fink install virtualenv-py27
virtualenv django-env --system-site-packages
source django-env/bin/activate
# django already installed
pip install django-analytical # or anything else you might want
deactivate # back to your normally scheduled programming

Question 55

Looking to do a very small, quick ‘n dirty side project. I like the fact that the Google App Engine is running on Python with Django built right in – gives me an excuse to try that platform… but my question is this:

Has anyone made use of the app engine for anything other than a toy problem? I see some good example apps out there, so I would assume this is good enough for the real deal, but wanted to get some feedback.

Any other success/failure notes would be great.

Question 56

I have tried app engine for my small quake watch application http://quakewatch.appspot.com/

My purpose was to see the capabilities of app engine, so here are the main points:

it doesn’t come by default with Django, it has its own web framework which is pythonic has URL dispatcher like Django and it uses Django templates So if you have Django exp. you will find it easy to use
- But you can use any pure python framework and Django can be easily added see http://code.google.com/appengine/articles/django.html google-app-engine-django (http://code.google.com/p/google-app-engine-django/) project is excellent and works almost like working on a Django project
You can not execute any long running process on server, what you do is reply to request and which should be quick otherwise appengine will kill it So if your app needs lots of backend processing appengine is not the best way otherwise you will have to do processing on a server of your own
My quakewatch app has a subscription feature, it means I had to email latest quakes as they happend, but I can not run a background process in app engine to monitor new quakes solution here is to use a third part service like pingablity.com which can connect to one of your page and which executes the subscription emailer but here also you will have to take care that you don’t spend much time here or break task into several pieces
It provides Django like modeling capabilities but backend is totally different but for a new project it should not matter.

But overall I think it is excellent for creating apps which do not need lot of background processing.

Edit: Now task queues can be used for running batch processing or scheduled tasks

Edit: after working/creating a real application on GAE for a year, now my opnion is that unless you are making a application which needs to scale to million and million of users, don’t use GAE. Maintaining and doing trivial tasks in GAE is a headache due to distributed nature, to avoid deadline exceeded errors, count entities or do complex queries requires complex code, so small complex application should stick to LAMP.

Edit: Models should be specially designed considering all the transactions you wish to have in future, because entities only in same entity group can be used in a transaction and it makes the process of updating two different groups a nightmare e.g. transfer money from user1 to user2 in transaction is impossible unless they are in same entity group, but making them same entity group may not be best for frequent update purposes…. read this http://blog.notdot.net/2009/9/Distributed-Transactions-on-App-Engine

Question 57

I am using GAE to host several high-traffic applications. Like on the order of 50-100 req/sec. It is great, I can’t recommend it enough.

My previous experience with web development was with Ruby (Rails/Merb). Learning Python was easy. I didn’t mess with Django or Pylons or any other framework, just started from the GAE examples and built what I needed out of the basic webapp libraries that are provided.

If you’re used to the flexibility of SQL the datastore can take some getting used to. Nothing too traumatic! The biggest adjustment is moving away from JOINs. You have to shed the idea that normalizing is crucial.

Ben

Question 58

One of the compelling reasons I have come across for using Google App Engine is its integration with Google Apps for your domain. Essentially it allows you to create custom, managed web applications that are restricted to the (controlled) logins of your domain.

Most of my experience with this code was building a simple time/task tracking application. The template engine was simple and yet made a multi-page application very approachable. The login/user awareness api is similarly useful. I was able to make a public page/private page paradigm without too much issue. (a user would log in to see the private pages. An anonymous user was only shown the public page.)

I was just getting into the datastore portion of the project when I got pulled away for “real work”.

I was able to accomplish a lot (it still is not done yet) in a very little amount of time. Since I had never used Python before, this was particularly pleasant (both because it was a new language for me, and also because the development was still fast despite the new language). I ran into very little that led me to believe that I wouldn’t be able to accomplish my task. Instead I have a fairly positive impression of the functionality and features.

That is my experience with it. Perhaps it doesn’t represent more than an unfinished toy project, but it does represent an informed trial of the platform, and I hope that helps.

Question 59

The “App Engine running Django” idea is a bit misleading. App Engine replaces the entire Django model layer so be prepared to spend some time getting acclimated with App Engine’s datastore which requires a different way of modeling and thinking about data.

Question 60

I used GAE to build http://www.muspy.com

It’s a bit more than a toy project but not overly complex either. I still depend on a few issues to be addressed by Google, but overall developing the website was an enjoyable experience.

If you don’t want to deal with hosting issues, server administration, etc, I can definitely recommend it. Especially if you already know Python and Django.

Question 61

I think App Engine is pretty cool for small projects at this point. There’s a lot to be said for never having to worry about hosting. The API also pushes you in the direction of building scalable apps, which is good practice.

app-engine-patch is a good layer between Django and App Engine, enabling the use of the auth app and more.
Google have promised an SLA and pricing model by the end of 2008.
Requests must complete in 10 seconds, sub-requests to web services required to complete in 5 seconds. This forces you to design a fast, lightweight application, off-loading serious processing to other platforms (e.g. a hosted service or an EC2 instance).
More languages are coming soon! Google won’t say which though :-). My money’s on Java next.

Question 62

This question has been fully answered. Which is good. But one thing perhaps is worth mentioning. The google app engine has a plugin for the eclipse ide which is a joy to work with.

If you already do your development with eclipse you are going to be so happy about that.

To deploy on the google app engine’s web site all I need to do is click one little button – with the airplane logo – super.

Question 63

Take a look the the sql game, it is very stable and actually pushed traffic limits at one point so that it was getting throttled by Google. I have seen nothing but good news about App Engine, other than hosting you app on servers someone else controls completely.

Question 64

I used GAE to build a simple application which accepts some parameters, formats and send email. It was extremely simple and fast. I also made some performance benchmarks on the GAE datastore and memcache services (http://dbaspects.blogspot.com/2010/01/memcache-vs-datastore-on-google-app.html ). It is not that fast. My opinion is that GAE is serious platform which enforce certain methodology. I think it will evolve to the truly scalable platform, where bad practices simply not allowed.

问题：使用Python 3从Jupyter Notebook中相对导入的另一个目录中的模块导入本地函数

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

映射模块文件所在的路径。就我而言，它是台式机

要么导入整个映射模块，要么然后使用.notation来映射诸如mapping.Shipping（）的类。

或从映射模块导入特定的类

map the path where the module file is located. In my case it was the desktop

Either import the whole mapping module BUT then you have to use the .notation to map the classes like mapping.Shipping()

Or import the specific class from the mapping module

回答 6

问题：Python-获取根项目结构的路径

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

回答 7

回答 8

回答 9

回答 10

项目本地代码

位置无关

Code local to project

Location independent

回答 11

回答 12

回答 13

回答 14

回答 15

问题：Python使用枚举内部列表理解

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

问题：我可以将Python Windows软件包安装到virtualenvs吗？

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

问题：在Mac上安装python模块的最兼容方法是什么？

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

回答 7

回答 8

回答 9

回答 10

回答 11

回答 12

问题：有关使用Google App Engine的反馈？[关闭]

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

回答 7

回答 8

回答 9

回答 10

问题：四肢瘫痪程序员的最佳编程工具

回答 0

回答 1

回答 2