Python 实用宝典

Question 1

I have two branches, Development and Production. Each has dependencies, some of which are different. Development points to dependencies that are themselves in development. Likewise for Production. I need to deploy to Heroku which expects each branch’s dependencies in a single file called ‘requirements.txt’.

What is the best way to organize?

What I’ve thought of:

Maintain separate requirements files, one in each branch (must survive frequent merges!)
Tell Heroku which requirements file I want to use (environment variable?)
Write deploy scripts (create temp branch, modify requirements file, commit, deploy, delete temp branch)

Question 2

You can cascade your requirements files and use the “-r” flag to tell pip to include the contents of one file inside another. You can break out your requirements into a modular folder hierarchy like this:

`-- django_project_root
|-- requirements
|   |-- common.txt
|   |-- dev.txt
|   `-- prod.txt
`-- requirements.txt

The files’ contents would look like this:

common.txt:

# Contains requirements common to all environments
req1==1.0
req2==1.0
req3==1.0
...

dev.txt:

# Specifies only dev-specific requirements
# But imports the common ones too
-r common.txt
dev_req==1.0
...

prod.txt:

# Same for prod...
-r common.txt
prod_req==1.0
...

Outside of Heroku, you can now setup environments like this:

pip install -r requirements/dev.txt

or

pip install -r requirements/prod.txt

Since Heroku looks specifically for “requirements.txt” at the project root, it should just mirror prod, like this:

requirements.txt:

# Mirrors prod
-r requirements/prod.txt

Question 3

A viable option today which didn’t exist when the original question and answer was posted is to use pipenv instead of pip to manage dependencies.

With pipenv, manually managing two separate requirement files like with pip is no longer necessary, and instead pipenv manages the development and production packages itself via interactions on the command line.

To install a package for use in both production and development:

pipenv install <package>

To install a package for the development environment only:

pipenv install <package> --dev

Via those commands, pipenv stores and manages the environment configuration in two files (Pipfile and Pipfile.lock). Heroku’s current Python buildpack natively supports pipenv and will configure itself from Pipfile.lock if it exists instead of requirements.txt.

See the pipenv link for full documentation of the tool.

Question 4

If your requirement is to be able to switch between environments on the same machine, it may be necessary to create different virtualenv folders for each environment you need to switch to.

python3 -m venv venv_dev
source venv_dev/bin/activate
pip install -r pip/common.txt
pip install -r pip/dev.txt
exit
python3 -m venv venv_prod
source venv_prod/bin/activate
pip install -r pip/common.txt
exit
source venv_dev/bin/activate
# now we are in dev environment so your code editor and build systems will work.

# let's install a new dev package:
# pip install awesome
# pip freeze -r pip/temp.txt
# find that package, put it into pip/dev.txt
# rm pip/temp.txt

# pretty cumbersome, but it works.

Question 5

I have an app who’s only dependency is flask, which runs fine outside docker and binds to the default port 5000. Here is the full source:

from flask import Flask

app = Flask(__name__)
app.debug = True

@app.route('/')
def main():
    return 'hi'

if __name__ == '__main__':
    app.run()

The problem is that when I deploy this in docker, the server is running but is unreachable from outside the container.

Below is my Dockerfile. The image is ubuntu with flask installed. The tar just contains the index.py listed above;

# Dockerfile
FROM dreen/flask
MAINTAINER dreen
WORKDIR /srv

# Get source
RUN mkdir -p /srv
COPY perfektimprezy.tar.gz /srv/perfektimprezy.tar.gz
RUN tar x -f perfektimprezy.tar.gz
RUN rm perfektimprezy.tar.gz

# Run server
EXPOSE 5000
CMD ["python", "index.py"]

Here are the steps I am doing to deploy

$> sudo docker build -t perfektimprezy .

As far as I know the above runs fine, the image has the contents of the tar in /srv. Now, let’s start the server in a container:

$> sudo docker run -i -p 5000:5000 -d perfektimprezy
1c50b67d45b1a4feade72276394811c8399b1b95692e0914ee72b103ff54c769

Is it actually running?

$> sudo docker ps
CONTAINER ID        IMAGE                   COMMAND             CREATED             STATUS              PORTS                    NAMES
1c50b67d45b1        perfektimprezy:latest   "python index.py"   5 seconds ago       Up 5 seconds        0.0.0.0:5000->5000/tcp   loving_wozniak

$> sudo docker logs 1c50b67d45b1
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat

Yep, seems like the flask server is running. Here is where it gets weird. Lets make a request to the server:

 $> curl 127.0.0.1:5000 -v
 * Rebuilt URL to: 127.0.0.1:5000/
 * Hostname was NOT found in DNS cache
 *   Trying 127.0.0.1...
 * Connected to 127.0.0.1 (127.0.0.1) port 5000 (#0)
 > GET / HTTP/1.1
 > User-Agent: curl/7.35.0
 > Host: 127.0.0.1:5000
 > Accept: */*
 >
 * Empty reply from server
 * Connection #0 to host 127.0.0.1 left intact
 curl: (52) Empty reply from server

Empty reply… But is the process running?

$> sudo docker top 1c50b67d45b1
UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
root                2084                812                 0                   10:26               ?                   00:00:00            python index.py
root                2117                2084                0                   10:26               ?                   00:00:00            /usr/bin/python index.py

Now let’s ssh into the server and check…

$> sudo docker exec -it 1c50b67d45b1 bash
root@1c50b67d45b1:/srv# netstat -an
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 127.0.0.1:5000          0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:47677         127.0.0.1:5000          TIME_WAIT
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node   Path
root@1c50b67d45b1:/srv# curl -I 127.0.0.1:5000
HTTP/1.0 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 5447
Server: Werkzeug/0.10.4 Python/2.7.6
Date: Tue, 19 May 2015 12:18:14 GMT

It’s fine… but not from the outside :( What am I doing wrong?

Question 6

The problem is you are only binding to the localhost interface, you should be binding to 0.0.0.0 if you want the container to be accessible from outside. If you change:

if __name__ == '__main__':
    app.run()

to

if __name__ == '__main__':
    app.run(host='0.0.0.0')

It should work.

Question 7

When using the flask command instead of app.run, you can pass the --host option to change the host. The line in Docker would be:

CMD ["flask", "run", "--host", "0.0.0.0"]

or

CMD flask run --host 0.0.0.0

Question 8

Your Docker container has more than one network interface. For example, my container has the following:

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
32: eth0@if33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever

if you run docker network inspect bridge, you can see that your container is connected to that bridge with the second interface in the above output. This default bridge is also connected to the Docker process on your host.

Therefore you would have to run the command:

CMD flask run --host 172.17.0.2

To access your Flask app running in a Docker container from your host machine. Replace 172.17.0.2 with whatever the particular IP address is of your container.

Question 9

To build on other answers:

Imagine you have two computers. Each computer has a network interface (WiFi, say), which is its public IP. Each computer has a loopback/localhost interface, at 127.0.0.1. This means “just this computer.”

If you listed on 127.0.0.1 on computer A, you would not expect to be able to connect to that via 127.0.0.1 when running on computer B. After all, you asked to listen on computer A’s local, private address.

Docker is similar setup; technically it’s the same computer, but the Linux kernel is allowing each container to run with its own isolated network stack. So 127.0.0.1 in a container is the same as 127.0.0.1 on a different computer than your host—you can’t connect to it.

Longer version, with diagrams: https://pythonspeed.com/articles/docker-connection-refused/

Question 10

First of all in your python script you need to change code from

app.run()

to

app.run(host="0.0.0.0")

Second, In your docker file, last line should be like

CMD ["flask", "run", "-h", "0.0.0.0", "-p", "5000"]

And on host machine if 0.0.0.0:5000 doesn’t work then you should try with localhost:5000

Note – The CMD command has to be proper. Because CMD command provide defaults for executing container.

Question 11

I’m a java developer/python beginner, and I’m missing my maven features, particularly dependency management and build automation (I mean you don’t build, but how to create a package for deployment?)

Is there a python equivalent to achieve these features?
Note: I use python 2.x

Thanks.

Question 12

Python uses distutils and setuptools for dependency and packaging.

Heres a tutorial which explains basics: http://docs.activestate.com/activepython/3.2/diveintopython3/html/packaging.html

In short, you will have setup.py file, which has dependency and script compilation/installation information, and you can build eggs, dist tarballs, binary tarballs, etc with it.

Question 13

There is no direct match. However, the closest you can get:

zc.buildout: It can setup closed environments, download/handle dependencies, initialize scripts, etc. It also builds on plugins (or “recipes”, as they call them). I used it a few years ago when it was in beta stages, probably it has evolved since then. There is learning curve, as with Maven, but it’s also the most powerful.

Other offerings are subsets of Maven/zc.buildout:

Setuptools: package creation / installation
Pip: dependency management
Virtualenv + virtualenvwrapper: Managing separate python environments (something you don’t need with Java)

You probably know Ant and shell scripting, so you could check also these Python tools:

Fabric or Paver: command-line task runners with added flavors. They wrap your traditional command-line execution in python, and allow to manage various tasks in a more powerful way (task dependencies, interpreting output, running commands in remote server, etc.). Basically nothing you couldn’t do with shell scripting, but in python, it’s much less cryptic.

Question 14

I’d like to point out PyBuilder which is heavily inspired by maven but uses python instead of XML for configuration, so it’s actually readable, IMHO.

There is a plugin for dependency management (uses pip under the hood and differentiates between build and runtime dependencies) and, not unlike maven, you can run through the full build lifecycle with a single command.

Question 15

For deployment, in addition to distutils/setuptoos, also take a look at the pip package (uses setuptools underneath). It can rollback failed installations and also uninstall (something missing from easy_install/setuptools). In addition, you can specify dependencies through a requirements text file.

Question 16

It’s good to use virtualenv to create standalone project environment and use pip/easy_install to management dependencies.

Question 17

Python works on multiple platforms and can be used for desktop and web applications, thus I conclude that there is some way to compile it into an executable for Mac, Windows and Linux.

The problem being I have no idea where to start or how to write a GUI with it, can anybody shed some light on this and point me in the right direction please?

Question 18

First you will need some GUI library with Python bindings and then (if you want) some program that will convert your python scripts into standalone executables.

Cross-platform GUI libraries with Python bindings (Windows, Linux, Mac)

Of course, there are many, but the most popular that I’ve seen in wild are:

Tkinter – based on Tk GUI toolkit (de-facto standard GUI library for python, free for commercial projects)
WxPython – based on WxWidgets (popular, free for commercial projects)
Qt using the PyQt bindings or Qt for Python. The former is not free for commercial projects. The latter is less mature, but can be used for free.

Complete list is at http://wiki.python.org/moin/GuiProgramming

Single executable (all platforms)

PyInstaller – the most active(Could also be used with PyQt)
fbs – if you chose Qt above

Single executable (Windows)

py2exe – used to be the most popular

Single executable (Linux)

Freeze – works the same way like py2exe but targets Linux platform

Single executable (Mac)

py2app – again, works like py2exe but targets Mac OS

Question 19

Another system (not mentioned in the accepted answer yet) is PyInstaller, which worked for a PyQt project of mine when py2exe would not. I found it easier to use.

http://www.pyinstaller.org/

Pyinstaller is based on Gordon McMillan’s Python Installer. Which is no longer available.

Question 20

An alternative tool to py2exe is bbfreeze which generates executables for windows and linux. It’s newer than py2exe and handles eggs quite well. I’ve found it magically works better without configuration for a wide variety of applications.

Question 21

There’s also PyGTK, which is basically a Python wrapper for the Gnome Toolkit. I’ve found it easier to wrap my mind around than Tkinter, coming from pretty much no knowledge of GUI programming previously. It works pretty well and has some good tutorials. Unfortunately there isn’t an installer for Python 2.6 for Windows yet, and may not be for a while.

Question 22

Since python is installed on nearly every non-Windows OS by default now, the only thing you really need to make sure of is that all of the non-standard libraries you use are installed.

Having said that, it is possible to build executables that include the python interpreter, and any libraries you use. This is likely to create a large executable, however.

MacOS X even includes support in the Xcode IDE for creating full standalone GUI apps. These can be run by any user running OS X.

Question 23

For the GUI itself:

PyQT is pretty much the reference.

Another way to develop a rapid user interface is to write a web app, have it run locally and display the app in the browser.

Plus, if you go for the Tkinter option suggested by lubos hasko you may want to try portablepy to have your app run on Windows environment without Python.

Question 24

I’m not sure that this is the best way to do it, but when I’m deploying Ruby GUI apps (not Python, but has the same “problem” as far as .exe’s are concerned) on Windows, I just write a short launcher in C# that calls on my main script. It compiles to an executable, and I then have an application executable.

Question 25

# I'd use tkinter for python 3

import tkinter

tk = tkinter.Tk()
tk.geometry("400x300+500+300")
l = Label(tk,text="")
l.pack()
e = Entry(tk)
e.pack()

def click():
    e['text'] = 'You clicked the button'

b = Button(tk,text="Click me",command=click)
b.pack()

tk.mainloop()

# After this I would you py2exe
# search for the use of this module on stakoverflow
# otherwise I could edit this to let you know how to do it

py2exe

Then you should use py2exe, for example, to bring in one folder all the files needed to run the app, even if the user has not python on his pc (I am talking of windows… for the apple os there is no need of an executable file, I think, as it come with python in it without any need of installing it.

Create this file

1) Create a setup.py

with this code:

from distutils.core import setup
import py2exe

setup(console=['l4h.py'])

save it in a folder

2) Put your program in the same folder of setup.py put in this folder the program you want to make it distribuitable: es: l4h.py

ps: change the name of the file (from l4h to anything you want, that is an example)

3) Run cmd from that folder (on the folder, right click + shift and choose start cmd here)
4) write in cmd:>python setup.py py2exe
5) in the dist folder there are all the files you need
6) you can zip it and distribute it

Pyinstaller

Install it from cmd

**

pip install pyinstaller

**

Run it from the cmd from the folder where the file is

**

pyinstaller file.py

**

Question 26

PySimpleGUI wraps tkinter and works on Python 3 and 2.7. It also runs on Qt, WxPython and in a web browser, using the same source code for all platforms.

You can make custom GUIs that utilize all of the same widgets that you find in tkinter (sliders, checkboxes, radio buttons, …). The code tends to be very compact and readable.

#!/usr/bin/env python
import sys
if sys.version_info[0] >= 3:
    import PySimpleGUI as sg
else:
    import PySimpleGUI27 as sg

layout = [[ sg.Text('My Window') ],
          [ sg.Button('OK')]]

window = sg.Window('My window').Layout(layout)
button, value = window.Read()

As explained in the PySimpleGUI Documentation, to build the .EXE file you run:

pyinstaller -wF MyGUIProgram.py

Question 27

You don’t need to compile python for Mac/Windows/Linux. It is an interpreted language, so you simply need to have the Python interpreter installed on the system of your choice (it is available for all three platforms).

As for a GUI library that works cross platform, Python’s Tk/Tcl widget library works very well, and I believe is sufficiently cross platform.

Tkinter is the python interface to Tk/Tcl

From the python project webpage:

Tkinter is not the only GuiProgramming toolkit for Python. It is however the most commonly used one, and almost the only one that is portable between Unix, Mac and Windows

Question 28

You can use appJar for basic GUI development.

from appJar import gui

num=1

def myfcn(btnName):   
    global num
    num +=1
    win.setLabel("mylabel", num)

win = gui('Test')

win.addButtons(["Set"],  [myfcn])
win.addLabel("mylabel", "Press the Button")

win.go()

See documentation at appJar site.

Installation is made with pip install appjar from command line.

Question 29

!!! KIVY !!!

I was amazed seeing that no one mentioned Kivy!!!

I have once done a project using Tkinter, although they do advocate that it has improved a lot, it still gives me a feel of windows 98, so I switched to Kivy.

I have been following a tutorial series if it helps…

Just to give an idea of how kivy looks, see this (The project I am working on):

And I have been working on it for barely a week now ! The benefits for Kivy you ask? Check this

The reason why I chose this is, its look and that it can be used in mobile as well.

Question 30

What is the recommended way of handling settings for local development and the production server? Some of them (like constants, etc) can be changed/accessed in both, but some of them (like paths to static files) need to remain different, and hence should not be overwritten every time the new code is deployed.

Currently, I am adding all constants to settings.py. But every time I change some constant locally, I have to copy it to the production server and edit the file for production specific changes… :(

Edit: looks like there is no standard answer to this question, I’ve accepted the most popular method.

Question 31

In settings.py:

try:
    from local_settings import *
except ImportError as e:
    pass

You can override what needed in local_settings.py; it should stay out of your version control then. But since you mention copying I’m guessing you use none ;)

Question 32

Two Scoops of Django: Best Practices for Django 1.5 suggests using version control for your settings files and storing the files in a separate directory:

project/
    app1/
    app2/
    project/
        __init__.py
        settings/
            __init__.py
            base.py
            local.py
            production.py
    manage.py

The base.py file contains common settings (such as MEDIA_ROOT or ADMIN), while local.py and production.py have site-specific settings:

In the base file settings/base.py:

INSTALLED_APPS = (
    # common apps...
)

In the local development settings file settings/local.py:

from project.settings.base import *

DEBUG = True
INSTALLED_APPS += (
    'debug_toolbar', # and other apps for local development
)

In the file production settings file settings/production.py:

from project.settings.base import *

DEBUG = False
INSTALLED_APPS += (
    # other apps for production site
)

Then when you run django, you add the --settings option:

# Running django for local development
$ ./manage.py runserver 0:8000 --settings=project.settings.local

# Running django shell on the production site
$ ./manage.py shell --settings=project.settings.production

The authors of the book have also put up a sample project layout template on Github.

Question 33

Instead of settings.py, use this layout:

.
└── settings/
    ├── __init__.py  <= not versioned
    ├── common.py
    ├── dev.py
    └── prod.py

common.py is where most of your configuration lives.

prod.py imports everything from common, and overrides whatever it needs to override:

from __future__ import absolute_import # optional, but I like it
from .common import *

# Production overrides
DEBUG = False
#...

Similarly, dev.py imports everything from common.py and overrides whatever it needs to override.

Finally, __init__.py is where you decide which settings to load, and it’s also where you store secrets (therefore this file should not be versioned):

from __future__ import absolute_import
from .prod import *  # or .dev if you want dev

##### DJANGO SECRETS
SECRET_KEY = '(3gd6shenud@&57...'
DATABASES['default']['PASSWORD'] = 'f9kGH...'

##### OTHER SECRETS
AWS_SECRET_ACCESS_KEY = "h50fH..."

What I like about this solution is:

Everything is in your versioning system, except secrets
Most configuration is in one place: common.py.
Prod-specific things go in prod.py, dev-specific things go in dev.py. It’s simple.
You can override stuff from common.py in prod.py or dev.py, and you can override anything in __init__.py.
It’s straightforward python. No re-import hacks.

Question 34

I use a slightly modified version of the “if DEBUG” style of settings that Harper Shelby posted. Obviously depending on the environment (win/linux/etc.) the code might need to be tweaked a bit.

I was in the past using the “if DEBUG” but I found that occasionally I needed to do testing with DEUBG set to False. What I really wanted to distinguish if the environment was production or development, which gave me the freedom to choose the DEBUG level.

PRODUCTION_SERVERS = ['WEBSERVER1','WEBSERVER2',]
if os.environ['COMPUTERNAME'] in PRODUCTION_SERVERS:
    PRODUCTION = True
else:
    PRODUCTION = False

DEBUG = not PRODUCTION
TEMPLATE_DEBUG = DEBUG

# ...

if PRODUCTION:
    DATABASE_HOST = '192.168.1.1'
else:
    DATABASE_HOST = 'localhost'

I’d still consider this way of settings a work in progress. I haven’t seen any one way to handling Django settings that covered all the bases and at the same time wasn’t a total hassle to setup (I’m not down with the 5x settings files methods).

Question 35

I use a settings_local.py and a settings_production.py. After trying several options I’ve found that it’s easy to waste time with complex solutions when simply having two settings files feels easy and fast.

When you use mod_python/mod_wsgi for your Django project you need to point it to your settings file. If you point it to app/settings_local.py on your local server and app/settings_production.py on your production server then life becomes easy. Just edit the appropriate settings file and restart the server (Django development server will restart automatically).

Question 36

TL;DR: The trick is to modify os.environment before you import settings/base.py in any settings/<purpose>.py, this will greatly simplify things.

Just thinking about all these intertwining files gives me a headache. Combining, importing (sometimes conditionally), overriding, patching of what was already set in case DEBUG setting changed later on. What a nightmare!

Through the years I went through all different solutions. They all somewhat work, but are so painful to manage. WTF! Do we really need all that hassle? We started with just one settings.py file. Now we need a documentation just to correctly combine all these together in a correct order!

I hope I finally hit the (my) sweet spot with the solution below.

Let’s recap the goals (some common, some mine)

Keep secrets a secret — don’t store them in a repo!
Set/read keys and secrets through environment settings, 12 factor style.
Have sensible fallback defaults. Ideally for local development you don’t need anything more beside defaults.
…but try to keep defaults production safe. It’s better to miss a setting override locally, than having to remember to adjust default settings safe for production.
Have the ability to switch DEBUG on/off in a way that can have an effect on other settings (eg. using javascript compressed or not).
Switching between purpose settings, like local/testing/staging/production, should be based only on DJANGO_SETTINGS_MODULE, nothing more.
…but allow further parameterization through environment settings like DATABASE_URL.
…also allow them to use different purpose settings and run them locally side by side, eg. production setup on local developer machine, to access production database or smoke test compressed style sheets.
Fail if an environment variable is not explicitly set (requiring an empty value at minimum), especially in production, eg. EMAIL_HOST_PASSWORD.
Respond to default DJANGO_SETTINGS_MODULE set in manage.py during django-admin startproject
Keep conditionals to a minimum, if the condition is the purposed environment type (eg. for production set log file and it’s rotation), override settings in associated purposed settings file.

Do not’s

Do not let django read DJANGO_SETTINGS_MODULE setting form a file.
Ugh! Think of how meta this is. If you need to have a file (like docker env) read that into the environment before staring up a django process.
Do not override DJANGO_SETTINGS_MODULE in your project/app code, eg. based on hostname or process name.
If you are lazy to set environment variable (like for setup.py test) do it in tooling just before you run your project code.
Avoid magic and patching of how django reads it’s settings, preprocess the settings but do not interfere afterwards.
No complicated logic based nonsense. Configuration should be fixed and materialized not computed on the fly. Providing a fallback defaults is just enough logic here.
Do you really want to debug, why locally you have correct set of settings but in production on a remote server, on one of hundred machines, something computed differently? Oh! Unit tests? For settings? Seriously?

Solution

My strategy consists of excellent django-environ used with ini style files, providing os.environment defaults for local development, some minimal and short settings/<purpose>.py files that have an import settings/base.py AFTER the os.environment was set from an INI file. This effectively give us a kind of settings injection.

The trick here is to modify os.environment before you import settings/base.py.

To see the full example go do the repo: https://github.com/wooyek/django-settings-strategy

.
│   manage.py
├───data
└───website
    ├───settings
    │   │   __init__.py   <-- imports local for compatibility
    │   │   base.py       <-- almost all the settings, reads from proces environment 
    │   │   local.py      <-- a few modifications for local development
    │   │   production.py <-- ideally is empty and everything is in base 
    │   │   testing.py    <-- mimics production with a reasonable exeptions
    │   │   .env          <-- for local use, not kept in repo
    │   __init__.py
    │   urls.py
    │   wsgi.py

settings/.env

A defaults for local development. A secret file, to mostly set required environment variables. Set them to empty values if they are not required in local development. We provide defaults here and not in settings/base.py to fail on any other machine if the’re missing from the environment.

settings/local.py

What happens in here, is loading environment from settings/.env, then importing common settings from settings/base.py. After that we can override a few to ease local development.

import logging
import environ

logging.debug("Settings loading: %s" % __file__)

# This will read missing environment variables from a file
# We wan to do this before loading a base settings as they may depend on environment
environ.Env.read_env(DEBUG='True')

from .base import *

ALLOWED_HOSTS += [
    '127.0.0.1',
    'localhost',
    '.example.com',
    'vagrant',
    ]

# https://docs.djangoproject.com/en/1.6/topics/email/#console-backend
EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend'
# EMAIL_BACKEND = 'django.core.mail.backends.dummy.EmailBackend'

LOGGING['handlers']['mail_admins']['email_backend'] = 'django.core.mail.backends.dummy.EmailBackend'

# Sync task testing
# http://docs.celeryproject.org/en/2.5/configuration.html?highlight=celery_always_eager#celery-always-eager

CELERY_ALWAYS_EAGER = True
CELERY_EAGER_PROPAGATES_EXCEPTIONS = True

settings/production.py

For production we should not expect an environment file, but it’s easier to have one if we’re testing something. But anyway, lest’s provide few defaults inline, so settings/base.py can respond accordingly.

environ.Env.read_env(Path(__file__) / "production.env", DEBUG='False', ASSETS_DEBUG='False')
from .base import *

The main point of interest here are DEBUG and ASSETS_DEBUG overrides, they will be applied to the python os.environ ONLY if they are MISSING from the environment and the file.

These will be our production defaults, no need to put them in the environment or file, but they can be overridden if needed. Neat!

settings/base.py

These are your mostly vanilla django settings, with a few conditionals and lot’s of reading them from the environment. Almost everything is in here, keeping all the purposed environments consistent and as similar as possible.

The main differences are below (I hope these are self explanatory):

import environ

# https://github.com/joke2k/django-environ
env = environ.Env()

# Build paths inside the project like this: os.path.join(BASE_DIR, ...)
BASE_DIR = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__))))

# Where BASE_DIR is a django source root, ROOT_DIR is a whole project root
# It may differ BASE_DIR for eg. when your django project code is in `src` folder
# This may help to separate python modules and *django apps* from other stuff
# like documentation, fixtures, docker settings
ROOT_DIR = BASE_DIR

# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/1.11/howto/deployment/checklist/

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = env('SECRET_KEY')

# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = env('DEBUG', default=False)

INTERNAL_IPS = [
    '127.0.0.1',
]

ALLOWED_HOSTS = []

if 'ALLOWED_HOSTS' in os.environ:
    hosts = os.environ['ALLOWED_HOSTS'].split(" ")
    BASE_URL = "https://" + hosts[0]
    for host in hosts:
        host = host.strip()
        if host:
            ALLOWED_HOSTS.append(host)

SECURE_SSL_REDIRECT = env.bool('SECURE_SSL_REDIRECT', default=False)

# Database
# https://docs.djangoproject.com/en/1.11/ref/settings/#databases

if "DATABASE_URL" in os.environ:  # pragma: no cover
    # Enable database config through environment
    DATABASES = {
        # Raises ImproperlyConfigured exception if DATABASE_URL not in os.environ
        'default': env.db(),
    }

    # Make sure we use have all settings we need
    # DATABASES['default']['ENGINE'] = 'django.contrib.gis.db.backends.postgis'
    DATABASES['default']['TEST'] = {'NAME': os.environ.get("DATABASE_TEST_NAME", None)}
    DATABASES['default']['OPTIONS'] = {
        'options': '-c search_path=gis,public,pg_catalog',
        'sslmode': 'require',
    }
else:
    DATABASES = {
        'default': {
            'ENGINE': 'django.db.backends.sqlite3',
            # 'ENGINE': 'django.contrib.gis.db.backends.spatialite',
            'NAME': os.path.join(ROOT_DIR, 'data', 'db.dev.sqlite3'),
            'TEST': {
                'NAME': os.path.join(ROOT_DIR, 'data', 'db.test.sqlite3'),
            }
        }
    }

STATIC_ROOT = os.path.join(ROOT_DIR, 'static')

# django-assets
# http://django-assets.readthedocs.org/en/latest/settings.html

ASSETS_LOAD_PATH = STATIC_ROOT
ASSETS_ROOT = os.path.join(ROOT_DIR, 'assets', "compressed")
ASSETS_DEBUG = env('ASSETS_DEBUG', default=DEBUG)  # Disable when testing compressed file in DEBUG mode
if ASSETS_DEBUG:
    ASSETS_URL = STATIC_URL
    ASSETS_MANIFEST = "json:{}".format(os.path.join(ASSETS_ROOT, "manifest.json"))
else:
    ASSETS_URL = STATIC_URL + "assets/compressed/"
    ASSETS_MANIFEST = "json:{}".format(os.path.join(STATIC_ROOT, 'assets', "compressed", "manifest.json"))
ASSETS_AUTO_BUILD = ASSETS_DEBUG
ASSETS_MODULES = ('website.assets',)

The last bit shows the power here. ASSETS_DEBUG has a sensible default, which can be overridden in settings/production.py and even that that can be overridden by an environment setting! Yay!

In effect we have a mixed hierarchy of importance:

settings/.py – sets defaults based on purpose, does not store secrets
settings/base.py – is mostly controlled by environment
process environment settings – 12 factor baby!
settings/.env – local defaults for easy startup

Question 37

I manage my configurations with the help of django-split-settings.

It is a drop-in replacement for the default settings. It is simple, yet configurable. And refactoring of your exisitng settings is not required.

Here’s a small example (file example/settings/__init__.py):

from split_settings.tools import optional, include
import os

if os.environ['DJANGO_SETTINGS_MODULE'] == 'example.settings':
    include(
        'components/default.py',
        'components/database.py',
        # This file may be missing:
        optional('local_settings.py'),

        scope=globals()
    )

That’s it.

Update

I wrote a blog post about managing django‘s settings with django-split-sttings. Have a look!

Question 38

The problem with most of these solutions is that you either have your local settings applied before the common ones, or after them.

So it’s impossible to override things like

the env-specific settings define the addresses for the memcached pool, and in the main settings file this value is used to configure the cache backend
the env-specific settings add or remove apps/middleware to the default one

at the same time.

One solution can be implemented using “ini”-style config files with the ConfigParser class. It supports multiple files, lazy string interpolation, default values and a lot of other goodies. Once a number of files have been loaded, more files can be loaded and their values will override the previous ones, if any.

You load one or more config files, depending on the machine address, environment variables and even values in previously loaded config files. Then you just use the parsed values to populate the settings.

One strategy I have successfully used has been:

Load a default defaults.ini file
Check the machine name, and load all files which matched the reversed FQDN, from the shortest match to the longest match (so, I loaded net.ini, then net.domain.ini, then net.domain.webserver01.ini, each one possibly overriding values of the previous). This account also for developers’ machines, so each one could set up its preferred database driver, etc. for local development
Check if there is a “cluster name” declared, and in that case load cluster.cluster_name.ini, which can define things like database and cache IPs

As an example of something you can achieve with this, you can define a “subdomain” value per-env, which is then used in the default settings (as hostname: %(subdomain).whatever.net) to define all the necessary hostnames and cookie things django needs to work.

This is as DRY I could get, most (existing) files had just 3 or 4 settings. On top of this I had to manage customer configuration, so an additional set of configuration files (with things like database names, users and passwords, assigned subdomain etc) existed, one or more per customer.

One can scale this as low or as high as necessary, you just put in the config file the keys you want to configure per-environment, and once there’s need for a new config, put the previous value in the default config, and override it where necessary.

This system has proven reliable and works well with version control. It has been used for long time managing two separate clusters of applications (15 or more separate instances of the django site per machine), with more than 50 customers, where the clusters were changing size and members depending on the mood of the sysadmin…

Question 39

I am also working with Laravel and I like the implementation there. I tried to mimic it and combining it with the solution proposed by T. Stone (look above):

PRODUCTION_SERVERS = ['*.webfaction.com','*.whatever.com',]

def check_env():
    for item in PRODUCTION_SERVERS:
        match = re.match(r"(^." + item + "$)", socket.gethostname())
        if match:
            return True

if check_env():
    PRODUCTION = True
else:
    PRODUCTION = False

DEBUG = not PRODUCTION

Maybe something like this would help you.

Question 40

Remember that settings.py is a live code file. Assuming that you don’t have DEBUG set on production (which is a best practice), you can do something like:

if DEBUG:
    STATIC_PATH = /path/to/dev/files
else:
    STATIC_PATH = /path/to/production/files

Pretty basic, but you could, in theory, go up to any level of complexity based on just the value of DEBUG – or any other variable or code check you wanted to use.

Question 41

For most of my projects I use following pattern:

Create settings_base.py where I store settings that are common for all environments
Whenever I need to use new environment with specific requirements I create new settings file (eg. settings_local.py) which inherits contents of settings_base.py and overrides/adds proper settings variables (from settings_base import *)

(To run manage.py with custom settings file you simply use –settings command option: manage.py <command> --settings=settings_you_wish_to_use.py)

Question 42

My solution to that problem is also somewhat of a mix of some solutions already stated here:

I keep a file called local_settings.py that has the content USING_LOCAL = True in dev and USING_LOCAL = False in prod
In settings.py I do an import on that file to get the USING_LOCAL setting

I then base all my environment-dependent settings on that one:

DEBUG = USING_LOCAL
if USING_LOCAL:
    # dev database settings
else:
    # prod database settings

I prefer this to having two separate settings.py files that I need to maintain as I can keep my settings structured in a single file easier than having them spread across several files. Like this, when I update a setting I don’t forget to do it for both environments.

Of course that every method has its disadvantages and this one is no exception. The problem here is that I can’t overwrite the local_settings.py file whenever I push my changes into production, meaning I can’t just copy all files blindly, but that’s something I can live with.

Question 43

I use a variation of what jpartogi mentioned above, that I find a little shorter:

import platform
from django.core.management import execute_manager 

computername = platform.node()

try:
  settings = __import__(computername + '_settings')
except ImportError: 
  import sys
  sys.stderr.write("Error: Can't find the file '%r_settings.py' in the directory containing %r. It appears you've customized things.\nYou'll have to run django-admin.py, passing it your settings module.\n(If the file local_settings.py does indeed exist, it's causing an ImportError somehow.)\n" % (computername, __file__))
  sys.exit(1)

if __name__ == "__main__":
  execute_manager(settings)

Basically on each computer (development or production) I have the appropriate hostname_settings.py file that gets dynamically loaded.

Question 44

There is also Django Classy Settings. I personally am a big fan of it. It’s built by one of the most active people on the Django IRC. You would use environment vars to set things.

http://django-classy-settings.readthedocs.io/en/latest/

Question 45

1 – Create a new folder inside your app and name settings to it.

2 – Now create a new __init__.py file in it and inside it write

from .base import *

try:
    from .local import *
except:
    pass

try:
    from .production import *
except:
    pass

3 – Create three new files in the settings folder name local.py and production.py and base.py.

4 – Inside base.py, copy all the content of previous settings.py folder and rename it with something different, let’s say old_settings.py.

5 – In base.py change your BASE_DIR path to point to your new path of setting

Old path-> BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))

New path -> BASE_DIR = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))

This way, the project dir can be structured and can be manageable among production and local development.

Question 46

In order to use different settings configuration on different environment, create different settings file. And in your deployment script, start the server using --settings=<my-settings.py> parameter, via which you can use different settings on different environment.

Benefits of using this approach:

Your settings will be modular based on each environment
You may import the master_settings.py containing the base configuration in the environmnet_configuration.py and override the values that you want to change in that environment.
If you have huge team, each developer may have their own local_settings.py which they can add to the code repository without any risk of modifying the server configuration. You can add these local settings to .gitnore if you use git or .hginore if you Mercurial for Version Control (or any other). That way local settings won’t even be the part of actual code base keeping it clean.

Question 47

I had my settings split as follows

settings/
     |
     |- base.py
     |- dev.py
     |- prod.py

We have 3 environments

dev
staging
production

Now obviously staging and production should have the maximum possible similar environment. So we kept prod.py for both.

But there was a case where I had to identify running server is a production server. @T. Stone ‘s answer helped me write check as follows.

from socket import gethostname, gethostbyname  
PROD_HOSTS = ["webserver1", "webserver2"]

DEBUG = False
ALLOWED_HOSTS = [gethostname(), gethostbyname(gethostname()),]


if any(host in PROD_HOSTS for host in ALLOWED_HOSTS):
    SESSION_COOKIE_SECURE = True
    CSRF_COOKIE_SECURE = True

Question 48

I differentiate it in manage.py and created two separate settings file: local_settings.py and prod_settings.py.

In manage.py I check whether the server is local server or production server. If it is a local server it would load up local_settings.py and it is a production server it would load up prod_settings.py. Basically this is how it would look like:

#!/usr/bin/env python
import sys
import socket
from django.core.management import execute_manager 

ipaddress = socket.gethostbyname( socket.gethostname() )
if ipaddress == '127.0.0.1':
    try:
        import local_settings # Assumed to be in the same directory.
        settings = local_settings
    except ImportError:
        import sys
        sys.stderr.write("Error: Can't find the file 'local_settings.py' in the directory containing %r. It appears you've customized things.\nYou'll have to run django-admin.py, passing it your settings module.\n(If the file local_settings.py does indeed exist, it's causing an ImportError somehow.)\n" % __file__)
        sys.exit(1)
else:
    try:
        import prod_settings # Assumed to be in the same directory.
        settings = prod_settings    
    except ImportError:
        import sys
        sys.stderr.write("Error: Can't find the file 'prod_settings.py' in the directory containing %r. It appears you've customized things.\nYou'll have to run django-admin.py, passing it your settings module.\n(If the file prod_settings.py does indeed exist, it's causing an ImportError somehow.)\n" % __file__)
        sys.exit(1)

if __name__ == "__main__":
    execute_manager(settings)

I found it to be easier to separate the settings file into two separate file instead of doing lots of ifs inside the settings file.

Question 49

As an alternative to maintain different file if you wiil: If you are using git or any other VCS to push codes from local to server, what you can do is add the settings file to .gitignore.

This will allow you to have different content in both places without any problem. SO on server you can configure an independent version of settings.py and any changes made on the local wont reflect on server and vice versa.

In addition, it will remove the settings.py file from github also, the big fault, which i have seen many newbies doing.

Question 50

Making multiple versions of settings.py is an anti pattern for 12 Factor App methodology. use python-decouple or django-environ instead.

Question 51

I think the best solution is suggested by @T. Stone, but I don’t know why just don’t use the DEBUG flag in Django. I Write the below code for my website:

if DEBUG:
    from .local_settings import *

Always the simple solutions are better than complex ones.

Question 52

I found the responses here very helpful. (Has this been more definitively solved? The last response was a year ago.) After considering all the approaches listed, I came up with a solution that I didn’t see listed here.

My criteria were:

Everything should be in source control. I don’t like fiddly bits lying around.
Ideally, keep settings in one file. I forget things if I’m not looking right at them :)
No manual edits to deploy. Should be able to test/push/deploy with a single fabric command.
Avoid leaking development settings into production.
Keep as close as possible to “standard” (*cough*) Django layout as possible.

I thought switching on the host machine made some sense, but then figured the real issue here is different settings for different environments, and had an aha moment. I put this code at the end of my settings.py file:

try:
    os.environ['DJANGO_DEVELOPMENT_SERVER'] # throws error if unset
    DEBUG = True
    TEMPLATE_DEBUG = True
    # This is naive but possible. Could also redeclare full app set to control ordering. 
    # Note that it requires a list rather than the generated tuple.
    INSTALLED_APPS.extend([
        'debug_toolbar',
        'django_nose',
    ])
    # Production database settings, alternate static/media paths, etc...
except KeyError: 
    print 'DJANGO_DEVELOPMENT_SERVER environment var not set; using production settings'

This way, the app defaults to production settings, which means you are explicitly “whitelisting” your development environment. It is much safer to forget to set the environment variable locally than if it were the other way around and you forgot to set something in production and let some dev settings be used.

When developing locally, either from the shell or in a .bash_profile or wherever:

$ export DJANGO_DEVELOPMENT_SERVER=yep

(Or if you’re developing on Windows, set via the Control Panel or whatever its called these days… Windows always made it so obscure that you could set environment variables.)

With this approach, the dev settings are all in one (standard) place, and simply override the production ones where needed. Any mucking around with development settings should be completely safe to commit to source control with no impact on production.

问题：如何为多个环境自定义requirements.txt？

回答 0

回答 1

回答 2

问题：在Docker中部署最小化Flask应用-服务器连接问题

回答 0

回答 1

回答 2

回答 3

回答 4

问题：相当于python的Maven

回答 0

回答 1

回答 2

回答 3

回答 4

问题：如何使用Python创建可直接执行的跨平台GUI应用程序？

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

回答 7

py2exe

建立这个档案

1）创建一个setup.py

将其保存在文件夹中

ps：更改文件名（从l4h更改为您想要的任何名称，这是一个示例）

py安装程序

从cmd安装

从文件所在的文件夹中的cmd运行它

py2exe

Create this file

1) Create a setup.py

save it in a folder

ps: change the name of the file (from l4h to anything you want, that is an example)

Pyinstaller

Install it from cmd

Run it from the cmd from the folder where the file is

回答 8

回答 9

回答 10

回答 11

问题：如何在Django中管理本地和生产设置？

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

让我们回顾一下目标（一些普通的，一些我的）

不

解

设置/.env

设置/ local.py

settings / production.py

设置/ base.py

Let’s recap the goals (some common, some mine)

Do not’s

Solution

settings/.env

settings/local.py

settings/production.py

settings/base.py

回答 6

更新资料

Update

回答 7

回答 8

回答 9

回答 10

回答 11

回答 12

回答 13

回答 14

回答 15

回答 16

回答 17