Python 实用宝典

Question 1

My Dockerfile is something like

FROM my/base

ADD . /srv
RUN pip install -r requirements.txt
RUN python setup.py install

ENTRYPOINT ["run_server"]

Every time I build a new image, dependencies have to be reinstalled, which could be very slow in my region.

One way I think of to cache packages that have been installed is to override the my/base image with newer images like this:

docker build -t new_image_1 .
docker tag new_image_1 my/base

So next time I build with this Dockerfile, my/base already has some packages installed.

But this solution has two problems:

It is not always possible to override a base image
The base image grow bigger and bigger as newer images are layered on it

So what better solution could I use to solve this problem?

EDIT##:

Some information about the docker on my machine:

☁  test  docker version
Client version: 1.1.2
Client API version: 1.13
Go version (client): go1.2.1
Git commit (client): d84a070
Server version: 1.1.2
Server API version: 1.13
Go version (server): go1.2.1
Git commit (server): d84a070
☁  test  docker info
Containers: 0
Images: 56
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Dirs: 56
Execution Driver: native-0.2
Kernel Version: 3.13.0-29-generic
WARNING: No swap limit support

Question 2

Try to build a Dockerfile which looks something like this:

FROM my/base

WORKDIR /srv
ADD ./requirements.txt /srv/requirements.txt
RUN pip install -r requirements.txt
ADD . /srv
RUN python setup.py install

ENTRYPOINT ["run_server"]

Docker will use cache during pip install as long as you do not make any changes to the requirements.txt, irrespective of the fact whether other code files at . were changed or not. Here’s an example.

Here’s a simple Hello, World! program:

$ tree
.
├── Dockerfile
├── requirements.txt
└── run.py   

0 directories, 3 file

# Dockerfile

FROM dockerfile/python
WORKDIR /srv
ADD ./requirements.txt /srv/requirements.txt
RUN pip install -r requirements.txt
ADD . /srv
CMD python /srv/run.py

# requirements.txt
pytest==2.3.4

# run.py
print("Hello, World")

The output of docker build:

Step 1 : WORKDIR /srv
---> Running in 22d725d22e10
---> 55768a00fd94
Removing intermediate container 22d725d22e10
Step 2 : ADD ./requirements.txt /srv/requirements.txt
---> 968a7c3a4483
Removing intermediate container 5f4e01f290fd
Step 3 : RUN pip install -r requirements.txt
---> Running in 08188205e92b
Downloading/unpacking pytest==2.3.4 (from -r requirements.txt (line 1))
  Running setup.py (path:/tmp/pip_build_root/pytest/setup.py) egg_info for package pytest
....
Cleaning up...
---> bf5c154b87c9
Removing intermediate container 08188205e92b
Step 4 : ADD . /srv
---> 3002a3a67e72
Removing intermediate container 83defd1851d0
Step 5 : CMD python /srv/run.py
---> Running in 11e69b887341
---> 5c0e7e3726d6
Removing intermediate container 11e69b887341
Successfully built 5c0e7e3726d6

Let’s modify run.py:

# run.py
print("Hello, Python")

Try to build again, below is the output:

Sending build context to Docker daemon  5.12 kB
Sending build context to Docker daemon 
Step 0 : FROM dockerfile/python
---> f86d6993fc7b
Step 1 : WORKDIR /srv
---> Using cache
---> 55768a00fd94
Step 2 : ADD ./requirements.txt /srv/requirements.txt
---> Using cache
---> 968a7c3a4483
Step 3 : RUN pip install -r requirements.txt
---> Using cache
---> bf5c154b87c9
Step 4 : ADD . /srv
---> 9cc7508034d6
Removing intermediate container 0d7cf71eb05e
Step 5 : CMD python /srv/run.py
---> Running in f25c21135010
---> 4ffab7bc66c7
Removing intermediate container f25c21135010
Successfully built 4ffab7bc66c7

As you can see above, this time docker uses cache during the build. Now, let’s update requirements.txt:

# requirements.txt

pytest==2.3.4
ipython

Below is the output of docker build:

Sending build context to Docker daemon  5.12 kB
Sending build context to Docker daemon 
Step 0 : FROM dockerfile/python
---> f86d6993fc7b
Step 1 : WORKDIR /srv
---> Using cache
---> 55768a00fd94
Step 2 : ADD ./requirements.txt /srv/requirements.txt
---> b6c19f0643b5
Removing intermediate container a4d9cb37dff0
Step 3 : RUN pip install -r requirements.txt
---> Running in 4b7a85a64c33
Downloading/unpacking pytest==2.3.4 (from -r requirements.txt (line 1))
  Running setup.py (path:/tmp/pip_build_root/pytest/setup.py) egg_info for package pytest

Downloading/unpacking ipython (from -r requirements.txt (line 2))
Downloading/unpacking py>=1.4.12 (from pytest==2.3.4->-r requirements.txt (line 1))
  Running setup.py (path:/tmp/pip_build_root/py/setup.py) egg_info for package py

Installing collected packages: pytest, ipython, py
  Running setup.py install for pytest

Installing py.test script to /usr/local/bin
Installing py.test-2.7 script to /usr/local/bin
  Running setup.py install for py

Successfully installed pytest ipython py
Cleaning up...
---> 23a1af3df8ed
Removing intermediate container 4b7a85a64c33
Step 4 : ADD . /srv
---> d8ae270eca35
Removing intermediate container 7f003ebc3179
Step 5 : CMD python /srv/run.py
---> Running in 510359cf9e12
---> e42fc9121a77
Removing intermediate container 510359cf9e12
Successfully built e42fc9121a77

Notice how docker didn’t use cache during pip install. If it doesn’t work, check your docker version.

Client version: 1.1.2
Client API version: 1.13
Go version (client): go1.2.1
Git commit (client): d84a070
Server version: 1.1.2
Server API version: 1.13
Go version (server): go1.2.1
Git commit (server): d84a070

Question 3

To minimise the network activity, you could point pip to a cache directory on your host machine.

Run your docker container with your host’s pip cache directory bind mounted into your container’s pip cache directory. docker run command should look like this:

docker run -v $HOME/.cache/pip-docker/:/root/.cache/pip image_1

Then in your Dockerfile install your requirements as a part of ENTRYPOINT statement (or CMD statement) instead of as a RUN command. This is important, because (as pointed out in comments) the mount is not available during image building (when RUN statements are executed). Docker file should look like this:

FROM my/base

ADD . /srv

ENTRYPOINT ["sh", "-c", "pip install -r requirements.txt && python setup.py install && run_server"]

Question 4

I understand this question has some popular answers already. But there is a newer way to cache files for package managers. I think it could be a good answer in the future when BuildKit becomes more standard.

As of Docker 18.09 there is experimental support for BuildKit. BuildKit adds support for some new features in the Dockerfile including experimental support for mounting external volumes into RUN steps. This allows us to create caches for things like $HOME/.cache/pip/.

We’ll use the following requirements.txt file as an example:

Click==7.0
Django==2.2.3
django-appconf==1.0.3
django-compressor==2.3
django-debug-toolbar==2.0
django-filter==2.2.0
django-reversion==3.0.4
django-rq==2.1.0
pytz==2019.1
rcssmin==1.0.6
redis==3.3.4
rjsmin==1.1.0
rq==1.1.0
six==1.12.0
sqlparse==0.3.0

A typical example Python Dockerfile might look like:

FROM python:3.7
WORKDIR /usr/src/app
COPY requirements.txt /usr/src/app/
RUN pip install -r requirements.txt
COPY . /usr/src/app

With BuildKit enabled using the DOCKER_BUILDKIT environment variable we can build the uncached pip step in about 65 seconds:

$ export DOCKER_BUILDKIT=1
$ docker build -t test .
[+] Building 65.6s (10/10) FINISHED                                                                                                                                             
 => [internal] load .dockerignore                                                                                                                                          0.0s
 => => transferring context: 2B                                                                                                                                            0.0s
 => [internal] load build definition from Dockerfile                                                                                                                       0.0s
 => => transferring dockerfile: 120B                                                                                                                                       0.0s
 => [internal] load metadata for docker.io/library/python:3.7                                                                                                              0.5s
 => CACHED [1/4] FROM docker.io/library/python:3.7@sha256:6eaf19442c358afc24834a6b17a3728a45c129de7703d8583392a138ecbdb092                                                 0.0s
 => [internal] load build context                                                                                                                                          0.6s
 => => transferring context: 899.99kB                                                                                                                                      0.6s
 => CACHED [internal] helper image for file operations                                                                                                                     0.0s
 => [2/4] COPY requirements.txt /usr/src/app/                                                                                                                              0.5s
 => [3/4] RUN pip install -r requirements.txt                                                                                                                             61.3s
 => [4/4] COPY . /usr/src/app                                                                                                                                              1.3s
 => exporting to image                                                                                                                                                     1.2s
 => => exporting layers                                                                                                                                                    1.2s
 => => writing image sha256:d66a2720e81530029bf1c2cb98fb3aee0cffc2f4ea2aa2a0760a30fb718d7f83                                                                               0.0s
 => => naming to docker.io/library/test                                                                                                                                    0.0s

Now, let us add the experimental header and modify the RUN step to cache the Python packages:

# syntax=docker/dockerfile:experimental

FROM python:3.7
WORKDIR /usr/src/app
COPY requirements.txt /usr/src/app/
RUN --mount=type=cache,target=/root/.cache/pip pip install -r requirements.txt
COPY . /usr/src/app

Go ahead and do another build now. It should take the same amount of time. But this time it is caching the Python packages in our new cache mount:

$ docker build -t pythontest .
[+] Building 60.3s (14/14) FINISHED                                                                                                                                             
 => [internal] load build definition from Dockerfile                                                                                                                       0.0s
 => => transferring dockerfile: 120B                                                                                                                                       0.0s
 => [internal] load .dockerignore                                                                                                                                          0.0s
 => => transferring context: 2B                                                                                                                                            0.0s
 => resolve image config for docker.io/docker/dockerfile:experimental                                                                                                      0.5s
 => CACHED docker-image://docker.io/docker/dockerfile:experimental@sha256:9022e911101f01b2854c7a4b2c77f524b998891941da55208e71c0335e6e82c3                                 0.0s
 => [internal] load .dockerignore                                                                                                                                          0.0s
 => [internal] load build definition from Dockerfile                                                                                                                       0.0s
 => => transferring dockerfile: 120B                                                                                                                                       0.0s
 => [internal] load metadata for docker.io/library/python:3.7                                                                                                              0.5s
 => CACHED [1/4] FROM docker.io/library/python:3.7@sha256:6eaf19442c358afc24834a6b17a3728a45c129de7703d8583392a138ecbdb092                                                 0.0s
 => [internal] load build context                                                                                                                                          0.7s
 => => transferring context: 899.99kB                                                                                                                                      0.6s
 => CACHED [internal] helper image for file operations                                                                                                                     0.0s
 => [2/4] COPY requirements.txt /usr/src/app/                                                                                                                              0.6s
 => [3/4] RUN --mount=type=cache,target=/root/.cache/pip pip install -r requirements.txt                                                                                  53.3s
 => [4/4] COPY . /usr/src/app                                                                                                                                              2.6s
 => exporting to image                                                                                                                                                     1.2s
 => => exporting layers                                                                                                                                                    1.2s
 => => writing image sha256:0b035548712c1c9e1c80d4a86169c5c1f9e94437e124ea09e90aea82f45c2afc                                                                               0.0s
 => => naming to docker.io/library/test                                                                                                                                    0.0s

About 60 seconds. Similar to our first build.

Make a small change to the requirements.txt (such as adding a new line between two packages) to force a cache invalidation and run again:

$ docker build -t pythontest .
[+] Building 15.9s (14/14) FINISHED                                                                                                                                             
 => [internal] load build definition from Dockerfile                                                                                                                       0.0s
 => => transferring dockerfile: 120B                                                                                                                                       0.0s
 => [internal] load .dockerignore                                                                                                                                          0.0s
 => => transferring context: 2B                                                                                                                                            0.0s
 => resolve image config for docker.io/docker/dockerfile:experimental                                                                                                      1.1s
 => CACHED docker-image://docker.io/docker/dockerfile:experimental@sha256:9022e911101f01b2854c7a4b2c77f524b998891941da55208e71c0335e6e82c3                                 0.0s
 => [internal] load build definition from Dockerfile                                                                                                                       0.0s
 => => transferring dockerfile: 120B                                                                                                                                       0.0s
 => [internal] load .dockerignore                                                                                                                                          0.0s
 => [internal] load metadata for docker.io/library/python:3.7                                                                                                              0.5s
 => CACHED [1/4] FROM docker.io/library/python:3.7@sha256:6eaf19442c358afc24834a6b17a3728a45c129de7703d8583392a138ecbdb092                                                 0.0s
 => CACHED [internal] helper image for file operations                                                                                                                     0.0s
 => [internal] load build context                                                                                                                                          0.7s
 => => transferring context: 899.99kB                                                                                                                                      0.7s
 => [2/4] COPY requirements.txt /usr/src/app/                                                                                                                              0.6s
 => [3/4] RUN --mount=type=cache,target=/root/.cache/pip pip install -r requirements.txt                                                                                   8.8s
 => [4/4] COPY . /usr/src/app                                                                                                                                              2.1s
 => exporting to image                                                                                                                                                     1.1s
 => => exporting layers                                                                                                                                                    1.1s
 => => writing image sha256:fc84cd45482a70e8de48bfd6489e5421532c2dd02aaa3e1e49a290a3dfb9df7c                                                                               0.0s
 => => naming to docker.io/library/test                                                                                                                                    0.0s

Only about 16 seconds!

We are getting this speedup because we are no longer downloading all the Python packages. They were cached by the package manager (pip in this case) and stored in a cache volume mount. The volume mount is provided to the run step so that pip can reuse our already downloaded packages. This happens outside any Docker layer caching.

The gains should be much better on larger requirements.txt.

Notes:

This is experimental Dockerfile syntax and should be treated as such. You may not want to build with this in production at the moment.
~~The BuildKit stuff doesn’t work under Docker Compose or other tools that directly use the Docker API at the moment.~~ There is now support for this in Docker Compose as of 1.25.0. See How do you enable BuildKit with docker-compose?
There isn’t any direct interface for managed the cache at the moment. It is purged when you do a docker system prune -a.

Hopefully, these features will make it into Docker for building and BuildKit will become the default. If / when that happens I will try to update this answer.

Question 5

I found that a better way is to just add the Python site-packages directory as a volume.

services:
    web:
        build: .
        command: python manage.py runserver 0.0.0.0:8000
        volumes:
            - .:/code
            -  /usr/local/lib/python2.7/site-packages/

This way I can just pip install new libraries without having to do a full rebuild.

EDIT: Disregard this answer, jkukul’s answer above worked for me. My intent was to cache the site-packages folder. That would have looked something more like:

volumes:
   - .:/code
   - ./cached-packages:/usr/local/lib/python2.7/site-packages/

Caching the download folder is alot cleaner though. That also caches the wheels, so it properly achieves the task.

Question 6

I have an app who’s only dependency is flask, which runs fine outside docker and binds to the default port 5000. Here is the full source:

from flask import Flask

app = Flask(__name__)
app.debug = True

@app.route('/')
def main():
    return 'hi'

if __name__ == '__main__':
    app.run()

The problem is that when I deploy this in docker, the server is running but is unreachable from outside the container.

Below is my Dockerfile. The image is ubuntu with flask installed. The tar just contains the index.py listed above;

# Dockerfile
FROM dreen/flask
MAINTAINER dreen
WORKDIR /srv

# Get source
RUN mkdir -p /srv
COPY perfektimprezy.tar.gz /srv/perfektimprezy.tar.gz
RUN tar x -f perfektimprezy.tar.gz
RUN rm perfektimprezy.tar.gz

# Run server
EXPOSE 5000
CMD ["python", "index.py"]

Here are the steps I am doing to deploy

$> sudo docker build -t perfektimprezy .

As far as I know the above runs fine, the image has the contents of the tar in /srv. Now, let’s start the server in a container:

$> sudo docker run -i -p 5000:5000 -d perfektimprezy
1c50b67d45b1a4feade72276394811c8399b1b95692e0914ee72b103ff54c769

Is it actually running?

$> sudo docker ps
CONTAINER ID        IMAGE                   COMMAND             CREATED             STATUS              PORTS                    NAMES
1c50b67d45b1        perfektimprezy:latest   "python index.py"   5 seconds ago       Up 5 seconds        0.0.0.0:5000->5000/tcp   loving_wozniak

$> sudo docker logs 1c50b67d45b1
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
 * Restarting with stat

Yep, seems like the flask server is running. Here is where it gets weird. Lets make a request to the server:

 $> curl 127.0.0.1:5000 -v
 * Rebuilt URL to: 127.0.0.1:5000/
 * Hostname was NOT found in DNS cache
 *   Trying 127.0.0.1...
 * Connected to 127.0.0.1 (127.0.0.1) port 5000 (#0)
 > GET / HTTP/1.1
 > User-Agent: curl/7.35.0
 > Host: 127.0.0.1:5000
 > Accept: */*
 >
 * Empty reply from server
 * Connection #0 to host 127.0.0.1 left intact
 curl: (52) Empty reply from server

Empty reply… But is the process running?

$> sudo docker top 1c50b67d45b1
UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
root                2084                812                 0                   10:26               ?                   00:00:00            python index.py
root                2117                2084                0                   10:26               ?                   00:00:00            /usr/bin/python index.py

Now let’s ssh into the server and check…

$> sudo docker exec -it 1c50b67d45b1 bash
root@1c50b67d45b1:/srv# netstat -an
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 127.0.0.1:5000          0.0.0.0:*               LISTEN
tcp        0      0 127.0.0.1:47677         127.0.0.1:5000          TIME_WAIT
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node   Path
root@1c50b67d45b1:/srv# curl -I 127.0.0.1:5000
HTTP/1.0 200 OK
Content-Type: text/html; charset=utf-8
Content-Length: 5447
Server: Werkzeug/0.10.4 Python/2.7.6
Date: Tue, 19 May 2015 12:18:14 GMT

It’s fine… but not from the outside :( What am I doing wrong?

Question 7

The problem is you are only binding to the localhost interface, you should be binding to 0.0.0.0 if you want the container to be accessible from outside. If you change:

if __name__ == '__main__':
    app.run()

to

if __name__ == '__main__':
    app.run(host='0.0.0.0')

It should work.

Question 8

When using the flask command instead of app.run, you can pass the --host option to change the host. The line in Docker would be:

CMD ["flask", "run", "--host", "0.0.0.0"]

or

CMD flask run --host 0.0.0.0

Question 9

Your Docker container has more than one network interface. For example, my container has the following:

$ ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
32: eth0@if33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever

if you run docker network inspect bridge, you can see that your container is connected to that bridge with the second interface in the above output. This default bridge is also connected to the Docker process on your host.

Therefore you would have to run the command:

CMD flask run --host 172.17.0.2

To access your Flask app running in a Docker container from your host machine. Replace 172.17.0.2 with whatever the particular IP address is of your container.

Question 10

To build on other answers:

Imagine you have two computers. Each computer has a network interface (WiFi, say), which is its public IP. Each computer has a loopback/localhost interface, at 127.0.0.1. This means “just this computer.”

If you listed on 127.0.0.1 on computer A, you would not expect to be able to connect to that via 127.0.0.1 when running on computer B. After all, you asked to listen on computer A’s local, private address.

Docker is similar setup; technically it’s the same computer, but the Linux kernel is allowing each container to run with its own isolated network stack. So 127.0.0.1 in a container is the same as 127.0.0.1 on a different computer than your host—you can’t connect to it.

Longer version, with diagrams: https://pythonspeed.com/articles/docker-connection-refused/

Question 11

First of all in your python script you need to change code from

app.run()

to

app.run(host="0.0.0.0")

Second, In your docker file, last line should be like

CMD ["flask", "run", "-h", "0.0.0.0", "-p", "5000"]

And on host machine if 0.0.0.0:5000 doesn’t work then you should try with localhost:5000

Note – The CMD command has to be proper. Because CMD command provide defaults for executing container.

Question 12

I have a directory apkmirror-scraper-compose with the following structure:

.
├── docker-compose.yml
├── privoxy
│   ├── config
│   └── Dockerfile
├── scraper
│   ├── Dockerfile
│   ├── newnym.py
│   └── requirements.txt
└── tor
    └── Dockerfile

I’m trying to run the following docker-compose.yml:

version: '3'

services:
  privoxy:
    build: ./privoxy
    ports:
      - "8118:8118"
    links:
      - tor

  tor:
    build:
      context: ./tor
      args:
        password: ""
    ports:
      - "9050:9050"
      - "9051:9051"

  scraper:
    build: ./scraper
    links:
      - tor
      - privoxy

where the Dockerfile for tor is

FROM alpine:latest
EXPOSE 9050 9051
ARG password
RUN apk --update add tor
RUN echo "ControlPort 9051" >> /etc/tor/torrc
RUN echo "HashedControlPassword $(tor --quiet --hash-password $password)" >> /etc/tor/torrc
CMD ["tor"]

that for privoxy is

FROM alpine:latest
EXPOSE 8118
RUN apk --update add privoxy
COPY config /etc/privoxy/config
CMD ["privoxy", "--no-daemon"]

where config consists of the two lines

listen-address 0.0.0.0:8118
forward-socks5 / tor:9050 .

and the Dockerfile for scraper is

FROM python:2.7-alpine
ADD . /scraper
WORKDIR /scraper
RUN pip install -r requirements.txt
CMD ["python", "newnym.py"]

where requirements.txt contains the single line requests. Finally, the program newnym.py is designed to simply test whether changing the IP address using Tor is working:

from time import sleep, time

import requests as req
import telnetlib


def get_ip():
    IPECHO_ENDPOINT = 'http://ipecho.net/plain'
    HTTP_PROXY = 'http://privoxy:8118'
    return req.get(IPECHO_ENDPOINT, proxies={'http': HTTP_PROXY}).text


def request_ip_change():
    tn = telnetlib.Telnet('tor', 9051)
    tn.read_until("Escape character is '^]'.", 2)
    tn.write('AUTHENTICATE ""\r\n')
    tn.read_until("250 OK", 2)
    tn.write("signal NEWNYM\r\n")
    tn.read_until("250 OK", 2)
    tn.write("quit\r\n")
    tn.close()


if __name__ == '__main__':
    dts = []
    try:
        while True:
            ip = get_ip()
            t0 = time()
            request_ip_change()
            while True:
                new_ip = get_ip()
                if new_ip == ip:
                    sleep(1)
                else:
                    break
            dt = time() - t0
            dts.append(dt)
            print("{} -> {} in ~{}s".format(ip, new_ip, int(dt)))
    except KeyboardInterrupt:
        print("Stopping...")
        print("Average: {}".format(sum(dts) / len(dts)))

The docker-compose build builds successfully, but if I try docker-compose up, I get the following error message:

Creating network "apkmirrorscrapercompose_default" with the default driver
ERROR: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

I tried searching for help on this error message, but couldn’t find any. What is causing this error?

Question 13

I’ve seen it suggested docker may be at its maximum of created networks. The command docker network prune can be used to remove all networks not used by at least one container.

My issue ended up being, as Robert commented about: an issue with openvpn service openvpn stop ‘solved’ the problem.

Question 14

I ran into this problem because I had OpenVPN running. As soon as I killed OpenVPN, docker-compose up fired right up, and the error disappeared.

Question 15

I ran in this problem with OpenVPN working as well and I’ve found a solution where you should NOT stop/start OpenVPN server.

Idea that You should specify what exactly subnet you want to use. In docker-compose.yml write:

networks:
  default:
    driver: bridge
    ipam:
      config:
        - subnet: 172.16.57.0/24

That’s it. Now, default network will be used and if your VPN did not assign you something from 172.16.57.* subnet, you’re fine.

Question 16

Following Peter Hauge‘s comment, upon running docker network ls I saw (among other lines) the following:

NETWORK ID          NAME                                    DRIVER              SCOPE
dc6a83d13f44        bridge                                  bridge              local
ea98225c7754        docker_gwbridge                         bridge              local
107dcd8aa889        host                                    host                local

The line with NAME and DRIVER as both host seems to be what he is referring to with “networks already created on your host”. So, following https://gist.github.com/bastman/5b57ddb3c11942094f8d0a97d461b430, I ran the command

docker network rm $(docker network ls | grep "bridge" | awk '/ / { print $1 }')

Now docker-compose up works (although newnym.py produces an error).

Question 17

I have the same problem. I ran docker system prune -a --volumes, docker network prune, but neither helped me.

I use a VPN, I turned off the VPN and, after it docker started normal and was able to create a network. After that, you can enable VPN again.

Question 18

As other answers mentioned, Docker’s default local bridge network only supports 30 different networks (each one of them uniquely identifiable by their name). If you are not using them, then docker network prune will do the trick.

However, you might be interested in establishing more than 30 containers, each with their own network. Were you interested in doing so then you would need to define an overlay network. This is a bit more tricky but extremely well documented here.

EDIT (May 2020): Link has become unavailable, going through the docs there’s not an exact replacement, but I would recommend starting from here.

Question 19

I had an identical problem with the same error message but the solution with removal of unused docker networks didn’t help me. I’ve deleted all non-default docker networks (and all images and containers as well) but it didn’t help – docker still was not able to create a new network.

The cause of the problem was in network interfaces that were left after OpenVpn installation. (It was installed on the host previously.) I found them by running ifconfig command:

...
tun0  Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
      inet addr:10.8.0.2  P-t-P:10.8.0.2  Mask:255.255.255.0
      UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
      RX packets:75 errors:0 dropped:0 overruns:0 frame:0
      TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:100 
      RX bytes:84304 (84.3 KB)  TX bytes:0 (0.0 B)

tun1  Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
      inet addr:10.8.0.2  P-t-P:10.8.0.2  Mask:255.255.255.0
      UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
      RX packets:200496 errors:0 dropped:0 overruns:0 frame:0
      TX packets:148828 errors:0 dropped:0 overruns:0 carrier:0
      collisions:0 txqueuelen:100 
      RX bytes:211583838 (211.5 MB)  TX bytes:9568906 (9.5 MB)
...

I’ve found that I can remove them with a couple of commands:

ip link delete tun0
ip link delete tun1

After this the problem has disappeared.

Question 20

Check if any other container is running, If yes, do: docker-compose down
If VPN is connected, then disconnect it and try again to up docker container:
```
docker-compose up -d container_name
```

Question 21

You can try

$sudo service network-manager restart

Worked for me.

Question 22

I encoutered the same problem, the reason why is that you reached the max of networks:

do an : docker network ls Choose one to remove using: docker network rm networkname_default

Question 23

This happened to me because I was using OpenVPN. I found a way that I don’t need to stop using the VPN or manually add a network to the docker-compose file nor run any crazy script.

I switched to WireGuard instead of OpenVPN. More specifically, as I am running the nordvpn solution, I installed WireGuard and used their version of it, NordLynx.

Question 24

Killing the vpn is not needed.

This other comment about using a new network comes pretty close to the solution for me, and was working for a while, but I found a better way thanks to some talk over in another question

Create a network with:

docker network create your-network --subnet 172.24.24.0/24

Then, at the bottom of docker-compose.yaml, put this:

networks:
  default:
    external: 
      name: your-network

Done. No need to add networks to all container definitions etc. and you can re-use the network with other docker-compose files as well if you’d like.

Question 25

TL;DR

Add

version: "3.7"
services:
  web:
    ...
    network_mode: "bridge"

Read about network_mode in the documentation.

Long version

Disclaimer: I am not very knowledgeable about Docker networking, but this did the trick for me. YMMV.

When I ran docker run my-image the networking gave me no problems, but when I converted this command to a docker-compose.yml file, I got the same error as the OP.

I read Arenim’s answer and some other stuff on the internet that suggested to re-use an existing network.

You can find existing networks like this:

# docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
ca0415dfa442        bridge              bridge              local
78cbbda034dd        host                host                local
709f13f4ce2d        none                null                local

I wanted to reuse the default bridge network, so I added

services:
  web:
    ...

networks:
  default:
    external:
      name: bridge

to the the root of my docker-compose.yml (so not inside one of my services, but at the root indentation).

I now got the following error:

ERROR: for your-container network-scoped alias is supported only for containers in user defined networks

This led met to this Docker Github issue, that plainly stated that I should add the network_mode object to my docker-compose:

version: "3.7"
services:
  web:
    ...
    network_mode: "bridge"

I was using Docker version 18.09.8, docker-compose version 1.24.1 and the compose file format 3.7.

Question 26

If you want lots of networks then you can control how much IP space docker hands out to each network via the default-address-pools deamon setting, so you could add this to your /etc/docker/daemon.json:

{
  "bip": "10.254.1.1/24",
  "default-address-pools":[{"base":"10.254.0.0/16","size":28}],
}

Here I’ve reserved 10.254.1.1/24 (254 IP addresses) for the bridge network.

For any other network I create, docker will partition up the 10.254.0.0 space (65k hosts), giving out 16 hosts at a time ("size":28 refers to the CIDR mask, for 16 hosts).

If I create a few networks and then run docker network inspect <name> on them, it might display something like this:

        ...
        "Subnet": "10.254.0.32/28",
        "Gateway": "10.254.0.33"
        ...

The 10.254.0.32/28 means this network can use 16 ip addresses from 10.254.0.32 – 10.254.0.47.

Question 27

I ran into the same problem

Creating network “schemaregistry1_default” with the default driver
ERROR: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

and nothing helped until I turned off the Cisco VPN. after that docker-compose up worked

Question 28

I fixed this issue by steps :

turn off your network (wireless or wired…).
reboot your system.
before turning on your network on PC, execute command docker-compose up, it’s going to create new network.
then you can turn network on and go on …

语言	创建者
Ada	Chris Moore
Ada #2	Nicolas Boulenguez
GNU Awk	Miutsuru Kariya
Bash 4	Joel Martin
BASIC(C64和QBASIC)	Joel Martin
BBC BASIC V	Ben Harris
C	Joel Martin
C #2	Duncan Watts
C++	Stephen Thirlwall
C#	Joel Martin
ChucK	Vasilij Schneidermann
Clojure(Clojure和ClojureScript)	Joel Martin
CoffeeScript	Joel Martin
Common Lisp	Iqbal Ansari
Crystal	Linda_pp
D	Dov Murik
Dart	Harry Terkelsen
Elixir	Martin Ek
Elm	Jos van Bakel
Emacs Lisp	Vasilij Schneidermann
Erlang	Nathan Fiedler
ES6(ECMAScript 2015)	Joel Martin
F#	Peter Stephens
Factor	Jordan Lewis
Fantom	Dov Murik
Fennel	sogaiu
Forth	Chris Houser
GNU Guile	Mu Lei
GNU Smalltalk	Vasilij Schneidermann
Go	Joel Martin
Groovy	Joel Martin
Haskell	Joel Martin
Haxe(Neko、Python、C++和JS)	Joel Martin
Hy	Joel Martin
Io	Dov Murik
Janet	sogaiu
Java	Joel Martin
Java(松露/GraalVM)	Matt McGill
JavaScript(Demo)	Joel Martin
jq	Ali MohammadPur
Julia	Joel Martin
Kotlin	Javier Fernandez-Ivern
LiveScript	Jos van Bakel
Logo	Dov Murik
Lua	Joel Martin
GNU Make	Joel Martin
mal itself	Joel Martin
MATLAB(GNU Octave&MATLAB)	Joel Martin
miniMAL(Repo，Demo)	Joel Martin
NASM	Ben Dudson
Nim	Dennis Felsing
Object Pascal	Joel Martin
Objective C	Joel Martin
OCaml	Chris Houser
Perl	Joel Martin
Perl 6	Hinrik Örn Sigurðsson
PHP	Joel Martin
Picolisp	Vasilij Schneidermann
Pike	Dov Murik
PL/pgSQL(PostgreSQL)	Joel Martin
PL/SQL(Oracle)	Joel Martin
PostScript	Joel Martin
PowerShell	Joel Martin
Prolog	Nicolas Boulenguez
Python(2.x和3.x)	Joel Martin
Python #2(3.x)	Gavin Lewis
RPython	Joel Martin
R	Joel Martin
Racket	Joel Martin
Rexx	Dov Murik
Ruby	Joel Martin
Rust	Joel Martin
Scala	Joel Martin
Scheme (R7RS)	Vasilij Schneidermann
Skew	Dov Murik
Standard ML	Fabian Bergström
Swift 2	Keith Rollin
Swift 3	Joel Martin
Swift 4	陆遥
Swift 5	Oleg Montak
Tcl	Dov Murik
TypeScript	Masahiro Wakame
Vala	Simon Tatham
VHDL	Dov Murik
Vimscript	Dov Murik
Visual Basic.NET	Joel Martin
WebAssembly(WASM)	Joel Martin
Wren	Dov Murik
XSLT	Ali MohammadPur
Yorick	Dov Murik
Zig	Josh Tobin

问题：为Python项目构建Docker映像时如何避免重新安装软件包？

编辑＃＃：

EDIT##:

回答 0

回答 1

回答 2

回答 3

问题：在Docker中部署最小化Flask应用-服务器连接问题

回答 0

回答 1

回答 2

回答 3

回答 4

问题：Docker“错误：在分配给网络的默认设置中找不到可用的，不重叠的IPv4地址池”

回答 0

回答 1

回答 2

回答 3

回答 4

回答 5

回答 6

回答 7

回答 8

回答 9

回答 10

回答 11

回答 12

TL; DR

长版

TL;DR

Long version

回答 13

回答 14

回答 15

MAL-做一个Lisp

描述

演示文稿

构建/运行实现

外部实现

HolyC

生锈

问：

其他MAL项目

实施详情

Ada

Ada.2

GNU awk

BASH 4

基本(C64和QBasic)

BBC Basic V

C

C.2

C++

C#

卡盘

封闭式

CoffeeScript

通用Lisp

水晶

D

省道

Emacs Lisp

灵丹妙药

榆树

二郎

ES6(ECMAScript 2015)

F#

因素

幻影

茴香

第四

GNU Guile 2.1+

GNU Smalltalk

去

时髦的

哈斯克尔

Haxe(Neko、Python、C++和JavaScript)

干草

IO

珍妮特