I recently had to work with a CentOS 7 system and a few docker containers. These containers were bound to different IP addresses on the server NIC and it was critical that they be available after a server reboot. CentOS 7 comes with docker 1.3.2 built into the base installation. This version supports the –restart switch as follows:
–restart=”” Restart policy to apply when a container exits (no, on-failure[:max-retry], always)
Setting the containers to restart=always, rebooting the box, and running docker ps, I discovered that the docker containers were not starting with the OS. After going through a few combinations of commands unsuccessfully I decided to take a look at the message log for the system and I noticed something odd:
Received an unexpected error during port allocation: Error starting userland proxy: listen … cannot assign requested address
Taking a closer look at the logs I noted that this was often followed later by something like:
kernel: e1000e: enp0s25 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx
It looks like the docker container engine was starting before the NIC was actually up. It was then trying to do its magic and create the bridge interface but the NIC with the given IP address wasn’t ready yet.
To solve this issue I looked around and found an article about systemd services specifically dealing with depending on the network interface to be up. Inside the systemd unit for docker, the service was set to depend on network.target:
Description=Docker Application Container Engine
Additionally, /usr/lib/systemd/system/docker.socket was not set to depend on network.target at all. To get everything rebooting with the system, I did the following:
- Added network-online.target to the Requires and Wants of both the docker.service unit and the docker.socket unit.
systemctl enable NetworkManager-wait-online.service
to stop boot processing until the network is available.
Both of these steps may not have been needed and you should play around with the configuration to see what works for you. My final .service and socket files were as follows:
[ddaeschler@docker ~]$ cat /usr/lib/systemd/system/docker.socket
[Unit] Description=Docker Socket for the API PartOf=docker.service After=network-online.target Wants=network-online.target [Socket] ListenStream=/var/run/docker.sock SocketMode=0660 SocketUser=root SocketGroup=docker [Install] WantedBy=sockets.target
[ddaeschler@docker ~]$ cat /usr/lib/systemd/system/docker.service
[Unit] Description=Docker Application Container Engine Documentation=http://docs.docker.com After=network-online.target docker.socket Requires=docker.socket Wants=network-online.target [Service] Type=notify EnvironmentFile=-/etc/sysconfig/docker EnvironmentFile=-/etc/sysconfig/docker-storage ExecStart=/usr/bin/docker -d $OPTIONS $DOCKER_STORAGE_OPTIONS LimitNOFILE=1048576 LimitNPROC=1048576 MountFlags=private [Install] WantedBy=multi-user.target
After making these changes, the containers start up normally at boot. I hope this can help someone else running into this issue.