CentOS 7/RHEL 7 and docker containers on boot

I recently had to work with a CentOS 7 system and a few docker containers. These containers were bound to different IP addresses on the server NIC and it was critical that they be available after a server reboot. CentOS 7 comes with docker 1.3.2 built into the base installation. This version supports the –restart switch as follows:

–restart=”” Restart policy to apply when a container exits (no, on-failure[:max-retry], always)

Setting the containers to restart=always, rebooting the box, and running docker ps, I discovered that the docker containers were not starting with the OS. After going through a few combinations of commands unsuccessfully I decided to take a look at the message log for the system and I noticed something odd:

Received an unexpected error during port allocation: Error starting userland proxy: listen … cannot assign requested address

Taking a closer look at the logs I noted that this was often followed later by something like:

kernel: e1000e: enp0s25 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx

It looks like the docker container engine was starting before the NIC was actually up. It was then trying to do its magic and create the bridge interface but the NIC with the given IP address wasn’t ready yet.

To solve this issue I looked around and found an article about systemd services specifically dealing with depending on the network interface to be up. Inside the systemd unit for docker, the service was set to depend on network.target:

/usr/lib/systemd/system/docker.service:

[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network.target docker.socket
Requires=docker.socket
Wants=network.target

Additionally, /usr/lib/systemd/system/docker.socket was not set to depend on network.target at all. To get everything rebooting with the system, I did the following:

  • Added network-online.target to the Requires and Wants of both the docker.service unit and the docker.socket unit.
  • Ran
    systemctl enable NetworkManager-wait-online.service

    to stop boot processing until the network is available.

Both of these steps may not have been needed and you should play around with the configuration to see what works for you. My final .service and socket files were as follows:
[ddaeschler@docker ~]$ cat /usr/lib/systemd/system/docker.socket

[Unit]
Description=Docker Socket for the API
PartOf=docker.service
After=network-online.target
Wants=network-online.target

[Socket]
ListenStream=/var/run/docker.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker

[Install]
WantedBy=sockets.target

[ddaeschler@docker ~]$ cat /usr/lib/systemd/system/docker.service

[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network-online.target docker.socket
Requires=docker.socket
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=-/etc/sysconfig/docker
EnvironmentFile=-/etc/sysconfig/docker-storage
ExecStart=/usr/bin/docker -d $OPTIONS $DOCKER_STORAGE_OPTIONS
LimitNOFILE=1048576
LimitNPROC=1048576
MountFlags=private

[Install]
WantedBy=multi-user.target

After making these changes, the containers start up normally at boot. I hope this can help someone else running into this issue.

5 thoughts on “CentOS 7/RHEL 7 and docker containers on boot

  1. Hi there,

    did you really modify the service files in /usr?
    There is a folder (maybe one has to create it) in /etc, where systemd looks for services, first.
    A service found in /etc overrides the service definition shipped with the distribution and is therefore
    update-resilient, e.g. an update of systemd will not wipe away your changes.

    But you have to maintain your changes on your own.

    Best

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s