Docker registry auth issues

Docker registry auth issues

I ran into an issue today when scripting a setup for a private docker registry and I ended up wasting a few hours on it. I’m hoping this post can save a few people the same headaches.

After setting up a docker registry with authentication following most of what was in https://docs.docker.com/registry/deploying/ I was confronted with a rather annoying message on my console:

Error response from daemon: no successful auth challenge for https://registry:5000/v2/ - errors: [basic auth attempt to https://registry:5000/v2/ realm "Registry Realm" failed with status: 401 Unauthorized]

Unauthorized seemed pretty clear, so I checked the user’s htpasswd again, and recreated the htpasswd file:

htpasswd -bc /opt/docker-registry/auth/htpasswd username password

That didn’t work. I knew that there’s no way I messed up the username/password again, so I took a look at the output from the docker container hosting the registry:

time="2016-01-18T00:04:59Z" level=warning msg="error authorizing context: basic authentication challenge for realm \"Registry Realm\": authentication failured" go.version=go1.5.2 http.request.remoteaddr="192.168.99.101:36020" http.request.uri="/v2/" http.request.useragent="docker/1.9.1 go/go1.4.3 git-commit/a34a1d5 kernel/4.1.13-boot2docker os/linux arch/amd64" instance.id=1b872c8e-90a7-4bea-9c59-fef4b2dfba43 version=v2.2.1

The “authentication failured” type was actually extremely useful, it lead me right to the go source code that produced the error:

access.go
and
htpasswd.go

So, it looked like this was definitely a run of the mill auth error. After confirming the process could indeed read my htpasswd file, I took a closer look at the source code..

e2239e830520fae9c56c6505c54c8e95

“Only bcrypt hash entries are supported”

…. CRAP! I forgot to use -B on my command line options for htpasswd..

htpasswd -cbB /opt/docker-registry/auth/htpasswd username password

Fixed my issue, and the world was lovely again!

CentOS 7/RHEL 7 and docker containers on boot

I recently had to work with a CentOS 7 system and a few docker containers. These containers were bound to different IP addresses on the server NIC and it was critical that they be available after a server reboot. CentOS 7 comes with docker 1.3.2 built into the base installation. This version supports the –restart switch as follows:

–restart=”” Restart policy to apply when a container exits (no, on-failure[:max-retry], always)

Setting the containers to restart=always, rebooting the box, and running docker ps, I discovered that the docker containers were not starting with the OS. After going through a few combinations of commands unsuccessfully I decided to take a look at the message log for the system and I noticed something odd:

Received an unexpected error during port allocation: Error starting userland proxy: listen … cannot assign requested address

Taking a closer look at the logs I noted that this was often followed later by something like:

kernel: e1000e: enp0s25 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx

It looks like the docker container engine was starting before the NIC was actually up. It was then trying to do its magic and create the bridge interface but the NIC with the given IP address wasn’t ready yet.

To solve this issue I looked around and found an article about systemd services specifically dealing with depending on the network interface to be up. Inside the systemd unit for docker, the service was set to depend on network.target:

/usr/lib/systemd/system/docker.service:

[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network.target docker.socket
Requires=docker.socket
Wants=network.target

Additionally, /usr/lib/systemd/system/docker.socket was not set to depend on network.target at all. To get everything rebooting with the system, I did the following:

  • Added network-online.target to the Requires and Wants of both the docker.service unit and the docker.socket unit.
  • Ran
    systemctl enable NetworkManager-wait-online.service

    to stop boot processing until the network is available.

Both of these steps may not have been needed and you should play around with the configuration to see what works for you. My final .service and socket files were as follows:
[ddaeschler@docker ~]$ cat /usr/lib/systemd/system/docker.socket

[Unit]
Description=Docker Socket for the API
PartOf=docker.service
After=network-online.target
Wants=network-online.target

[Socket]
ListenStream=/var/run/docker.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker

[Install]
WantedBy=sockets.target

[ddaeschler@docker ~]$ cat /usr/lib/systemd/system/docker.service

[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network-online.target docker.socket
Requires=docker.socket
Wants=network-online.target

[Service]
Type=notify
EnvironmentFile=-/etc/sysconfig/docker
EnvironmentFile=-/etc/sysconfig/docker-storage
ExecStart=/usr/bin/docker -d $OPTIONS $DOCKER_STORAGE_OPTIONS
LimitNOFILE=1048576
LimitNPROC=1048576
MountFlags=private

[Install]
WantedBy=multi-user.target

After making these changes, the containers start up normally at boot. I hope this can help someone else running into this issue.