Docker registry auth issues

Docker registry auth issues

I ran into an issue today when scripting a setup for a private docker registry and I ended up wasting a few hours on it. I’m hoping this post can save a few people the same headaches.

After setting up a docker registry with authentication following most of what was in https://docs.docker.com/registry/deploying/ I was confronted with a rather annoying message on my console:

Error response from daemon: no successful auth challenge for https://registry:5000/v2/ - errors: [basic auth attempt to https://registry:5000/v2/ realm "Registry Realm" failed with status: 401 Unauthorized]

Unauthorized seemed pretty clear, so I checked the user’s htpasswd again, and recreated the htpasswd file:

htpasswd -bc /opt/docker-registry/auth/htpasswd username password

That didn’t work. I knew that there’s no way I messed up the username/password again, so I took a look at the output from the docker container hosting the registry:

time="2016-01-18T00:04:59Z" level=warning msg="error authorizing context: basic authentication challenge for realm \"Registry Realm\": authentication failured" go.version=go1.5.2 http.request.remoteaddr="192.168.99.101:36020" http.request.uri="/v2/" http.request.useragent="docker/1.9.1 go/go1.4.3 git-commit/a34a1d5 kernel/4.1.13-boot2docker os/linux arch/amd64" instance.id=1b872c8e-90a7-4bea-9c59-fef4b2dfba43 version=v2.2.1

The “authentication failured” type was actually extremely useful, it lead me right to the go source code that produced the error:

access.go
and
htpasswd.go

So, it looked like this was definitely a run of the mill auth error. After confirming the process could indeed read my htpasswd file, I took a closer look at the source code..

e2239e830520fae9c56c6505c54c8e95

“Only bcrypt hash entries are supported”

…. CRAP! I forgot to use -B on my command line options for htpasswd..

htpasswd -cbB /opt/docker-registry/auth/htpasswd username password

Fixed my issue, and the world was lovely again!

Xcode handy key bindings

This is a tiny postlet on some key combinations I have found handy while using Xcode. I’ll update this as I run into more invaluable shortcuts.

  • shift + cmd + o – Quickly open file
  • shift + cmd + j – Reveal in project navigator
  • ctrl + cmd + up / down – Jump to next counterpart (ex. .h/.cpp file)

A ton more are listed here: http://nshipster.com/xcode-key-bindings-and-gestures/

Message ordering in the face of failure

Message ordering in the face of failure

Another day learning and coding and another interesting problem!

While designing a distributed message queue, I decided to use vector clocks to preserve message ordering when producers are talking to a different quorum of nodes. You can see some of my conclusions in a previous post titled Vector clocks and quorum consensus.

Vector clocks do seem to solve my ordering problems pretty well and don’t require me to have perfectly synchronized wall clock times between nodes to accomplish it. However, a problem arises with my original implementation in the case of a node failure.

Originally, I intended to associate counter values in vector clocks with a particular queue. This would mean that the vector clock’s value would be representative of the state of just the given queue, and when that queue was collected due to a lack of messages, all counter values would be reset to zero. After finally getting to the point where I was coding these portions of the design, I realized that this creates a problem in even the most basic and obvious scenarios:

  • Producer 1 (p1) queues message “Hi” to nodes A and B. Resulting clock is (A:1, B:1)
  • Consumer 1 (c1) consumes message “Hi” and sets its clock to (A:1, B:1)
  • [… TTL passes…]  Queue is garbage collected since all messages have expired. Clocks are reset.
  • p1 queues message “How are you?” to nodes A and B. Resulting clock is again (A:1, B:1)
  • c1 asks for all messages since “Hi” (A:1, B1) and gets nothing.
  • Oops.

Since the client is using the vector clock to ask for new messages, we can’t simply arbitrarily reset the clock. So instead we move the values of the clock to the server instance using a 64 bit unsigned integer to ensure we don’t get overflow. This solves the problem.. Well kinda, in a perfect world. Lets examine what is still wrong.

  • Producer 1 (p1) queues message “Hi” to nodes A and B. Resulting clock is (A:1, B:1)
  • Consumer 1 (c1) consumes message “Hi” and sets its clock to (A:1, B:1)
  • [… TTL passes…]  Queue is garbage collected since all messages have expired. But the clocks are not reset since they are based on the count for the node.
  • p1 queues message “How are you?” to nodes A and B. Resulting clock is (A:2, B:2)
  • c1 asks for all messages since “Hi” (A:1, B1) and gets “How are you?” with clock (A:2, B:2)
  • Yay!

So far, so good. But now what happens when A dies and then comes back with its clock reset? Well.. Nothing good.

  • Node A dies
  • Node A comes back to life with clocks reset
  • p1 queues message “I’m doing well!”. Resulting clock is (A:1, B:3)
  • c1 asks for all messages since “How are you?” with clock (A:2, B:2)
  • Uhoh. Clock (A:1, B:3) “I’m doing well!” does not follow (A:2, B:2) “How are you?”. We get no messages 😦

What do we do? We can keep our vector clock up to date every time we receive a message and for example write the value to disk, but that involves a lot of writes on the disk as well as being prone to not being up to date due to a crash before the updated clock can be written.

What I’ve decided to do is keep tract of the number of times a node is restarted and send that number with each node clock. That way, if a node restarts and the event clock is reset, we can use the restart count to know that any message with a lower restart count comes before a message with a greater restart count, even if the event clock is greater. Lets examine the above scenario with the new rules:

  • We set up clocks as (restart count/event count since last restart)
  • Producer 1 (p1) queues message “Hi” to nodes A and B. Resulting clock is (A:1/1, B:1/1)
  • Consumer 1 (c1) consumes message “Hi” and sets its clock to (A:1/1, B:1/1)
  • [… TTL passes…]  Queue is garbage collected since all messages have expired. But the clocks are not reset since they are based on the count for the node.
  • p1 queues message “How are you?” to nodes A and B. Resulting clock is (A:1/2, B:1/2)
  • c1 asks for all messages since “Hi” (A:1/1, B1/1) and gets “How are you?” with clock (A:1/2, B:1/2)
  • Yay!
  • Node A dies
  • Node A comes back to life with its event clock reset
  • p1 queues message “I’m doing well!”. Resulting clock is (A:2/1, B:1/3)
  • c1 asks for all messages since “How are you?” with clock (A:1/2, B:1/2)
  • Success! Clock (A:2/1, B:1/3) follow clock (A:1/2, B:1/2) since A’s restart count is 2 vs 1 and B’s event count is 2 vs 1.

In this way we can still determine message ordering after a node restart and all is right in the world.. until the next challenge.

Staying on the horse

Staying on the horse

Software development can be exciting. We learn new things. We get to explore new ideas and see them come to life.

Software allows us to share our talents with people from all over the world. We get to see the things we create  put to use sometimes in ways that we never considered.

But software isn’t all fun and games. Sometimes software development can be downright monotonous and annoying. For any project of sufficient size, we seem to start off excited, but then once we get far enough into it, it becomes work and takes a lot of willpower to want to continue. Time keeps ticking by, the project gets larger and larger. It gets more difficult to fit it all into your head, and development becomes “real”.

I was thinking about this today, and trying to enumerate all the times that I felt like I didn’t want to touch a particular project ever again. I came up with a few suggestions on preventing project burnout that I hope can be useful to some of you out there working long hours in software.

Change code that is causing repetitive typing

If you find yourself frequently having to type some kind of boilerplate code, or typing the same basic pattern of code over and over again, it’s time to take a look at a refactoring to remove the cause of your pain. Over time, this will sap your willpower to continue on the project since most of your time will be spent pounding out pages and pages of nothing but supporting structure, without getting any forward moving work done.

If refactoring doesn’t work for one reason or another, or the cause for the repetition is beyond the builtin power of your language to solve, try code generation. Use something like Cog to make your life easier and get back to actually making progress on your code instead of doing the boring work yourself.

Refactor confusing code

If you keep bumping into a chunk of code that is confusing or that you always have to think about before you can make changes, it might be time to refactor and come up with a simpler way of doing the task. For example, you might have a class where in each method callback you need to check the status of a connection or other object before proceeding. Over time when adding new methods eventually this is going to be forgotten introducing subtle bugs into your code. If you have to keep remembering things, find a way so that those memories are embedded in your code.

Encapsulate behaviors

If you have to remember to “do X” before each call to a certain library, see if you can encapsulate the doing of X into an easy to reuse chunk. For example, if you’re using a C library and you keep having to remember to do your own cleanups, use a unique_ptr with a custom deleter to create RAII objects. Just because a library didn’t come with an easier way to use it, doesn’t mean you shouldn’t build one.

Don’t just code. Write unit tests

If you’re not into test driven development, or are adverse to the extra coding that unit testing creates, you are missing out on an opportunity to pull yourself from “programmers writers block”. Completing and executing unit tests regularly while creating software  is a great way to pull yourself out of the code code code code cycle for a bit and creates a higher quality product in the end. It also will end up saving you debugging time.

I hope this post can help you to prevent coding burnout. Remember that if you’re starting to feel like you don’t want to work on a project, many times there is a problem causing it that you can take steps to resolve. Do your best to finish the projects that you start. Programming is more rewarding that way.

Discovering C++11

It seems that while I was off in the land of C# and C++/CLI, something really awesome happened in the C++ world. The C++0x standard was ratified and became C++11. Not only that, but there seems to already be wide support for the vast majority of features in modern compilers!

I think C++11 puts C++ much closer to the productivity levels of more modern programming languages. This is something that C++ desperately needed to stay relevant for more than just “performance oriented” programming.

Changes to the language are too vast to write about in their entirety, and since I’m late to the party, most of the best bits have already been covered extensively. However, I’d like to go over a few of the features that I have found vastly boost my productivity.

The improved auto keyword

We can now use auto in place of very long type names. The compiler figures out the type from the expression and everything stays type safe.

//instead of
std::vector<std::string>::iterator start = vecOfStrings.begin();

//we can now write:
auto start = vecOfStrings.begin();

Range based for loops

We can more easily iterate through C++ collections

MyClass::iterateThings()
{
	std::vector vecOfStrings;
	vecOfStrings.push_back("Hello");
	vecOfStrings.push_back("my");
	vecOfStrings.push_back("name");
	vecOfStrings.push_back("is");
	vecOfStrings.push_back("C++11!");

	for (auto sayStr : vecOfStrings) {
		std::cout << sayStr << " " <<;
	}
	//instead of
	for (std::vector<std::string>::iterator i = vecOfStrings.begin(); 
		i != vecOfStrings.end(); ++i) 
	{ 
		std::string& str = (*i); //... 
	} 
}

Lambda Functions

We can define small, one off functions that perform a task at the call site rather than away from it

void function() {
	std::vector numbers;
	numbers.push_back(1);
	numbers.push_back(2);
	numbers.push_back(3);

	std::for_each(numbers.begin(), numbers.end(), 
		[](int i){ std::cout << ' ' << i; });
}

//
//instead of this
//
struct funky {           // function object type:
  void operator() (int i) {std::cout << ' ' << i;}
} functor;

void function() {
	std::vector numbers;
	numbers.push_back(1);
	numbers.push_back(2);
	numbers.push_back(3);

	std::for_each(numbers.begin(), numbers.end(), functor);
}

There are many more features present in C++11 that are worth your time. If you haven’t used C++ in a while and you have a project that demands staticly typed, native object oriented programming, C++11 is worth a close look.