Docker registry auth issues

Docker registry auth issues

I ran into an issue today when scripting a setup for a private docker registry and I ended up wasting a few hours on it. I’m hoping this post can save a few people the same headaches.

After setting up a docker registry with authentication following most of what was in I was confronted with a rather annoying message on my console:

Error response from daemon: no successful auth challenge for https://registry:5000/v2/ - errors: [basic auth attempt to https://registry:5000/v2/ realm "Registry Realm" failed with status: 401 Unauthorized]

Unauthorized seemed pretty clear, so I checked the user’s htpasswd again, and recreated the htpasswd file:

htpasswd -bc /opt/docker-registry/auth/htpasswd username password

That didn’t work. I knew that there’s no way I messed up the username/password again, so I took a look at the output from the docker container hosting the registry:

time="2016-01-18T00:04:59Z" level=warning msg="error authorizing context: basic authentication challenge for realm \"Registry Realm\": authentication failured" go.version=go1.5.2 http.request.remoteaddr="" http.request.uri="/v2/" http.request.useragent="docker/1.9.1 go/go1.4.3 git-commit/a34a1d5 kernel/4.1.13-boot2docker os/linux arch/amd64" version=v2.2.1

The “authentication failured” type was actually extremely useful, it lead me right to the go source code that produced the error:


So, it looked like this was definitely a run of the mill auth error. After confirming the process could indeed read my htpasswd file, I took a closer look at the source code..


“Only bcrypt hash entries are supported”

…. CRAP! I forgot to use -B on my command line options for htpasswd..

htpasswd -cbB /opt/docker-registry/auth/htpasswd username password

Fixed my issue, and the world was lovely again!

The further consequences of free and cheap

The further consequences of free and cheap

I touched a little bit on this subject on the InWorldz blog in a post titled The hard thing about hosting things, but lately this has been coming up more and more, and I’m starting to get annoyed.

There are some very talented people in the OpenSim space. People with a wide range of skills from C++ development, C#, web development in PHP, node.js and other languages. The thing that all of them seem to have in common is that they have a passion for working on 3d software and simulations, but unfortunately we’ve found more and more that this passion and drive is being taken advantage of, and just like hosting prices in OpenSim, a lowest bidder mentality has emerged.

As a business owner and employee of small businesses, I learned a long time ago that when people don’t have to worry about paying their bills, they are more likely to be creative and solve problems in effective ways. They are loyal to your vision, and want to help see it through. They don’t have to seek out work elsewhere to make ends meet and they are appreciative of the sacrifices that you make for them.

Unfortunately the stories I’m hearing from people getting paid for OpenSim based work and from OpenSim grids paints a really disturbing picture of the way people are being used and led astray with promises.

The most recent I’ve heard is from an extremely talented software developer that is also serving as a systems administrator, a devops person, and someone that is constantly on call anytime something breaks. This person tells me that he calculated his effective hourly rate with this schedule and came up with a figure of $0.70/hr. $0.70/hr to be a software developer and systems administrator on call 24/7. No thanks, i’ll pass no matter what vague promises about the future you make to me.

Really? Is this the best that we can do? No wonder we have a hard time getting anything done and this space is considered a joke to many outsiders. I don’t think this particular joke is funny at all.

On a related note, I was just recently made aware of a comment that claimed an $800 bid to implement export, which requires BOTH viewer side and server side changes was “not great”. As if this bid to do custom software development on two separate platforms were easy and should pay minimum wage? What?

If you think paying $800 for software implementation is too much, maybe your business should be charging more to free up real money for the software development that it depends on. People’s time is not a charity to be exploited.

Software development charges from companies and independent contractors typically land in the $70 – $120/hr range. In my experience these are fair numbers, and when you pay a much much lower price to try to get the best deal, you end up getting what you pay for, and many times it’ll have to be completely redone later on. When you have to do something two and three times, the cheap rates end up not meaning very much.

So my final message is pretty simple. As we free up more cash to pay more people for their time, expect that the contractors you’re used to paying pennies to may end up finding their way over here. I won’t make any vague promises. I’ll ask them flat out what kind of compensation they think they need to complete a project and pay accordingly. I won’t put anyone on call 24/7 unless they’re being paid real wages. If they go over budget due to a bad estimate, we’ll work to make it right.

I actually like people. I want to see them succeed and be happy. Only when I see happy people working for us and alongside us do I know that we’ve truly succeeded.

Xcode handy key bindings

This is a tiny postlet on some key combinations I have found handy while using Xcode. I’ll update this as I run into more invaluable shortcuts.

  • shift + cmd + o – Quickly open file
  • shift + cmd + j – Reveal in project navigator
  • ctrl + cmd + up / down – Jump to next counterpart (ex. .h/.cpp file)

A ton more are listed here:

Message ordering in the face of failure

Message ordering in the face of failure

Another day learning and coding and another interesting problem!

While designing a distributed message queue, I decided to use vector clocks to preserve message ordering when producers are talking to a different quorum of nodes. You can see some of my conclusions in a previous post titled Vector clocks and quorum consensus.

Vector clocks do seem to solve my ordering problems pretty well and don’t require me to have perfectly synchronized wall clock times between nodes to accomplish it. However, a problem arises with my original implementation in the case of a node failure.

Originally, I intended to associate counter values in vector clocks with a particular queue. This would mean that the vector clock’s value would be representative of the state of just the given queue, and when that queue was collected due to a lack of messages, all counter values would be reset to zero. After finally getting to the point where I was coding these portions of the design, I realized that this creates a problem in even the most basic and obvious scenarios:

  • Producer 1 (p1) queues message “Hi” to nodes A and B. Resulting clock is (A:1, B:1)
  • Consumer 1 (c1) consumes message “Hi” and sets its clock to (A:1, B:1)
  • [… TTL passes…]  Queue is garbage collected since all messages have expired. Clocks are reset.
  • p1 queues message “How are you?” to nodes A and B. Resulting clock is again (A:1, B:1)
  • c1 asks for all messages since “Hi” (A:1, B1) and gets nothing.
  • Oops.

Since the client is using the vector clock to ask for new messages, we can’t simply arbitrarily reset the clock. So instead we move the values of the clock to the server instance using a 64 bit unsigned integer to ensure we don’t get overflow. This solves the problem.. Well kinda, in a perfect world. Lets examine what is still wrong.

  • Producer 1 (p1) queues message “Hi” to nodes A and B. Resulting clock is (A:1, B:1)
  • Consumer 1 (c1) consumes message “Hi” and sets its clock to (A:1, B:1)
  • [… TTL passes…]  Queue is garbage collected since all messages have expired. But the clocks are not reset since they are based on the count for the node.
  • p1 queues message “How are you?” to nodes A and B. Resulting clock is (A:2, B:2)
  • c1 asks for all messages since “Hi” (A:1, B1) and gets “How are you?” with clock (A:2, B:2)
  • Yay!

So far, so good. But now what happens when A dies and then comes back with its clock reset? Well.. Nothing good.

  • Node A dies
  • Node A comes back to life with clocks reset
  • p1 queues message “I’m doing well!”. Resulting clock is (A:1, B:3)
  • c1 asks for all messages since “How are you?” with clock (A:2, B:2)
  • Uhoh. Clock (A:1, B:3) “I’m doing well!” does not follow (A:2, B:2) “How are you?”. We get no messages 😦

What do we do? We can keep our vector clock up to date every time we receive a message and for example write the value to disk, but that involves a lot of writes on the disk as well as being prone to not being up to date due to a crash before the updated clock can be written.

What I’ve decided to do is keep tract of the number of times a node is restarted and send that number with each node clock. That way, if a node restarts and the event clock is reset, we can use the restart count to know that any message with a lower restart count comes before a message with a greater restart count, even if the event clock is greater. Lets examine the above scenario with the new rules:

  • We set up clocks as (restart count/event count since last restart)
  • Producer 1 (p1) queues message “Hi” to nodes A and B. Resulting clock is (A:1/1, B:1/1)
  • Consumer 1 (c1) consumes message “Hi” and sets its clock to (A:1/1, B:1/1)
  • [… TTL passes…]  Queue is garbage collected since all messages have expired. But the clocks are not reset since they are based on the count for the node.
  • p1 queues message “How are you?” to nodes A and B. Resulting clock is (A:1/2, B:1/2)
  • c1 asks for all messages since “Hi” (A:1/1, B1/1) and gets “How are you?” with clock (A:1/2, B:1/2)
  • Yay!
  • Node A dies
  • Node A comes back to life with its event clock reset
  • p1 queues message “I’m doing well!”. Resulting clock is (A:2/1, B:1/3)
  • c1 asks for all messages since “How are you?” with clock (A:1/2, B:1/2)
  • Success! Clock (A:2/1, B:1/3) follow clock (A:1/2, B:1/2) since A’s restart count is 2 vs 1 and B’s event count is 2 vs 1.

In this way we can still determine message ordering after a node restart and all is right in the world.. until the next challenge.

Staying on the horse

Staying on the horse

Software development can be exciting. We learn new things. We get to explore new ideas and see them come to life.

Software allows us to share our talents with people from all over the world. We get to see the things we create  put to use sometimes in ways that we never considered.

But software isn’t all fun and games. Sometimes software development can be downright monotonous and annoying. For any project of sufficient size, we seem to start off excited, but then once we get far enough into it, it becomes work and takes a lot of willpower to want to continue. Time keeps ticking by, the project gets larger and larger. It gets more difficult to fit it all into your head, and development becomes “real”.

I was thinking about this today, and trying to enumerate all the times that I felt like I didn’t want to touch a particular project ever again. I came up with a few suggestions on preventing project burnout that I hope can be useful to some of you out there working long hours in software.

Change code that is causing repetitive typing

If you find yourself frequently having to type some kind of boilerplate code, or typing the same basic pattern of code over and over again, it’s time to take a look at a refactoring to remove the cause of your pain. Over time, this will sap your willpower to continue on the project since most of your time will be spent pounding out pages and pages of nothing but supporting structure, without getting any forward moving work done.

If refactoring doesn’t work for one reason or another, or the cause for the repetition is beyond the builtin power of your language to solve, try code generation. Use something like Cog to make your life easier and get back to actually making progress on your code instead of doing the boring work yourself.

Refactor confusing code

If you keep bumping into a chunk of code that is confusing or that you always have to think about before you can make changes, it might be time to refactor and come up with a simpler way of doing the task. For example, you might have a class where in each method callback you need to check the status of a connection or other object before proceeding. Over time when adding new methods eventually this is going to be forgotten introducing subtle bugs into your code. If you have to keep remembering things, find a way so that those memories are embedded in your code.

Encapsulate behaviors

If you have to remember to “do X” before each call to a certain library, see if you can encapsulate the doing of X into an easy to reuse chunk. For example, if you’re using a C library and you keep having to remember to do your own cleanups, use a unique_ptr with a custom deleter to create RAII objects. Just because a library didn’t come with an easier way to use it, doesn’t mean you shouldn’t build one.

Don’t just code. Write unit tests

If you’re not into test driven development, or are adverse to the extra coding that unit testing creates, you are missing out on an opportunity to pull yourself from “programmers writers block”. Completing and executing unit tests regularly while creating software  is a great way to pull yourself out of the code code code code cycle for a bit and creates a higher quality product in the end. It also will end up saving you debugging time.

I hope this post can help you to prevent coding burnout. Remember that if you’re starting to feel like you don’t want to work on a project, many times there is a problem causing it that you can take steps to resolve. Do your best to finish the projects that you start. Programming is more rewarding that way.

Programming for.. relaxation?

Programming for.. relaxation?

Sometimes after a long 12 hour day of software development I like to sit back, relax, and… do more software development.

I never said I was normal.

I’ve always had a huge interest in embedded and constrained systems. I have an Arduino microcontroller that I’ve done a bunch of projects on to teach myself more about the crossroads between software and hardware. This past halloween I put together a pumpkin powered by a microcontroller, a few LEDs, and an ultrasonic distance sensor. You can see it here, and it is spooky.

On a side note, it just so happens that my desire to learn more about electronics actually brought about the InWorldz InShape project. I was planning on building a motion tracking setup out of some very cheap hardware that would connect to your tablet to send the InShape motion data. Thankfully resident grid monkey Jim Tarber talked me out of building the device and just write an app for smartphones. We really don’t want to become a hardware manufacturing company, but as a virtual worlds company, we will always be looking at the latest hardware to try to make the immersive experience more compelling.


That brings me to the point of this post. A few weeks ago someone I follow on twitter and have met on InWorldz landed a well deserved career with the pebble smartwatch company. @KatharineBerry always has something interesting to say and really seemed to be enjoying the software development she was doing for the pebble devices. So as usual I got curious and I wondered just how capable a device the size of a watch could be nowadays.

It turns out the watch is very capable. Utilizing a bluetooth link to your smartphone, the watch can transfer data and act as a quick wrist-flip conduit to your information. Without taking your phone out of your pocket you can see who is calling you, switch to the next track while listening to music and read incoming text messages.

Being me though, I was more interested in what I could make the watch do than what the watch could do for me. So I took a few hours to go through and learn about the pebble platform. The 2.0 SDK is straightforward without too many gotchas, and it was pretty easy to utilize the UI elements to provide a decent interface. After running through some docs, I wrote my first app. Pebble TipCalc. A watch app where you put in the total for a restaurant bill and the percentage you would like to tip and it returns the tip amount and bill total.

Since the first version, I’ve updated the app to utilize fixed point math (even with the small amounts and totals, I was seeing annoying accuracy problems with floating point) and enabled a more intuitive input of the the bill amount. It was a nice weekend distraction for when my brain was fried from too much VR work. I’m hoping to be able to somehow incorporate the device into future InWorldz projects.

Now you know what programmer types do “for fun”.