Almond Chocolate Banana Protein Shake

Almond Chocolate Banana Protein Shake

In a bit of a break from the geekery, I am happy to present to you a newly discovered protein shake! I came up with this recipe myself while trying to think of delicious ways to use the chocolate protein powder I was forced to get when the all natural vanilla powder I usually buy was sold out.

I started off just using crushed almonds and the chocolate powder as the primary flavors, but that didn’t turn out so good. I discovered that adding a banana gave the shake just the right amount of body and sweetness. A hint of cinnamon and almond extract and this thing went from “meh.” to “woot!”.

From my blender to yours. I present the almond chocolate banana protein shake!

Almond Chocolate Banana Protein Shake

  • Small handful (about 3/4 cup) of chopped/slivered almonds. You will need to further crush these up into a powder-like consistency. I use a mortar and pestle.
  • 1/3 cup plain greek yogurt
  • 2 cups of plain almond milk
  • 1 Banana
  • 2 scoops of chocolate protein powder
  • 1 – 2 ice cubes
  • A few good shakes of cinnamon (maybe 3/4 tsp)
  • A drop of pure almond extract

Put it all in a blender and blend very well. Leave this one blending for longer than normal since the pureed banana tends to want to float on top of the almond milk.

The solar power box

I finally got around to completing a project I got started on months ago, and I’m pretty happy with the way it turned out.

I am a huge fan of alternative energy, especially solar and wind power. I think that in the future, these technologies will be affordable enough that everyone that has the spare roof space for panels or land to put up windmills will be able to get enough clean energy to power their every day lives. Given enough future improvements in battery technology, even cloudy and windless days here and there won’t be enough to cause much of a concern.

I wanted to be able to take a setup camping with me that could charge multiple cellphones in a reasonable amount of time, and have some spare capacity to be able to charge walkie-talkies, mobile ham gear, or even the batteries that we’re using in flashlights. For this task, I needed a bit more than the hand held portable solar chargers you see cheaply available on the market.

One day I walked into a West Marine store, and happened upon a set of 3 – 20W, weatherproofed, amorphous silicon solar panels (see here for a description of the different types of panels). These panels were on clearance and were being replaced with newer technology, but they were packaged very nicely with all sorts of different connectors as well as the ability to easily daisy chain the panels together by design.

You can see one of the panels below. I put an american quarter dollar on it to give some sense of scale. The panels are about 3ft (1m) long and about 1/3 as wide.


This single panel puts out 20 watts of power and enough voltage to charge a 12v lead acid battery of decent capacity. I decided that I wanted this setup to be portable so I opted out of a deep cycle marine battery for this application. Though consumer marine batteries have very good energy capacity (measured in amp hours), they are very bulky and heavy. I didn’t want this setup to be a large burden to lug around. The size of the panel makes it a bit cumbersome already, so adding 30 lbs of battery weight didn’t fit my idea of making this a portable setup.

Instead, I decided to purchase a couple sealed lead acid batteries like the ones that you can find in popular uninterruptible power supplies. The ones I purchased have a 7 amp hour capacity. I wired two of them together in parallel to offer 14 ah of capacity and a 12v output.

I had the batteries and the panel set up for a few months just sitting in the window with no case or anything enclosing any of it. It was just a hodge-podge of wires and looked like a science project. I knew I wanted something I could carry, so I tried attaching the panel and batteries to a cheap plastic rifle case with no luck. Since that failed, it was on to actually making a box out of wood. I enlisted the help of my father-in-law and my uncle and we came up with a pretty decent setup!


Here you can see the inside of the box with three 12v cigarette lighter style plugs. The first one on the left takes the input from the panel which they nicely supplied with a male plug end for just this use. The other two on the right are for whatever we want to plug into this thing. Currently the next time we go camping and deploy this box, I plan to bring a 120v inverter and a USB charging adaptor.

The large size of the box is so that the solar panel can sit on the top and carried along with the setup. Then when on-site, the panel can be detached and moved anywhere within about 8 ft of the box. You’ll note the batteries are wired up together with detachable cables and I’m using ground bars as bus bars for the positive and negative power feeds from the batteries. This makes everything on the inside a bit cleaner. The batteries can be detached by opening a latch and swinging that piece of wood out from above them.

In the picture below you can see the finished product charging an iPhone using the power of the sun!


Solar power appeals to me on multiple levels. This is not the first project that I’ve done using solar panels and lead acid batteries. In fact I completed a much larger one last year that I have a draft post just waiting to get finished here on wordpress. I guess I’ll have to try to get this up soon in between some of the other geek stuff!

Vector clocks and quorum consensus

While designing a distributed message queue, I noted that it would be difficult to properly order messages between a quorum of nodes if the system clocks between the nodes are not perfectly in sync or if messages came in faster than the system’s minimum tick resolution.

For example, on Windows 2008 R2 virtual machines running under Hyper-V, we’ve noted that the lowest resolution we can achieve without registry hacks is 15.6 ms and is related to the system minimum timer period. If two messages are received in under 15.6 ms, even on a single node,  they will effectively have the same tick count and it will be impossible to order them by wallclock time.

If the clocks on nodes aren’t perfectly in sync, Node A with a clock that is 3 seconds behind may see a message come in 2 seconds before Node B even though Node B’s message actually came in first. When working with virtual machines, clock skew, and large clusters, even protocols like NTP won’t keep clocks within acceptable limits to guarantee ordering.

Because of these issues with wallclock time, to accomplish a partial ordering of messages between nodes, I have looked into vector clocks which I see mentioned a lot in distributed systems and I think I have come up with a plan that lets them work with the hash ring quorum based messaging system I am developing.

Vector clocks track the number of events that have been processed by actors in a distributed system. Each actor tracks not only its current event count, but also holds knowledge of the number of events that it is aware of that have occurred with other actors. Using this knowledge, it is possible to deduce causality, and thus order between a series of messages.

There are a few methods for updating and passing vector clock data between actors in a system, and this actually seems to be the most interesting part of utilizing them. You can read more in Distributed Computing [Kindle Edition].

While searching for a way to utilize vector clocks in my system, I first found a way that didn’t work which I will demonstrate first, then I’ll explain the method that I plan to use which utilizes a quorum consensus to update the vector clock on a message.

Let’s take a look at the first example of using vector clocks that didn’t quite turn out the way I needed it to in order to decide on message order. See the crude diagram below.


In the diagram above, we are sending the message “hi how are you” in 4 pieces coming from two different producers. The first “hi” message comes from producer 1 (“p1”). At the time the first message is sent, the producer has no knowledge of the vector clock associated with the queue. In fact, the queue is brand new and all the nodes give this queue an initial value of [0,0,0] corresponding to the initial state of all three nodes involved in managing the range for this queue. When the first message “hi” is send from p1, it sent to nodes A and B that form a quorum for this queue. Both nodes are contacted and update their vector clocks to show that they have received an event for this queue. The return clocks for nodes A and B are [1,0,0] (the first 1 corresponding to node A, the first node in the range) and [0,1,0] (the 1 in the second position corresponding to node B, the second node in the range) respectively. Note that this is also the value of the vector clock recorded with the message “hi” for each of the nodes. As far as node A is concerned message “hi” has a clock of [1,0,0], and as far as node B is concerned, message “hi” has a different value of [0,1,0] for its clock.

When the producer gets these clocks back from the quorum, they are combined with max(vector clock A, vector clock B) and the producer now knows that the resultant clock from this operation is [1,1,0]. The producer p1 then sends “how” to nodes A and B again forming a quorum, and tags the message with its knowledge of the combined vector clock [1,1,0] to A and B. Node A receives this message, increments its part in the vector clock and ALSO updates its knowledge of the vector clock for Node B yielding [2,1,0]. Node B gets the same message, updates its value in the vector clock and ALSO updates its knowledge of the vector clock value for node A yielding [1,2,0]. The producer then combines these values again to yield a resulting clock of [2,2,0].

This process continues through p2 producing the message “are” and then p1 producing the message “you”. Then on the bottom right hand side of the diagram, we see consumer c1 connect to nodes A and C, and try to read the message stream.

Message “hi” is consumed from Node A, and its clock value there is [1,0,0]. Message “how” is also consumed from Node A, and its clock value is [2,1,0]. From this data we can deduce that message “hi” came before “how” because “how” has two values that are greater than those of “hi” and none that are less than it. So far so good, but not for long.


When we then consume the value “are” we have a problem. The clock value obtained from Node C on that message is [0,0,1]. While the third count is greater than the third count on any other message, the clock values for Nodes A and B are lower than any other previous message. This leaves us with an ambiguity in how to order the messages.

What I noticed while performing this mental exercise was that the producer DOES know the proper order at each of the steps. Since it is combining the vector clocks returned from the quorum, it has a better idea of the current clock for the messages than the nodes do. What if we take an extra step once we have contacted a quorum, and instruct the nodes to write the combined clock values back to the message?

You can see this sequence in the diagram below:


Now we can see a completely different (but familiar) series of events. p1 writes the value “hi” to nodes A and B. A’s resultant clock is [1,0,0] while B’s is [0,1,0]. p1 gets the quorum response back and then proceeds to tell A and B to update the vclock for the message “hi” to [1,1,0]. p1 then writes “how” to nodes A and B, and gets a resulting clock of [2,2,0], again instructing both nodes to write this clock value back to the message. This continues until we reach the end of the series of messages in the queue.


When consumer c1 connects to nodes A and C this time, it has a different view of the clocks associated with the messages. “hi” is now read from Node A, and has a clock of [1,1,0]. “how” is also read from Node A and has a clock of [2,2,0]. “are” is read from Node C and has a clock of [2,3,1] and finally “you” is read from Node A again with a clock value of [3,4,1] producing a series of clocks [1,1,0] -> [2,2,0] -> [2,3,1] -> [3,4,1]. There are no ambiguities remaining in how to order the messages, and the proper order of the resulting messages is maintained when they are returned.

As I learn more about distributed algorithms, I may discover a more efficient way to accomplish the same thing. For example, I haven’t looked into using matrix clocks yet to see if I might be able to skip the extra write-back step given the additional information that would be available. However, for now I am happy that this will at least solve the ordering issue.

Distributed messaging fault tolerance

In a previous post, I was left with a question of how to ensure that messages were not dropped in the face of a specific type of failure. The problem that I was faced with at the end of the article was the case where the consumer side loses its link to the common node where a producer is sending messages to in a quorum.


In this scenario, the producer continues pumping out messages to a quorum of nodes, unaware of the link failure on the client side. The client does not receive them until it makes a new connection to achieve a read quorum. Without adapting this design, messages will be lost while the client reconnects.


I have chosen to go with a design that allows the caller to specify a TTL on messages even if they’re not going to be sent to persistent storage. I will then implement an in-memory queue for messages that are received on a node and not claimed. Any message that is not claimed by an active subscriber will go into this queue for a configurable amount of time (I’m thinking 5- 10 seconds max) that will give the consumer time to recover from a temporary failure like the one above without losing messages. Though this wasn’t very important for my use case, it felt wrong to lose messages during ANY kind of failure on what is supposed to be a fault tolerant system.

The other option that I had entertained in the previous article was to just commit messages to all nodes that are up that handle the range, and only return success if the producer was able to contact at least a quorum of them. This would handle the above client side link failure case, but I threw an additional monkey wrench into the mix.

I want to borrow the idea of the coordinator node from Apache Cassandra. That is, any client connects to a single node and doesn’t have to worry about ring topology to begin passing messages with the system. I want to keep the design of the client as simple as I can so that it is easy to port to multiple languages.


In this configuration, even if a producer were to write to all the nodes that handle a range on the ring, we could still lose messages if the coordinator node went down. Writing to a quorum of nodes and holding unpiped messages temporarily in a time and space limited queue allows the coordinator to die without a loss of messages, as long as the recovery happens within the TTL specified by the producer of the message.

Disadvantages of this idea are additional latency and additional memory usage. Because I have to wait for the given TTL to expire before pronouncing a message claimed or not, when a client isn’t available to read a message there will be a minimum of TTL seconds in latency before I can respond back to the coordinator node that a message could not be piped. Because I have to keep a queue of messages for the given TTL, there will be memory used for that data.

However, one piece of good news is that because sopmq is optionally persistent, if a message is flagged to be persisted and it can’t be piped, I can store it to the Cassandra backend and return immediately once a quorum of nodes responds that the message could not be piped. In the persistent case, the latency will be minimal, and for us this is the case that is most important because it means we can quickly tell the user that their message was not delivered but was saved.

We’ll see in time how these decisions impact the design and development of the project.