Reality check on opensim stats

As I suspected would happen, a recent article on hypergrid business has people speculating that the opensim metaverse is crowding out and outgrowing commercial grids and will now all of a sudden take off without them.

So I did some fact finding and found something very interesting.

There has indeed been growth in opensim grids over the past 3 years, but it has been dismally small. Until people come to terms with this and start advertising their grids (be them free or commercial) this trend will continue. Of the growth that has occurred since 2011, the majority of it has happened on the back of InWorldz.

Using the statistics provided to me by Maria Korolov, I tracked total active user growth for all known opensim grids between December 2011 and December 2014. Are you ready for this?

Total opensim active users growth 2011-2014:

  • 5,043 users (yuck. for comparison over 7700 people have purchased minecraft in the past 24 hours)

Total InWorldz active users growth 2011-2014:

  • 2,803 or about 56% of the total for all new active users

If I go back to our peak a few months ago of over 8300 active users, we account for an even larger percentage.

I understand there are many out there that for one reason or another don’t like InWorldz, but between our continuing small marketing pushes on facebook, and our largest advertising campaign ever coming in the next few weeks, we contribute to a great degree to the inflow of new users to the opensim platform. Our servers and staff have taken care of over 100,000 registered users and we have a lot of lessons learned because of it, which we’re sharing as we can. Starting more infighting over dismal numbers isn’t going to grow the opensim VR space.

CentOS 7/RHEL 7 and docker containers on boot

I recently had to work with a CentOS 7 system and a few docker containers. These containers were bound to different IP addresses on the server NIC and it was critical that they be available after a server reboot. CentOS 7 comes with docker 1.3.2 built into the base installation. This version supports the –restart switch as follows:

–restart=”” Restart policy to apply when a container exits (no, on-failure[:max-retry], always)

Setting the containers to restart=always, rebooting the box, and running docker ps, I discovered that the docker containers were not starting with the OS. After going through a few combinations of commands unsuccessfully I decided to take a look at the message log for the system and I noticed something odd:

Received an unexpected error during port allocation: Error starting userland proxy: listen … cannot assign requested address

Taking a closer look at the logs I noted that this was often followed later by something like:

kernel: e1000e: enp0s25 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: Rx/Tx

It looks like the docker container engine was starting before the NIC was actually up. It was then trying to do its magic and create the bridge interface but the NIC with the given IP address wasn’t ready yet.

To solve this issue I looked around and found an article about systemd services specifically dealing with depending on the network interface to be up. Inside the systemd unit for docker, the service was set to depend on


Description=Docker Application Container Engine
Documentation= docker.socket

Additionally, /usr/lib/systemd/system/docker.socket was not set to depend on at all. To get everything rebooting with the system, I did the following:

  • Added to the Requires and Wants of both the docker.service unit and the docker.socket unit.
  • Ran
    systemctl enable NetworkManager-wait-online.service

    to stop boot processing until the network is available.

Both of these steps may not have been needed and you should play around with the configuration to see what works for you. My final .service and socket files were as follows:
[ddaeschler@docker ~]$ cat /usr/lib/systemd/system/docker.socket

Description=Docker Socket for the API



[ddaeschler@docker ~]$ cat /usr/lib/systemd/system/docker.service

Description=Docker Application Container Engine
Documentation= docker.socket

ExecStart=/usr/bin/docker -d $OPTIONS $DOCKER_STORAGE_OPTIONS


After making these changes, the containers start up normally at boot. I hope this can help someone else running into this issue.

Xcode handy key bindings

This is a tiny postlet on some key combinations I have found handy while using Xcode. I’ll update this as I run into more invaluable shortcuts.

  • shift + cmd + o - Quickly open file
  • shift + cmd + j – Reveal in project navigator
  • ctrl + cmd + up / down – Jump to next counterpart (ex. .h/.cpp file)

A ton more are listed here:

Relay for Life on InShape!

Relay for Life on InShape!

On November 1st 2014, something profound is going to happen, and I’m super excited to be a part of it.

For the first time, avatars and the people behind them are going to run the American Cancer Society Relay for Life, for real, virtually.

What in the world does that mean?

RFL_Snapshot_001Using the InWorldz InShape system, beta testers from the virtual world will get together on a virtual track that has been designed by Relay For Life of InWorldz volunteers. They will log into the virtual world, and by using acceleration data supplied by the cell phone in their pockets, they will transmit walking, biking, and running forces from their real life into their avatar. That data will be used to move them through the Relay for Life track.

From exercise bikes, to rowers, to treadmills, the harder they run in real life, the faster their avatar will go. Their dedication will show through form the real to the virtual. Their avatar will become an extension of the good we all want to see in the world.

Business stamina

We will gather from around the world. We will run the same track for the same great cause, and we will help to give hope where hope is needed.

This is what virtual worlds are all about. Fighting for good causes and using virtual worlds technology to augment the human experience. If you are an InShape beta tester, please join me on November 1st to help kick cancer right in the backside. Let’s get together and sweat to draw attention to a disease that affects us all, and needs to be eradicated before it can claim any more lives.

Run for life.

(Watch the livestream here starting at 9 AM PDT)

.@subquark’s Mint Tin Pirates!

.@subquark’s Mint Tin Pirates!

Mint Tin Games

I had the fortune of getting an early copy of SubQuark’s Mint Tin Games, and we’ve now spent a bit of time playing them. I have to say I’m super impressed with the quality of the games, as well as their playability and replay value.

Mint Tin Games are quick and easy to learn card games that fit in your pocket. They’re designed to provide fast action when you need to kill 10 minutes of time and don’t feel like staring at the internet. They’re super fun and even playing the game with adults has proven extremely satisfying. We’ll be keeping these around for snowed in days.

You can see the kickstarter for mint tin games here. If you like the idea, hurry up and pitch in a few bucks! It’s only running for one more week!

Mint Tin Pirates

Min Tin Pirates starts off rather innocently. You’re a pirate doing pirate like things on the high seas. Your days of plundering have been going well and your crew is rowdy and ready for action. Just when you think you’ll be off to take another merchant ship filled with booty, you encounter your arch nemesis Capt. Red or Capt. Black (depending on which one you happen to be).

A battle ensues, each player draws 5 cards from the deck. When you get a pair of matching attack cards, you roll the dice to determine if your attack is successful or not. Being pirates, you have a wealth of deadly weapons on hand. Your arsenal includes cannons, bombs, knives, and pistols. Only cannons can directly damage the opponent’s ship. The other attacks send his crew mates to Davey Jones’ locker. You both know full well that without a crew, the ship is as good as sunk.

Mint Tin Pirates - Game in progress

Mint Tin Pirates – Game in progress

Amidst the fighting, you notice that one of your opponent’s crew has decided desert and join your band of buccaneers. Clearly you are the superior captain. However, much to your shock and dismay, some sorcery has brought back one of your opponents crew from the murky depths. Between the desertion, and the resurrection, you consider this round a draw. But the battle isn’t over until the other captain has no crew, or his ship is sunk!

Once the smoke has cleared, the score is tallied. The pirate who’s ship is still floating, or who has the most resources left (including possibly the gold!) wins the game.

Mint Tin Pirates - Black wins

Mint Tin Pirates – Black wins


Mint Tin Pirates has enough twists and turns to keep the game fun and interesting through the very fast gameplay. Even though the game mechanics are very simple, you will never play the same game twice. Even better, I noticed that due to the speed of gameplay, though I was on the majority losing side, it didn’t sting so much and kept me wanting to come back for another chance to get even, and get the gold.

You definitely need to check this game out. It makes a great distraction and a lot of fun on camping trips, or long flights.


Subquark’s blog:

The kickstarter:

Pumpkin Pie Protein Shake

Pumpkin Pie Protein Shake

I happened to have some leftover pumpkin from making paleo pumpkin pancakes the previous night, and I wanted to figure out a way to use some more of it. Those cans of pureed pumpkin are absolutely huge and it would be a shame to let most of it go to waste.

I started thinking how much I really love pumpkin pie during this time of year, but I don’t want to eat all that sugar. Maybe there is a better way to conquer my craving.

Thus the pumpkin pie breakfast smoothie/protein shake was invented

Pumpkin Pie Protein Shake

1/2 Cup of pumpkin puree
1/3 Cup crushed pecans
1/2 Cup greek yogurt
1/2 Teaspoon pumpkin pie spice
1/4 Teaspoon vanilla extract
1 Cup milk
2 Scoops vanilla whey protein powder
2 Ice cubes
Sweeten to taste with ~1/4 tsp pure stevia or other sweetener

The pumpkin pie spice is sold premixed in a container and is a blend of cinnamon, ginger, nutmeg, and allspice. Crush the pecans in a mortar and pestle until they have a powdered consistency. Add the pecans, yogurt, pumpkin pie spice, vanilla extract, and sweetener before the milk. Then add the milk, then the protein powder. This makes sure the protein powder sticks to the sides less. Finally, add the ice cubes and blend well.

Message ordering in the face of failure

Message ordering in the face of failure

Another day learning and coding and another interesting problem!

While designing a distributed message queue, I decided to use vector clocks to preserve message ordering when producers are talking to a different quorum of nodes. You can see some of my conclusions in a previous post titled Vector clocks and quorum consensus.

Vector clocks do seem to solve my ordering problems pretty well and don’t require me to have perfectly synchronized wall clock times between nodes to accomplish it. However, a problem arises with my original implementation in the case of a node failure.

Originally, I intended to associate counter values in vector clocks with a particular queue. This would mean that the vector clock’s value would be representative of the state of just the given queue, and when that queue was collected due to a lack of messages, all counter values would be reset to zero. After finally getting to the point where I was coding these portions of the design, I realized that this creates a problem in even the most basic and obvious scenarios:

  • Producer 1 (p1) queues message “Hi” to nodes A and B. Resulting clock is (A:1, B:1)
  • Consumer 1 (c1) consumes message “Hi” and sets its clock to (A:1, B:1)
  • [… TTL passes…]  Queue is garbage collected since all messages have expired. Clocks are reset.
  • p1 queues message “How are you?” to nodes A and B. Resulting clock is again (A:1, B:1)
  • c1 asks for all messages since “Hi” (A:1, B1) and gets nothing.
  • Oops.

Since the client is using the vector clock to ask for new messages, we can’t simply arbitrarily reset the clock. So instead we move the values of the clock to the server instance using a 64 bit unsigned integer to ensure we don’t get overflow. This solves the problem.. Well kinda, in a perfect world. Lets examine what is still wrong.

  • Producer 1 (p1) queues message “Hi” to nodes A and B. Resulting clock is (A:1, B:1)
  • Consumer 1 (c1) consumes message “Hi” and sets its clock to (A:1, B:1)
  • [… TTL passes…]  Queue is garbage collected since all messages have expired. But the clocks are not reset since they are based on the count for the node.
  • p1 queues message “How are you?” to nodes A and B. Resulting clock is (A:2, B:2)
  • c1 asks for all messages since “Hi” (A:1, B1) and gets “How are you?” with clock (A:2, B:2)
  • Yay!

So far, so good. But now what happens when A dies and then comes back with its clock reset? Well.. Nothing good.

  • Node A dies
  • Node A comes back to life with clocks reset
  • p1 queues message “I’m doing well!”. Resulting clock is (A:1, B:3)
  • c1 asks for all messages since “How are you?” with clock (A:2, B:2)
  • Uhoh. Clock (A:1, B:3) “I’m doing well!” does not follow (A:2, B:2) “How are you?”. We get no messages :(

What do we do? We can keep our vector clock up to date every time we receive a message and for example write the value to disk, but that involves a lot of writes on the disk as well as being prone to not being up to date due to a crash before the updated clock can be written.

What I’ve decided to do is keep tract of the number of times a node is restarted and send that number with each node clock. That way, if a node restarts and the event clock is reset, we can use the restart count to know that any message with a lower restart count comes before a message with a greater restart count, even if the event clock is greater. Lets examine the above scenario with the new rules:

  • We set up clocks as (restart count/event count since last restart)
  • Producer 1 (p1) queues message “Hi” to nodes A and B. Resulting clock is (A:1/1, B:1/1)
  • Consumer 1 (c1) consumes message “Hi” and sets its clock to (A:1/1, B:1/1)
  • [… TTL passes…]  Queue is garbage collected since all messages have expired. But the clocks are not reset since they are based on the count for the node.
  • p1 queues message “How are you?” to nodes A and B. Resulting clock is (A:1/2, B:1/2)
  • c1 asks for all messages since “Hi” (A:1/1, B1/1) and gets “How are you?” with clock (A:1/2, B:1/2)
  • Yay!
  • Node A dies
  • Node A comes back to life with its event clock reset
  • p1 queues message “I’m doing well!”. Resulting clock is (A:2/1, B:1/3)
  • c1 asks for all messages since “How are you?” with clock (A:1/2, B:1/2)
  • Success! Clock (A:2/1, B:1/3) follow clock (A:1/2, B:1/2) since A’s restart count is 2 vs 1 and B’s event count is 2 vs 1.

In this way we can still determine message ordering after a node restart and all is right in the world.. until the next challenge.