Why a virtual world?

I’d like to start this post off with an interesting graphic.

The term "flappy bird", vs virtual worlds, and Second Life®

The term “flappy bird”, vs virtual worlds, and Second Life®  | Google Trends

A simple iPhone game called “flappy bird” has attracted explosive attention in the past few months. Created by an independent game studio in Vietnam, this game went viral before the publisher removed it from the app store. If the game wouldn’t have been removed, you can see it was well on its way to garner more search engine attention in a few months than virtual worlds have had in six years.

These days mobile apps and games are hot items. There is a lot of competition, but there doesn’t seem to be a lack of demand for new apps based on new ideas, and challenging games that you can pass some of your free time on while waiting for the bus. Challenging old style platformers seem to be back on the rise again, and a simple one can be completed in less than a few thousand lines of code. The market is ripe and there is an extremely large set of users of mobile devices that play games. Make a decent/fun/aggravating or innovative game, advertise it, and you have a chance of making a living. It is not easy money, nothing is, but there are much more difficult ways to spend your time as an independent software developer.

Speaking of which, developing a virtual world platform is not easy. If you get the opportunity to grow, you will end up running into some of the most challenging problems in computer science, where there is still much ongoing research to try to solve problems, or make operations faster and scalable. At scale, a virtual world will store billions of large binary assets and related records and must be able to search and analyze the data, often in realtime. You’ll have to solve many of the problems of modern MMOs, plus you’ll need to be able to handle user generated content that is not optimized for fast 3d rendering. You’ll have a community that is depending on your work to keep them in business creating content, and any deviation in behavior no matter how small it may seem to your team may well end up being a big deal to them, causing you to have to go back to the drawing board. You’ll do all this, commit hundreds of thousands to millions of lines of code, a ton of money and man hours to your platform, and you may end up doing all this for less income than what you might make on a single release of an iOS game. When you look at the graph above it seems that we’re fighting a losing battle.

So with much of the current opportunity for freelance development gone mobile, what is keeping some of us working on virtual worlds?

We enjoy the challenge.

I think first and foremost developers are attracted to virtual worlds because we enjoy the challenge of creating a stable scalable platform, but we don’t want to get involved in integrating and creating the actual content. When I was asked to join InWorldz, what really pulled me in was that I knew I’d be dealing with large ever growing datasets (with a user expectation of zero downtime), 3d mathematics, physics simulations, script engines, and a bunch of other topics that require a lot of learning to master. I really wanted to dive deep into solving hard problems and InWorldz was the perfect opportunity.

We believe 3d environments have a bright future

The true potential of 3d hasn’t yet been realized. Many of us watched Star Trek as kids, teens, and young adults, and we loved the idea of the holodeck. The holodeck was a place where books, and games appear before your eyes in full 3d and are fully interactive. More importantly, the holodeck allowed you to create your own sequences, turning your imagination into a 3d canvas. We believe with the acceleration of development of 3d hardware and appropriate sensors, general purpose 3d environments will provide a place where businesses and individuals can express their ideas in a realtime, fully interactive, 3d scene.

We like seeing what you create

We love seeing the creativity that is unleashed when people from all over the world come together and generate their own purpose and content on the blank 3d canvas we have provided. We see everything from naval battles, large and diverse elven colonies, to sci-fi role play with ideas that rival the greatest authors. You create games, communities, and entire cities full of interesting things to see and do. We get to watch the world come alive.

We believe virtual world tech can be used for good

Open ended virtual worlds give us the opportunity to use our skills to do some real good in the world. We’ve already seen this with Relay for Life and the countless other charitable contributions that have been donated by members of virtual worlds. This Christmas alone InWorldz and residents raised $4,538.00 US dollars for the Toys for Tots foundation.

We want to get our technology involved in directly helping people as well. InShape is an idea we believe can motivate people to get up and be healthy. We think it can make exercise fun again and provide an additional benefit of allowing people to design their own exercise environments and equipment. We’re trying to take the strengths of virtual worlds and apply them to real problems, and this is only the beginning. Maybe some day we’ll be fortunate enough to see or be part of projects that can help rehabilitation, and maybe we’ll even influence more distant future advances like what we see in the video entitled World Builder below.

Virtual environments let you build your imagination. Let’s build the future together.

C++11 Reference Capture

In a recent post I began exploring some of the features that have popped up in C++11 since I’ve been on a mostly .NET and C++/CLI path lately with InWorldz projects. One of the features I mentioned was the new lambda syntax available in C++11.

C++11 allows you to automatically capture variables that you use inside a lambda function by using special qualifiers in the beginning of the lambda definition. I began using lambdas after quickly glancing over the syntax thinking that I completely understood them from my previous experience in other languages. I hadn’t taken into account that C++ is not a garbage collected language, and the differences this would make when referencing variables in a lambda turned out to be extremely important.

There is one principle gotcha in lambda reference capture. This issue is also a common gotcha in C++ programming in general. If you guessed that I’m talking about object lifetime, you are correct.

I had the following code running in test mode on my development machine:

try {
	this->performAssetRequest(std::get(work), std::get(work));

} catch (const std::exception& e) {
			<< "[CLOUDFILES] Caught exception while trying to get asset: " 
			<< e.what() << std::endl; 

Randomly during execution of a long string of requests I would run into an access violation. The access violation would always happen inside the boost ioservice while it was calling my lambda. For a while I was quite confused since the exception message never seemed to be valid. It seemed like some kind of corruption was happening upon throwing the exception. But then I looked closer at the lambda and capture code.. What exactly does [&] mean to C++ besides “capture any variables mentioned in the lambda”.

It turns out that [&] tells the C++11 compiler to capture the std:: exception e by reference. What this means is that we are taking a reference to the memory that contains the exception. As long as that memory is guaranteed to be valid by the time the lambda is executed, this is an efficient way to capture a variable.

However, in my case there were no guarantees. In fact, after the execution of the catch block, e will be destroyed. After e was destroyed, sometime later boost::asio would call into my error reporting lambda and blammo, we’re accessing freed memory.

C++11 allows you to specify capture by value. Essentially, instead of being dependent on the lifetime of the original captured object, we instead make a copy of it that will be passed to the lambda function later. This is exactly what we need in this case because we know that e will go out of scope and be destroyed before we execute the lambda function.

To specify capture by value instead of by reference we change [&] to [=] as shown

try {
	this->performAssetRequest(std::get(work), std::get(work));

} catch (const std::exception& e) {
			<< "[CLOUDFILES] Caught exception while trying to get asset: " 
			<< e.what() << std::endl; 

Now this code will do what we want, and as an extra added bonus, it won’t crash. Finally, all is well in the world.

Virtual machine state transfer

On the InWorldz grid to run user created dynamic content we designed a script engine called Phlox. Phlox executes scripts that are written in the LSL scripting language. The script engine consists of a bytecode compiler, a virtual machine, and a runtime environment that provides scripts with access to fast execution and functions that manipulate the state of the world simulation.

Some questions have come up recently about how running scripts attached to live objects on the InWorldz Grid move between simulators. An object on the InWorldz grid is not static. It can move around freely using physics or kinematic movement within the confines of the permissions assigned by region owners. This means that objects may potentially cross between regions either under their own control, or under the control of an avatar driver. Load sharding on the InWorldz grid at the simulation level is done via spatial partitioning. Each region is 256 x 256 meters and within that area all the scripts on the region run in the same process/address space. This means that in order to let an object move from one partition to another, which are potentially running on completely separate servers, we need a way to pack up the script no matter where it happens to be in its execution. We can’t wait for safe points since such a thing does not exist in every script. Take the following LSL fragment as an example. If you don’t know LSL, don’t worry, it is very similar to other C based languages:


  touch_start(integer total_number)
    integer a = 0;
    integer b = 100;
    for(; a < b; ++a)
      llOwnerSay("Hello Avatar! " + (string)a);

When touched, this script will count from 0 to 99 and will output the current result to the owner of the object. The script will continue execution inside the touch_start() event until the count completes in about 100 seconds. This is fine when you are in an environment that will stay running and where you wont have to worry about being moved around, but what happens when an avatar is wearing this script in an object on their wrist as some kind of watch and they wish to move to another region? As mentioned previously, regions are spatially partitioned. The region they want to go to may not even be on the same server. How can we pack up a script like this, or another that may be deep inside nested function calls with no end in sight? We run into the same problem when we simply want to save an object to storage. The script is constantly changing and could be anywhere along its execution when we come by to persist the object data. We can’t simply hope and wait that eventually the script will exit a loop or that a script will be generally well behaved.

We need access to the full call stack as well as all global and local variables involved in the call chain. We need to be able to pack that data up so that when we return to the call on a different machine we can unwind the stack the same way as each call returns. We need access to this data NOW, not in 1 second, not in 10 seconds, and not in 10 minutes.

Since we knew we would need to support full runtime data collection, Phlox was designed to provide this data quickly and on demand at just about any point in a script’s execution. Let’s take a look at some key data structures that govern and track an executing script.

The script once compiled and associated with a running virtual machine gets its very own instance of a RuntimeState class. This tracks the current state of a script as it executes inside the phlox VM. A partial snippet of the structure is listed below:


The components shown are:

  • IP – The instruction pointer. This variable tracks the current memory address of the instruction that is executing on the phlox virtual machine. This variable is incremented as each instruction in a script is executed, and can vary wildly for jumps and function calls.
  • LSLState – This is the current runstate of the script. If the script is disabled, sleeping, etc it will be indicated here.
  • Globals – A script’s global variable values.
  • Operands – The operand stack. This is used for intermediate values when we’re doing long strings of calculations like x = 1 + 2 * 3 + 4
  • Calls – A set of callstack objects that can be reconstructed that provide us with a current view of all function calls, even nested or recursive ones.

Given access to this information and a LIFO stack of StackFrame objects, we now have everything that we need to reconstruct a script. Each stack frame represents a function call and its structure is shown below.


The components of the StackFrame are:

  • FunctionInfo – Information about a called function such as the number of parameters it takes and its memory address and name
  • ReturnAddress – When this function returns, this member contains the address of the instruction in the script we should return to
  • Locals – The list of variables that were passed to this function

With all of this information, a simplified procedure to capture the full state information becomes pretty straightforward

  1. Stop the script from executing any more instructions on this Phlox VM
  2. Save the IP to identify where in the script we are executing
  3. Save the current global variable values for the script
  4. Save the stackframes list to get a full view of where we’re at in the execution of our script
    1. Along with the stack frame we now have a full view of all functions involved in the current call and all their local variables
  5. Save the operand stack so that even if we’re in the middle of executing a single line of script we can still pause the script and resume its execution without error.

We then serialize this into a compact format, transport it to a new machine and inject it into a running VM. It fires up right where it left off milliseconds later, faster than a single simulation frame, and the script continues to do whatever it was doing on a new server. All thanks to the power of a machine that is completely defined in software.

It should be noted that something like this happens but on a far more massive and exciting scale each time you migrate a running virtual machine to new hardware on your favorite virtual machine platform. See Hyper-V Live Migration FAQ and Chapter 21. Xen live migration for some really interesting information at how the same techniques are used in big VMs.

Programming in isolation

When I was a kid, I used to love coming home from school and messing around with my computer. I remember being 12 years old sitting in front of an IBM PS/2 model 25. 640k of ram was all I would need to explore the machine and learn as much as I could about how it worked. There were much more advanced 286 and 386 machines available at the time, but I didn’t care.

DOS and BASIC were my friends. With these two tools, I was able to tell the machine anything I wanted it to do and watch the instructions be executed in the exact same manner each time. With each run and iteration I would tweak and adjust the program to do what I wanted, fixing problems as I went along. Some bugs actually became features as I designed an e-comic called “Barfield” (a garfield parody) with my friends. The comic was pretty twisted at times, but it wasn’t always on purpose.  At one point, a yellow fill call ended up escaping the line drawn object it was supposed to be contained in. This became part of the plot. I won’t explain exactly how this incorporated into the story, but think of a group of almost teenage boys laughing at a bunch of yellow filling a living room. I’m sure you’ll follow right along.

Bliss! I could compile down the comic with quickbasic into an exe and share it with my friends on a floppy. Even those that had not been involved in the development of the comic could see it in all of its glory. We shared it with everyone. We had become artists and writers. Nothing could stop our comic from conquering the world. It turns out that this was also a great way to spread the then prevalent boot sector viruses AntiCMOS, and stoned.

As the years went on, programming in my childhood tended to be fun, engaging, a great learning experience, and often times very social. But then something changed.

While in my senior year of high school I began working as a professional software developer. Developing business software in Visual Basic and C++, I started to understand programming wasn’t always all about fun and games, but was actually hard work. Gathering and understanding business requirements, translating user needs and wants into code, bugs actually meaning something more than an occasional yellow filling of the screen. Someone was counting on you to make sure that their vision was properly executed, tracked, and catalogued.

Though I had interaction with numerous employees throughout the day and had co-workers on and off throughout my experience at that first programming job, the vast majority of my time involved sitting alone in an office staring at code on a screen. Following that first experience, I worked a few other jobs, and I ran my own consulting business for a while. Sometimes I would get lucky and have a peer programming or software architecture project where I was constantly interacting with other programmers, but the vast majority of my time was spent in isolation. Just myself and the machine.

Fast forward to InWorldz. What an awesome project for a programmer. 3d work, lots of important data, huge performance challenges, internet scale architecture. I get to meet tons of interesting people, I get to try to help them build out their dreams in realtime. I get to see vast flowing landscapes, amazing artwork, and avatars of all shapes and sizes. When I’m able to actually log into the world that I helped to build, I am always greeted by friendly people doing some fascinating things.

But there is a downside. I work from my home office, and I get even less real life interaction with the people I work with, admire, and care very deeply about. InWorldz employees, owners, and contractors are spread literally throughout the world. We’ve had to send equipment to Guatemala,  I try to travel to Nova Scotia once a year to see my right hand man, and I’ve made plenty of trips to Texas.

I know many programmers in the same boat as me, so what do I do when I start to feel disconnected from the world when most of my time is spent staring at a screen?

I log into InWorldz and see what our customers are up to

This is one of my favorite activities when I’m feeling isolated and disconnected. It gives me a sense of purpose and shows me the value of my work and why I am working so hard. Seeing and hearing about people enjoying themselves and being productive due to your efforts reconnects you to the people you’re doing the work for.

I start or get involved in some twitter conversations

There are plenty of people on twitter that will join into a conversation about pretty much anything. You can find everything from geek talk to food. I often try to spark up a day or two worth of food picture postings. The food some people make at home looks like something from a gourmet restaurant! The world is chock-full of amazing, and sometimes it is the simplest things that strike the right chord.

I got a pet

My cat Chester reminds me daily that life is not all about code and bugs. Sometimes life needs to be about being silly, sleeping in, and getting some attention from the people that are close to you.

I sit on Skype or debug in-world with my co-workers

While not a substitute for working alongside people, an open group Skype audio session can really make your home office feel like a real office with all the banter and distraction you can handle for a while. Because InWorldz is a virtual world I can also join my compadres in 3d while we all attempt to break things and reproduce problems. Finding some bumper cars on the beta grid and crashing into each other is a great way to remember we’re all human regardless of distance and lack of physical presence.

I go to a coffee shop and work

Sometimes it is enough to just be around people and have some conversation and good coffee in the background. If you need to get out, a coffee shop where they’ll let you sit and work while ordering drinks all day may be the ticket.

I hope this post can help others in the same situation I am. Don’t forget to reconnect with the world when you need it. Remember that you are not and will never be an island. Reconnect.

Discovering C++11

It seems that while I was off in the land of C# and C++/CLI, something really awesome happened in the C++ world. The C++0x standard was ratified and became C++11. Not only that, but there seems to already be wide support for the vast majority of features in modern compilers!

I think C++11 puts C++ much closer to the productivity levels of more modern programming languages. This is something that C++ desperately needed to stay relevant for more than just “performance oriented” programming.

Changes to the language are too vast to write about in their entirety, and since I’m late to the party, most of the best bits have already been covered extensively. However, I’d like to go over a few of the features that I have found vastly boost my productivity.

The improved auto keyword

We can now use auto in place of very long type names. The compiler figures out the type from the expression and everything stays type safe.

//instead of
std::vector<std::string>::iterator start = vecOfStrings.begin();

//we can now write:
auto start = vecOfStrings.begin();

Range based for loops

We can more easily iterate through C++ collections

	std::vector vecOfStrings;

	for (auto sayStr : vecOfStrings) {
		std::cout << sayStr << " " <<;
	//instead of
	for (std::vector<std::string>::iterator i = vecOfStrings.begin(); 
		i != vecOfStrings.end(); ++i) 
		std::string& str = (*i); //... 

Lambda Functions

We can define small, one off functions that perform a task at the call site rather than away from it

void function() {
	std::vector numbers;

	std::for_each(numbers.begin(), numbers.end(), 
		[](int i){ std::cout << ' ' << i; });

//instead of this
struct funky {           // function object type:
  void operator() (int i) {std::cout << ' ' << i;}
} functor;

void function() {
	std::vector numbers;

	std::for_each(numbers.begin(), numbers.end(), functor);

There are many more features present in C++11 that are worth your time. If you haven’t used C++ in a while and you have a project that demands staticly typed, native object oriented programming, C++11 is worth a close look.

Peanut Butter and Jelly Smoothie

Let’s get this blog started off right. With some food!

I’ve been making breakfast smoothies of all different kinds off and on for years now. Smoothies are an easy and quick meal, require very little prep, and are pretty easy to clean up after. This makes them ideal for my mornings where I wake up and just want to get to work as quickly as I can.

I present you with the approximate ingredients to the smoothie. I don’t normally measure anything out but this should get you started. Make one and then adjust to taste.

photo (1)

Peanut Butter and Jelly Smoothie

  • 3/4 cup nonfat greek yogurt
  • 1 cup frozen three berry mix (raspberry, strawberry, blackberry)
  • 1 cup nonfat milk
  • 4 tbsp unsweetened creamy peanut butter (smuckers natural is really tasty)
  • 2 scoops vanilla whey protein powder
  • a small amount of agave nectar (to taste, depending on the ripeness of the berries)

Put it all in a blender and set it for the smoothie setting if you have one. My blender does a bunch of pulses on high to mix things up then runs a blend cycle for a few seconds and repeats.

It should be noted this is not a low fat, or low anything smoothie for that matter. The peanut butter will add a significant amount of calories and fat. I drink this on mornings when I’m going to do weight training at the gym in the afternoon. That being said, it is probably better for you and more nutritious than many other breakfast options. An added bonus is that it will keep you filled up for quite a while.

I hope you have a great day! Let me know if you try this.

Getting started

I’ve wanted to start a blog for a long time. I love twitter, but sometimes there is just too much coming out of my head to put in 140 characters.

I plan to use this space for both technical/engineering posts as well as my personal views of the world and things that I see going on around me. My goal is to write at least one post a week derived from something I’m working on.

I hope this blog may be of some use to others.