For this post, let's take a look at Amazon's EC2. We've used it here at Engine Yard, our friends at RightScale use it, and there's generally a pretty good amount of information about it floating around the web. EC2 has a very distinct and interesting behavior profile. Let's take a moment to see what consensus there is about that behavior. The salient points of most coverage I can find is roughly:

1. EC2 is fairly variable in terms of absolute performance.
2. EC2 is at least within the same ballpark as most commodity equipment.
3. EC2's EBS performs slightly less well than native disks.

I think everyone with any experience would have guessed the above. Machines are going to have little differences. Small differences in the randomized layout generated by randomized resource allocations have noticeable effects on the behavior of applications. EBS, being some sort of network attached disk, runs slower than a normal disk because it's more than just a disk. All of this makes complete sense.

Given the above, the question is "Why are these benchmarks important?" Some more pedantic types would chime in that seeing what you expect on a benchmark is good confirmation. Others with a need-for-speed will claim that this is evidence that rolling your own infrastructure is always preferable because of the aggregate speed benefits. However, I think that these benchmarks only serve to show what is most important to Amazon. I also humbly suggest that what's good for Amazon is good for you. At least, it should benefit you if your sites grow at all.

In general, these benchmarks show that EC2 is designed for making scalable applications. Their performance isn't top-of-the-line, but it's not abysmal either. An EC2 instance is appreciably slower than bare metal, but it's instantly replaceable. EBS isn't crazy fast, but it's a portable, durable data store.

This is what most of the benchmarks out there are missing. In interpreting these benchmarks, I immediately realized that these are not the metrics that matter to scalable applications. Raw performance has at most an instantaneous, linear effect on your application. Every one of these metrics is only concerned with raw performance. As such, they're not really useful.

A noteworthy suite of benchmarks would be done on metrics like "cost-to-grow per request." Such metrics are slippery and difficult to nail down, but they are where EC2 really shines. It's clear that the message from EC2's benchmarks is that performance isn't king, scalability is, and effective application architecture isn't easily benchmarked. EC2 is designed for applications that can scale.  Such applications don't demand the highest performance, they demand moderate performance. They don't demand the utmost high-availability, they demand tolerance of failures.

From that angle, it's pretty clear that both camps are right, sort of. EC2 benchmarks exactly as we'd expect (making the pedants happy), and it's built to deliver applications at the top end of throughput (theoretically making the speed-addicted happy). The catch is that there is a fundamental switch in how you view throughput. Rather than being about performance, it's about aggregate performance (sometimes called scalability). That has everything to do with the type of applications on which Amazon was built.

So, how should we interpret the benchmarks of EC2? We should interpret that the Cloud is about building applications in different ways than previously possible. We should see that in a world where hardware is commoditized, squeezing out the last drops of performance plays second fiddle to adding hardware. We should notice that making applications tolerate failures is the new "high availability."

In my opinion, it's an interpretation that's long overdue.

I've been working on a solid, cross-platform installer script for all of the components. Kevin Smith and Kirk Haines have all been tearing into the integration testing. John Hornbeck has been visiting to get more information on the project. He (and his Vertebra pristine laptop) have been sussing out the system dependencies that we developers take for granted. Sam has been directing the work and working towards a some good demo material for the upcoming screencast.

All in all, great work. As an added bonus, I brought my camera. I've posted "a set on Flickr":http://www.flickr.com/photos/kagato/sets/72157608035805222/ that contains the latest photos. I may add a few more batches before the end of the sprint.

Internally, we've used Subversion a lot. It works well enough, but, particularly with development on projects as large and hot as Rubinius, flexibility is king.

So, let me set the stage. I have this project that I've been working on. It's been the focus of most of my coding effort for about a month. We are also a geographically distributed organization. Finally, we've recently been moving things from SVN to Git.

I woke up this morning and, to my surprise, one of our very talented developers in the UK had taken in upon himself to move my code to Git from it's comfy home in SVN. After a brief bout of unhappiness, I set out to "git" the source and continue development unabated.

To my chagrin, he had put the source into a repository without giving me access to said repository. Specifically, he had given access to everyone who was in IRC at the time, and I was not among the lucky ones.

At this point, our boy Ezra came to the rescue. He let me pull from his repository and I worked. After a while, I got access and committed my changes. As time progressed, I was able to pull and push to his repository to get and provide updates.

Had this been CVS or SVN, I'd have been out of luck. Pushing to a new repository would have been a pain too. With git, I was free to work and everything else sorted itself out. Changing to the official source was a single git-config command.

I don't know about you, but I just think that's pretty neat.