Archive for 2006

I just wanted to say Happy New Year to everyone. And a special thanks to our customers. It has been great working with all of you to get your apps deployed here at EY. We were only getting started in 2006. 2007 will be a big year for us!

I also wanted to give a staus update on the control panel. I have had most of my time consumed with wearing multiple hats here at EY and haven’t been able to work on the control panel for about a month now. But it has been nice and quiet for the last few days with everyone enjoying the holidays. I was able to spend this time working on the backend of the control panel. It’s a custom capistrano/backgroundrb worker setup that will allow us to run any capistrano task right from the panel. Yesterday I was able to deploy an app for the first time solely through the control panel. This was a nice hurdle to get over and means that I will be calling for beta testers sooner then I thought. Stay tuned ;)

Now go out and party!

I’ve just completed upgrading our Subversion cluster to 1.4.2

I did this because a new customer wanted to mirror their repository to another server, a great idea that we completely support. You can never be too careful with your data!

Subversion 1.4+ includes mirroring support, but both servers must be running the 1.4+, so we happily performed the upgrade.

During the next maintenance window, I’ll likely reload the repositories from svnadmin dumps, to update the repositories to the new delta format employed by 1.4+. There’s no immediate need to do so as this version is entirely backward compatible with the older format.

We had noticed that occasionally, requests were a tad slower than at other times. We’ve been tuning here, and tuning there, and each little twist made a little difference.

I’m a big believer in tuning systems over time, and don’t like to twist each and every knob available to me when using a new piece of hardware, because it doesn’t allow you to understand why the company that built the product chose to ship it the way they did.

Today, I spent some time digging into the algorithms our load balancers use to distribute requests between slices.

As it turns out, the default algorithm the load balancer uses is rather naive.

I think that our vendor defaulted to the simplest algorithm, one that new users could immediately grasp: Deliver requests to each slice serially in round-robin fashion. This make it easy to determine that the load balancer is working, because all systems will get requests in a predictable and orderly fashion.

However, it’s far from ideal. Imagine a particularly resource intensive page on your site. If a slice is already serving one of those expensive pages, why ask it to serve another? It makes more sense to ship it to another slice instead.

The load balancer doesn’t really know the page is expensive, of course, but it can see the side effects of something like that, by noticing the response time. If a server is a bit slow on requests, new requests will get immediately routed to slices that are not bogged down. Very cool!

In any case, all customers will immediately see an improvement in average response time, and that’s a good thing for everyone!

I just finished automating Subversion repository backups!

We use svnadmin dump on each repository, tar and bzip2 them, then shoot them off to Amazon’s very cool S3 service.

Marcel Molina’s for the aws-s3 gem that makes handling S3 very simple.

Thanks Marcel!

Next project: use the same methods to backup user generated content to our customer’s Amazon S3 account.