Archive for June, 2009

Sharding is usually the final strategy to reach for when scaling a Ruby on Rails app: caching, offloading, and data segmentation are usually the first strategies to implement when scaling your application (they’re usually easier).

It probably sounds obvious, but it’s always important to find out what part of your application needs help before you start re-architecting. If you’re having issues with your database, and you build a spiffy disk sharding scheme, you’ve just fixed a problem that doesn’t exist. So, doing the proper discovery will allow you to allocate your efforts for best effect.

Finding your performance hotspots is very important in this process. A hotspot is a point in the architecture where you’re running at high percentages of capacity, or where your application is spending a lot of time. Hotspots are where the flames start. Knowing your points of pain allow you to triage correctly, and to know how to best spend your developers’ time. Using a combination of resource monitoring (like nagios) and performance introspection (like New Relic) is essential to identifying your Ruby on Rails hotspots.

One of the things to keep in mind is that this process is ongoing. When you clear out one hotspot generally another one will pop up to take its place as you grow. You might be optimizing disk reads and writes one week, and be neck deep in a SQL re-write the next.

If you have a proper staging setup, you can build estimates against generated traffic. This can give you a (blurry) view into what the next hotspot might be. A good process is to capture an hour or so worth of traffic on the live site, and replay it two, three, or more times faster against the staging environment. You want the traffic to be as real as possible. You can even go further and do a formal load test using a tool like browsermob.

When to go deep, and when to go long

After you have killed all of the hotspots you can, and added all the resources you can afford, it’s time to look at the next level. Usually this is when you start to see people thinking about sharding of some manner. There are three major types of sharding at the moment – File System, Database and Application. I’ll touch on each of these topics, starting with the highest level, and hardest.

Ruby on Rails Application Sharding?

Application sharding is the most extreme, provides the most benefit and is the hardest to accomplish. There are several ways to accomplish application sharding.

If you can split your users amongst several vertical groups, you can basically install copies of the application for each segment. This method assumes that users in each group will not need to interact.

For example, if you can segment your user base into three groups who do not really interact, you can simply provision 3 environments and install 3 separate copies of the application. An example of this might be a site hosting application. Each site hosted will not need much (if any) interaction with the other sites hosted. This is by far the easiest method of sharding your application.

You can also look at abstracting any shared logic into a back end service accessible via API. The rule of thumb there is to have each back end application do one thing, and do one thing very quickly. Service oriented architectures (SOA) get this by design.

Alternatively, you can also look at this from a business logic viewpoint. If you can cut your application into portions (say, photos, chat and games for a social site) you can create smaller applications to handle photos, chat and games as well as the shared authentication and user information storage parts. Have the photos, chat and games applications leverage the back end authentication and user information applications to read and write shared information.

This gives us several advantages. For the back end application you can remove all unneeded code (i.e., if you are not going to need provide views, then remove ActionView), plugins and gems. Keep the app as light as possible, and give each of the application shards on dedicated resources (i.e., their own databases).

Another advantage of this approach is that you can start to optimize your hardware spend. If your chat application is 1/2 as intensive as your photo and games applications, it’s far easier to assign resources in a targeted fashion and maximize returns. In a monolithic application, if the photo application breaks, or needs more resources the entire stack is affected. With sharding, you get some buffering from some site wide issues, and the ability to assign resources exactly where they’re needed. The big drawback is that it’s not easy.

Database Sharding

This is another step that can be looked at in certain circumstances. If the amount of data you need to process is so large, or the number of transactions is sinking your Database, you can look into database sharding. Basically, you take your database and break the schema up among several Database servers. There are tools in most major RDBMS’s which will allow you to take care of this. Informing the application where the data is might be complex depending on which RDBMS you use.

Filesystem Sharding

If your application is file system IOPS heavy, file system sharding might be the route that you want to look at. Basically you add more hardware disk arrays, and split the reads and writes between them. You need to inject some logic into the save and open functions in your application so that it knows which file system each file is to be saved to and opened from. Usually you can create a hash of the file name, and key off the first couple of characters in the hash. If you’re interested, you can read our more detailed dive into file system sharding.

That’s No Moon!

Scaling can be a daunting task if you put it off too long. It can mean the difference between a successful business and one that dies. Don’t let that scare you however. Taken in small, bite sized chunks it’s certainly an achievable goal. Make sure that you are working on the right problems, and make sure that you are doing a little throughout the lifespan of your application.

And keep in mind, Scaling is a Discipline, not a Goal. What works great for 20 users:

users = User.find(:all)
for user in users
if user.name = "fred"
user.make_happy
end
end

may not work as well with 2000, or 20,000. So do the work it takes to make your application work today, and keep in mind the changes you’ll have to make, and the challenges you’ll face tomorrow.