Let's start off with a simple, two-part application. Part 1 is a Sinatra app that puts JSON-serialized messages into an AMQP queue. Part 2 is a daemon that consumes those messages.

Here's what a Gemfile for this application might look like:

gem 'json'
gem 'sinatra'
source 'http://gems.github.com'
gem 'famoseagle-carrot', :require_as => 'carrot'

gem 'eventmachine'
gem 'amqp'

The Sinatra app starts off with these lines:

# This makes sure the bundled gems are in our $LOAD_PATH
require File.expand_path(File.join(File.dirname(__FILE__), 'vendor', 'gems', 'environment'))

# This actually requires the bundled gems
Bundler.require_env

class MyApp < Sinatra::Base
  # stuff

The daemon starts like this:

# This makes sure the bundled gems are in our $LOAD_PATH
require File.expand_path(File.join(File.dirname(__FILE__), 'vendor', 'gems', 'environment'))

# This actually requires the bundled gems
Bundler.require_env

AMQP.start do
  # stuff

However, this loads too many gems. The Sinatra app synchronously publishes its messages using carrot, so it doesn't need the EventMachine gem or the AMQP gem. Likewise, the daemon doesn't serve HTTP requests, so it doesn't need Sinatra, and it's processing messages asynchronously using EventMachine, so it doesn't need Carrot.

This isn't the end of the world for this set of gems; it just makes all the Ruby processes use a little more memory than they otherwise might. However, if the Sinatra app uses one gem that defines Array#foo one way, and the daemon uses another gem that defines Array#foo an entirely different way, then each component must require only its own dependencies.

Fortunately, the bundler makes this really easy: just change the Gemfile to look like this:

gem 'json'

only :app do
  gem 'sinatra'
  source 'http://gems.github.com'
  gem 'famoseagle-carrot', :require_as => 'carrot'
end

only :daemon do
  gem 'eventmachine'
  gem 'amqp'
end

In the Sinatra app, change Bundler.require_env to Bundler.require_env(:app). This loads JSON, Sinatra, and Carrot, but not EventMachine or AMQP.

Similarly, in the daemon, Bundler.require_env becomes Bundler.require_env(:daemon).

But wait, it gets better!

Bundler.require_env can be called more than once with a different environment name each time. Consider tests (which, of course, you wrote first, right?): you want to run them. Now your Gemfile might look like this:

gem 'json'

only :app do
  gem 'sinatra'
  source 'http://gems.github.com'
  gem 'famoseagle-carrot', :require_as => 'carrot'
end

only :daemon do
  gem 'eventmachine'
  gem 'amqp'
end

only :test do
  gem 'rspec'
  gem 'webrat'
end

Then, in your spec, you start like this:

require 'my_app'       # load the Sinatra app so we can test it
Bundler.require_env(:test)    # get rspec and webrat in here

describe "the main page" do
  # stuff

Now each part of your app has its dependencies neatly contained. Each part only gets what it needs, and the different dependencies can't step on each others' toes.

Sweet.

However, when you have jobs that use resources from multiple machines, lockrun isn't adequate. Consider a cron job that builds a local index of a bunch of NFS-mounted files. You don't want more than one consumer of that NFS volume to index it at a time, otherwise performance will be degraded.

Enter distlockrun: it works similarly to lockrun except that it talks to a central server for mutual exclusion instead of locking a file.

First of all, you need the lock server running. It's really lightweight, so you can run it on any machine you've already got. For example, on Engine Yard Cloud, it should be run on the database master in a screen session.

Start the server like so: distlockrun-server

By default, it starts on port 7890, but that can be changed with the --port option.

Then, on the client, run your job as follows: distlockrun --server {server's hostname} big-expensive-indexer.rb

If there's already a big-expensive-indexer.rb running, then distlockrun will just exit. Otherwise, it runs the job and then tells the lock server that it's finished.

That's all it takes to get cluster-wide mutual exclusion; short but sweet!