I'm going to walk you through some key benefits of Bundler. Then, I'll answer some Bundler questions and concerns to leave you with a better understanding of Bundler so you can use it better day to day.

Key Benefits

1. Version Consistency
   Ensuring your developers are all working with the same version of things can be very challenging. One of the great things about Bundler is that it allows you to ensure that all machines are running the same versions of gems AND their dependency while at the same time giving you a simple path to update one or two of your gems.
2. Dependency Resolution
   If you've ever come across an error like this:
   can't activate rack (~> 1.0.1, runtime) for [], already activated rack-1.0.0 for [], then you know how hard it can be to debug. You need to figure out which gem requires the older version, which gem requires the newer version, and then (usually through trial and error) determine which rack version satisfies the dependency of both. Bundler solves this challenge.
3. Development Freedom
   Does your application depend on a gem that you are developing? You used to stick it in vendor/gems or maybe even used git submodules. Now it’s as simple as forking the gem and passing the :git parameter to your gem. No more committing large swaths of files to your repo just because your app relies on them.

Now that I've given the brief overview of some of Bundler’s benefits I hope it’s apparent how useful it is. It's as simple as adding a gem to your gemfile and running bundle install. If you're using Rails 3 you're already using Bundler, so go right ahead.

Your Questions

A few weeks ago we asked you to suggest challenges encountered using Bundler, or common hesitations to using Bundler. Here’s what you came up with.

How do I run code in one bundle from code in a different bundle? (via @joshsusser)

Good question. This isn’t heavily documented.

It all comes down to this method: Bundler.with_clean_env { #do stuff }.

This resets your env to what it was before you called the require to bundler/setup. This will clear out the env['BUNDLE_GEMFILE'] and allow you to use a different Gemfile. Here Andre Arko shows how this works, and what to expect from running code within a Bundler.with_clean_env block.

How do I avoid fetching source index every time? (via @Pistos)

There is a flag you can pass to bundle install that does this: bundle install --local.
According to the docs, the local flag does this:

Do not attempt to connect to rubygems.org, instead using just the gems located in vendor/cache. Note that if a more appropriate platform-specific gem exists on rubygems.org, this will bypass the normal lookup.

When should I run bundle install vs. bundle update? (via @jamiecobbett)

Always run bundle install. Run bundle update only if you actually wish to update a gem, or if bundle install warns you that you need to update a gem. You can also conservatively update gems with bundle update #{gemname} which will keep everything else at the same version but re-resolve the dependency with the newer version of the gem you want to update. The implication of bundle install is that you want to change only the things that you changed in the Gemfile and any non-shared dependencies. If a dependency of a gem you change is a dependency of another gem, the bundle install will fail and actually warn you to update.

Is Bundler compatible with Ruby 1.9.2? (via @vandrijevik)

Yes. Bundler is fully compatible with Ruby 1.9.2.

How can I use Capistrano with Bundler? (via @smathy)

You can add require 'bundler/capistrano' to the top of your deploy.rb. This adds the bundle install task to run after every deployment. By default this runs a bundle install with the --deployment and --quiet flags as well as without the development and test groups. It installs the bundle to shared/bundle. You can override these defaults by setting any of these in your deploy.rb.

set :bundle_gemfile,  "Gemfile"
set :bundle_dir,      File.join(fetch(:shared_path), 'bundle')
set :bundle_flags,    "--deployment --quiet"
set :bundle_without,  [:development, :test]
set :bundle_cmd,      "bundle" # e.g. "/opt/ruby/bin/bundle"
set :bundle_roles,    #{role_default} # e.g. [:app, :batch]

What can I do about the indirection Bundler adds to the command line? For example, `bundler exec foo` instead of simply `foo` (via @peteraronoff)

Unfortunately you need at least a little indirection or there wouldn’t be a way to keep track of which gems you need to activate. Here are some tricks you can use to minimize the indirection as much as possible.

1. alias b=”bundle exec”
   This makes it easy to run most bundled commands, for example "b rails c".
2. bundle exec bash
   You can bundle exec into bash (or zsh or whatever you favorite shell is) and have the shell executed within the bundle environment. This allows you to run binary gems from the command line without having to bundle exec.
3. bundle install --binstubs
   This will install bin files into the bin/ directory of your app. Now you can run bin/rspec or bin/executable and it will be run within the bundle environment.

Why is there is a platform option for Windows, but not Darwin? (via @soederpop)

Windows is a Ruby platform. Darwin is an operating system that the ruby interpreter runs on. For now, use groups.

I'm interested in some examples for 'bundle pack.' I find this command useful to deploy to machines off the network. (via @hoxworth)

Simple Example:
bundle pack && git add vendor/cache && git commit -am “Packed gems” && git push && cap deploy

You can use this as a drop in replacement for vendor/gems, with the added benefit of knowing that even gems depended on will be consistent across platforms. Keep in mind this doesn’t work with gems from git yet, though.

Any performance issues with switching from Bundler 0.9.26 to 1.0.7? I’m seeing an odd behavior with higher CPU usage on prod server. (via @ttdonovan)

Currently there are no other reports of performance issues. Please file an issue with the steps to reproduce and the developers can look into it for you. The simpler the issue is to reproduce, the easier it will be to track down.

Does Bundler work on Windows? (via @drnic)

Yes, Bundler does work on Windows. There is a known issue regarding lockfiles and multi-platform use of Bundler. If you use Bundler on Windows and Mac or Linux systems, and run bundle install on the Windows machine the lockfile gets generated specifically for Windows. For now, the recommendation is to run bundle install on a machine that closely resembles your deployment setup and commit that lockfile into the repository.

Other Questions?

What other hesitations or frequently encountered issues have we missed? Leave them in the comments section and we’ll do our best to tackle them too.

To learn more about Bundler visit the Bundler website. You can also fork Bundler on GitHub.

This is where dedicated search engines come into play, and Sphinx is our favorite tool for the job. Written in C++ by Andrew Aksyonoff, and originally released to open source in 2001, Sphinx is a blazing fast search engine. Considering that fast and complex full-text searching is a somewhat frequent need, I've put together this post with my top five tips for implementing Sphinx.

1) Use thinking_sphinx

There are several plugins out there for Sphinx integration into Rails. UltraSphinx, ThinkingSphinx, and acts_as_sphinx (no longer under active development) are the most commonly used plugins.

We recommend ThinkingSphinx over UltraSphinx for several reasons. Ultrasphinx, unfortunately, can throw cryptic errors because of the way it preloads your indexed model. If you have patches that are in your lib directory, they must be explicitly required. And while the UltraSphinx way of defining blocks is simpler for simple cases, in more advanced cases it can become far less readable, and you'll hit those advanced cases before you know it.

2) Know When to Index, and How Often

This next tip is really three tips all bundled into one (because who wants to read seven tips for anything? ;) )

1. Know your requirements
   What's an acceptable lag between the data being updated and becoming available in search results? Does it need to be instantaneous, or is it acceptable to wait 5, 10, 15 or even 30 minutes? The longer you wait between index updates, the less resources (and CPU time) your search engine consumes.
2. Know how long your indexing takes
   If it takes three minutes to complete an index run, but you kick off a full re-index every 60 seconds, that's not going to work out well. If you absolutely need an updated index every 60 seconds, then you need to consider alternatives like a bigger instance for your search engine or other strategies like delta indexing (below).
3. Test your new indexes or new data in staging
   When making changes to data being indexed or adding new indexes, do a test run of your indexing in a staging environment with snapshot of production data. Sometimes small changes to data or indexing result in an enormous increase in index size. If your changes create gigantic indexes, it's best to learn that on staging, instead of running out of space in your production environment.

3) Use Delta Indexes (When You Need To)

Without a reindex, your search won't be up to date; the question is when to reindex.

When indexing small data-sets, a full reindex can be done frequently. But as size grows, so does the index, and with it the time it takes to index. This is when delta indexing comes into play. A delta index is nothing more than a second index containing indexes for only the documents that changed since your last index. There are three main methods of delta indexing built into ThinkingSphinx: the default behavior, timestamped deltas, and delayed_job integration.

The first method—the default behavior of thinking_sphinx—is big on convenience. On every save it fires off the delta indexer and you get near instantaneous index updating. However, while this works great on development environments, and most staging environments, in production this can be problematic.

One problem is that the indexer is now part of the request cycle, which means that with each save comes a reindex. This method will cause scaling problems—with increased traffic, the indexer will fire more frequently. This puts increased load on the database as well as the filesystem.

Another problem is that in a production environment with many instances, the delta index is only created on the instance that handles the request. This results in instances with out of date information until the next full index. We deal with this by adding a cron entry to run the delta index cron task on all machines that run the search daemon. This has the effect of keeping your indexes in sync to the interval that the cron job runs at.

The second method—the timestamped version—works by adding a time threshold to the define_index block e.g.

set_property :delta => :datetime, :threshold => 1.hour

This is the frequency with which you run your rake task to reindex the delta. This means your deltas are updated every hour (in this example). While a nice improvement to the built-in default, this means that your indexes are out of date until the next rake tasks run, so you need to set the frequency according to user expectations (or reset expectations).

The third method uses the delayed_job gem and pushes a job onto the delayed_job queue that tells the indexer to run as needed. This is more immediate than the threshold option, while still running outside of the request cycle. This is the most promising setup, although it lends itself to a single searchd server setup. Specifically, a single machine running the indexer and search daemon with each instance sending reindex tasks to the queue when needed.

The drawbacks to this third approach are:

1. You lose availability. If the searchd server goes down, your search goes down. Allowing each instance to have its own instance of searchd builds redundancy into the setup.
2. As mentioned in the official documentation,"because the delta indexing requests are queued, they will not be processed immediately—and so your search results will not not be accurate straight after a change. Delayed_Job is pretty fast at getting through the queue though, so it shouldn’t take too long."

4) Know Your Bottleneck: Database or Filesystem

When maintaining your indexes, you have a choice of merging delta indexes into your main index or doing a full reindex. Merging can save you a database hit, but require twice the I/O of the two indexes to be merged, and hits the filesystem hard. On the other hand, reindexing hits the database hard. So you have to know your bottleneck. Most Rails developers are acutely aware of their database load. We optimize queries, we index tables, and we even use methods to read exclusively from the replica and write only to the master. So, instinctively, most developers select to merge their delta index into the main, rather than perform a full reindex, in order to take load off the database. But this isn't always right.

If your application processes a lot of uploads or your application has poor cacheability (and you're serving direct from filesystem a lot) then you probably want to avoid putting more load on the filesystem. In these cases, reindexing will make more sense then merging delta indexes.

5) If you're on ultrasphinx, switch to thinking_sphinx

Rein Henrichs wrote a great blog post which included the steps to make the switch. I'll expand on those here, and include some real world code samples.

Switching is actually relatively simple and in these four steps you can convert an Ultrasphinx application to a ThinkingSphinx one.

1. Uninstall UltraSphinx and install ThinkingSphinx:

Run:

script/plugin remove ultrasphinx

and add this line to you environment.rb:

config.gem('freelancing-god-thinking-sphinx', :lib => 'thinking_sphinx')

2. Translate your is_indexed declaration into a define_index block and change your search actions to use the ThinkingSphinx API:

class Post < ActiveRecord::Base
belongs_to :blog
belongs_to :category

is_indexed :conditions => "posts.state = 'published'",
  :fields     => [{:field => 'title', :sortable => true},
  {:field => 'body'},
  {:field => 'cached_tag_list'}],
  :include    => [{:association_name => "blog",
    :field            => "title",
    :as               => "blog",
    :sortable         =>  true},
    {:association_name => "blog",
  :field            => "description",
  :as               => "blog_description"},
  {:association_name => "category",
  :field            => "title",
  :as               => "category",
  :sortable         =>  true}]
end

class Post < ActiveRecord::Base
  belongs_to :blog
  belongs_to :category

  define_index do
    indexes title, :sortable => true
    indexes body, cached_tag_list

    indexes blog.description, :as => :blog_description
    indexes blog.title,       :as => :blog,     :sortable => true
    indexes category.title,   :as => :category, :sortable => true

    where "posts.state = 'published'"
  end
end

Your old search task might look like:

Ultrasphinx::Search.new(:query => params[:query])

Where your new one would look like (assuming you've indexed the model Post):

Post.search(params[:query])

3. Rewrite your deployment tasks to run the ThinkingSphinx rake tasks:

namespace :sphinx do
  desc "Stop the sphinx server"
  task :stop, :roles => [:app], :only => {:sphinx => true} do
    run "cd #{latest_release} && RAILS_ENV=#{rails_env} rake thinking_sphinx:stop"
  end

  desc "Reindex the sphinx server"
  task :index, :roles => [:app], :only => {:sphinx => true} do
    run "cd #{latest_release} && RAILS_ENV=#{rails_env} rake thinking_sphinx:index"
  end

  desc "Configure the sphinx server"
  task :configure, :roles => [:app], :only => {:sphinx => true} do
    run "cd #{latest_release} && RAILS_ENV=#{rails_env} rake thinking_sphinx:configure"
  end

  desc "Start the sphinx server"
  task :start, :roles => [:app], :only => {:sphinx => true} do
    run "cd #{latest_release} && RAILS_ENV=#{rails_env} rake thinking_sphinx:start"
  end

  desc "Restart the sphinx server"
  task :restart, :roles => [:app], :only => {:sphinx => true} do
    run "cd #{latest_release} && RAILS_ENV=#{rails_env} rake thinking_sphinx:running_start"
  end
end

and you'll probably want to add these as well to automate the reindexing and starting on deploy:

after "deploy:symlink_configs", "new_sphinx:configure"
after "sphinx:configure", "sphinx:index"
after "sphinx:index", "sphinx:restart"

If you're not running on Engine Yard Slices, you can still get the benefit of prewritten Capistrano tasks by adding:

require "vendor/plugins/thinking-sphinx/lib/thinking_sphinx/deploy/capistrano"

to the top of your deploy.rb file. If you're running on the latest release from GitHub, as a plugin, the tasks should be included automatically. This is of course assuming that you are running Capistrano from a working repository and by itself (a non-developer deploying code for example—thanks for the tip, commenter Josh!)

Because of a custom script that exists on Engine Yard Slices, if you are using our eycap gem, these tasks are included for you as:

sphinx:configure
sphinx:reindex
sphinx:restart
sphinx:start
sphinx:stop
thinking_sphinx:configure
thinking_sphinx:reindex

4. Stop searchd and then run your new configure, index and start start tasks:

cap sphinx:stop && cap sphinx:configure && cap sphinx:index && cap sphinx:start

Solid searching is key in numerous applications; Sphinx is a great tool for many cases, and I hope this post helped convince you!