Customer Stories: Resolving a Deploy Issue Due to Load Error

  

Customer StoriesThe following post is about a Support ticket recently filed by a customer. And how Engine Yard Support saved the day!

The customer is trying to deploy their app in production but it's failing on rake db:migrate.

rake aborted!
cannot load such file -- net/ssh
/data/myapp/shared/bundled_gems/ruby/1.9.1/gems/activesupport-3.2.16/lib/active_support/dependencies.rb:251:in `require'
/data/myapp/shared/bundled_gems/ruby/1.9.1/gems/activesupport-3.2.16/lib/active_support/dependencies.rb:251:in `block in require'
/data/myapp/shared/bundled_gems/ruby/1.9.1/gems/activesupport-3.2.16/lib/active_support/dependencies.rb:236:in `load_dependency'
/data/myapp/shared/bundled_gems/ruby/1.9.1/gems/activesupport-3.2.16/lib/active_support/dependencies.rb:251:in `require'
/data/myapp/shared/bundled_gems/ruby/1.9.1/gems/fog-1.26.0/lib/fog/joyent/compute.rb:3:in `<top (required)>'

The keen reader might notice that this app uses old versions. Ruby is 1.9.3. Rails is 3.2. Fog 1.26.0 was released on December 12, 2014. Engine Yard recommends that customers update their apps to latest versions but we understand that in some cases they're not able to in a timely manner.

We still provided stellar support as can be attested by the feedback of this customer. Here's how Engine Yard Support debugged the problem.

The error showed that the fog gem requires net/ssh but it can't find it. We looked at the dependencies of fog 1.26.0 and indeed net-ssh isn't there.

This is a red flag right away but we got sidetracked when we saw that net-ssh is on Gemfile.lock. We ran irb and require 'net/ssh' didn't fail. We thought it can be a Bundler or Rubygems issue. Net-ssh is there but it's not loaded for some reason.

Next we turned to the tool that any good sysadmin should know how to use. Google. Yep! Engine Yard Support Engineers don't claim to know everything. But a good problem solver should always know how to find answers on the internet.

The search resulted in this GitHub issue on the fog-core gem opened in 2015. In the comments and this commit, we learned that fog-core already dropped support for Ruby 1.9.3 previously but loosened the requirements and supported Ruby 1.9.3 again.

These are in interesting lines from the commit.

        begin
          require "net/ssh"
        rescue LoadError
          Fog::Logger.warning("'net/ssh' missing, please install and try again.")

If require "net/ssh" fails, then a warning that tells you to install it should be displayed. We didn't see this on the deploy log but nevertheless, this explains the issue. The net-ssh gem is needed but is not installed. This is what we suspected in the first place.

So why did the deploy fail when net-ssh is already on Gemfile.lock? The engineyard gem which our customers use to deploy their app depends on net-ssh. Since the engineyard gem is not needed in production, it is added to the development group in Bunder, as it should be.

And why did irb not fail when we ran require 'net/ssh'? One Support Engineer installed net-ssh on the system while debugging. If we ran bundle exec irb we would have seen the problem there.

Lessons learned. Check that the gem is included in the production group and always use bundle exec.

The customer added net-ssh to the Gemfile and the next deploy was successful. Another happy customer.

Free Ebook:
Should I Hire DevOps or Outsource to a Provider?

You have to invest in your infrastructure: Do you hire DevOps for this critical function, assign it to your already overworked engineers, or outsource to a provider that offers full-stack capabilities?

Should I Hire DevOps?

Christopher Rigor

 
DevOps Support Manager, Asia-Pacific at Engine Yard. Organizer of @RubyConfPH. Speaker. Interested in automation, Kubernetes, Docker, Deis, ops, Ruby.
Find me on:

Comments

Subscribe Here!