Customer Stories: Resolving a Deploy Issue Due to Load Error


Customer StoriesThe following post is about a Support ticket recently filed by a customer. And how Engine Yard Support saved the day!

The customer is trying to deploy their app in production but it's failing on rake db:migrate.

rake aborted!
cannot load such file -- net/ssh
/data/myapp/shared/bundled_gems/ruby/1.9.1/gems/activesupport-3.2.16/lib/active_support/dependencies.rb:251:in `require'
/data/myapp/shared/bundled_gems/ruby/1.9.1/gems/activesupport-3.2.16/lib/active_support/dependencies.rb:251:in `block in require'
/data/myapp/shared/bundled_gems/ruby/1.9.1/gems/activesupport-3.2.16/lib/active_support/dependencies.rb:236:in `load_dependency'
/data/myapp/shared/bundled_gems/ruby/1.9.1/gems/activesupport-3.2.16/lib/active_support/dependencies.rb:251:in `require'
/data/myapp/shared/bundled_gems/ruby/1.9.1/gems/fog-1.26.0/lib/fog/joyent/compute.rb:3:in `<top (required)>'

The keen reader might notice that this app uses old versions. Ruby is 1.9.3. Rails is 3.2. Fog 1.26.0 was released on December 12, 2014. Engine Yard recommends that customers update their apps to latest versions but we understand that in some cases they're not able to in a timely manner.

We still provided stellar support as can be attested by the feedback of this customer. Here's how Engine Yard Support debugged the problem.

The error showed that the fog gem requires net/ssh but it can't find it. We looked at the dependencies of fog 1.26.0 and indeed net-ssh isn't there.

This is a red flag right away but we got sidetracked when we saw that net-ssh is on Gemfile.lock. We ran irb and require 'net/ssh' didn't fail. We thought it can be a Bundler or Rubygems issue. Net-ssh is there but it's not loaded for some reason.

Next we turned to the tool that any good sysadmin should know how to use. Google. Yep! Engine Yard Support Engineers don't claim to know everything. But a good problem solver should always know how to find answers on the internet.

The search resulted in this GitHub issue on the fog-core gem opened in 2015. In the comments and this commit, we learned that fog-core already dropped support for Ruby 1.9.3 previously but loosened the requirements and supported Ruby 1.9.3 again.

These are in interesting lines from the commit.

          require "net/ssh"
        rescue LoadError
          Fog::Logger.warning("'net/ssh' missing, please install and try again.")

If require "net/ssh" fails, then a warning that tells you to install it should be displayed. We didn't see this on the deploy log but nevertheless, this explains the issue. The net-ssh gem is needed but is not installed. This is what we suspected in the first place.

So why did the deploy fail when net-ssh is already on Gemfile.lock? The engineyard gem which our customers use to deploy their app depends on net-ssh. Since the engineyard gem is not needed in production, it is added to the development group in Bunder, as it should be.

And why did irb not fail when we ran require 'net/ssh'? One Support Engineer installed net-ssh on the system while debugging. If we ran bundle exec irb we would have seen the problem there.

Lessons learned. Check that the gem is included in the production group and always use bundle exec.

The customer added net-ssh to the Gemfile and the next deploy was successful. Another happy customer.

Free Ebook:
Top 10 Advantages of PaaS

Looking to develop, deploy, and manage applications in the cloud? Learn how PaaS empowers you to innovate faster, more cost-effectively, and with less risk

PaaS Is Dead

Christopher Rigor

Christopher Rigor is a Senior Technical Evangelist at Engine Yard. He’s a long time Rails user, system administrator, and recently became a contributor of RailsInstaller. Previously, he was the DevOps Support Manager for Asia-Pacific at Engine Yard.
Find me on:


Subscribe Here!