The following post is about a Support ticket recently filed by a customer. And how Engine Yard Support saved the day!
The customer is trying to deploy their app in production but it's failing on
rake aborted! cannot load such file -- net/ssh /data/myapp/shared/bundled_gems/ruby/1.9.1/gems/activesupport-3.2.16/lib/active_support/dependencies.rb:251:in `require' /data/myapp/shared/bundled_gems/ruby/1.9.1/gems/activesupport-3.2.16/lib/active_support/dependencies.rb:251:in `block in require' /data/myapp/shared/bundled_gems/ruby/1.9.1/gems/activesupport-3.2.16/lib/active_support/dependencies.rb:236:in `load_dependency' /data/myapp/shared/bundled_gems/ruby/1.9.1/gems/activesupport-3.2.16/lib/active_support/dependencies.rb:251:in `require' /data/myapp/shared/bundled_gems/ruby/1.9.1/gems/fog-1.26.0/lib/fog/joyent/compute.rb:3:in `<top (required)>'
The keen reader might notice that this app uses old versions. Ruby is 1.9.3. Rails is 3.2. Fog 1.26.0 was released on December 12, 2014. Engine Yard recommends that customers update their apps to latest versions but we understand that in some cases they're not able to in a timely manner.
We still provided stellar support as can be attested by the feedback of this customer. Here's how Engine Yard Support debugged the problem.
The error showed that the fog gem requires
net/ssh but it can't find it. We looked at the dependencies of fog 1.26.0 and indeed net-ssh isn't there.
This is a red flag right away but we got sidetracked when we saw that net-ssh is on Gemfile.lock. We ran
require 'net/ssh' didn't fail. We thought it can be a Bundler or Rubygems issue. Net-ssh is there but it's not loaded for some reason.
Next we turned to the tool that any good sysadmin should know how to use. Google. Yep! Engine Yard Support Engineers don't claim to know everything. But a good problem solver should always know how to find answers on the internet.
The search resulted in this GitHub issue on the fog-core gem opened in 2015. In the comments and this commit, we learned that fog-core already dropped support for Ruby 1.9.3 previously but loosened the requirements and supported Ruby 1.9.3 again.
These are in interesting lines from the commit.
begin require "net/ssh" rescue LoadError Fog::Logger.warning("'net/ssh' missing, please install and try again.")
require "net/ssh" fails, then a warning that tells you to install it should be displayed. We didn't see this on the deploy log but nevertheless, this explains the issue. The net-ssh gem is needed but is not installed. This is what we suspected in the first place.
So why did the deploy fail when net-ssh is already on Gemfile.lock? The engineyard gem which our customers use to deploy their app depends on net-ssh. Since the engineyard gem is not needed in production, it is added to the development group in Bunder, as it should be.
And why did
irb not fail when we ran
require 'net/ssh'? One Support Engineer installed net-ssh on the system while debugging. If we ran
bundle exec irb we would have seen the problem there.
Lessons learned. Check that the gem is included in the production group and always use
The customer added net-ssh to the Gemfile and the next deploy was successful. Another happy customer.