fog is a Ruby gem that allows you to control a number of cloud infrastructure services through a common API. fog provides the basis for building applications and cloud platforms that are infrastructure-independent, abstracting such functions as spinning up servers, connecting storage, managing DNS, etc. Internally, fog uses models to represent cloud structures analogously to the way ActiveRecord represents relational structures. Using fog makes your application or platform much more easily portable across infrastructures such as Amazon, Rackspace, and many others.

In addition to Engine Yard, fog is in use at other platform providers as well as in many core community projects such as Chef, Puppet, CarrierWave, and Paperclip. fog now supports 18 compute, 5 storage, 9 DNS, 2 CDN, and a variety of other infrastructure services. With 13 core committers, 121 contributors, and nearly 1,500 watchers, we're seeing really nice growth in participation and attention.

We believe that this release of fog is a really important step in making the cloud an easier and more powerful environment for developers to run great apps. We hope you'll give it a try, share your feedback, spread the word, and (most importantly) enjoy some great success with fog.

Get fog with a simple gem install fog. Read docs at fog.io, or grab source off GitHub at geemus/fog. Happy fogging!

Since Last Episode

[caption id="attachment_9987" align="alignright" width="300" caption="Photo courtesy of flickr user nplove"][/caption]

Since the last update in mid-March a lot has changed. A new, stronger focus on documentation has helped clarify things alongside many, many fixes and improvements. In addition to fixing the old stuff we have also added support for a number of new services:

 
• AWS Autoscaling was added (tests)
• AWS CloudWatch was added (tests)
• AWS Simple Notification Service was added (tests)
• AWS Simple Queue Service was added (tests)
• DNSMadeEasy DNS service was added (examples)
• Linode Compute was expanded with models (tests)
• Ninefold Compute (tests)
• Ninefold Storage (tests)
• OpenStack Compute Support (utilizing Rackspace Implementation)
• OpenStack Rackspace Support (utilizing Rackspace Implementation)
• vCloud 1.0 (tests)

Stats

In addition to the codebase, growth and stability have also dominated the userbase. Since the beginning of the year there have been huge gains across the board. We started the year at 50,000 downloads and have now reached 180,000 downloads. Our 860 followers have increased to 1360, 100 forks to 225, and 40 contributors to 105.

The most important change may be the number of commiters. At the start of the year I was the sole committer. Since then I have kept a careful eye out for dedicated contributors. There are now 8 of us. These awesome commiters are: Brian Hartsock, Christopher Oliver, Dylan Egan, Henry Addison, Aaron Suggs, Lincoln Stoll, Luqman Amjad, Nick Ricketts, Nick Osborn, Paul Thornthwaite. I thank them all and encourage you to do the same (HINT: high fives are awesome).

fog 2011 World Tour

I just returned from OSCON, but there are still plenty of chances to catch up this year. I'll be speaking at Lone Star Ruby Conf V August 11-13th, Golden Gate Ruby Conference September 16-17th, and Strange Loop September 18-19th. Finally for the cherry on top of a busy conference season, I'll be attending RubyConf September 29-October 1st. So, be sure to find me to help answer fog questions, or to offer high fives.

NEXT!

The hits keep coming and I expect we will continue to see new providers appear as new competitors emerge and people become interested in trying the latest and greatest. Beyond that there is still plenty of work to be done to improve the consistency between some of the providers and consolidating the shared feature set. You can expect more great stuff that you have grown accustomed to, but what else might await?

A lot of that is in your hands. What other services would you like to see abstracted? Should there be a CLI? How about RESTFUL web service? Should we convert the shindo tests to minitest? These are just a few of the many questions I am focused on answering as we continue to move the project forward. This is a great time to get involved and cast your vote. You can find all kinds of information about getting involved on the ever expanding fog.io. Let me know what you think!

Computation is the lifeblood of the cloud, providing the raw resources for innovation. Unfortunately, with great power comes great responsibility complexity. The actual process of refining these resources into services is serious business, further complicated by the huge variance between offerings. Thankfully flexibility and power no longer have to be out of reach. fog simplifies the process of utilizing these resources and smooths the differences between providers so that you can focus on creating the next revolutionary cloud service.

Installing fog

fog is distributed as a RubyGem:

gem install fog

Or for bundler users, you can add it in your Gemfile:

gem "fog"

fog => Rackspace Cloud Servers

We can start our exploration of cloud computing with Rackspace's Cloud Servers. You can sign up here and copy down your api key and username from here. We are about to get into the code samples, so be sure to fill in anything in ALL_CAPS with your own values!

# create a connection
connection = Fog::Compute.new({
  :provider           => 'Rackspace',
  :rackspace_username => RACKSPACE_USERNAME,
  :rackspace_api_key  => RACKSPACE_API_KEY
})

Servers the Rackspace way

Creating a server on Rackspace is very easy if you are willing to accept the defaults (the smallest server size, using Ubuntu 10.04 LTS).

server = connection.servers.create

You can then list your servers to see that it now appears.

connection.servers

Or you can fetch the latest data for your server in particular.

servers.get(server.identity)

As you might imagine, this is a pretty common use case, so fog simplifies it by providing the `reload` method to refresh the state of a model to the freshest available.

server.reload

But this too can get tedious quickly, especially when servers can take several minutes to boot. Fog uses `wait_for` in cases like this to periodically reload a model until either the block returns true or a timeout occurs (by default the timeout is 600 seconds). We can combine wait_for with `ready?` to check when a server has finished booting without needing to know the specifics of each service.

server.wait_for { ready? }

Once we are done with that we can shut it down.

server.destroy

Bootstrap: Servers the fog Way

Cycling servers is great, but in order to actually ssh to a server on Rackspace you need to place ssh keys (and ideally disable password authentication for root). Rather than worrying about the nitty gritty, we can utilize `bootstrap`.

server = connection.servers.bootstrap({
  :private_key_path => '~/.ssh/id_rsa',
  :public_key_path => '~/.ssh/id_rsa.pub'
})

Bootstrap will create the server, but it will also make sure that port 22 is open for traffic and has ssh keys setup. In order to get all the pieces put together the server will have to be running, so we can skip checking `ready?` since it should already be true. Now we can send commands directly to the server.

server.ssh('pwd')
server.ssh(['pwd', 'whoami'])

These return an array of results, where each has stdout, stderr and status values so you can check out what your commands accomplished. Now just shut it down to make sure you don't continue getting charged.

server.destroy

Using Amazon EC2 and fog

Sign up for an account here and copy down your secret access key and access key id from here.

First, create a connection with your new account:

require 'rubygems'
require 'fog'

# create a connection
connection = Fog::Compute.new({
  :provider                 => 'AWS',
  :aws_secret_access_key    => YOUR_SECRET_ACCESS_KEY,
  :aws_access_key_id        => YOUR_SECRET_ACCESS_KEY_ID
})

With that in hand we are ready to start making EC2 calls!

We should be able to reuse all our old code, except for one small exception. fog uses the official Canonical Ubuntu image as a default on AWS (official Canonical Releases). This image uses 'ubuntu' as the username, rather than 'root'. So the bootstrap call will look slightly different.

server = connection.servers.bootstrap({
  :private_key_path => '~/.ssh/id_rsa',
  :public_key_path => '~/.ssh/id_rsa.pub',
  :username => 'ubuntu'
})

Just like on Rackspace, this will boot a server and place our keys so that we can ssh in, run commands and eventually shut down the server with destroy.

Mocking out Compute

You can also start any of these scripts with

Fog.mock!

or start the fog interactive tool from the command line with

$ FOG_MOCK=true fog

to run in mock mode. In this mode commands are run as local simulation, so no cloud resources are ever consumed and things operate much faster. Not everything has mocks written for it, but if you run up against these edges errors will be raised to quickly alert you that you are entering not-yet-mocked territory. The functionality that has been mocked gets exercised by the same test suite as the real code, so you should feel confident that its behavior should be consistent.

Cleaning up

To cover your tracks it is a good idea to check for running servers and shut them down, here is one way you might do that.

connection.servers.select {|server| server.ready? && server.destroy}

Other Providers

Rackspace and Amazon are just the tip of the iceberg. fog also supports compute offerings from BlueBox, Brightbox, Terremark, GoGrid, Linode, Slicehost, Storm on Demand and Voxel. Once you wrap your head around that you can also check out fog's many supported storage, DNS and other services. With familiar commands throughout it is easy to pick up and try new services. Quickly your toolbox runneth over and you should have all the power you need to innovate.

Summary

Compute can be tricky, but the abstractions in fog make it much easier to get started. With your servers up and running you can then focus on the task at hand and get some work done. Congratulations on adding a new tool to your arsenal. Let us know what we can do better.

Adoption

Adoption continues at an amazing pace. We went from 40 contributors two short months ago up to 60, crossed 80,000 downloads from rubygems.org and 1,000 followers on github [ed: Wes was very giddy on that day]. All the extra attention has helped to weed out even more bugs and grow the available services to meet ever more diverse use cases.

New Providers and Services

Once again, several new services have joined the fog family, as well as a few notable additions to the existing services:

• AWS Cloudformation basics allow for configuring whole clusters at once (tests)
• AWS IAM has been fleshed out considerably from it's humble origins in 0.4.0 (tests)
• AWS RDS basics add the ability to manage hosted database servers (tests)
• AWS S3 gained static website support (tests)
• AWS SES basics support for cloud email delivery (example)
• DNSimple DNS service was added (examples)
• VirtualBox basics support managing local virtual machines (example)
• Voxel Compute service was added (tests)

Contribution Overhaul

I am always looking for ways to make it easier for people to get involved, so let me know if you have suggestions. First off I made the standing t-shirt offer more explicit, but that didn't have too much impact.

After that I decided I ought to better groom the issues so that they would be more accessible. The clearest thing is that must issues are now labeled as either easy, medium or hard. Rather than just plucking these values out of thin air I consider each issue and add notes on what needs to be done to accommodate them before adding the label. So if you would like to help out you can pick a difficulty matching the time you have available and familiarity with the code and feel confident that everything you need to complete the task will be available. You can also feel free to skip the non-categorized tickets, since I either haven't figured out how to do them or they will require my attention directly (i.e. accepting pull requests).

Integrations

In addition to working inside fog I have been working to help others better use it. Both carrierwave and paperclip have received patches with fog support. Both of these libraries provide ways to handle files for users in your web apps and by adding fog support they no longer have to worry about supporting numerous different providers and they can focus on their specialties. We will have more details about these integrations and how you can take advantage of them in future posts.

On the Road

I have also been privileged to be accepted to speak at a number of conferences and meetups. I spoke in January at Silicon Valley Ruby on Rails Group (recorded talk video here) and in February at Magic Ruby. Yesterday I presented at MountainWest RubyConf, and next month I'll be at Ruby Nation April 1-2, and Red Dirt RubyConf April 21-22. If you are out and about at these conferences come say hello and let me know how you use fog.

Toward 1.0

With all these improvements and bug fixes I can't help but feel like we must be nearing 1.0. It will be great to reach that point but I still have a couple things I want to sort out along the way.

Although DNS and storage provide fairly universal access across providers, compute still has a few rough edges. One key focus will be smoothing differences between the compute providers.

Another goal is to better sort out testing. There is a working set of tests now, but it is split between rspec and shindo and there are some discrepancies between style and usage throughout. I'd like to solidify how these will work, as well as making it easier for contributors to run them without have credentials for everything.

Last, but certainly not least is a stronger focus on documentation. There is a bit here and a bit there currently and I'd like to consolidate things on fog.io and make them easier to explore, as well as reworking the generated in-code documentation to be more accessible.

Over all I see 1.0 outside of that as being more of the same great stuff, from bug fixes to new services as things are released and the needs of users change, and beyond 1.0 will be no different. It is just a stepping stone for us to acknowledge the great level of support and stability we have already reached.

Setup

First, make sure you have fog installed:

gem install fog

For this first example we will use Zerigo (see below for how to use other providers). You can signup for Zerigo DNS here. Gather up your new credentials to initialize a connection to the service:

require 'rubygems'
require 'fog'

# create a connection to the service
dns = Fog::DNS.new({
  :provider     => 'Zerigo',
  :zerigo_email => ZERIGO_EMAIL,
  :zerigo_token => ZERIGO_TOKEN
}

Getting in the Zone

The first thing you need to do to prepare for your DNS excursion is create a zone for your domain. The zone will contain all of the more specific records that you will create later. You will just need to specify the domain, which should be your url without the 'http' or 'www' parts, and an email address. Then you can create the zone with your DNS connection:

zone = @dns.zones.create(
  :domain => 'example.com',
  :email  => 'admin@example.com'
)

Now that you have a zone you will need to update your registrar to let them know what DNS servers are responsible for your domain. You can ask the zone what values to use:

zone.nameservers

Spinning Records

With your new zone in hand you can add records as needed. First and foremost you will probably want the 'www' version of your site to point to whatever your ip might be:

record = @zone.records.create(
  :ip   => '1.2.3.4',
  :name => 'example.com',
  :type => 'A'
)

Adding other records is similarly easy, for instance if we want 'www.example.com' to go to the same place, we can use a cname record:

record = @zone.records.create(
  :ip   => 'example.com',
  :name => 'www',
  :type => 'CNAME'
)

Or, similarly you might want to have your blog elsewhere:

record = @zone.records.create(
  :ip   => '4.3.2.1',
  :name => 'blog.example.com',
  :type => 'A'
)

You can add more specifics if you need to, but reasonable defaults make it just that easy. You can also add any other kind of DNS record you might need for mail or other purposes, you can find a nice overview of record options and types on Wikipedia.

No Zerigo? No Problem

If you already have an account with another service you can just as easily use this same code with different credentials. fog currently supports AWS Route 53, Blue Box, DNSimple, Linode, Slicehost and Zerigo; so you can have your pick. As an example you can connect to AWS instead of Zerigo:

dns = Fog::DNS.new({
  :provider               => 'AWS',
  :aws_access_key_id      => AWS_ACCESS_KEY_ID,
  :aws_secret_access_key  => AWS_SECRET_ACCESS_KEY
}

Go Forth and Resolve

You can see an example of reusing code like this in the examples folder. Using this makes it easier to give yourself shortcuts to your cloud servers and manage how clients and users access them as well. It is great to have this flexibility so that you can modify your cloud infrastructure as needed while keeping everything ship shape. It also provides a nice way to create custom subdomains for users and just generally round out your cloud solution.

Faster websites are better. Better experience, better sales, you name it. Unfortunately, making a website faster can be tough. Thankfully a content distribution network, or CDN, can give you great performance bang for your buck. A CDN helps speed things up by putting copies of your files closer to your users. It's like the difference between pizza delivery from across the street and pizza delivery from the next town over.

The ease and deliciousness are the good news, but until recently CDN's were only available in the big leagues via 'my business guys will talk to your business guys' deals. Fortunately for us, Amazon recently updated CloudFront, their CDN service, to allow us to get these benefits with just a credit card and an API call. So now we'll see how you can spend a few minutes to save your users countless hours of load time.

Preliminaries

First, make sure you have fog installed:

gem install fog

Now you'll need to sign up for Cloudfront. Gather up the credentials your new credentials to initialize a connection to the service:

require 'rubygems'
require 'fog'

# create a connection to the service
cdn = Fog::CDN.new({
  :provider               => 'AWS',
  :aws_access_key_id      => AWS_ACCESS_KEY_ID,
  :aws_secret_access_key  => AWS_SECRET_ACCESS_KEY
}

Setting Up Your CDN

Now you'll need to create a 'distribution' which represents a mapping from the CDN to your domain. For the examples we'll pretend we are working on 'http://www.example.com', but you can just switch it to your actual domain. Some other options are available, but the only other one we need to fill in is OriginProtocolPolicy. This sets what to do about http vs https. We will use 'match-viewer' which returns the same protocol as the request, but you can also choose 'http-only' which always returns http responses.

# let the cdn know about your site
data = cdn.post_distribution({
  'CustomOrigin' => {
    'DNSName'               => 'www.example.com',
    'OriginProtocolPolicy'  => 'match-viewer'
  }
})

# parse the response for stuff you'll need later
distribution_id   = data.body['Id']
caller_reference  = data.body['CallerReference']
etag              = data.headers['ETag']
cdn_domain_name   = data.body['DomainName']

# wait for the updates to propogate
Fog.wait_for {
  cdn.get_distribution(distribution_id).body['Status'] == 'Deployed'
}

Getting Served

With the domain name from the distribution in hand you should now be ready to serve content from the edge. All you need to do is start replacing urls like http://www.example.com/stylesheets/foo.css with #{cdn_domain_name}/stylesheets/foo.css. Just because you can do something doesn't always mean you should though. Dynamic pages are not really well suited to CDN storage, since CDN content will be the same for every user. Fortunately some of your most used content is a great fit. By just switching over your images, javascripts and stylesheets you can have an impact for each and every one of your users.

Congrats, your site is faster! By default the urls aren't very pretty, something like http://d1xdx2sah5udd0.cloudfront.net/stylesheets/foo.css. Thankfully you can use CNAME config options to utilize something like http://assets.example.com/stylesheets/foo.css, if you are interested in learning more about this let me know in the comments.

Cleaning Up

But, just in case you need to update things I'll run through how you can make changes. In my case I just want to clean up after myself, so I'll use the distribution_id and ETag from before to disable the distribution. We need to use the ETag as well because it provides a way to refer to different versions of the same distribution and ensures we are updating the version that we think we are.

data = cdn.put_distribution_config(
  distribution_id,
  etag,
  {
    'CustomOrigin' => {
      'DNSName'               => 'www.example.com',
      'OriginProtocolPolicy'  => 'match-viewer'
    },
    'CallerReference' => caller_reference,
    'Enabled' => 'false'
  }
)

# parse the updated etag
etag = data.headers['ETag']

Now you just need to wait for the update to happen like before and once its disabled we can delete it:

Fog.wait_for {
  cdn.get_distribution(distribution_id).body['Status'] == 'Deployed'
}
cdn.delete_distribution(distribution_id, etag)

Thats it, now go forth and speed up some load times!

Contributors, Contributors, Contributors

One of the most exciting changes has been the reaching the height of over 40 contributors, keep up the awesome work everybody! We have fixed tons of bugs, improved docs and added loads of features. So without further ado I'll summarize the new hotness.

New Providers and Services

First off, several new services have joined the fog family, as well as a few notable additions to the existing services:

• AWS IAM service for access control of Amazon credentials was added
• AWS storage (S3) gained CDN custom origin support and multipart uploads
• AWS compute (EC2) gained support for tags and searching resources by filters
• New DNS API with support for AWS, Linode, Slicehost and Zerigo
• Brightbox contributed a ton of code for their new (still in beta) compute offering
• GoGrid compute has just gotten off the ground, with plenty left to work on

Smaller Memory Usage and Safer Credentials

As the number of services continues to grow I have been working hard to keep things nicely organized. Toward that goal we have a few additions. Now more than ever fog tries not to load parts of itself (or other gems) unless it needs them, allowing fog to use even less memory and start up faster.

To encourage the best practice of keeping your credentials safely outside your code, it is now easier to setup and use a yaml credential file. You can now use Fog::AWS::Compute.new with no arguments if you have a valid credential file supplied (default is ~/.fog). To use a different credential file just call Fog.credentials_path = 'path/to/credentials'.

Focus on Services

Another big change has been focusing on services more than providers, as mentioned above. New service level initializers simply take a :provideralong with the normal credentials, and if you have your credentials file setup all you'll need to specify is the provider:

  Fog::Compute.new(:provider => 'AWS')
  Fog::DNS.new(:provider => 'AWS')
  Fog::Storage.new(:provider => 'AWS')

Along with the initializers I have done a great deal of work to smooth out even more differences between the services (especially for storage and DNS). You can see the recommended interoperable way of working with dns and storage in the examples directory. These examples and many tests are now shared between providers to ensure compatibility and interoperability moving forward.

Community Grows Up

There were a few non-code changes as well. We gained a mailing list where you can see the release notes for all the patch versions, the #ruby-fog irc channel, and we're @fog on Twitter if you have questions. We gave out shirts, and added directions on how to score them to the README (in case you are curious). And we started on a webpage for fog, it still needs a lot of work but you can help by contributing to the gh-pages branch.

Sponsors

Since September Engine Yard has given me the awesome opportunity to work full time on fog. I can't thank them enough for the whirlwind, hectic, but ultimately amazing experience. Engine Yard goes above and beyond paying the bills, helping me get to conferences and covering the awesome stickers and shirts.

Also thank you to Bluebox, Brightbox and GoGrid who all graciously donate resources so that I can run tests. For the other providers I would ask that you please consider getting in touch about donating resources so that I can keep my costs down.

Thanks again to everyone supporting and using fog, looking forward to another great year with my head in the clouds! ;)

In this blog post you will learn to use fog - the cloud computing library - to upload your resume to Amazon's Simple Storage Service (S3), Rackspace's CloudFiles or Google's Storage for Developers.

Here's my out of date resume stored on S3, CloudFiles and Google Storage; programmatically stored in the cloud using this tutorial. NOTE: my boss would like me to add that I'm not currently looking for a new gig ;)

Check out those cloud-specific URLs! You could put all three in your job application, add the Ruby source for how you did it, and have your choice of Ruby jobs for being so awesome!

How? The all-clouds-in-one library of choice is fog.

Installing fog

fog is distributed as a RubyGem:

gem install fog

Or add it in your application's Gemfile:

gem "fog"

Using Amazon S3 and fog

Sign up for an account here and copy down your secret access key and access key id from here. We are about to get into the code samples, so be sure to fill in anything in ALL_CAPS with your own values!

First, create a connection with your new account:

require 'rubygems'
require 'fog'

# create a connection
connection = Fog::Storage.new(
  :provider                 => 'AWS',
  :aws_secret_access_key    => YOUR_SECRET_ACCESS_KEY,
  :aws_access_key_id => YOUR_SECRET_ACCESS_KEY_ID
)

# First, a place to contain the glorious details
directory = connection.directories.create(
  :key    => "fog-demo-#{Time.now.to_i}", # globally unique name
  :public => true
)

# list directories
p connection.directories

# upload that resume
file = directory.files.create(
  :key    => 'resume.html',
  :body   => File.open("/path/to/my/resume.html"),
  :public => true
)

If you are anything like me, you will continually tweak your resume. Pushing updates is easy:

file.body = File.open("/path/to/my/resume.html")
file.save

As you can see, cloud storage files in fog are a lot like an ActiveRecord model. Attributes that can be changed and a `#save` method that creates or updates the stored file in the cloud.

But if it took you longer to realize the mistake you might not still have file around, but you've got options.

directory = connection.directories.get("proclamations1234567890")

# get the resume file
file = directory.files.get('resume.html')
file.body = File.open("/path/to/my/resume.html")
file.save

# also, create(attributes) is just new(attributes).save, so you can also do:
file = directory.files.new(
  :key => 'resume.html',
  :body => 'improvements',
  :public => true
)
file.save

Alright, so you (eventually) become satisfied enough to send it off, what is the URL endpoint to your resume?

puts file.public_url

Pop that link in an email and you should be ready to cruise job ads and send your resume far and wide (Engine Yard is hiring, so check us out!). Now you are set, unless you are interviewing for Google, or Rackspace... Both of these companies have their own cloud storage services, so using Amazon S3 might not be the foot in the door you hoped for.

More clouds? How much extra stuff will you have to do for these services!?! Hardly anything needs to change, you just have to pass slightly different credentials in, but I'm getting ahead of myself.

Google Storage for Developers

Sign up here and get your credentials here.

connection = Fog::Storage.new(
  :provider => 'Google',
  :google_storage_secret_access_key => YOUR_SECRET_ACCESS_KEY,
  :google_storage_access_key_id => YOUR_SECRET_ACCESS_KEY_ID
)

Rackspace CloudFiles

Rackspace has Cloud Files and you can sign up here and get your credentials here.

connection = Fog::Storage.new(
  :provider => 'Rackspace',
  :rackspace_username => RACKSPACE_USERNAME,
  :rackspace_api_key => RACKSPACE_API_KEY
)

Then create, save, destroy as per fog-for-AWS. The `:public => true` option when creating directories (see above) is important for Rackspace; your folder and files won't be shared to Rackspace's CDN and hence your users without it. Similarly the `:public => true` on files is important for AWS and Google or they will be private.

Local Storage

While you are working out the kinks you might not want to do everything live though, ditto for while you are running tests, so you have a couple options to try before you buy. First, you can use a local provider to store things in a directory on your machine.

connection = Fog::Storage.new(
  :provider => 'Local',
  :local_root => '~/fog'
)

Mocking out Cloud Storage

Of course when you are testing or developing you can always just use the mocks (at least for AWS and Google, Rackspace still needs mocks implemented if you are looking for somewhere to contribute). They emulate the behavior of the external systems without actually using them. It is as simple as:

Fog.mock!
connection = Fog::Storage.new(config_hash)

Cleaning up

Fog takes care of the rest so you can focus on your cover letter. And with the awesome cover letter and cloud delivered resume you are probably a shoe-in. So all that is left is to cleanup that leftover job hunt residue.

file.destroy
directory.destroy

Summary

All done. Try out all the different options and let me know if you have any bugs or issues. I also wrote up a more consolidated example as a script that you can use for reference.

Bonus, note the `Fog.mock!` command. In your tests you can easily mock out calls to cloud providers.

Please let me know in the comments if you got a new Ruby job because you hosted your CV on 3 different Cloud Stores without getting your hands dirty.

Have questions or comments? Hop into #ruby-fog on freenode, ping @fog or @geemus.

And please always remember that I accept high fives and contributions!

Micro instances have no ephemeral storage, which also means they do not support instance-store images. Instance-store images are stored in snapshots and copied to the ephemeral root device when instances are booted. The alternative is to use volume-store images, which mount the image snapshot on an Elastic Block Store (EBS) volume. Making a choice between instance and volume store involves some trade offs. Instance-store boots slower but costs less. Volume-store boots faster, runs slower at first while data finishes copying and costs more (the price difference comes from EBS transfer/storage costs). Publicly available, supported AMIs do not necessarily provide volume-store versions. A conversion process will allow the transformation of an instance-store image into a volume-store image, but this effectively forks away from the supported version, and it is not clear that it's worth the trouble if your only goal is running micros.

The characteristic of micro instances that probably sticks out the most is the allotted CPU. EC2 describes its machines in terms of Elastic Compute Units, or ECU, each unit equivalent to a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor. Instances prior to micro were listed with a number of allotted ECUs, with small providing a reference point at 1 ECU. Micro instances break the trend though, and are listed as providing "Up to 2 EC2 Compute Units (for short periodic bursts)". The fine print provides some guidelines but no hard numbers. Bursting is everything above some background level much lower than 1 ECU, and is allowed for some percentage of each time period. Unfortunately the specifics are not revealed and bursting out of turn will both be throttled and increase the time before the next burst. To provide context, micro instances are are described as serving tens of requests per minute, with the fine print containing a number of graphs showing recommended CPU usage patterns.

So, at the end of the day it turns out that micro really is not the new small. They are smaller and have fewer, weirder resources which are allotted in a non-specific way that seems counter to the way many people might use them. Load balancing, proxies, cron or monitoring systems might be workable, but running web or database servers would be pushing your luck. If you need something cheap but consistent, your best bet probably remains small instances (if not a vps or different cloud). If you need affordable, bursty processing, you're probably better served using non-micro spot instances. Based on these considerations, we are not currently rushing to add micro support to Engine Yard AppCloud, where it could easily provide a poor user experience for many customers. However, we will keep these things in mind as they become more tried and as we develop new features.

Anecdotal evidence aside, I welcome you to try them out for yourself:

  require 'rubygems'
  require 'fog'

  # NOTE: You should replace ALL_CAPS things, manually or by defining constants
  compute = Fog::AWS::Compute.new(
    :aws_access_key_id => AWS_ID,
    :aws_secret_access_key => AWS_KEY
  )

  # bootstrap:
  # uses or creates fog_default keypair from private key to boot server with
  # opens port 22 of the first security specified or 'default' group.
  # an 'EBS boot' ubuntu image list is available: http://alestic.com/
  # waits for the server to be ssh-able, which can take a minute or two
  server = compute.servers.bootstrap(
    :flavor_id => 't1.micro',
    :image_id => 'ami-6c06f305',
    :private_key_path => PRIVATE_KEY_PATH,
    :public_key_path => PUBLIC_KEY_PATH,
    :username => 'ubuntu'
  )

  # run some commands over ssh and return results
  server.ssh(ARRAY_OF_COMMANDS_TO_RUN)

  # shutdown the server
  server.destroy

Please let us know what you find out!

Library What, Huh?

What makes libraries so special? They save time and effort by providing code you don't want to write. Skip the nitty gritty of this authentication scheme or that request protocol and get on with the task at hand. There are many ways for a library to capture wily domain knowledge and not all are created equal. After working on some libraries and reading many more, what follows are my personal guidelines for creating lovely encapsulating bits.

First, Do No Harm

It should be easier to work with a library than without it. People will grudgingly use libraries that are painful, but will recommend and rave about libraries that feel good. You should start with some idea of what an ideal interaction with the library might feel like. Your first pass at implementation will tend to be rough work, but keep utopia in mind. Subsequent iterations can work to unify reality and awesome.

This happened when I wrote shindo, a test framework that I wrote because I wanted something short and sweet. I have a love/hate relationship with testing. I like the confidence that tests give me, but I'm just not a big fan of the process. Shindo lets me easily understand what's actually happening when tests run. It looks like this:

# Each test is a Ruby block, with success determined by return value
Shindo.tests('examples') do
  test('successful test') { true }
  test('failing test')    { false }
  test('pending test')    # no block provided
end

My first pass at implementation was basic, but then I started iterating. I wanted the runner to be more interactive than what I was used to. So I added a prompt system that halts the test run when it runs into an error and asks you what you want to do. This led to rapid iterations:

1. I needed backtraces, but I didn't want to raise errors. So I wrote a nice backtrace object and added some code to view its output.
2. I wanted a better way to sift through files referenced in the backtrace, so I added the ability to choose a line from the backtrace and get an excerpt of the file containing it.
3. I wanted a better way to debug errors, so I added the choice to launch irb with the same binding as the test block. That's my kind of digging in.
4. There was no need to run all the tests all the time, so I added tags to help narrow a run down to the relevant ones.
5. Finally, I added the option to reload files and restart the test run.

Keep It Simple

Libraries should be less complicated than the problems they address. Simple libraries are easier to understand, improve and maintain. Abstractions are a tempting place to start, but they are easier to build on a solid foundation. Start with the simplest thing that could work, staying close to the problem you are trying to solve. For instance, when wrapping public APIs, match the names of attributes and functions to the API documentation. It makes it easier to refer to things and provides documentation from the start.

When I found out I would be joining the Engine Yard Cloud team, it occurred to me that I should have more than a passing knowledge of AWS. I started by reading tons of docs. Equipped with a rough idea of what was involved, I wanted an easy way to experiment and code AWS. Such was the genesis of fog.  AWS interactions can be super complicated, so I started by mapping the docs almost directly to code. Using it looks like this:

s3 = Fog::AWS::S3.new(
 :aws_access_key_id => id,
 :aws_secret_access_key => key
)

# get a list of buckets
s3.get_service

# create a bucket
s3.put_bucket('bucketname')

# get the contents of that bucket
s3.get_bucket('bucketname')

# delete that bucket
s3.delete_bucket('bucketname')

I'll be the first to admit that its not the prettiest, but it worked! And when in doubt, the AWS documentation could remind me what in the world some method I wrote a month ago was supposed to do. Recently someone asked if I had generated this code somehow. I wrote it all by hand (unless you count copy/paste as code generation), but for once it was oddly comforting to be mistaken for a robot.

Stay Focused

Stick to capturing stuff for one problem at a time. If sub-sections start getting distracting, consider extracting them into their own libraries. Avoid monkey patching; use inheritance or something that is not monkey patching. Premature optimization is the root of all evil, iteration is the trunk of all good. Do not undermine your tree, and do furnish it with leaves of documentation so that it does not wither and die.

While working on shindo, I ended up adding lots of backtrace related code. Since I was not relying on raised errors, I needed some other way to control starting and stopping tracing. At first it was small and self-contained, like this:

class Annals

  attr_accessor :buffer, :max

  def initialize(max = 20, &block)
    @max = max
    start
  end

  def lines
    @buffer.map do |event|
      "#{event[:file]}:#{event[:line]} #{event[:method]}"
    end
  end

  def start
    @buffer = []
    @size   = 0
    Kernel.set_trace_func(
      lambda { |event, file, line, id, binding, classname|
        if event == 'call'
          unshift(:file => file, :line => line, :method => "in #{id}")
        end
      }
    )
  end

  def stop
    Kernel.set_trace_func(lambda {})
    @buffer.shift # remove Annals#stop from buffer
  end

  def unshift(line)
    @buffer.unshift(line)
    if @size == @max
      @buffer.pop
    else
      @size += 1
    end
  end

end

But it just kept growing. So I pulled it out into its own project, annals. After extraction I could proceed to add other handy backtrace related features, even ones I had no real need for just then. Small libraries like this make great building blocks for newer, crazier stuff.

Add Value

Once equipped with more knowledge about the problem than is healthy for any one person, abstraction is a go. Tackle low hanging fruit first. Ruby is, afterall, object-oriented, so consider providing some nice models that encapsulate the nitty gritty bits. Write mocks. No one knows better what mocks should do. Add some command line tools or an interactive mode. Focus on eliminating pain points, increasing usability and generally making the project awesome to work with.

With a good foundation of low level functionality in fog, it was time to start adding value. I added easy mocking, because running tests against AWS can be expensive and slow. Then I started building objects around the various entities. Here is how the objects replace the low level calls from earlier:

# this is optional and great for testing
Fog.mock!

# connection
s3 = Fog::AWS::S3.new(
 :aws_access_key_id => id,
 :aws_secret_access_key => key
)

# get a list of buckets
s3.buckets.all

# create a bucket
bucket = s3.buckets.create(:name => 'bucketname')

# get the contents of that bucket
bucket.objects.all

# delete that bucket
bucket.destroy

The new interface mimics DataMapper, an interface myself and many others are already familiar with. I did my best to abstract away the nitty gritty captured in the low level functionality while leaving the ability to get stuff done. I think it is better, but it is still a work in progress. I dedicate a little time every day to filling in the blanks, but libraries have a way of never truly being finished.

Lather, Rinse, Repeat

Making libraries that capture a domain and feel good to use is hard. Implement the simplest thing first, but never stop iterating. Be absurdly consistent. Keep fiddling with the look and feel. If something seems painful, it probably is, so fix it. Make your code a joy to use and people will reward you by using it, and submitting patches (hint, hint)!

What do you think makes a good library? How can I improve mine? Comment and let me know!