Click to view full size.

We are pleased to say that as of a recent deploy, customers with many environments can now enjoy a much snappier (70% faster) dashboard.

For the curious, the fix was done by discovering an I/O bottleneck via New Relic and getting rid of it by appropriately using fragment caching.

Hubris is a gem that allows you to call out to natively compiled Haskell shared objects. In this post, I'll talk a bit more about why you might want to do that, how you'd go about doing that, and share some real world examples.

So Why Would I Want To Use Haskell?

It's easy for any decent programmer to spot the many striking differences between Ruby and Haskell, but why a Rubyist might be interested in  Haskell is a question worth asking. There are two primary scenarios:

1. Native Extensions

When CPU-intensive algorithms are needed for a task in Ruby, you usually drop down to the C or C++ level to take advantage of the highly optimized code yielded by a C/C++ compiler. Unfortunately, for someone without significant experience in the language, it can be easy to run into memory management woes, not to mention the frustration of dealing with a more hands-on (with respect to memory and explicit typing) language like C.

Haskell is a managed memory language with one of the most powerful type inference capabilities around. What this means is that you can avoid being bothered with explicitly denoting types at function call-sites, and choose whether or not to add types to definitions for the particular function you're working on. Additionally, GHC (Haskell's most prominent compiler for production applications) produces compiled native code that has performance in the same league as C/C++.

2. Concurrency

Many popular languages (especially Object Oriented ones) allow unconstrained mutation of data (e.g. mutator! methods in Ruby). This has always had implications for the difficulty of reasoning in the resulting code. But lately, this has become a bigger problem because it makes reasoning about concurrent program execution a nightmare.

What differentiates Haskell from many other functional languages is that Haskell isolates side effects statically in types rather than dynamically in language constructs. In effect, this means you can statically determine a list of functions/places to look (and eliminate many of the places you don't have to look) when problems related to side effects do come up.

3. And More

Haskell  as a whole has many other useful properties, such as implicit laziness, implicit argument currying, a typeclass system for generic programming, various concurrency constructs including an STM and a wonderful concurrency debugger... just to name a few. Rubyists who enjoyed Why's Poignant Guide to Ruby may enjoy parsing through Learn You a Haskell for Great Good.

Okay, I Buy It: Now How Can I Do It?

Now that we've seen why you might want to use Haskell, we'll take a look at how to use Hubris from Ruby. There are two main modes of operation: a way to embed a Ruby string containing Haskell code inline, and a way to load code from a plain Haskell .hs file. Like so:

class Example
  hubris :inline => "foo :: Integer -> Integer; foo x= 1330 + x"
end

Example.new.foo # => 1337

class Example
  hubris :source => "MyHaskell.hs"
end

Example.new.my_awesome_function "go ye forth into a world unknown"

Note that due to support for the :inline option, you can even experiment with adding new functions dynamically in an irb session (compile errors like type conflicts get spit out for you to read)! In the background, Hubris will compile code to /var/hubris/cache and then link it in to your running Ruby program.

The first time you run the code, it will be a bit slower as it compiles the Haskell, but subsequent runs will load the cached .so files directly. Another nice property is that the already compiled code is compared against a saved hash in order to only recompile when changes have been made.

As the Hubris README demonstrates, a third option is load modules already installed on the system:

hubris :module => "Data.ByteString", :packages => ["bytestring"]

Hubris comes with an example Rack app that exposes a Haskell implementation of the Fibonacci function. I'll show it to you here as a compact Sinatra app. A likely use case for Rubyists will be to leverage Ruby's mature web development library ecosystem by exposing an API for a compact and optimized Haskell algorithm/program .

require 'sinatra'
require 'hubris'

helpers do
  hubris :inline => <<-HASKELL
fibonacci :: Int -> Int
fibonacci n = fibs !! n
  where fibs = 0 : 1 : zipWith (+) fibs (tail fibs)
HASKELL
end

get '/:number' do
  fibonacci(params[:number].to_i).inspect
end

As you can see, sensible conversions between data types in both languages are done automatically. Below is a table of the mappings between Ruby and Haskell types (see the Hubris specs for example uses). Try  searching Hoogle (no, that's not a typo) to find Haskell functions supporting types you're interested in.

Hubris currently lacks support for calling functions with multiple arguments (or tuples or custom algebraic types)—which is a serious drawback. However, this is not by design and you can track the Github issues for those missing features via the links in the table above.

In the future, fine-grained interaction between both languages will be possible. For example, unit testing Haskell functions with RSpec, including those with multiple arguments. For the time being, you can take a more coarse-grained approach by passing a hash of many values to a single interface function in Haskell.

A Real-World Example

I'll end my overview by linking to a more complicated example that exposes a Ruby webservice to a coarsely-grained Haskell function. Here, a Sinatra web service is passed a JSON string, which is parsed into a Ruby Hash (actually using C via the JSON gem), then passed to a Haskell function, which is converted on the Haskell side to a Map and then returns an [[Int]] List that gets converted into a nested Ruby Array of integers to finally return the web response as a String. (That made for a long and hard to read sentence, but go through it slowly: it makes sense.)

Rather than calling out to a simple Fibonacci function, you call a basic implementation of a Multi-layer Perceptron Neural Network. HUnit is used for unit testing in Haskell. These tests correspond to values in a handwritten derivation of the first epoch of learning the XOR problem. A Ruby integration test drives the whole stack via rack-test on the XOR problem. A rake task runs both sets of tests to conveniently validate expected behavior. Explaining the algorithm would be enough content for at least a separate post, but hopefully Neurosis can serve as a useful Hubris reference implementation to fiddle around with in the meantime.

Conclusion

There are some serious limitations that prevent fine-grained interaction between Ruby and Haskell. Nevertheless, Hubris has strong potential as a useful library and can be used today via coarse-grained interfaces. Note that Hubris has been tested to work on MRI 1.8 and 1.9, but should work with any Ruby that conforms to MRI's C API.

Installation of the Haskell-side of the library is a bit of a pain right now, but this wiki page is recording the team's progress as they work on fixing it.

So here we are: it's time to go out and write your own Haskell code, to be called from Ruby (or find a favorite Haskell library to reuse)... that is, if you can make it past installation ;)

As always, leave any questions and comments!

Problem: Step Definition Arguments Captured as Strings

As you write step definitions that capture variable data in the provided regex, you'll remain happy if the target data is naturally represented as a string. The following example uses DataMapper as its ORM and dm-sweatshop (found in dm-more) for fixture factories:

# step definition file
Given /^a new user with username (.+)$/ do |username|
  @user = User.make(:username => username)
end

Then /^the user is valid$/ do
  @user.should be_valid
end

# feature file
Scenario: valid username
  Given a new user with username larrytheliquid
  Then the user is valid

However, there are many occasions where the natural data-type for a captured argument may be something other than a string:

# step definition file
Given /^a new user with age (\d+)$/ do |age_string|
  @user = User.make(:age => age_string.to_i)
end

Then /^the user is valid$/ do
  @user.should be_valid
end

# feature file
Scenario: valid age
  Given a new user with age 22
  Then the user is valid

The key thing to notice is that it was necessary to convert age_string to an integer because the concept of age is naturally represented here as an integer. While the conversion does not look like much work now, the actual problem lies in modularity. If we would like to refer to age in any subsequent step definitions, we must duplicate the code to coerce the string everywhere... certainly not DRY.

The code we've seen so far is able to get away with going from captured argument string, to the argument with its natural data-type, to an ORM instance representation, without too much pain. This is possible because the structure of the scenarios being written is fairly simple and setup can be performed in a Given that creates an instance variable to be used in a Then. However, there are more complex scenarios where this pattern is not possible.

# step definition file
Given /^a customer user named (\w+)$/ do |username|
  User.gen(:customer, :username => username)
end

Given /^a support user named (\w+)$/ do |username|
  User.gen(:support, :username => username)
end

When /^user (\w+) is assigned user (\w+)$/ do |support_username, customer_username|
  support_user = User.first(:username => support_username)
  customer_user = User.first(:username => customer_username)
  support_user.assign(customer_user)
end

Then /^user (\w+) should be in user (\w+)'s work queue$/ do |customer_username, support_username|
  support_user = User.first(:username => support_username)
  customer_user = User.first(:username => customer_username)
  support_user.work_queue.should include(customer_user)
end

# feature file
Scenario: support assigned a customer
  Given a support user named stoltini
  And a customer user named larrytheliquid
  When user stoltini is assigned user larrytheliquid
  Then user larrytheliquid should be in user stoltini's work queue

The example above needs to reference specific users in both a When and a Then, and cannot rely upon a single instance variable that sets up state in a Given. The unfortunate result is some pretty nasty duplication of the ORM finder code to lookup each user by their respective usernames in the database.

Solution: Step Argument Transforms

The historical problem with Cucumber has always been that it was restricted to yielding strings as step definition arguments, but this is no longer the case. With a new Transform method, we are able to register regular expressions with Cucumber that it will check against arguments before they are yielded to step definitions. In addition to a regex, Transform expects a block that will be passed the raw argument, and whose return value will be used in place of it.

First, lets revisit our original modularity problem in the age example:

# support file
Transform /^age \d+$/ do |step_arg|
  /(\d+)$/.match(step_arg)[0].to_i
end

# step definition file
Given /^a new user with (age \d+)$/ do |age|
  @user = User.make(:age => age)
end

Then /^the user is valid$/ do
  @user.should be_valid
end

If you look at the Given, you'll see that we expanded the capture group to include "age" as valuable contextual information. With step argument transforms, such contextual information is important to avoid overly general transforms that affect every argument.

The first argument to Transform uses a regex that anchors the beginning and end. This means that we will only match that specific entire string, rather than accidentally matching other step arguments that happen to contain a partial piece of our regex (of course, you could have more general versions if you wanted to, just tread carefully).

If a registered transform matches an argument of a step definition, that argument will be passed to the block supplied with the transform definition. In the Transform example above, we anchor at the end and just capture the digit, because we already know the structure of our input based on the initial match.

After pulling the information we want out of the match group, we apply our transform, to_i, which is yielded to the Given as the variable age. Note that we chose the name age in our new Given instead of age_string because we are expecting the transform to be applied. Most importantly, any other step definition that captures a group of the form (age \d+) will happily transform age into its natural type, keeping our code nice and DRY. Let's see how step argument transforms change our previous more complex scenario.

# support file
Transform /^user \w+$/ do |step_arg|
  User.first :username => /(\w+)$/.match(step_arg)[0]
end

# step definition file
Given /^a customer user named (\w+)$/ do |username|
  User.gen(:customer, :username => username)
end

Given /^a support user named (\w+)$/ do |username|
  User.gen(:support, :username => username)
end

When /^(user \w+) is assigned (user \w+)$/ do |support_user, customer_user|
  support_user.assign(customer_user)
end

Then /^(user \w+) should be in (user \w+)'s work queue$/ do |customer_user, support_user|
  support_user.work_queue.should include(customer_user)
end

Here we use a similar strategy to capture groups of the form (user \w+). The transform applied looks up the user by their username and returns a DataMapper instance. The neat thing is that we can reuse our capture group across multiple different step definitions (the When and the Then), and the more involved duplicated boilerplate code gets packed away in the call to Transform.

Tips and Tricks

Scenario outlines and example tables are really cool features of Cucumber that let you specify a lot of different permutations of data in a compact way. However, the feature is also somewhat limited, because it can only yield string data. With step argument transforms, you'll find yourself using the awesome tables more because duplicated transform code is removed so there is less friction to write additional step definitions.

# feature file
Scenario Outline: username validity
  Given a new user with age <age>
  Then the user is <validity>

Scenarios: valid
| age | validity |
| 18  | valid    |
| 21  | valid    |
| 49  | valid    |
| 120 | valid    |
Scenarios: invalid
| age | validity|
| 0   | invalid |
| 1   | invalid |
| 12  | invalid |
| 17  | invalid |

#... plus different steps using age

Sometimes it may be more convenient to pass a string version of a regex to Transform, so this is supported. Below is an example where the goal is to test properties of a Unix system. Any capture groups that contain path followed by a Unix path are desired to be expanded into their absolute system filepath. The UNIX_PATH_CAPTURE pattern is designed to be regex-interpolated into other regex capture groups, so it is defined as a string to prevent unintentional use as a standalone regex.

# support file
UNIX_PATH_CAPTURE = 'path (?:\w+|\/|\.|-|~)+'

Transform UNIX_PATH_CAPTURE do |step_arg|
  File.expand_path /^path (.*)/.match(step_arg)[0]
end

To avoid overly confusing dependencies, a step argument may only be transformed once. The Transform defined last gets matching order precedence over previously defined transforms, giving you the ability to "override" previous transforms. As a rule of thumb, define general transforms first and get more specific last. More importantly, appropriately including contextual data in capture groups prevents potentially unexpected transforms.

Conclusion

Cucumber has been a fantastic and innovative tool thus far. With step argument transforms, another bit of frustration is removed and your step definitions stay DRY.

As a final note, I'd like to point out how awesome it was to hack out the first version of this with Dave Astels while pairing, test-driven, and in less than an hour... given his BDD/RSpec/Cucumber background =) As mentioned before, we were pairing and ran into a problem that unearthed this feature. Before we knew it the console flashed from red to green.

Happy hacking!