Blog

Archive for January, 2009

A Clarification on Public API

By | January 8th, 2009 at 4:01PM

I’ve heard more than a few people express some confusion about what is meant by “public API“, especially since I have lauded Rails for having an excellent user-facing API. How does that jive with the agenda of creating a defined, documented API?

To clarify, I am applauding Rails for having created a developer interface to the framework that is pleasant to use. A lot of code that is created using the Rails DSL is immensely understandable, even to relatively new users, because of a passionate focus on making it so.

When I talk about “public API” I mean something much more boring and technical. In particular, I mean dividing up the parts of the Rails internals that are meant to be used, documenting them as public, and writing tests that are specifically marked as public and cannot be changed without a darn good reason. At present, while the public-facing API is relatively stable, there is no infrastructure around making it so.

But perhaps more important is the idea of doing the same for plugin developers that Rails has already done for end-users, by forming known, defined, and tested mechanisms for adding or modifying Rails behavior. In the same was as for public-facing functionality, we should document the methods that are intended to be used by plugin developers, create hooks where appropriate, and write tests to ensure that these contracts with plugin developers remain so.

Once this is done, and the private functionality of Rails is well cordoned off from the parts that are intended for public or plugin use, the Rails team will have significantly more latitude to keep the internals of the framework from getting crufty too quickly. We found this approach to be quite useful in Merb, and the plan is to do the same for Rails.

Of course, it can hardly be said that by creating a public/plugin API we will have completely removed the need for anyone to ever monkey-patch Rails internals directly. However, it will be more clear which modifications are likely to survive from version to version, and which modifications will require constant vigilance (and should probably be the subject of a conversation with the Rails core team itself to determine whether a more explicit hook could be added).

In short, when I say “public API“, I mean documentation, testing and commitment.

Popularity: 1% |

This Week’s Changes at Engine Yard, With More Detail

By | January 8th, 2009 at 4:01PM

Hi,

I was on our weekly customer call this morning and I heard directly from customers that we haven’t done the best job of communicating some changes that happened earlier this week at Engine Yard.

First, let me apologize for this. We’ve tried to be open about important things, but sometimes it’s not always clear how much or how little detail is right.

To our customers, partners, and the Ruby and Rails communities at large, I want to reassure you that we are financially very sound and we intend to be here for years and years.

Our new CEO and everyone at Engine Yard are making the company better and making our long-term goals more attainable.

I hope the following timeline will help clarify things:

  • Tom and I started Engine Yard in 2006. We brought in Ezra and Jayson as founders #3 and #4 a little bit later in 2006.
  • Tom and I shared the CEO role in 2007.
  • I was CEO alone in 2008.
  • During 2008, we took investments from Benchmark, New Enterprise Associates, and Amazon. We were growing fast and this sped things up even more.
  • By the middle of 2008, all 4 founders agreed that Engine Yard was growing so fast that we needed someone with the experience to grow the company WAY beyond my/our abilities. I have experience running companies with simple operations in Support, Finance, Sales, etc. Engine Yard grew to 80+ people by the end of 2008 and the organization has grown more complex right along with our customers and the market.
  • Someone with prior experience scaling a business from small to medium to large can benefit Engine Yard and its customers a great deal, and I can learn a lot from such a person. Thus, I see this as a good problem to solve.
  • We started a search and interviewed a number of fantastic candidates in the Summer, Fall, and Winter of 2008.
  • The 4th quarter of 2008 saw a lot of change and analysis. We brought in some great consultants to help us better understand our market and the internals of our business (how to run a financial organization in addition to bookkeeping, where we could operate more efficiently, etc). Add to this the changing landscape with things like cloud computing and we found ourselves needing our first formal budgets and plans for the coming year.
  • The overall plan encompasses everything from slightly changing our support, sales, and account management organizations; to bringing out new AWS-based services; to open-sourcing our Vertebra software. The plan took months of input from every part of the company. With regards to layoffs, we decided it was best to wait until after the holidays.
  • Back on the CEO search, we were getting into the holiday season and John Dillon stood out as a superb candidate. He’s a great guy and has fantastic experience. At earlier points in his life, John was CEO of Salesforce.com, CEO of Hyperion, and he held sales & engineering positions at Oracle & EDS. John earned a degree in engineering from the U.S. Naval Academy at Annapolis and an MBA from Golden Gate University.
  • We made a deal with John over the Christmas break and it was decided that he’d start when the holidays were over.
  • Which leads to this week. We did the layoffs on Monday – the same day John started. I want to clarify that there’s no connection between John and the layoffs, nor between my stepping down as CEO and the layoffs. The timing is simply because Monday was the first full work week in 2009.
  • Some have asked about my role moving forward. My role is supporting John as he learns the details of Engine Yard, its customers, and its community. I will define a new role for myself where I can use my 3 years at Engine Yard to help us make the best possible decisions.

I hope this helps!

If you have any questions or comments, please feel free to email me at: lwalley (at) engineyard (dot) com

Thanks!

Popularity: 1% |

Passively Multiplayer Online Game

By | January 7th, 2009 at 4:01PM

GameLayers, the people behind the awesome online game, PMOG, just posted a nice article about their experience hosting at Engine Yard.

If you want to imagine how just a few folks build a big fun game, it’s with the help of companies like Engine Yard.

Thanks for kind words, GameLayers!

Popularity: 1% |

Today’s Communique

By | January 7th, 2009 at 4:01PM

During work today, I made more progress in unifying the ActionController=>ActionView API:

The API into ActionView is now ActionView::Base#_render_for_template. This is where the refactor was going all along, and with Josh’s help, we were able to really solidify how this should work.

At the moment, there are a few stragglers that may or may not be cleaned up: AV::Base#_render_text and AV::Base#_render_inline. Using _render_inline in particular ensures that render :inline does the same thing in both ActionController and ActionView. This is the code in ActionController that delegates to ActionView for :inline:

if inline = options[:inline]
  render_for_text(@template._render_inline(inline, _pick_layout(layout_name), options))

This is the code in ActionView for rendering inline:

elsif inline = options[:inline]
  _render_inline(inline, layout, options)

As you can see, having a single method allows us to be sure that the code does the same thing in both places. You can compare with the code from before the refactor. ActionController:

elsif inline = options[:inline]
  render_for_text(@template.render(options.merge(:layout => layout)), options[:status])

ActionView:

elsif options[:inline]
  InlineTemplate.new(options[:inline], options[:type]).render(self, options[:locals])

Not particularly terrible, but the mechanism is a bit more obscure, and it’s less obvious that both pieces of code do the same thing. Before the refactor, you can see that ActionController delegated to ActionView’s render method, after twiddling with the options and passing them through uncritically.

Another interesting item in today’s refactor: previously, layout selection happened at the beginning of the render method, before the options were examined. As a result, it was necessary for the template chooser to scan through all the options to determine whether it should fall back to the default layout if no layout was specified. For instance, render :action assumes that a layout should be used by default, while render :text assumes that a layout should not be used. Of course, :layout => “…” can be specified, but the default behavior is different.

In the refactored version, the _pick_layout method is called with information about whether it should fall back to the default, so it no longer has to scan the options. It also means that a layout is never selected if it is not required (render :nothing or RJS, for instance).

After work, I hung out with a friend from New York, but when I got back I decided to do a bit more work on Rails.

Some interesting stuff: I’m experimenting with ConcurrentHash and SafelyMemoizable, which exploits the fact that Ruby implicitly locks instance variable mutations to create a cache that should both be threadsafe and not require locks for almost all cases. This also allows a very simple threadsafe memoization technique for global caches. Again, this is a pretty early experiment, so the details may change some. Note that it’s not as expensive as it looks, because Hash#dup is a shallow clone, so the memory-expensive values will exist only once in memory.

I also cleaned up layouts a bit more, including replacing a somewhat complex mechanism for dealing with default layouts that was implemented via an inherited hook that involved the rules for whether or not a missing layout should raise an error. It still is fairly arcane stuff, but I’m pretty pleased with the revised code. I needed to add some tests for layout :except and layout :only, as there were no tests for that functionality.

The very last thing tonight was to put the final piece in place to have a single place for determining whether a response is exempt from layout. Now that all options that involve rendering templates go through the same method, all of the logic for skipping layout is encapsulated in layout = _pick_layout(*layout) unless tmp.exempt_from_layout?. Putting this piece in place was very personally satisfying.

At this point, I’m pretty sure I’m posting simply to create a personal record of my work. If some of you are enjoying it, that’s a bonus!

Popularity: 1% |

Another Dispatch: Step 1 of 2 Complete!

By | January 6th, 2009 at 4:01PM

A couple of notes of interest for the work over the past few days:

Git is holding up surprisingly well. I’ve done daily merges from rails/rails/master into wycats/rails/master and then into wycats/rails/action_view, and have so far only run into a single conflict, which was easy to resolve. Moving Rails to git made the work we’re doing possible!

As far as I can tell, all current cases of using render() as an API now use _render_for_parts(). This is a huge milestone. Now that the four-element tuple is the standard across the board, we can push back the path lookup to the callers and unify behind something like _render_for_template(). There’s still quite a bit of work to do in general (most notably around partials and layouts), but things are coming along quite nicely.

I was purposely holding off keeping ActionMailer in sync as I did the refactor, because things were too much in flux for that to really be viable. As it turns out, it was relatively simple to move ActionMailer over to the new approach after everything was done. The only real irritation is the repeated use of the render() options to do mostly the same thing. I’ve moved the code that takes a partial path and converts it into the requisite paths into action_pack/common.rb so it can be used in both ActionController and ActionView. I’m not sure if this will be the final resting place but this refactor definitely pulls out some AC/AV common functionality.

I am more convinced than ever that a common AbstractController superclass for both ActionController and ActionMailer is the way to go. This was actually the thing that originally brought me to Merb (that firm conviction), and I’m pretty convinced it’s the right approach for Rails as well. Of course, there’s still quite a bit of work to do before it’s possible to really evaluate it, so stay tuned.

Lori has been working hard on the ORM adapter for Rails. Effectively, the adapter will work like: ActiveORM.for(@sequel_obj). If the object is compatible with the ActiveORM API (note that the names et al are still very much in flux), it will be passed through. For instance, ActiveORM.for(@ar_obj) will just return the ActiveRecord object. ActiveORM.for(@sequel_obj), however, will return a wrapper around the object that is compatible with Rails expectations. Example: ActiveORM.for(@sequel_obj).new_record?. Note that this paragraph is speculative and represents work that is still very much in flux.

I’m really looking forward to finishing the work that _render_for_parts was a placeholder for. As I go through the codebase, I’m definitely noticing some obvious perf opportunities (non-trivial perf that should show up very clearly on macro-benches) and I’m looking forward to finishing the initial refactor so I can get into some perf optimizations. I should have some initial numbers within a week or so.

Popularity: 1% |