Everyone can still sleep well at night: SHA1 was brutally assaulted, but its walls remain inviolate. The winning Hamming Distance was 30 -- which given the time and resources available was a terrific achievement. That means the winner generated a hash that matched 130 bits of the 160 bits of the challenge hash!

@seibert leaped into an early lead on Monday afternoon and held it alone until, with 75 minutes left in the contest, @hashbreaker crashed in with a HD31 entry. The leaderboard was tied. Then with a few minutes left in the contest, @CodingCrypto jumped in with the sole HD30 entry FTW! @seibert won the coin-toss vs. @hashbreaker for second place -- congrats to all the finalists!

First Place:  @CodingCrypto with BuGS bugs bUgs BUGs BLAnK BlAnK blANK BlaNK BuGS BUgS buGS bugS FkCjV using a 10 machine cluster of mixed CPU and GPU execution. A Hamming Distance of 30. @CodingCrypto wins the iPhone 3GS and $2k in Engine Yard Cloud credit.

Second Place: @seibert with lINeS lInUX ligHTtpD lIb leheY LIBRaRY libcRyPt linK layer LeSK linus laYOuT j|39 using 4 high end NVIDIA cards and a CUDA program written by Steve Worley. A Hamming Distance of 31. @seibert gets the second place iPhone 3GS!

[Greg Lehey is a FreeBSD contributor and Michael Lesk wrote lex and uucp for Unix]

@CodingCrypto was an international team from the Technical University of Eindhoven, San Diego Super Computer Center, University of Illinois at Chicago, National Taiwan University, and Academia Sinica, Taiwan. @seibert was Stan Seibert, a postdoctoral researcher at Los Alamos National Laboratory.

There were a bunch of neat things we learned during the contest:

• A clutch of desktop GPUs is silly fast for these kinds of calculations. Most of the higher scorers were using some kind of GPU rig from Nvidia or ATI.
• Using browser-based javascript engines to crowd-source your computations delivers better than average performance, but couldn't compete with a multi-GPU rig.
• If you're crowd-sourcing, better check the inputs! One of the crowd-sourcers was thrilled to receive a result with a reported hamming distance of zero! Only to discover one of his crowd-sourcers had hacked his javascript to report back this (incorrect) result.

Shout-Outs and Miscellanea!

• http://jazzychad.com/engineyard/ and http://digitalworkboxlabs.com/ did very cool automated leaderboards
• The excellent and creative crowdsourcers
• The Nvidia CUDA hivemind collaboration with some gnarly CUDA code
• @antirez who did the math on hash collision probabilities
• @rkneufeld with his HD34 entry "ruby ruBy RuBY RuBy rUBy ruBY ruby ruby ruby ruby ruby ruby 3:OID"
• And finally, the OCD-like HD33 entry from @hashbreaker: "CoWS coWS CowS cOWS DiRTY DIrtY DirTy DIRtY COWS COWS cOwS cOWS bvpDq"

We thought our sample challenge phrase would give Ruby on Rails folks a small advantage since it was a DHH quote, as was our second sample phrase. It wasn't impossible to figure out what other possible phrases we might use, and start crunching away on the dictionary excerpt last week (it had enough entropy). But alas, it did not appear anyone took advantage of the head-start.

The dictionary was made up of notable programmers, computer scientists, cryptologists, major internet RFC's, Ruby and Rails community Twitter handles, ruby and rails function calls, and a mishmash of random things like xkcd.

Hope you enjoyed the contest! We'll be doing another related blog post next week with a bit more information on some of the creative solutions people came up with, complete with code and stories.

Stay tuned for our September contest, but no more with the brute-forcing! The next contest will entail some creative Ruby programming!

The challenge phrase is:

I would much rather hear more about your whittling project

And here is the phrase dictionary.

Remember:

• The cut-off time/date for the contest is 6pm PDT tomorrow July 21st
• You must be following @engineyard for your entries to count
• Max of five entries per person

While your CPU's are burning through this, you might want to check out our developer webinar on the Engine Yard Cloud at 11am PDT tomorrow; after all, if you're going to win the cloud credit, don't you want to learn all about it? ;)

Good luck!

Note: If you liked this contest, stay tuned for our September contest! Similar levels of trickery + programming will prevail.

What is the contest?

You must tweet a sequence of twelve words that when hashed is bit-wise closest to a hash of a challenge phrase that we will announce the morning of July 20th.  All words must be from a 1,000 word dictionary we will provide at that same time. You are allowed to append up to five random characters to the end of your entry. We're pretty confident you'll want to write a program to automate the finding of close matches, so announcing this a week in advance should give you enough time to get your programs up and running.

How do I enter?

To enter the contest, follow @engineyard on Twitter and tweet your best candidate word sequence before 6pm Pacific Time on July 21st. This means you have about 30 hours between the availability of the challenge phrase and dictionary, and the entry submission cut-off time.

As previously mentioned, the winner of the contest will get an iPhone 3GS AND $2,000 of Cloud (Flex or Solo) credit). [You can also choose an alternative of load test credits worth $2,000 from browsermob]. Second prize is another iPhone 3GS.

So how does it work exactly? (Update! Example Now Clearer!)

Let's take an example: a dictionary excerpt is: "Cloud, Ruby, DHH, one, eight, six, active, record, controller, data, rspec, mongrel, MySQL, postgresSQL, tokyo, MRI, jruby, rubinius, memcached, exception, metaprogramming, reflection." Let's also say we announce that the challenge phrase is I am not a big believer in fortune telling

To submit a contest entry, you would follow us on Twitter and tweet your best entry, e.g:
"@engineyard Rubinius one eight six active active record memcached exception JRuby DHH TOKYO sdfe3"

We will take the SHA-1 hash of this phrase: Rubinius one eight six active active record memcached exception JRuby DHH TOKYO sdfe3 which hashes to cd36b6dc8d4ed51b36dd7fce08f500392a7fb782 and compare it to the SHA-1 hash of I am not a big believer in fortune telling (which hashes to: 6cac827bae250971a8b1fb6e2a96676f7a077b60).

When we say "compare," we mean that we will take the Hamming distance between the two hashes; the sum of the count of dissimilar bits when the hex hashes are converted to binary.

For example, here the binary of cd36...etc. is:
1100110100110110...etc.
and the binary of 6cac...etc. is:
0110110010101100...etc.

So calculating the Hamming distance is done as follows:
- first two bits (1 vs 0) don't match -> +1 to Hamming distance
- second two bits (1 vs. 1) do match -> no change to Hamming distance
- third two bits (0 vs. 1) don't match -> +1 to Hamming distance
etc.

In the case of the complete example hashes above, the Hamming difference is 74. If you are the submitter with the lowest Hamming distance, you win the prizes - it's that simple ;)

Extra Prize: If you manage to achieve a Hamming distance of zero, we'll throw in a MacBook Pro: you are either highly improbable, have mad algorithm cracking skills or you work for the NSA, any of which makes you cool enough to deserve random goodness and recognition. Note: we know the probability of anyone getting to a zero Hamming distance is truly vanishingly small, but we wanted to acknowledge anyone making it there!

There are some obvious brute force strategies to win this prize. We'd suggest building a really fast word permutation algorithm, and finding a fast SHA-1 hash algorithm. Then find a way to get your hands on a whole bunch of computation for the 24 hours that the contest will run (hmm... perhaps the cloud would be useful).

More details and conditions:

1. Only US ASCII printable characters in your custom five character string please (we really want to avoid Unicode rat-holes)
2. The words in your string must be single space separated; no other punctuation is allowed
3. Spamming new entries as you find better ones is a fail whale; limit your contest tweets to a maximum of five
4. The dictionary will be a Macintosh TextEdit file in RTF format, with each word on a separate line, and no white-space.
5. In the case of a tied Hamming distance (entirely possible), the winner will be chosen by lottery among people with the best distance
6. You may permute capitalization for the dictionary words (i.e. you may use Ruby, rUby, RUBY, and RUBy)
7. Please scrub your custom five character string for the five words you can't say on television
8. If the exact same string is submitted multiple times, only the first submission counts
9. Employees and contractors of Engine Yard, and their family members are not eligible

Okay, so maybe "It's that simple" was a bit, well, over-simplified, but we're confident we'll get some great submissions. The Ruby community is nothing if not persistent, creative and intelligent -- so show us what you've got!

Miscellaneous clarifications

1) When you convert the hexadecimal hash to binary -- you need to convert the hex to the equivalent number in binary e.g. "c" = "12" in decimal = "1100" in (big endian) binary. Make sure that your binary conversion function is NOT treating "c" as ASCII letter "c" -- which gets you the completely different answer of "63" in decimal or "01100011" in binary.
2) Be careful if you end up using string hashing functions in C. Remember that C strings are null terminated, and from reports we're getting, at least some string functions out there take the Null string terminator ( "�") as an input to the hash function. This will get you in trouble because we will not be including a null terminator when we calculate the hash of your tweets. Naturally, we will treat your tweet as a (sane) Ruby string.

Another example for folks

People have asked for another example to test their hashing algorithms so here it is:

Example challenge phrase #2:
What you write today will become legacy
which hashes to:
7f83e6b422af5ca4e3112486aea3e702e98a894e or in hex to binary (big-endian):
0111 1111 1000 0011 1110 0110 1011 0100 0010 0010 1010 1111 0101 1100 1010 0100 1110 0011 0001 0001 0010 0100 1000 0110 1010 1110 1010 0011 1110 0111 0000 0010 1110 1001 1000 1010 1000 1001 0100 1110

Example contest entry tweet #2:
@engineyard RuBy one eight six rspec mongrel MRI jruby jruby memcached exception reflection utf8E

We will take the hash of RuBy one eight six rspec mongrel MRI jruby jruby memcached exception reflection utf8E
which hashes to:
075a32acb1816b570607189475ebbbaccce8b79f or in hex to binary (big-endian):
0000 0111 0101 1010 0011 0010 1010 1100 1011 0001 1000 0001 0110 1011 0101 0111 0000 0110 0000 0111 0001 1000 1001 0100 0111 0101 1110 1011 1011 1011 1010 1100 1100 1100 1110 1000 1011 0111 1001 1111

The hamming distance between these two hashes (again, remember treating the hash as HEXADECIMAL, NOT ASCII) is 80.

Example dictionary file: sampledictionary

"Last week, Engine Yard announced they would soon support running JRuby in their cloud environment. I think I speak for the whole JRuby community when I say how excited we are about this new possibility. JRuby has proven itself a top-notch, production-quality Ruby implementation, and the Engine Yard announcement really made us feel proud of what we've accomplished. It also got us thinking about what JRuby really means for Engine Yard customers.

JRuby is, simply put, Ruby on top of the Java virtual machine. While this means you get the benefits of the JVM's world-class garbage collectors, libraries, and optimizations, it does not mean you have to know Java to use JRuby. We've worked very hard to make JRuby look and feel "just like Ruby." So much so, that these days basically all pure-Ruby libraries should "just work" out of the box. Rails runs great, and there's dozens of production users out there reaping the benefits of JRuby's outstanding memory management, native threads (actually running in parallel!), and excellent performance...all of which we continue to improve with every release. JRuby at Engine Yard means you'll also be able to take advantage of Engine Yard Ruby and Rails expertise, along with the assurances that your application will "just work" in their cloud.

So how do you get started with JRuby? Easy!

• Download JRuby from http://www.jruby.org. JRuby 1.3.0 is the current release, but you can feel comfortable testing out either 1.3.0 or 1.2.0: the previous release several folks already have in production.
• Unpack it somewhere convenient. You don't have to install it as root, but you can if you like. And you can have as many separate JRuby installs as you want, alongside any standard Ruby installs already on your system.
• Put JRuby's "bin" directory somewhere in your PATH, so you can run the "jruby" command easily.

That's it! You're ready to try it out!

Depending on how you have your PATH set up, you can either run "gem" to install RubyGems (if JRuby's "gem" comes earlier in the path than standard Ruby), or you can run "jruby -S gem" to force JRuby's copy to run (this will work for anything else in JRuby's bin dir, too). A few gems you'll probably want to know about:

• Rails works great! Just "gem install rails" or "jruby -S gem install rails" and you're nearly there.
• ActiveRecord is also well-supported in JRuby, but you'll need to install the JDBC-based (Java's DB API) adapters. So for mysql, you want "gem install activerecord-jdbcmysql-adapter" and "jdbcmysql" as your adapter type in database.yml. For sqlite3 there's "activerecord-jdbcsqlite3-adapter", and so on.
• Gems that have native extensions (C code, usually indicated by RubyGems trying to build something on install) will not work in JRuby. Usually there's an equivalent library, either a port of the C version, a wrapper around the actual C library, or a Java version. jruby.org provides a good list of commonly-used extensions and their equivalents on JRuby, or you can check isitjruby.org to see if your favorite extension is supported or has an alternative.

We'll be publishing more information about how to get up and running with JRuby, especially if you have native dependencies, over the next couple weeks. And of course you can join the JRuby mailing lists or visit our IRC channel #jruby on FreeNode any time. There's never been a better time to try JRuby!"

Please read the full posting on the Rails Security Group for more details and the appropriate workaround to implement in your code, until the official fix is available in the 2.3.3 release.

(Engine Yard customers have already been contacted via email about this vulnerability).

1) Cache, cache, cache and more cache.
Cache at the client and use Ajax libraries like JQuery to stream in data to the browser on demand. Use gateway /reverse proxy caches to cache HTTP responses at your website, and learn how to use expiration and etags. Take full advantage of Rails' built-in action, page and fragment caching. Use memcache to cache results that you'd otherwise pull from your database.

2) Segregate data and data serving
Don't munge all your data storage into a single database "for convenience." Datasets that are independent should go into separate databases. Serve static assets from a separate tier, or use Amazon S3 or a CDN like Akamai to serve those assets. It's more expensive, but it simplifies scaling. Relational databases scale up, not out, so sit down and have a heart to heart talk with your DBA over whether you really need a relational data model for all your data stores. Maybe you can get away with a simpler key-value data store for some of your simpler data. There are ruby clients, so use Hadoop for scaling the storage and analysis of large amounts of unstructured data. Also know the scalability limitations of whatever file system you're using. If you have heavy data reporting needs, do your reporting from a copy of your main database, not from your production database!

3) Minimize & handle external dependencies
Watch for dependencies on external services like ad serving networks or RSS feeds. If a service isn't responding or can't handle your growing request load, make sure that you have a fallback strategy.

4) Tend your database and your job handlers
Any ORM, including Rails' ActiveRecord can generate SQL queries that cause database performance issues. Make sure you're looking at your slow query log after each major integration to make sure you don't have "missing" database indices, and haven't written inappropriate find-all's in your Rails code. Scrub your database periodically for indices that are no longer being used. Similarly, watch the resource consumption of your background and scheduled jobs. As your user base grows jobs can start to overlap, and daily log processing can start to take more than 24 hrs! This kind of thing can sneak up on you easily. Ideally, segregate your jobs in a separate tier. And, as you grow, look at moving to a message based job handler.

5) Shard your unavoidably relational data
At high scaling levels, your MySQL database will have to be sharded. Sharding involves breaking up your datasets into independent pieces based on a key. For many consumer-oriented Rails sites, that can mean sharding based on userid's, but other sharding schemes use data-age, or access-frequency (if that's predictable.)