Unless you’ve been living under a rock, you’ve probably heard about the new hotness for JRuby: Java 7’s support for dynamic languages. You may also have heard about the huge perf gains that JRuby’s seeing when running on Java 7. How can you try it yourself?

Get Java 7

The reference implementation for Java is OpenJDK, and OpenJDK 7 has been out for almost six months now. The current version is 7u2 (’u’ stands for ‘update’), and includes a number of improvements over the GA (‘General Availability’) release.

Most platforms have easy access to OpenJDK builds. I’ll summarize the steps here.

Linux, Windows, and Solaris

Oracle provides binary downloads for Windows, Linux, and Solaris on its site. The JavaSE Downloads page links to both JDK and JRE download pages. You’ll probably want the JDK, since it includes other JVM-related dev tools, but the JRE will work too. Download, install, and you’re ready.

Additionally, package managers for Linux and Solaris will likely soon have OpenJDK 7u2 packages available, if they don’t already.

OS X

The official preview of OpenJDK for OS X lags behind, but you can get the current builds from theopenjdk-osx-build project. The build you want is currently labeled “OpenJDK-OSX-1.7-x64-u2-b21”, but any build labeled “1.7” and “u2” in the future will get what you need. The .dmg provides either a self-contained JDK for you to drop onto your system or a .pkg installer that does it for you.

Update: Henri Gomez, the primary guy behind openjdk-osx-build, has set up a page specifically for the update builds. Grab the JDK or JRE from his OpenJDK7JDK7UOSX page.

*BSD

The OS X work is based off the “bsd-port” branch of OpenJDK. There are links to Java 7 package information for FreeBSD, NetBSD, DragonFly BSD, and OpenBSD on the BSD Port wiki. These may not be updated to 7u2 yet.

Why Update 2?

We haven’t previously made a lot of noise about Java 7 and JRuby, nor assembled a blog post/tutorial like this, primarily because Java 7 GA was missing key optimizations in the invokedynamic subsystem. JRuby 1.7 will make heavy use of invokedynamic, and if we had released it before those optimizations were in place, it would have given people a bad impression of the power of invokedynamic.

Update 2 now has a small set of optimizations that make a very large difference. If you intend to start playing with JRuby 1.7 builds, we strongly recommend you use OpenJDK 7u2 or higher.

Update: Your jruby -v output should look something like this:

jruby 1.7.0.dev (ruby-1.8.7-p352) (2011-12-19 f404f75) (OpenJDK 64-Bit Server VM 1.7.0-u2-b21) [darwin-amd64-java]

2011: The Year We Made Contact

We've come a long way since JRubyConf 2009.

That year, JRubyConf was simply an "extra day" attached to RubyConf, in the same hotel and with many of the same attendees. It was a great success, but pretty low-key compared to JRubyConf 2010. In 2010, Joe O'Brien and the EdgeCase folks offered to merge JRubyConf with eRubyCon, the "enterprise" Ruby conference. It marked the first time JRubyConf was a real, live conference, and we managed to draw in 150+ attendees and a great bunch of speakers.

JRuby has come a long way too. Since JRubyConf, we've released two major versions of JRuby: 1.5 and 1.6. Both versions represented a tremendous number of commits and bugfixes...the two combined add up to almost as much work as all prior versions of JRuby. All that work has meant JRuby's achieved an extremely high level of compatibility with Ruby 1.8.7 (we almost never get 1.8.7-related bugs anymore), and in 1.6 we even officially started supporting Ruby 1.9.2 features. Also in 1.6, we included experimental support for running C extensions...something we never believed we'd be able to do.

The JRuby updates and yearly JRubyConf are joined by the JRuby book, "Using JRuby", available from Pragmatic Programmers. After two years of on-again, off-again work, we managed to finish the book early this year. The response has been excellent, with new and existing users alike telling us how much they've enjoyed reading it.

All this adds up to a big 2011 for JRuby, and it feels like we're starting to get our stride in both the Ruby and Java worlds. Almost every Rubyist has at least tried JRuby, and it seems like every other Rubyist has something running on JRuby today, even if it's not their primary platform. For me, that means people find JRuby useful, either as "just another Ruby", or because it has a number of features above and beyond Ruby (like Java integration and novel deployment options). For you, maybe it's time to check out JRuby, buy the book and attend JRubyConf.

A Whole New JRubyConf

For 2011, we've made JRubyConf into its own event, not attached to or taking over any other conference. The show will be in Washington, D.C. August 3-4, at the beautiful Kellogg Conference Center Auditorium on the campus of Gallaudet University. The conference registration price has been slashed, thanks to support from Engine Yard and EdgeCase, to only $199! We'll have the requisite after-hours events, repeating our whisky tasting from last year's show and adding a few other fun activities. But the focus of the show is squarely on the excellent speakers and presentations you'll see. I'm looking forward to all of them, but here's a few talks I think Rubyists and Java-heads alike will find interesting, especially if you're new to JRuby.

Adopting JRuby and Making It Work

If you're not yet sold on JRuby, Marc Peabody might be able to convince you. He's presenting A JRuby Tale, the story of a nearly-complete JRuby project being deployed into a Java organization. You'll learn not just when to use JRuby, but also when JRuby might not be the best fit. We also have Jason Gilman presenting Building the MagicWebService. Jason works on a NASA project called ECHO, which had a large SOAP web service they wanted to expose through a RESTful interface. JRuby came to the rescue, providing access to high-performance Java libraries, wrapped with a little Ruby sugar and wired into the Ruby libraries you know and love.

Along similar lines comes the JRuby at LinkedIn talk by Baq Haidri. LinkedIn is one of the premier JRuby users, with several of their key applications using JRuby for the web front-end. Baq will explain the reasons for using JRuby on the front-end compared to other JVM language options, and hopefully we'll see some of the awesome polyglot integration the LinkedIn folks are doing.

Desktop Applications

If you're a desktop application developer, you can't miss Dan Lucraft's talk on Cross Platform Desktop Apps with JRuby. Dan is the creator of the Redcar Editor, a Textmate clone written in Ruby and running on JRuby. Redcar uses the SWT GUI framework for its UI, but wraps it with Ruby code and combines it with a powerful plugin system for all of Redcar's functionality. The result is a beautiful, fast, powerful editor that works across platform and requires no compilation on any of them. It's a great example of how to write good Ruby and take advantage of what JRuby has to offer.

Big Data

Big internet means big data, and Jeremy Hinegardner will present the timely Big Data and JRuby. If you're going to process big data, you need a solid runtime that can manage memory well, a concurrent runtime that allows you to completely utilize your hardware, and the right tools and libraries. JRuby, running on the JVM, gives you easy access to all three. Jeremy will talk specifically about Hadoop and Hadoop-related projects and how you can use them from JRuby today.

Sematic Web and JRuby

If you're interested in the semantic web, Carin Meier will present Semantic Web and JRuby and show how to leverage the Jana semantic web framework from your Ruby applications. I know at least one JRuby committer who's always had a think for the semantic web. Maybe you do too?

Web Deployment

Once you've got your JRuby application written, you're going to need to deploy it. To answer that need, we have David Calavera presenting Trinidad is Still My Name and Ben Browning presenting Why I Love Torquebox (And Why You Should Too. Trinidad (an embedded command-line Tomcat for running Rack apps) and Torquebox (JBoss AS plugins and standalone server for Rack, messaging, and more) are the two top web deployment options for JRuby these days. This is your chance to talk to the folks behind them and learn why you might choose one over the other.

Beyond JRuby

Every good conference needs talks that are either non-technical or which push the boundaries of current tech. For that, we have Justin Gehtland, Michael Feathers, and Neal Ford presenting on aspects of the development process that plague us from day to day. Justin's talk Never Settle will help you learn to keep fighting for a better experience in both development and day-to-day life. Michael will present Your Software, It's Alive, emphasizing something too few developers realize: software projects are born, grow up, and die just like organisms. And the always excellent Neal Ford will present Abstraction Distractions, a survey of what sorts of designs and abstractions work and which ones don't (with examples ranging from the iPad to Maven) and how you can apply that to your own software.

On the more technical side, Ian Dees will present JRuby, Not Just for Hard-Headed Pragmatists Anymore. Ian tackles the other side of JRuby, where you can utilize libraries from languages like Clojure or Scala and build your own languages using tools like BiteScript. JRuby's more than just a Ruby implementation...and I'm sure Ian will show some good reasons why.

And finally, there's Russ Olsen presenting Eloquent JRuby. Russ will show something I've been dying to see: best practices for a JRuby application that keeps Javaisms from leaking into your beautiful Ruby world.

JRuby Core Team

Of course we'll have a session to update you on JRuby's status and future. Tom Enebo and I will talk about: invokedynamic and its impact on JRuby; a new intermediate representation compiler we're working on; JRuby 1.7, Ruby 1.9.x, and Java 7 plans; and the status of JRuby adoption, platforms (like cloud, mobile, embedded), and community.

More to Come!

We're still ironing out the last few speakers, and will try to post an update in the next week. But do you really need more reasons to attend?

One Last Thing

With that, I'd like to personally invite you to attend JRubyConf 2011. JRuby has really come into focus over the past year, and I really love hearing from current and potential JRuby users. We're building this thing for you, and we're putting this conference on for you, so come out to D.C. and tell us what you're doing with JRuby or what you'd like to see from us in the future.

JRuby is an important tool every Rubyist should have in their toolbox. Come to JRubyConf 2011 and let our excellent speakers tell you why!

We've just wrapped up the longest, largest JRuby dev cycle ever: JRuby 1.6. It's time to recap what we've accomplished and talk about what comes next...both for you and for JRuby.

A Mountain of Work

Boy, and I thought JRuby 1.5 would never be topped. That release cycle last 5 months from the end of JRuby 1.4 maintenance in late 2009 until the 1.5 final release in May of 2010. We fixed an resolved an incredible 456 bugs and had around 1250 commits, making it our largest release at the time.

For JRuby 1.6, we "only" resolved 420 issues. But during the roughly 9-month dev time, we logged nearly 2500 commits (twice as many as JRuby 1.5). What the heck did we accomplish?

New and Notable

I'm going to direct you to my JRuby 1.6.0 RC1 post to get more details, but here's the "short" list of major release bullets. I will call out the fact that since the 1.6.0.RC1 announcement and today, we fixed an additional 150+ issues, most of them reported against our release candidates. Thank you for helping us make this an outstanding release!

• Ruby 1.9.2 API and language features: JRuby 1.6 is the first release where we recommend people start trying 1.9 mode. We've put in months of work making sure the new APIs function, we've updated the standard library to the current MRI 1.9.2 copy, and we're actively looking for missing or buggy features to add or fix. If you've been waiting for 1.9 support in JRuby, now's your time to give it a try...pass --1.9 at the command line or put it in the JRUBY_OPTS environment variable and you're ready to go!
• Improved Ruby performance: As with every release, there's tons of incremental perf improvements in JRuby 1.6. We never get as much time to focus on performance as we'd like, but this time there's been more attention paid to execution performance specifically. Small benchmarks are running 1.5 to 2x faster, and CPU-bound applications should perform better as a result. We've also laid groundwork to enable our own optimizing compiler modes and to make it easier for the JVM to optimize things for us. It's even better now, with enormous potential for JRuby 1.7.
• RubyGems updates and enhancements: We've bumped RubyGems up to version 1.5.1, which greatly improves startup time. We've disabled (by popular demand) the generation of ri and rdoc during gem installation. We've added support for installing Java libraries directly from Maven as if they're regular gems. And you get all this in both 1.8 and 1.9 modes.
• Profiling and debugging: JRuby 1.6 improves the flat profiler from 1.5.6 (--profile or --profile.flat) and adds a graph-based profiler for more detail (--profile.graph). We've also resolved release issues for the ruby-debug gems, so you can gem install them from JRuby without hassle.
• Platform and native: This release marks the first time we've called Windows an "officially supported" platform. All that really means is we've got a Windows CI server and we're keeping it green for all the suites we run on Linux and OS X. We're also previewing experimental support for installing C extensions written to MRI's C API. Not every extension works, and there's certainly some caveats...but it's a hell of a thrill seeing sqlite3_ruby or rubysdl installing and working.
• JVM and JVM languages: For 1.6, I wired up a lot more Ruby logic to the new "invokedynamic" JVM bytecode, which may enable us to run Ruby code many times faster than we do today. We also included enhancements to make it easier to call from Ruby into Scala libraries, which has enabled the Lift web framework to offer support for Ruby. The JVM is is becoming an amazing polyglot environment, and JRuby's leading the charge.
• All-around improvements: JRuby starts up faster, runs faster, uses less memory, and integrates with the JVM and JVM libraries better. It behaves consistently across platforms, runs anywhere the JVM runs (including on Android!), and lets you remain a Rubyist while you take advantage of an amazing VM and solid libraries. It's the best of both the Ruby and the JVM worlds, together in one package.

What's Next?

I won't go all cliché on you and say this is just the beginning for JRuby. We're widely accepted as an outstanding Ruby implementation (and an outstanding JVM language), and JRuby runs applications for banks, governments, airports, space exploration, and more. JRuby is no newcomer...so we're well past the "beginning" stage.

What I will say is JRuby is about to cross a threshold. The release of JRuby 1.6 marks the first time we're largely caught up with the released version of Matz's Ruby, 1.9.2. Now of course development continues unabated on Ruby 1.9.3, and we'll continue to chase MRI forever. But we've made up a lot of ground in a very short time...which means we're going to have more freedom to work on things other than Ruby compatibility.

Now it gets exciting.

Looking Toward the Future

We love working on JRuby because we love working atop the JVM. Say what you will about Java the language...the JVM is very hard to beat. There are JVMs for just about every platform out there. The JVM boasts some of the best garbage collection and optimizing JIT capabilities in the world. The platform plays host to tens of thousands of libraries, increasing the chance that you'll find what you need...and that it will actually do what it claims. And best of all, when we work on JRuby we don't have to worry about the JVM at all...there are dozens or hundreds of folks around the world ensuring every JVM is moving forward at a breakneck pace.

So is the JVM perfect? Of course not!

JRuby stands out among JVM languages for having done the most to accommodate the JVM. We've hacked around aging, cumbersome APIs (process management, IO, JDBC). We've gone "to the metal", using native libraries and OS features to improve your JVM experience (posix bindings, FFI, C exts). I personally have gone to great lengths to make Ruby code not just compile to JVM bytecode, but compile in such a way that the JVM can optimize it well. So it may come as a surprise to hear how excited we are that Java 7 is going to make a lot of that work unnecessary and irrelevant.

JRuby 1.7 and Java 7

I have a vision for JRuby 1.7. With our release cycles these days, we'll probably see a JRuby 1.7 release some time around the release of Java 7. And I want JRuby to take as much advantage as possible of the features Java 7 brings to the table. What are these features, you ask?

InvokeDynamic

Probably the biggest and most exciting change in Java 7 is the new support for dynamic invocation. Now I'm not just talking about being able to dispatch from A to B dynamically, like we have in Ruby. Obviously we and other languages have been doing that for years. What I'm talking about is finally being able to tell the JVM what the hell we mean by dynamic invocation...and teaching the JVM how to optimize it.

InvokeDynamic has been underway for three to four years now. It brings to the JVM two key features:

• The new "invokedynamic" bytecode, for making calls that are wired up at runtime rather than at compile time.
• The MethodHandle API, which provides optimized "function pointers" to wire up those runtime calls.

Optimizing VMs like the JVM do their best work when calls are as direct as possible. The JVM takes a call from A to B, sees that you're always calling B, and optimizes both methods as if they were one. This is called "inlining", and it can be done ahead-of time by compilers like GCC or at runtime by optimizing runtimes like the JVM. All the most important optimizations grow out of inlining. Conversely, if you can't inline B into A, you're not going to get the fastest-possible execution.

On Java 6 and earlier, JRuby has a hard time making dynamic calls inline. Because there's a lot of method lookup logic between A and B, we confuse the JVM; it can't see that A's dynamic call eventually gets to B, and the two are never apart, so it treats them separately. We still get a great boost from the JVM's optimizer, since A and B both make a lot of calls into the JRuby runtime. But we're defeating most optimizations because we don't have a good way to explain to the JVM "hey...ignore all the logic between A and B, and treat them as a direct call."

That's where invokedynamic comes in. With invokedynamic, we're explicitly saying "this call is special". The JVM sees the invokedynamic call and asks JRuby "ok...what do I do?" JRuby responds by wiring up a series of MethodHandle objects that take A's call and wire it all the way to B. But instead of that intermediate logic confusing the JVM, it now knows all that logic is just in service of A's dynamic call. It knows that B is being called from A, and the two should be optimized together. That's really amazing.

I've seen this in action many times now. I've seen the JVM inline Ruby code into Ruby code, without any crazy compiler tricks in JRuby. Even more amazing, I've seen Ruby inline into Java and Java inline into Ruby. Yes, that's right; with invokedynamic, you'll see optimization happen across all JVM languages; it's very likely some application out there will see Ruby, Clojure, Scala, and Java all inlined and optimized together. And there's another incredible benefit for JRuby users: thousands of lines code and hundreds of thousands of bytes of bytecode specifically crafted to support dynamic calls will simply disappear...since invokedynamic eliminates the need for many tricks we utilized in the past.

There's a very real chance that invokedynamic could improve JRuby performance many times, putting us on par with our statically-typed brothers like Java and Scala. And that means you can write Ruby code without fear. Awesome.

New IO 2: Eclectic Bugaboo

The addition of InvokeDynamic also fits another theme: things that should have been in the JVM in the first place. NIO2 takes that theme and runs with it.

The "New IO" APIs, added in Java 1.4, fixed a long standing problem for Java developers: matching native IO performance and features. NIO added the ability to read from IO "Channels" directly into native buffers, rather than lifting bytes up to JVM objects. It made real nonblocking IO calls possible by abstracting the standard nonblocking APIs across all platforms (think eselect, kqueue, etc). And it finally gave us direct control over sockets, file streams, memory-mapped regions of disk, raw datagram traffic, and more. It made the JVM a solid platform for high-performance IO.

That was nine years ago. And little has changed since then.

In addition to wanting better control over IO, JVM users have screamed for better access to the filesystem itself. We've wanted to query and set permissions. We've wanted to monitor for filesystem events, like modified files and directories, without constantly polling. We wanted to be able to manipulate symlinks -- seriously...SYMLINKS...totally unsupported in the JVM (except by intrepid upstarts like JRuby that have routed around the JVM.) NIO2 finally does that.

So, what do we get with NIO2?

• Platform-agnostic filesystem events: You read that right...JRuby users will be able to use filesystem event APIs that look and behave the same across all major platforms. No more futzing around with C extensions, no more platform-specific logic. Write once, run anywhere.
• Filesystem attributes: NIO2 will enable us to manage a much broader range of filesystem attributes across platforms. We'll have all the permissions for UNIX plus APIs for managing Windows ACLs. We'll be able to manage symbolic and hard links directly. And we'll be able to read directories just like POSIX users have done for years.
• Filesystem inspection: There are several new APIs for walking the filesystem quickly, where previously a lot of cycles were wasted constructing Java File objects, examining them, and then proceeding. There's APIs for quickly reading in file contents, reading in all lines from a file, or getting a direct channel to the file's bytes on disk. And there's an entire set of filesystem-general APIs for querying available space, attributes (remote, read-only, etc), and more.

Speedier Adoption?

Java 7 also marks the first time that a major Java release will be available in open source from day 1.

OpenJDK is the GPLed version of Sun's "Hotspot" JVM and class libraries. When the Sun piñata broke open, OpenJDK is one of the jewels that fell out. Because of OpenJDK, we now have Oracle, IBM, RedHat, Apple, Azul, and others all collaborating on the same JVM, ensuring it will run great on all platforms. We have top-notch JVMs for every Linux and BSD flavor. And I believe OpenJDK will mean Java 7 adoption moves much more quickly than Java 5 or 6.

All the best development these days are done by folks like you and me that aren't afraid of new things. Give us a buildable, open-source project, and we'll start exploring how we can use it. Because JVM releases used to lag behind on many platforms, and because it wasn't possible to know...really know...what changed in the underlying codebase, people were usually terrified to make a move. Now that OpenJDK is out there, you can build the JVM on your own, you can understand how the platform works. And you can be confident in making a move to Java 7, taking advantage of everything it offers.

For JRuby users, this means you can start playing with solid OpenJDK7 preview releases right now, with a final release coming out this fall. You can start to take advantage of invokedynamic and NIO2. The platform is finally moving again...and it's going to be a hell of a ride.

A Bright Future

JRuby's future is looking, well, amazing. The Java patform has emerged from a long slumber, and JRuby users will soon have an amazing array of features at their fingertips. You can still be a true Rubyist, writing Ruby code, and stand shoulder to shoulder with any language or runtime on performance and features. Most importantly: you can still have fun, and get work done at the same time!

It's a great time to try JRuby.

Ruby 1.9.2 Language and API Compatibility

Ruby 1.9.2 compatibility is largely complete. There are a few features that we plan to add in a follow-up release: Encoding::Converter, non-ASCII identifiers, and the 'ripper' and 'fiddle' libraries. We would like users to test JRuby in 1.9 mode to shake out remaining issues before 1.6.0 final.

Support for Ruby 1.9.2 represents a tremendous amount of work by many contributors. We are very proud to include it in JRuby 1.6.

Improved Ruby Performance

A key factor for performance is how quickly methods can be called. In 1.6 we have improved the performance of Ruby method calls, reducing their overhead substantially. We have also made smaller optimizations across the board in response to user reports and profiling.

Please continue to report performance issues you come across in your own applications. We consider poor performance a bug to be fixed!

Built-in Graph Profiler

In addition to the "flat" profile support added in JRuby 1.5.6 (via the --profile flag) we have added "graph" profiling support (--profile.graph) contributed by Dan Lucraft of the Redcar project. Graph profiles will help you localize performance bottlenecks, making them easier to fix.

~/projects/jruby ➔ jruby --profile.flat -e "def foo; 100000.times { (2 ** 200).to_s }; end; foo"
Total time: 0.99

    total        self    children       calls  method
----------------------------------------------------------------
     0.99        0.00        0.99           1  Object#foo
     0.99        0.08        0.90           1  Fixnum#times
     0.70        0.70        0.00      100000  Bignum#to_s
     0.21        0.21        0.00      100000  Fixnum#**
     0.00        0.00        0.00         145  Class#inherited
     0.00        0.00        0.00           1  Module#method_added

Check out my blog post on the new JRuby graph profiler.

RubyGems Changes

This release updates our bundled RubyGems to 1.4.2. RubyGems 1.4.2 has eliminated several bottlenecks impacting startup speed, and fixed several issues from previous releases. Both 1.8 and 1.9 modes use RubyGems 1.4.2 in JRuby. Also, in response to user requests, we have disabled the generation of ri and rdoc when installing gems. See the 'gem rdoc' command to generate docs offline.

We also unbundled rspec and ruby-debug in this release. You should be able to gem install them normally.

C Extension Support (Experimental)

For Ruby Summer of Code 2010, Tim Felgentreff helped us build out support for Ruby's C extension API. JRuby 1.6 includes that support as an experimental feature. The first release candidate only ships C extension support prebuilt for OS X, but we hope to include it prebuilt for Windows and Linux by 1.6.0 final. If you need to build it for your platform, follow these instructions for building JRuby C extension support.

We recommend users implement JRuby extensions in Java or use a Java library when possible, and consider using FFI to wrap C libraries, but we hope C extension support will make it easier to migrate to JRuby.

RubyGems Maven Support (Preview)

Continuing our efforts to integrate Ruby and Java ecosystems, JRuby 1.6 includes a preview of integration between RubyGems and Maven. Any artifact published to a Maven repository can be installed with the "gem" command, and gems targeting JRuby can depend on Maven artifacts directly. This support requires Maven 3 to be installed, or alternatively you can install the ruby-maven gem.

Here's an example of installing the Quartz library for scheduled jobs. The Quartz library has a Maven group ID of "quartz" and an artifact ID of "quartz":

~/projects/jruby ➔ gem install quartz:quartz
Successfully installed quartz.quartz-1.5.1-java
1 gem installed

You can also specify Maven artifacts in gemspecs, and the appropriate jars will be fetched and installed. The entire world of Java libraries is now at your fingertips!

This is a technology preview, so we may revise it based on user reports and feature requests.

Improved Compatibility and User Experience on Windows

The JRuby team has always valued Windows as a user platform, and with this release we have promoted Windows to being a primary supported platform. We added Windows to our continuous integration servers, and have been maintaining a green test suite there. We encourage all Ruby users on Windows to try JRuby and let us know if there are remaining bugs or usability issues we need to fix.

Look for Tom Enebo's post on JRuby's excellent Windows support tomorrow.

Over 2000 Commits and 265 Issues Resolved

JRuby 1.6 is our largest release to date. We have fixed more bugs, committed more revisions, and accepted contributions from more users than ever before. Expect to see more detailed blog posts from us and others on the above features in the coming weeks.

We thank the JRuby community for their continued support, and Engine Yard for their sponsorship!

Hello friends! With 2010 wrapping up, we figured it might be interesting to publish a "year in review" post that summarizes key events, milestones, and people from this past year. It's been a spectacular year for JRuby, with adoption rapidly increasing, folks accepting JRuby as both a top-notch Ruby implementation and a top-notch JVM language, and JRuby itself improving by leaps and bounds. Let's dive in and see what happened to JRuby in 2010!

The People

JRuby has always been a project centered around bringing Ruby to more and more people. To that end, we've traveled the world, written blog posts, added contributors, accepted patches, and always tried to support users online. Here's a few key folks that made a huge difference to JRubyists in 2010

Hiro Asari

Hiro had been a contributor to JRuby throughout 2009, and had become a regular fixture on the mailing lists and IRC channel, helping users through day-to-day issues running JRuby-based apps. In early 2010, Engine Yard needed to hire a JRuby support expert, and Hiro was the man for the job. He came on full time, we added him as a committer, and late in the year he even started presenting JRuby on the conference circuit.

Hiroshi Nakamura

"Nahi" is a longtime "Ruby core" (the C Ruby/MRI core) team member, specializing in crypto (OpenSSL), IO, and various other system-level features. If you poke around Ruby's standard library, chances are you'll see Nahi pop up several times. In mid-2010, Nahi started helping us with jruby-openssl, JRuby's support for the "openssl" library. He took Ola Bini's epic port of OpenSSL features and brought it much closer to 100% compatibility with C Ruby's version--a stunning feat, considering how low-level the "openssl" library actually is. Since then, he's helped us with some of our trickiest subsystems, and become a great JRuby advocate.

David Calavera

David was also a heavy contributor in 2009, and he continued his contributions in 2010. He has worked hard on improving JRuby's support for Ruby 1.9 features, fixed dozens of user-reported bugs, and helped support users on email and IRC. He also assembled the amazing Trinidad gem, a JRuby-based wrapper around embedded Tomcat that's rapidly become a server of choice for JRubyists everywhere.

Joe O'Brien

Joe is one of the founders of EdgeCase, a Ruby/Rails (and more) consultancy based in Ohio. Joe has been a great advocate for JRuby, having dealt with the "enterprise" and knowing how difficult it can be to get them to adopt new technology. In 2010, Joe helped us host the first standalone JRubyConf, turning over his own eRubyCon to JRuby presenters and users from around the world. We're thrilled to have had such an opportunity to spread the JRuby word...and thrilled with the amazing event Joe set up for us.

Engine Yard

Of course we have to thank Engine Yard for continuing to support us and the JRuby community. Engine Yard has provided a welcoming home, marketing help, travel and technical resources, and moral support to the core JRuby folks who work here: me, Tom Enebo, Nick Sieger, and Hiro Asari. We've been proud to launch JRuby support and professional services offerings, JRuby is in production for xCloud, and you can now trial JRuby on AppCloud through the beta program. It's been great to work at Engine Yard, and even greater to help contribute to the bottom line!

Users and Contributors Like You

Of course JRuby would be nothing without users and contributors just like you. This past year, we've had dozens of new JRuby-based sites reported, Ruby gem authors have started to test and explicitly support JRuby, and David Heinemeier Hansson (creator of Rails) noted that Rails 3.0 was "designed to work with" JRuby, in addition to C Ruby. Twitter shows hundreds of JRuby tweets every day, the JRuby mailing lists fielded nearly 14000 messages, and we met thousands more users and potential users at dozens of conferences around the world. We thank our users, contributors, and Ruby library and application authors for making 2010 the best JRuby year yet!

JRuby Itself

Of course we've also done a tremendous amount of work on JRuby proper. In 2010, we logged over 2300 commits (on master), resolved 567 bug reports (with another 868 filed...work continues!), published eight JRuby releases (1.4.1, 1.5.0-1.5.6), and improved JRuby's "RubySpec" pass rates to over 99% in 1.8 mode and over 90% in 1.9 mode. This work will culminate with a JRuby 1.6 release early in 2011...our biggest and most important release yet. Here's a few of the top areas JRuby has improved in 2010.

Ruby 1.9 Support

With the release of Ruby 1.9.2 this year, we've made supporting all its features in JRuby a top priority. We had some support in place already, thanks to contributors like Marcin Mielzynski, Ola Bini, David Calavera, and Hiro Asari, but had put much of that work on hold until Ruby's core team put out a final 1.9.2 release. Since that release this summer, we've made hundreds of commits and fixed hundreds of issues, helping bring JRuby to better than 90% pass rates on RubySpec and over 95% passing running Rails' ActiveSupport tests in 1.9 mode. JRuby will be officially 1.9-compatible in the coming months.

Windows Support

We believe JRuby is the best path to getting "Ruby everywhere", and to that end we made another big push in 2010 to improve JRuby's support for running on Windows. Of course the core VM is already solid; the various JVMs do an excellent job here. But Ruby's feature set includes a number of system-level features that can be tricky to support uniformly across platforms. Even with those features in place, Rubyists frequently encounter native libraries that don't work or can't be built on Windows...and often have had trouble getting up-to-date builds of Ruby itself. We continued to help the community port key extensions to Java (or Ruby + Java integration) to make it easier for Windows users to get started, and JRuby is now frequently considered the easiest path to Ruby on Windows. Expect to see us continue this work in 2011.

Green Rails 3 Tests

We've "run Rails" longer than any other alternative Ruby implementation (since 2006!), but during all that time we were never 100% green on Rails' own tests. Contributors like Hiro, David, Nick Sieger, and Tom Enebo changed all that in 2010, fixing remaining JRuby issues (and bad Rails tests) so that JRuby now runs green on all Rails 3 tests. Even more impressive is the fact that of all the Ruby implementations, JRuby is the only one that maintains a nightly continuous integration build against Rails. We're dedicated to ensuring JRuby users have an excellent experience running Rails.

C Extension Support

As part of the 2010 Ruby Summer of Code, Tim Felgentreff built out prototype work by Wayne Meissner to add support for Ruby C extensions to JRuby. It was an outstanding effort, well above and beyond expectations (I swore we'd never be able to do it...Tim and Wayne proved me wrong), resulting in many common extensions "just working" on JRuby. We still recommend using Java equivalents or Java ports, since that's the best path to portable, stable, concurrency-friendly extensions...but this new work will at least provide a short-term migration path. We'll include C extension support in a JRuby release as soon as possible.

Performance

Behind the scenes, we've also continued to move JRuby forward as an implementation. Subbu Sastry and Tom have continued work on a new IR (intermediate representation) and optimizing compiler for Ruby code that has a preliminary interpreter now and will hopefully have drastic impacts on JRuby performance during 2011. I worked on improving JRuby's potential for dynamic optimization, first by experimenting with using interpreter profiling to eliminate dynamic calls, and later by eliminating much of the per-call overhead for most Ruby methods. Preliminary tests show it's possible for us to approach Java performance when "dynopts" are enabled, and we've come to realize JRuby's far from being fully-optimized (even while being one of the best already). Look for a lot more of this work in early 2011.

Peripherals

While the core JRuby folks continued to improve JRuby, dozens of other folks improved JRuby-related libraries, platforms, and servers. Here's a few of our favorites.

win32ole

One of the oldest bugs in JRuby's tracker documented our lack of support for the "win32ole" library. win32ole provides a clean Ruby API wrapping OLE components on Windows, allowing you to call into Office or IE or most other key Windows-based apps and libraries without writing any native code. As part of an Engine Yard Professional Services contract for Audentes Technology, we finally managed to cleanup, rework, and tightly wrap the Jacob (Java-COM Bridge) library, providing full win32ole support for JRubyists. Tom Enebo put in a Herculean effort over several weeks, ultimately producing the most solid, concurrency-friendly Ruby win32ole library available.

ActiveRecord JDBC

Along with fixing JRuby issues for Rails 3.0, we needed JRuby's version of Rails' database adapters to be 100% solid as well. Nick Sieger, always the primary force behind "ActiveRecord-JDBC", made good on that goal in 2010, bringing AR-JDBC to 1.0 and 100% green across a half-dozen databases. He continued that work, making it solid on several pure-Java databases like Derby (JavaDB), Hypersonic SQL, and the H2 database. Nick's efforts have helped make JRuby the easiest way to build Rails apps across a wide range of platforms, and we have the users to prove it &mdash running some of the oddest combinations of database and host platform the Ruby world has ever seen.

Deployment Options

Several contributors have also made it even easier to run JRuby server apps this year. Nick Sieger brought his "warbler" and "jruby-rack" libraries to 1.0. Warbler makes it possible to bundle up any Rack-friendly app (including Rails) as a "plain old" WAR (Web ARchive) file, allowing them to be deployed on any Java web container. Nick also added a built-in server called WinStone to Warbler, allowing you to execute a WAR file directly with no target server. He went even farther, making it possible to "warble" any arbitrary Ruby application into an executable JAR file...amazing! No other deployment option has the potential to bring Ruby to as many enterprises as Warbler, and reaching 1.0 was a great milestone. On the other side of the coin, jruby-rack provides a rack-compatible interface to the Java "servlet" API, serving as the backbone of all servlet-hosted Ruby apps. For command-line users, Trindad made its debut. David Calavera wrapped "embedded" Tomcat with an excellent command-line tool, utilizing jruby-rack, allowing JRubyists to have a fast, concurrency-friendly server in a 7MB gem. Hiro Asari helped us "adopt" the GlassFish gem, which serves a similar purpose by wrapping the fast HTTP server "Grizzly" at the core of Oracle's GlassFish server. These gems are the easiest path to production Rails, bar none. And if you're a JBoss fan, Bob McWhirter and his team at RedHat have built perhaps the best JRuby support in any mainstream Java application server: TorqueBox. TorqueBox uses jruby-rack to deploy JRuby apps, but goes far beyond that, providing Ruby APIs and plugins for the EE subsystem, scheduled jobs, web services, and a whole lot more. It's the most complete "Ruby for JavaEE" option available, and we recommend it to all JBoss or EE users looking to explore Ruby (or anyone who wants the excellent services it provides out of the box). Finally, John Woodell and Ryan Brown (two of JRuby's friends inside Google) continued to improve support for JRuby on Google AppEngine for Java. GAE/J offers users a unique environment for small services, and has proven to be an excellent host for JRuby-based apps. John and Ryan continue to support the JRuby-on-GAE community, and JRubyists are happy to have GAE as another deployment option.

Tooling and Management

JRuby has always been able to leverage standard JVM/JDK tools and management interfaces, but many of these are foreign (and perhaps a bit scary) to day-to-day Rubyists. To support these users better, we've done a lot of work in 2010 to make monitoring and managing JRuby apps easier. We've added a built-in instrumenting profiler (--profile) that only considers Ruby code, and "call graph" profiling is nearly ready as well. We worked with the maintainers of the ruby-debug gem to finally release the JRuby-backed as an official gem, meaning ruby-debug now installs cleanly. We've contributed patches to RubyGems to improve startup, and modified our own copy (with lots of help from "mkristian") to support installing and depending on Maven libraries as Ruby gems, improving how JRuby users incorporate all those amazing Java libraries into their apps. And we shipped two-way integration between Rake and Ant this year, making it trivial (and beautiful) to replace Ant build scripts with Rakefiles or seamlessly combine the two worlds.

Pure-Java Nokogiri

During 2009, the Nokogiri XML library became the de-facto standard for anyone doing serious XML processing. It was clean, fast, and wrapped the widely-used libxml library. JRubyists were able to play too, but because of the libxml dependency they had to accept an FFI (foreign function interface) version that did not perform as well as the "raw" C extension. We hosted a Google Summer of Code student, Sergio Arbeo, who made a valiant initial effort at producing a pure-Java Nokogiri by wrapping key Java libraries like Xerces. Unfortunately, the summer ended, and there remained a lot of work. During 2010, our own Yoko Harada, creator of JRuby's embedding, JSR-223, and BSF implementations, took on much of the task of finishing "nokogiri-java". As a result, we now have a nearly-perfect version of Nokogiri that works on all platforms, with or without libxml, and with no native bindings. It's not only the most portable version of Nokogiri...it's also a lot faster than the FFI version.

Community

Where would we be without the users, events, and projects our community members have put together? Nowhere, that's where! We're proud point out a few key community-related wins from 2010.

JRubyConf 2010

Thanks to Joe O'Brien and Engine Yard, we ran the first ever standalone JRubyConf! In early October, over 150 people from around the world descended on Columbus, Ohio, to hear about JRuby, JRuby on Rails, JRuby on Android, JRuby-related projects, and a whole lot more (not all of it JRuby-specific, either.) After the great sucesses of JRubyConf 2009 (the day after RubyConf 2009) and JRubyConf 2010, we're sure to set up a JRubyConf 2011. We've also been kicking around the idea of a JRubyConf EU. Will we see you at a JRubyConf this year?

Blog Posts

JRuby community members have written hundreds of blog posts this year...many more than we can list in this review. We'll try to assemble an aggregated post soon, but I strongly recommend you do a bit of searching and see what you can find. JRuby is being used in some surprisingly novel ways!

Conferences and User Groups

We are in an interesting position with JRuby, needing to cover (ideally) both Ruby and Java-related events. As a result, we've done a lot of traveling to bring JRuby "to the people". These days, there's a JRuby presentation at just about every Ruby and Java conference around the globe, and most Ruby or Java user groups have had at least one JRuby talk. Many of them are run by core JRuby folks, but many more are done by actual JRuby users helping to spread the word. JRuby's community is doing more and more to help us cover events around the globe, and we hope in 2011 to see that trend continue. 2010 also saw the birth of the first JRuby-specific user group, JRuby Users JP. They've hosted a half-dozen meetings already, and we got to meet the organizer and several members at RubyKaigi 2010. Have you done a presentation on JRuby yet? Why not?

RubyConf 2010 "JRuby Track"

My first JRuby presentation (and first public speaking gig) was at RubyConf 2005. At the time, JRuby couldn't even run IRB, and other presenters actually made jokes about JRuby. Fast-forward to 2010, where JRuby had a half-dozen dedicated sessions at RubyConf and played a part in many more. We're grateful to the RubyConf organizers and humbled by the Ruby community's acceptance of JRuby. It really makes us feel welcome to know our efforts are helping people be better Rubyists.

LinkedIn's "Signal"

LinkedIn has always been a hotbed of polyglot development, and were one of the earlier users of JRuby for user-facing applications (LinkedIn Polls ran--and may still run--atop JRuby). This year, they launched the ambitious Signal, a social-site aggregator for your LinkedIn profile. It uses JRuby for the front-end with other JVM languages like Scala providing the backend services. The project went so well, they've been green-lighted to do several more in the same mold.

RedCar

If you're a Mac user, chances are your GUI text editor of choice is TextMate. It's so good, in fact, that Dan Lucraft made his own version called RedCar, and wrote it in Ruby so he could easily hack on it and add or tweak the features he never got from TextMate. Late in 2009, Dan made the decision to move RedCar to JRuby from a C Ruby + GTK backend, to make cross-platform support easier. And during 2010, RedCar has added many incremental releases (up to the current 0.9.2) and tons of new features, users, and contributors. I even switched from TextMate to RedCar as my GUI text editor of choice. We're very proud to show off RedCar during JRuby presentations, and we're thrilled to see a packaged desktop app based on JRuby becoming popular.

Ruboto

2010 also saw an explosion in another area: the popularity of the Android platform. Android is a very rich Java-like system, and where Java goes (J)Ruby can follow! Building off my experiments in 2009 to bring JRuby to Android, community members like Jan Berkel and Scott Moyer made massive improvements to my "Ruboto IRB" app, finally releasing Ruboto to the Android marketplace in 2010. Ruby Summer of Code student Daniel Jackoway started the "Ruboto Core" project, an app generator and set of utility APIs for building standalone JRuby apps for Android. And users like Jay McGavren have been promoting and experimenting with Ruboto, showing the world how beautiful and powerful mobile Ruby really can be.

"Using JRuby" Completed

After two years of work by five authors, four of them from JRuby core (me, Nick, Tom, and Ola) and the fifth our outstanding lead author Ian Dees, we finally completed "Using JRuby", the definitive guide for JRuby users. As of this writing, the book has gone to typesetting and should ship in January 2011, thanks to the patience and encouragement of the folks at Pragmatic Programmers. Along with having a dedicated conference, a published book has always indicated to me that a project has really arrived. Finally, JRuby has its book!

The Bossie

JRuby also arrived in a different way during 2010 by winning an InfoWorld "Bossie" award for "Best Open Source Application Development Software." Now we can say "JRuby, the award-winning Ruby implementation"!

Your Turn

Now it's time for you to join the JRuby community! Get out to your local Java, Ruby, OO, or other tech group and do a talk or host a hackfest. Present at a nearby conference. Try out JRuby for your own applications and let us know what we can improve. Write a blog post or record a screencast. Tell your friends and neighbors how awesome JRuby is. Scan recent JRuby bug reports, failing RubySpecs, or mailing list items to see if you can contribute. Jump on the JRuby wiki and contribute improvements, new articles, or translations of existing ones. Help us keep bringing Ruby to the world, and we'll keep doing everything we can to make JRuby great! And finally...why not tell us your JRuby 2010 story in the comments or in a blog post? We'd love to hear from you!

The Eclipse Memory Analyzer provides a very robust set of tools for analyzing JVM heap dumps. Along with being able to simply browse the heap, like jhat, or run Object Query Language queries against it, like VisualVM, it also provides features to search for GC roots, identify probable leaks, and search for common bad practices. It also seems (in my experience) to use less memory than jhat or VisualVM, so for really large heap dumps it's often the best option.

In this post, I'll show how to use MAT to look for memory leaks (or wasteful memory use).

MAT, like many Eclipse-based applications, starts up with a "for dummies" page linking to various actions.

The most interesting use of MAT is to analyze a heap dump in a bit more interactive way than with the "jhat" tool. The analysis supports the "jmap" dump format, so we'll proceed to get a jmap dump of a "leaky" Rails application.

I've added this controller to a simple application:

class LeakyController < ApplicationController
  class MyData
    def initialize(params)
      @params = params
    end
  end

  LEAKING_ARRAY = {}
  def index
    LEAKING_ARRAY[Time.now] = MyData.new(params)
    render :text => "There are #{LEAKING_ARRAY.size} elements now!"
  end
end

Some genius has decided to save all recent request parameters into a constant on the LeakyController, keyed by time, wrapped in a custom type, and never cleaned out. Perhaps this was done temporarily for debugging, or perhaps we have a moron on staff. Either way, we need to find this problem and fix it.

We'll run this application and crank 10000 requests through the /leaky index, so the final request should output "There are 10000 elements now!"

~ ➔ ab -n 10000 http://localhost:3000/leaky
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 1000 requests
Completed 2000 requests
...

After 10000 requests have completed, we notice this application seems to grow and grow until it maxes out the heap (JRuby, being on the JVM, automatically limits heap sizes for you). Let's start by using jmap to investigate the problem.

~ ➔ jps -l
61976 org/jruby/Main
61999 sun.tools.jps.Jps
61837

~ ➔ jmap -histo 61976 | grep " ruby\." | head -5
 37:         11685         280440  ruby.TZInfo.TimezoneTransitionInfo
 40:         10000         240000  ruby.LeakyController.MyData
133:           970          23280  ruby.Gem.Version
137:           914          21936  ruby.Gem.Requirement
170:           592          14208  ruby.TZInfo.TimezoneOffsetInfo

We can see our old friend TimezoneTransitionInfo in there, but of course we've learned to accept that one. But what's this LeakyController::MyData object we've apparently got 10000 instances of? Where are they coming from? Who's holding on to them?

At this point, we can proceed to get a memory dump and move over to MAT, or have MAT acquire and open the dump in one shot, similar to VisualVM. Let's have MAT do it for us.

Getting Our Heap Into MAT

(Caveat: While preparing this post, I discovered that the jmap tool for the current OS X Java 6 (build 1.6.0_20-b02-279-10M3065) is not properly dumping all information. As a result, many fields and objects don't show up in dump analysis tools like MAT. Fortunately, there's a way out; on OS X, you can grab Soylatte or OpenJDK builds from various sources that work properly. In my case, I'm using a local build of OpenJDK 7.)

From the File menu, we select Acquire Heap Dump.

The resulting dialog should be familiar, since it lists the same JVM processes the "jps" command listed above. (If you had to specify a specific JDK home, like me, you'll need to click the "Configure" button and set the "jdkhome" flag" for "HPROF jmap dump provider".)

We'll pick our Rails instance (pid 61976) and proceed.

MAT connects to the process, pulls a heap dump to disk, and immediately proceeds to parse and open it.

Once it has completed parsing, we're presented with a few different paths to follow.

On other days, we might be interested in doing some component-by-component browsing to look for fat objects or minor leaks, or we might want to revisit the results of previous analyses against this heap. But today, we really need to figure out this MyData leak, so we'll run the Leak Suspects Report.

Leak Suspects?

Are you kidding? A tool that can search out and report possible leaks in a running system? Yes, Virginia, there is a Santa Claus!

This is the good side of the "plague of choices" we have on the JVM. Because there's so many tools for almost every basic purpose (like the dozen - at least - memory inspection tools), tool developers have moved on to more specific needs like leak analysis. MAT is my favorite tool for leak-hunting (and it uses less memory than jhat for heap-browsing, which is great for larger dumps).

Once MAT has finished chewing on our heap, it presents a pie chart of possible leak suspects. The logic used essentially seeks out data structures whose accumulated size is large in comparison to the rest of the heap. In this case, MAT has identified three suspects that in total comprise over half of the live heap data.

Scrolling down we start to get details about these leak candidates.

So there's a Hash, a Module, and 711 Class objects in our list of suspects. The Class objects are probably just loaded classes, since the JRuby core classes and additional classes loaded from Rails and its dependent libraries will easily number in the hundreds. We'll ignore that one for now. There's also an unusually large Module taking up almost 4MB of memory. We'll come back to that.

The Hash seems like the most likely candidate. Let's expand that.

The first new block of information gives us a list of "shortest paths" to the "accumulation point", or the point at which all this potentially-leaking data is gathering. There's more to this in the actual application, but I'm showing the top of the "path" here.

At the top of this list, we see the RubyHash object originally reported as a suspect, and a tree of objects that lead to it. In this case, we go from the Hash itself into a ConcurrentHashMap (note that we're hiding nothing here; you can literally browse anything in memory) which in turn is referenced by the "constants" field of a Class. So already we know that this hash is being referenced in some class's constant table. Pretty cool, eh?

Let's make sure we've got the right Hash and not some harmless data structure inside Rails. If we scroll down a bit more, we see a listing of all the objects this Hash has accumulated. Let's see what's in there.

Ok, so it's a hashtable structure with a table of entries. Can we get more out of this?

Of course like most of these tools, just about everything is clickable. We can dive into one of the hash entries and see what's in there. Clicking on an entry gives us several new ways to display the downstream objects we've managed to aggregate. In this case, we'll just do "List Objects", and the suboption "With Outgoing References" for downstream data.

Now finally in the resulting view of this particular RubyHashEntry, we can see that our MyData object is happily tucked away inside.

Ok, so we definitely have the right data structure. Not only that, but we can see that the entry's "key" is a Time object (org.jruby.RubyTime). Let's go back to the "Shortest Paths" view and examine the ConcurrentHashMap entry that's holding this Hash object. Each entry in this hash maps a constant name to a value, so we should be able to see which constant is holding the leaking references.

(At this point you'll see the side effects of my switch to OpenJDK 7; the memory addresses have changed, but the structure is the same.)

We'll do another "List Objects" "with outgoing references" against the the HashEntry object immediately referencing our RubyHash.

And there it is! In the "key" field of the HashEntry, we see our constant name "LEAKING_ARRAY".

What About That Module?

Oh yeah, what about that Module that showed up in the leak suspects? It was responsible for almost 4MB of the heap. Let's go back and check it out.

A-ha! Eclipse MAT has flagged the Gem module as being a potential leak suspect. But why? Let's go back to the suspect report and look at the Accumulated Objects by Class table, toward the bottom.

Ok, so the Gem module eventually references nearly 249 Gem::Specification objects, which makes up the bulk of our 3.8MB. Perhaps we've just used JRuby and Eclipse MAT to discover either a leak or wasteful memory use in RubyGems?

In this case, the retained size seems a bit suspect. Could there really be 3.8MB of data kept alive by Gem::Specification objects? It seems like a bit much, to be sure, but digging through the tree of references from the Gem module down shows there's several references to classes and modules, which in turn reference constant tables, method tables, and so on. How can we filter out that extra noise?

First we'll return to the view of the Gem module (two screenshots up) by going back to leak suspect #2, expanding "Shortest Paths". The topmost RubyModule in that list is the Gem module, so we're all set to calculate a Customized Retained Set.

The resulting dialog provides a list of options through which you can specify classes or fields to ignore when calculating the retained set from a given starting point. In this case, it's simple enough to filter out org.jruby.RubyClass and org.jruby.RubyModule, so that references from Gem::Specification back into the class/module hierarchy don't get included in calculations.

Which results in a similar view to those we've seen, but with objects sorted by retained heap.

Well what the heck? It looks like it's all String data?

JRuby's String implementation is an org.jruby.RubyString object, aggregating an org.jruby.util.ByteList object, aggregating a byte array, so the top three entries there in total are essentially all String memory. The best way to investigate where they're coming from is to do "List Objects" on RubyString, but instead of "with outgoing references" we'll use "with incoming references" to show where all those Strings are coming from.

Finally we have a view that lets us hunt through all these strings and see where they're coming from. Poking at the first few shows they're stored in constant tables of the Gem module (that last RubyModule I haven't expanded in). That's probably not a big deal. But if we sort the the list of RubyString objects by their retained sizes, we get a different picture of the system.

If we dig into the *largest* String objects, they appear to be referenced by Gem::Specification instance variables! So there's probably something worth investigating here.

It's also worth noting that any Ruby application is going to have a lot of Strings in it, so this isn't all that unusual to see. But it's nice to have a tool that lets you investigate potential inefficiencies (even down to the raw bytes!), and it's nice to know that at least some of that retained data for the Gem module is "real" and not just references back into the class hierarchy.

(And I'm not convinced all those Strings really *need* to be alive...but you're welcome to take it from here!)

Your Turn

Eclipse MAT is probably one of the nicest of the free JVM memory analysis tools. In addition to object browsing, leak detection, GC root analysis, and object query language support, there's a ton of other features, both in the main distribution and available from third parties. If you're hunting for memory leaks, or just want to investigate the memory usage of your (J)Ruby application, MAT is a tool worth playing with (and as always, I hope you will blog and report your experiences!)

Because JRuby runs on the JVM, we benefit from the dozens of tools that have been written for the JVM. Among these tools are numerous memory inspection, profiling, and reporting tools, some built into the JDK itself. Want a heap dump? Check out the jmap (Java memory map) and jhat (Java heap analysis tool) shipped with Hotspot-based JVMs (Sun, OpenJDK). Looking for a bit more? There's the Memory Analysis Tool based on Eclipse, the YourKit memory and CPU profiling app, VisualVM, now also shipped with Hotspot JVMs...and many more. There's literally dozens of these tools, and they provide just about everything you can imagine for investigating memory.

In this post, I'll show how you can use two of these tools: VisualVM, a simple, graphical tool for exploring a running JVM; and the jmap/jhat combination, which allows you to dump the memory heap to disk for inspection offline.

Getting JRuby Prepared

All these tools work with any version of JRuby, but as part of JRuby 1.6 development I've been adding some enhancements. Specifically, I've made some modifications that allow Ruby objects to show up side-by-side with Java objects in memory profiles. A little explanation is in order.

In JRuby, all the core classes are represented by "native" Java classes. Object is represented by org.jruby.RubyObject, String is org.jruby.RubyString, and so on. Normally, if you extend one of the core classes, we don't actually create a new "native" class to represent it; instead, all user-created classes that extend Object simply show up as RubyObject in memory. This is still incredibly useful; you can look into RubyObject and see the metaClass field, which indicates the actual Ruby type.

Let's see what that looks like, so we know where we're starting from. We'll run a simple script that creates a custom class, instantiates and saves 10000 instances of it, and then sleeps.

~/projects/jruby ➔ cat foo_heap_example.rb
class Foo
end

ary = []
10000.times { ary << Foo.new }

puts "ready for analysis!"
sleep

~/projects/jruby ➔ jruby foo_heap_example.rb
ready for analysis!

So we have our test subject ready to go. To use the jmap tool, we need the pid of this process. Of course we can use the usual shell tricks to get it, but the JDK comes with a nice tool for finding all JVM pids active on the system: jps

~/projects/jruby ➔ jps -l
52862 sun.tools.jps.Jps
52857 org/jruby/Main
48716 com.sun.enterprise.glassfish.bootstrap.ASMain

From this, you can see I have three JVMs running on my system right now: jps itself; our JRuby instance; and a GlassFish server I used for testing earlier today. We're interested in the JRuby instance, pid 52857. Let's see what jmap can do with that.

~/projects/jruby ➔ jmap
Usage:
    jmap [option] <pid>
        (to connect to running process)
    jmap [option] <executable <core>
        (to connect to a core file)
    jmap [option] [server_id@]<remote server IP or hostname>
        (to connect to remote debug server)

where <option> is one of:
    <none>               to print same info as Solaris pmap
    -heap                to print java heap summary
    -histo[:live]        to print histogram of java object heap; if the "live"
                         suboption is specified, only count live objects
    -permstat            to print permanent generation statistics
    -finalizerinfo       to print information on objects awaiting finalization
    -dump:<dump-options> to dump java heap in hprof binary format
                         dump-options:
                           live         dump only live objects; if not specified,
                                        all objects in the heap are dumped.
                           format=b     binary format
                           file=<file>  dump heap to <file>
                         Example: jmap -dump:live,format=b,file=heap.bin <pid>
    -F                   force. Use with -dump:<dump-options> <pid> or -histo
                         to force a heap dump or histogram when <pid> does not
                         respond. The "live" suboption is not supported
                         in this mode.
    -h | -help           to print this help message
    -J<flag>             to pass <flag> directly to the runtime system

<

The simplest option here is -histo, to print out a histogram of the objects on the heap. Let's run that against our JRuby instance.

~/projects/jruby ➔ jmap -histo:live 52857

 num     #instances         #bytes  class name
----------------------------------------------
   1:         22677        3192816  <constMethodKlass>
   2:         22677        1816952  <methodKlass>
   3:         35089        1492992  <symbolKlass>
   4:          2860        1389352  <instanceKlassKlass>
   5:          2860        1193536  <constantPoolKlass>
   6:          2798         739264  <constantPoolCacheKlass>
   7:          5861         465408  [B
   8:          5399         298120  [C
   9:          3042         292032  java.lang.Class
  10:          4037         261712  [S
  11:         10002         240048  org.jruby.RubyObject
  12:          3994         179928  [[I
  13:          5474         131376  java.lang.String
  14:          1661          95912  [I
...

The resulting output is a listing of literally every object in the system...not just Ruby objects even! The value of this should be apparent; not only can you start to investigate the memory overhead of code you've written, you'll also be able to investigate the memory overhead of every library and every piece of code running in the same process, right down to byte arrays (the "[B" above) and "native" Java strings ("java.lang.String" above). And so far we haven't had to do anything special to JRuby. Nice, eh?

So, back to the matter at hand: the Foo class from our example. Where is it?

Well, the answer is that it's right there; 10000 of those 10002 org.jruby.RubyObject instances are our Foo objects; the other two are probably objects constructed for JRuby runtime purposes. But obviously, there's nothing in this output that tells us how to find our Foo instances. This is what I'm remedying in JRuby 1.6.

On JRuby master, there's now a flag you can pass that will stand up a JVM class for every user-created Ruby class. Among the many benefits of doing this, we also get a more useful profile. Let's see how to use the flag (which will either be default or very easy to access by the time we release JRuby 1.6).

~/projects/jruby ➔ jruby -J-Djruby.reify.classes=true foo_heap_example.rb
ready for analysis!

If we run jmap against this new instance, we see a more interesting result.

 num     #instances         #bytes  class name
----------------------------------------------
   1:         22677        3192816  <constMethodKlass>
   2:         22677        1816952  <methodKlass>
   3:         35089        1492992  <symbolKlass>
   4:          2860        1389352  <instanceKlassKlass>
   5:          2860        1193536  <constantPoolKlass>
   6:          2798         739264  <constantPoolCacheKlass>
   7:          5863         465456  [B
   8:          5401         298208  [C
   9:          3042         292032  java.lang.Class
  10:          4037         261712  [S
  11:         10000         240000  ruby.Foo
  12:          3994         179928  [[I
  13:          5476         131424  java.lang.String
  14:          1661          95912  [I

A-ha! There's our Foo instances! The "reify classes" option generates a JVM class of the same name as the Ruby class, prefixed by "ruby." to separate it from other JVM classes. Now we can start to see the real power of the tools, and we're just at the beginning. Let's see what a simple Rails application looks like.

~/projects/jruby ➔ jmap -histo:live 52926 | grep " ruby."
  29:         11685         280440  ruby.TZInfo.TimezoneTransitionInfo
  97:           970          23280  ruby.Gem.Version
  98:           914          21936  ruby.Gem.Requirement
 122:           592          14208  ruby.TZInfo.TimezoneOffsetInfo
 138:           382           9168  ruby.Gem.Dependency
 159:           265           6360  ruby.Gem.Specification
 201:           142           3408  ruby.ActiveSupport.TimeZone
 205:           118           2832  ruby.TZInfo.DataTimezoneInfo
 206:           118           2832  ruby.TZInfo.DataTimezone
 273:            41            984  ruby.Gem.Platform
 383:            14            336  ruby.Mime.Type
 403:            13            312  ruby.Set
 467:             8            192  ruby.ActionController.MiddlewareStack.Middleware
 476:             8            192  ruby.ActionView.Template
 487:             7            168  ruby.ActionController.Routing.DividerSegment
 508:             6            144  ruby.TZInfo.LinkedTimezoneInfo
 523:             6            144  ruby.TZInfo.LinkedTimezone
 810:             4             96  ruby.ActionController.Routing.DynamicSegment
2291:             2             48  ruby.ActionController.Routing.Route
2292:             2             48  ruby.I18n.Config
2293:             2             48  ruby.ActiveSupport.Deprecation.DeprecatedConstantProxy
2298:             2             48  ruby.ActionController.Routing.ControllerSegment
...

This time I've opted to grep out just the "ruby." items in the histogram, and the results are pretty impressive! We can see the baffling fact that there's 970 instance of Gem::Version, using at least 23280 bytes of memory. We can see the even more depressing fact that there's 11685 live instances of TZInfo::TimezoneTransitionInfo, using at least 280440 bytes.

Now that we're getting useful data, let's look at the first of our tools in more detail: jmap and jhat.

jmap and jhat

As you might guess, I do a lot of profiling in the process of developing JRuby. I've used probably a dozen different tools at different times. But the first tool I always reach for is the jmap/jhat combination.

You've seen the simple case of using jmap above, generating a histogram of the live heap. Let's take a look at an offline heap dump.

~/projects/jruby ➔ jmap -dump:live,format=b,file=heap.bin 52926
Dumping heap to /Users/headius/projects/jruby/heap.bin ...
Heap dump file created

That's how easy it is! The binary dump in heap.bin is supported by several tools: jhat (obviously), VisualVM, the Eclipse Memory Analysis Tool, and others. It's not officially a "standard" format, but it hasn't changed in a long time. Let's have a look at jhat options.

~/projects/jruby ➔ jhat
ERROR: No arguments supplied
Usage:  jhat [-stack <bool>] [-refs <bool>] [-port <port>] [-baseline <file>] [-debug <int>] [-version] [-h|-help] <file>

 -J<flag>          Pass <flag> directly to the runtime system. For
     example, -J-mx512m to use a maximum heap size of 512MB
 -stack false:     Turn off tracking object allocation call stack.
 -refs false:      Turn off tracking of references to objects
 -port <port>:     Set the port for the HTTP server.  Defaults to 7000
 -exclude <file>:  Specify a file that lists data members that should
     be excluded from the reachableFrom query.
 -baseline <file>: Specify a baseline object dump.  Objects in
     both heap dumps with the same ID and same class will
     be marked as not being "new".
 -debug <int>:     Set debug level.
       0:  No debug output
       1:  Debug hprof file parsing
       2:  Debug hprof file parsing, no server
 -version          Report version number
 -h|-help          Print this help and exit
 <file>            The file to read

For a dump file that contains multiple heap dumps,
you may specify which dump in the file
by appending "#<number>" to the file name, i.e. "foo.hprof#3".

All boolean options default to "true"

Generally you can just point jhat at a heap dump and away it goes. Occasionally if the heap is large, you may need to use the -J option to increase the maximum heap size of the JVM jhat runs in. Since we're running a Rails app, we'll bump the heap up a little bit.

~/projects/jruby ➔ jhat -J-Xmx200M heap.bin
Reading from heap.bin...
Dump file created Fri Jul 09 02:07:46 CDT 2010
Snapshot read, resolving...
Resolving 604115 objects...
[much verbose logging elided for brevity]

Chasing references, expect 120 dots........................................................................................................................
Eliminating duplicate references........................................................................................................................
Snapshot resolved.
Started HTTP server on port 7000
Server is ready.

"Server is ready"? Damn you Java people! Does everything have to be a server with you?

In this case, it's actually an incredibly useful tool. jhat starts up a small web application on port 7000 that allows you to click through the dump file. Let's see what that looks like.

Here's the front page of the tool. We see a listing of all JVM classes in the system. If you scroll to the bottom, there's a few more general functions.

Let's go with what we know and view the heap histogram again.

Here we can see that there's lots of objects taking up memory, and they're a mix of JVM-native types, JRuby implementation classes, and actual Ruby classes. In fact, here we can see our friend TZInfo::TimezoneTransitionInfo again. Let's click through.

Pretty mundane stuff so far; basically just information about the class itself. But you see at the bottom of this screenshot that we can go from here to viewing all instances of TimezoneTransitionInfo. Let's try that.

Ahh, that's more like it! Now we can see that there's a heck of a lot of these things floating around. Let's investigate a bit more and click through the first instance.

Now this is some cool stuff!

We can see that the JVM class generated for TimezoneTransitionInfo has three fields: metaClass, which points at the Ruby Class object; varTable, which is an array of Object references used for instance variables and other "internal" variables; and a flags field containing runtime flags for the object, like whether it's frozen, tainted, and so on. We can see that this object has no special flags set, and we can dig deeper into those fields if we like. We'll skip that today.

Moving further down, we see a few more amazing links. First, there's a list of all references to this object. Ahh, now we can start to investigate why they're staying in memory, even though we're not using them. We can even have jhat show us the full chains of references keeping these objects alive; a series of objects leading all the way back to one "rooted" by a thread or by global JVM state. And we can explore the other direction as well, walking all objects reachable from this one.

This is only a small part of what you can do with jmap and jhat, and they're so simple to use it feels almost criminal. But what if we want to inspect an application while it's running? Dumping heaps and analyzing them offline can tell you much of the story, but sometimes you just want to see the objects coming and going yourself. Let's move on to VisualVM.

VisualVM

VisualVM spawned out of the NetBeans profiling tools. One of the biggest complaints about the JVMs of old were that all the built-in tooling seemed to be designed for JVM engineers alone. Because Sun had the foresight to build and own their own IDE and related modules, it eventually became a natural fit to pull out the profiling tools for use by everyone. And so VisualVM was born.

On most systems with Java 6 installed, you should have a "jvisualvm" command. Let's run it now.

When you start up VisualVM, you're presented with a list of running JVMs, similar to using the 'jps' command. You can also connect to remote machines, browse offline heap and core dump files, and look through memory and CPU profiling snapshots from previous runs. Today, we'll just open up our running Rails app and see what we can see.

VisualVM connects to the running process and brings up a basic information pane with process information, JVM information, and so on. We're interested in monitoring heap usage, so let's move to the "Monitor" tab.

Already we're getting some useful information. This view shows CPU usage (currently zero, since it's an idle Rails app), Heap usage over time, and the number of JVM classes and threads that are active. We can trigger a full GC, if we'd like to tidy things up before we start poking around. But most importantly, we can do the jmap/jhat dance in one step, by clicking the Heap Dump button. Tantalizing, isn't it?

Initially, we see a basic summary of the heap: total size, number of classes and GC roots, and so on. We're looking for our friend TimezoneTransitionInfo, so let's look for it in the "Classes" pane.

Ahh, there it is, just a little ways down the list. The counts are as we expect, so let's double-click and dig a bit deeper.

Here we have a lot of the same information about object instances that we did with jhat, but presented in a much richer format. Almost everything is active; you can jump around the heap and do analysis that would take a lot of manual work very easily. Let's try another tool: the Retained Size calculator.

Because our JVM tools see all objects equally, the reported size for a Ruby object on the heap is only part of the story. There's also the variable table, the object's instance variables, and objects they reference to consider. Let's jump to a different object now, Gem::Version.

We don't want to have to scroll through the list of classes to find ruby.Gem.Version, so let's make use of the Object Query Language console. With the OQL console, you can write SQL-like queries to retrieve listings of objects in the heap. We'll search for all instances of ruby.Gem.Version.

The query runs and we get a listing of Gem::Version objects. Let's dig deeper and see how much retained memory each Version object is keeping alive.

Clicking on the "Compute Retained Sizes" link in the "Instances" pane prompts us with this dialog. We're tough...we can take it.

Reticulating splines...

So it looks like each of the Version objects take from 125 to 190 bytes for a total of 19400 bytes, most of which is from the variable table. What's in there?

Ahh...looks like there's a String and an Array. And of course we can poke around the heap ad infinatum, into and out of "native" JRuby and JVM classes, and truly get a complete picture of what our running applications look like. Now you're playing with power.

Your Turn

This is obviously only the tip of the iceberg. Tools like Eclipse Memory Analysis Tool include features for detecting leaks; VisualVM and NetBeans both allow you to turn on allocation tracing, to show where in your code all those objects are being created. There's tools for monitoring live GC behavior, and many of these tools even allow you to dig into a running heap and modify live objects. If you can dream it, there's a tool that can do it. And you get all that for free by using JRuby.

If you'd like to play with this, it all works with JRuby 1.5.1 but you won't get the nice JVM classes for Ruby classes. For that, you can pull and build JRuby master, download a 1.6.0.dev snapshot, or just wait for JRuby 1.6. And if you do play with these or other tools, I hope you'll let us know and blog about your experience!

In the future, I'll try to show some of the other tools plus some of the CPU profiling capabilities they bring to the table. For now, rest assured that if you're using JRuby, you really do have the best tools available to you.

In this article, we'll explore some of the memory monitoring beans and how you can use them to analyze the memory characteristics of your application. I'll be using the default Java 6 version for OS X 10.6.3, which is based on Hotspot, the JVM inside OpenJDK.

We start by requiring 'java' and importing the ManagementFactory from the JDK.

require 'java'

import java.lang.management.ManagementFactory

Now we have access to the four primary beans associated with memory management on the JVM: GarbageCollector, MemoryManager, Memory, and MemoryPool MXBeans. We'll grab them all first.

gc_beans = ManagementFactory.garbage_collector_mxbeans
memman_beans = ManagementFactory.memory_manager_mxbeans
mem_bean = ManagementFactory.memory_mxbean
mempool_beans = ManagementFactory.memory_pool_mxbeans

Let's start with the simplest of these beans, the Memory bean.

The Memory bean provides the most basic information: current heap usage, current non-heap usage, and the count of objects waiting to be finalized. There's also methods to force a GC run and to enable verbose GC logging. Let's check current stats, run some code with verbose logging on, and force a GC run:

puts "Current heap usage: #{mem_bean.heap_memory_usage}"
puts "Current non-heap usage: #{mem_bean.non_heap_memory_usage}"
puts "Waiting to finalize: #{mem_bean.object_pending_finalization_count}"

puts "Enabling verbose GC logging..."
mem_bean.verbose = true
1_000_000.times { "foo" + "bar" }
puts "Forcing a full GC run..."
mem_bean.gc
mem_bean.verbose = false
puts "Disabling verbose GC logging"
puts

Here's the output so far:

Current heap usage: init = 268435456(262144K) used = 1960416(1914K) committed = 270073856(263744K) max = 520093696(507904K)
Current non-heap usage: init = 12750848(12452K) used = 13608880(13289K) committed = 21921792(21408K) max = 100663296(98304K)
Waiting to finalize: 0
Enabling verbose GC logging...
[GC 17029K->1642K(263744K), 0.0014821 secs]
[GC 17066K->1653K(263744K), 0.0122954 secs]
[GC 17077K->1654K(263744K), 0.0012559 secs]
[GC 17078K->1654K(263744K), 0.0006338 secs]
[GC 17078K->1658K(263744K), 0.0050983 secs]
[GC 17082K->1646K(263744K), 0.0003545 secs]
[GC 17070K->1646K(263744K), 0.0005288 secs]
[GC 17070K->1644K(263744K), 0.0006449 secs]
[GC 17068K->1644K(263744K), 0.0090768 secs]
Forcing a full GC run...
[Full GC 11674K->1621K(263744K), 0.0617646 secs]
Disabling verbose GC logging

The GarbageCollector beans provide information about all the garbage collectors in use on a given JVM. They allow you to query collection counts and total collection time.

Below, we'll do some work that should trigger at least a couple GC runs, and then print out stats.

1_000_000.times { "foo" + "bar" }

total_collections = 0
total_time = 0.0
for gc_bean in gc_beans
  puts "Name: #{gc_bean.name}"
  puts "  Pools: #{gc_bean.memory_pool_names.to_a.join(', ')}"
  puts "  Collections: #{gc_bean.collection_count}"
  puts "  Time: #{gc_bean.collection_time}"
  total_collections += gc_bean.collection_count
  total_time += gc_bean.collection_time
end
puts "Total collections: #{total_collections}"
puts "Total time: #{total_time}"
puts

And here's the output for my JVM:

Name: ParNew
  Pools: Par Eden Space, Par Survivor Space
  Collections: 18
  Time: 38
Name: ConcurrentMarkSweep
  Pools: Par Eden Space, Par Survivor Space, CMS Old Gen, CMS Perm Gen
  Collections: 2
  Time: 149
Total collections: 20
Total time: 187.0

You may notice that each garbage collector has several memory pools associated with it. Let's look at the memory pools now.

Memory pools represent areas of memory used to isolate objects with differing characteristics. For example, some JVMs will list a "Par Eden Space", which represents the memory pool containing the newest and youngest objects. Some JVMs list a "Perm Gen" which will contain objects only collected during full GC runs.

Let's iterate over all memory pools and get information on each:

for pool_bean in mempool_beans
  puts "Name: #{pool_bean.name}"
  puts "  Type: #{pool_bean.type}"
  puts "  Peak usage: #{pool_bean.peak_usage}"
  puts "  Current usage: #{pool_bean.usage}"
  puts "  Usage after last collection: #{pool_bean.collection_usage}"
  puts "  Managers: #{pool_bean.memory_manager_names.to_a.join(', ')}"
end
puts

Here's the listing of memory pools for me:

Name: Code Cache
  Type: Non-heap memory
  Peak usage: init = 167936(164K) used = 1004992(981K) committed = 1019904(996K) max = 33554432(32768K)
  Current usage: init = 167936(164K) used = 1004992(981K) committed = 1019904(996K) max = 33554432(32768K)
  Usage after last collection:
  Managers: CodeCacheManager
Name: Par Eden Space
  Type: Heap memory
  Peak usage: init = 12582912(12288K) used = 15794176(15424K) committed = 15794176(15424K) max = 25165824(24576K)
  Current usage: init = 12582912(12288K) used = 10244904(10004K) committed = 15794176(15424K) max = 25165824(24576K)
  Usage after last collection: init = 12582912(12288K) used = 0(0K) committed = 15794176(15424K) max = 25165824(24576K)
  Managers: ConcurrentMarkSweep, ParNew
Name: Par Survivor Space
  Type: Heap memory
  Peak usage: init = 2097152(2048K) used = 49824(48K) committed = 2621440(2560K) max = 4194304(4096K)
  Current usage: init = 2097152(2048K) used = 2048(2K) committed = 2621440(2560K) max = 4194304(4096K)
  Usage after last collection: init = 2097152(2048K) used = 2048(2K) committed = 2621440(2560K) max = 4194304(4096K)
  Managers: ConcurrentMarkSweep, ParNew
Name: CMS Old Gen
  Type: Heap memory
  Peak usage: init = 251658240(245760K) used = 1682040(1642K) committed = 251658240(245760K) max = 490733568(479232K)
  Current usage: init = 251658240(245760K) used = 1662536(1623K) committed = 251658240(245760K) max = 490733568(479232K)
  Usage after last collection: init = 251658240(245760K) used = 1659912(1621K) committed = 251658240(245760K) max = 490733568(479232K)
  Managers: ConcurrentMarkSweep
Name: CMS Perm Gen
  Type: Non-heap memory
  Peak usage: init = 12582912(12288K) used = 12861856(12560K) committed = 21250048(20752K) max = 67108864(65536K)
  Current usage: init = 12582912(12288K) used = 12861856(12560K) committed = 21250048(20752K) max = 67108864(65536K)
  Usage after last collection: init = 12582912(12288K) used = 12749352(12450K) committed = 20967424(20476K) max = 67108864(65536K)
  Managers: ConcurrentMarkSweep

Note that the "usage" values are actually instances of MemoryUsage, which provides four different values for the pool:

• init: the amount of memory the JVM initially reserved from the OS
• used: the amount of memory currently used
• committed: the amount of memory actually available
• max: the maximum memory that will ever be available

These will change over time as objects are collected and the heap grows to meet application needs. Let's show the current usage for the Par Eden Space, as an example:

par_eden = mempool_beans.find {|b| b.name == "Par Eden Space"}
usage = par_eden.usage
puts "Initially requested: #{usage.init}"
puts "Currently used: #{usage.used}"
puts "Total available: #{usage.committed}"
puts "Max potential: #{usage.max}"
puts

The output for "Par Eden Space":

Initially requested: 12582912
Currently used: 10575384
Total available: 15794176
Max potential: 25165824

Memory pools also allow you to set usage thresholds. When a threshold is crossed, the JVM will increment a counter. This allows you to monitor whether or not your application is regularly exceeding a particular threshold, so you can adjust memory settings or work to reduce your application's demands.

Here's what that might look like for the "Par Survivor Space," the "old" pool. We'll monitor the collection usage threshold, or the amount of space in use after the most recent GC run. Some JVMs will support this or the general usage threshold, but neither are required to be supported.

We'll set a limit that's 5MB larger than the last collection usage and then run another loop that keeps memory alive a bit longer.

par_old = mempool_beans.find {|b| b.name == "Par Survivor Space"}
if par_old.collection_usage_threshold_supported?
  par_old.collection_usage_threshold =
    par_old.collection_usage.used + 5_000

  1_000.times do
    ary = []
    10_000.times do
      ary << "foo" + "bar";
    end
  end

  puts "Crossed it: #{par_old.collection_usage_threshold_count} times"
end

And the result:

Crossed it: 104 times

Crossing the threshold here means that the minimum usage of the pool after a collection went over this size that many times. In this example, we create a 10,000 element array 1000 times and then throw it away; but while it's alive, it gets promoted into this "Survivor" space. So we repeatedly create it, promote it, and release it, resulting in the threshold being crossed several times.

The final bean we haven't covered, the MemoryManager bean, is usually just the superclass for other beans like GarbageCollector, so there's not much more to say about them. On my system, the list of MemoryManager beans is the same as the list of GarbageCollector beans.

Hopefully this walkthrough has whetted your appetite for the JVM's management and monitoring interfaces. In future posts, we'll cover how you can use the other builtin monitors, how you can track JRuby's internal monitors, and even how you can register your own monitors for any application. As always, leave questions and comments!

Accessing Java

Because JRuby always strives to be a solid Ruby implementation first, we don't normally expose nonstandard features like Java integration. That said, it's easy to pull it in... just require the 'java' library:

require 'java'

Alternatively, you can -rjava on the command line. And yes. That's all there is to it. You might want to consider taking a break at this point, so your manager doesn't realize just how simple it was to make this happen. Grab a snack, play some video games, and once you're done with all that hard work, we'll be ready to pull in some Java APIs!

Importing Java Classes

Once you've loaded Java support, all the Java classes that the JVM can see are available to you as though they were normal Ruby classes. You can access them directly:

java.lang.System.out.println "hello, world"

This "package" access works for any Java classes in packages starting with "java", "javax", "org", or "com." In order to avoid polluting the top-level namespace (we actually define Kernel methods for "java", "javax", "org", and "com" to provide this feature), other packages need to be scoped under the Java module:

reader = Java::jline.ConsoleReader.new
obj = Java::SomeClassWithNoPackage.new

You can actually "import" these classes in a couple different ways. First, you can simply assign them to a constant:

System = java.lang.System

You can also import them with the "java_import" Kernel method (also available as "import", but that frequently conflicts with libraries like Rake):

# This is basically the same as "JButton = javax.swing.JButton"
java_import javax.swing.JButton

Calling Methods

Once you have access to the Java classes, there's really nothing tricky about using them. Call the methods you want to call, and you'll get back the results you'd expect to get:

# display current system ns-granularity timer value
puts System.nanos

# Java constructors are just Class#new, as in Ruby
frame = JFrame.new "My window!"

# notice we allow setSize to work as set_size
frame.set_size 300, 300

# set* and get* properties work like Ruby attributes
frame.always_on_top = true
frame.visible = true

button = JButton.new "Press me!"
frame.add button

A MIDI Keyboard

Ok, let's put it all together using a more entertaining library: Java's MIDI API. The MIDI API is located under the javax.sound.midi package, which contains all the classes you'd need to load, construct, and play MIDI sequences.

In this first MIDI example, we'll listen for a line of text at a time and convert that text into a playable MIDI sequence. First we'll load Java support and import the classes we need:

require 'java'

midi = javax.sound.midi
import midi.MidiSystem
import midi.Sequence
import midi.MidiEvent
import midi.ShortMessage

Notice I assigned the javax.sound.midi package to the 'midi' local variable in order to shorten the import lines a bit. It's all just objects, after all...

Next up we'll want a simple gets loop with an exit clause:

while (line = gets) !~ /exit/

Ok, now we get to the meat of our app: building a sequence. For every line, we'll construct a new sequence. It's not the most efficient way, but it's nice and simple for now. Then for each character in the sequence we'll add a NOTE_ON event using the character code as the note. Finally we'll add a STOP event to stop playing the sequence.

  # Construct a new sequence using 2 beats per quarter note
  seq = Sequence.new Sequence::PPQ, 2
  track = seq.create_track

  # add each character into track as a note
  for i in 0...line.length do
    msg = ShortMessage.new
    # args here are (message, data1, and data2 according to MIDI spec)
    # NOTE_ON, character code, and a mid-range note velocity of 64
    msg.set_message(ShortMessage::NOTE_ON, line[i], 64)
    track.add MidiEvent.new(msg, i)
  end

  # add a STOP to end the track
  msg = ShortMessage.new
  msg.message = ShortMessage::STOP
  track.add MidiEvent.new msg, i + 1

Finally we'll play the sequence and end the loop. There's also a hard JVM System.exit call here, to shut down the internal event threads that MIDI support starts.

  # open the sequencer and play the sequence
  seqer = MidiSystem.sequencer
  seqer.open
  seqer.sequence = seq
  seqer.start
end
java.lang.System.exit(0)

Give it a try! Here's the full source to our first MIDI script.

Making it More Interactive

Given what we've learned, perhaps we can make our MIDI keyboard app a little more interactive. Ideally we'd like it to respond to keystrokes live,turning the note on when the key is pressed and turning it off when the key is released. For that, we'll launch a GUI window to receive keystroke events.

Our preamble this time only imports three classes, since we won't be constructing sequences this time.

# MIDI keyboard take two!
require 'java'

import javax.sound.midi.MidiSystem
import javax.swing.JFrame
import java.awt.event.KeyListener

In order to make this interactive, we're going to access the system synthesizer directly. Here we retrieve the default synthesizer, open it, and grab its channel number 0:

# Prepare the synth, get channel 0
synth = MidiSystem.synthesizer
synth.open
channel = synth.channels[0]

There are a few ways to receive keystroke events directly at the command line, but a lot of them are system-specific or require special libraries. For simplicity, we'll launch a GUI frame and just set it up to receive keystroke events.

# Prepare a frame to receive keystrokes
frame = JFrame.new("Music Frame")
frame.set_size 300, 300
frame.default_close_operation = JFrame::EXIT_ON_CLOSE

Here we're going to use another feature of JRuby: implementing an interface using a block of code. Every Java interface has a class method "impl" that receives a block and produces an object that implements that interface using the given block. The block itself is called for every interface method, receiving the name of the method and the original arguments as parameters.

In this case, we're going to handle the keyPressed and keyReleased methods on the KeyListener interface by turning notes on and off, again using the character code as the note:

# Listen for keystrokes, play notes
frame.add_key_listener KeyListener.impl { |name, event|
  case name
  when :keyPressed
    channel.note_on event.key_char, 64
  when :keyReleased
    channel.note_off event.key_char
  end
}

And finally, we'll show the frame to get the whole thing started:

# Show the frame
frame.visible = true

Here's the source for our MIDI keyboard.

Using JRuby, a whole new world of libraries are at your fingertips, and you never have to write a line of Java code. This is just one use case, but there really are an unlimited number of great ways JRuby allows you to leverage technologies that are otherwise out of the picture for Rubyists.

In the coming weeks and months we'll be working on a series of posts about how you can use JRuby, why we love it, and what you can look forward to in the future. If you've got a hankering for something specific, leave a comment—we aim to please :D

But why Engine Yard?

Engine Yard is, by almost all measures, the leading Ruby and Rails company in the world. They've literally written the book on how to make Rails deploy and scale easily, and if you have a tough Rails infrastructure issue, they're the company you talk to. But in order for Rails to penetrate the large organizations of the world, many of which run Java in some form, JRuby is often the answer. So out of JRuby, Engine Yard and the Ruby community get a broad new landscape of opportunities.

For JRuby, the move to Engine Yard means we've got a dedicated Ruby and Rails company backing our project. We've always believed in Ruby and Rails, but have long realized there's more we need to do. We need to finish support for the newer Ruby versions like Ruby 1.8.7 and 1.9.1. We need to build out the "last mile" of JRuby's Java integration logic, to allow seamless two-way integration with Java frameworks and libraries. And we need to work with Rubyists to reach out to the Java community, to expand Ruby into a much larger world. All of this will be easier to do with Engine Yard backing us up.

So, what does the future hold for JRuby?

Well there's the aforementioned work on Ruby 1.8.7/1.9.1 support, which is already very far along. We're looking at making a move to 1.8.7 for base compatibility in JRuby 1.4 right now, and will start filling in the remaining 1.9.1 bits very soon. I also mentioned Java integration work; our goal is that JRuby will become a "first class citizen" on the JVM, so you can swap out any piece of Java for a bit of Ruby with as few penalties as possible.

Already we've begun exploring how to make libraries like Hibernate, JAX-RS, and many more integrate well with JRuby and Rails. Of course we already have excellent performance characteristics, but that work will also continue. And we plan to work very closely with the Rails core team to ensure Rails fits seamlessly into classic and modern Java platform stacks.

As far as Engine Yard goes, we'll be working with the Engine Yard Cloud team to make sure our JRuby support is top-notch, and we'll be exploring possible paths for JRuby's future. We want to deliver a full set of cloud, hosting and support options, and will be talking to users and studying alternative platforms to figure out what those options should look like.

Ok, I'm hooked... how can I get started with JRuby?

Your first stop should be either my previous Engine Yard blog post on Getting Started with JRuby or the JRuby home site at jruby.org. You'll find instructions on how to download, install, and get up and running with JRuby. If you're interested in the project itself, check out JRuby's kenai.com project page and wiki or our mirror repository on GitHub. And of course you'll want to connect with the rest of the JRuby community, by joining our mailing lists, hopping on the #jruby IRC channel, or following us and @engineyard on Twitter.

What can I do to help?

There's a lot of work to do in the JRuby project, and we certainly can't do it all ourselves. So here are a few tips for folks interested in helping Ruby succeed on the Java platform:

• Try using JRuby!  The more folks that use JRuby, the more feedback we'll get and the better we'll be able to make it. If you can't use JRuby, then at least check it out and let us know what we're missing. User feedback is taken very seriously by the JRuby team.
• Join the JRuby mailing lists and IRC channels, and help us support the JRuby community and the new Rubyists we're bringing in.
• Submit bug reports for bugs you find, feature requests for features you think are missing, wiki updates and articles for topics you think need better coverage, and blog, blog, blog about your experiences. We need to work together to improve JRuby and get the word out.
• If you're hacker-minded, check out JRuby's source and see if you can help us fix a bug, add a feature, or improve performance. JRuby is a product of many, many external contributions, and would not exist today without community help. You can be part of the team too.
• Study alternative solutions to the problems Ruby and Rails solve so well, and let us know what you want out of JRuby. What do you need to make JRuby your web platform of choice? What are we missing compared to the alternatives? What would your perfect JRuby cloud or support offering look like? Don't hold back...we're eager to meet your challenges.

And finally, we would like to bid a fond farewell to our friends and colleagues at Sun Microsystems. We're really going to miss you guys, and hope we can continue to work together on improving the JVM as a language platform. Thanks for everything.