But in the process, the author made the same mistake I’ve seen time and time again.

Let’s assume you have a project with resources that you want to list by “popularity”. It can be a Reddit-like site where people like or dislike posts or comments. It can be an e-commerce where people like or dislike products.

It can be anything where people like or dislike something.

The biggest error people make is to consider a simple score like this:

popularity = positive_votes - negative_votes

There is an old article titled “How Not To Sort By Average Rating” and I quote:

“Why it is wrong: Suppose one item has 600 positive ratings and 400 negative ratings: 60% positive. Suppose item two has 5,500 positive ratings and 4,500 negative ratings: 55% positive. This algorithm puts item two (score = 1000, but only 55% positive) above item one (score = 200, and 60% positive). WRONG.”

Then you may think, I know how to fix it:

Score = average_rating = positive_votes / total_votes

Again, this is wrong, and again I quote:

“Why it is wrong: Average rating works fine if you always have a ton of ratings, but suppose item 1 has 2 positive ratings and 0 negative ratings. Suppose item 2 has 100 positive ratings and 1 negative rating. This algorithm puts item two (tons of positive ratings) below item one (very few positive ratings). WRONG.”

Correct Solution: Lower Bound of Wilson Score Confidence Interval for a Bernoulli

And I quote again:

“Say what: We need to balance the proportion of positive ratings with the uncertainty of a small number of observations. Fortunately, the math for this was worked out in 1927 by Edwin B. Wilson. What we want to ask is: Given the ratings I have, there is a 95% chance that the “real” fraction of positive ratings is at least what? Wilson gives the answer. Considering only positive and negative ratings (i.e. not a 5-star scale), the lower bound on the proportion of positive ratings is given by:”

I recommend you read the original article but let’s cut to the chase. If we follow the original post I linked in the beginning, I have a simple blog post Rails app, but instead of a visits_count field I need to add a positive:integer and negative:integer fields and user interface to post votes.

And I will replace the PostWithPopularityQuery class with the following:

class PostWithPopularityQuery
  def self.call
    Post.find_by_sql ['SELECT id, title, body, positive, negative,
        ((positive + 1.9208) / (positive + negative) -
        1.96 * SQRT((positive * negative) / (positive + negative) + 0.9604) /
        (positive + negative)) / (1 + 3.8416 / (positive + negative))
        AS ci_lower_bound
      FROM posts 
      WHERE positive + negative > 0
      ORDER BY ci_lower_bound DESC']
  end
end

And this is what I expect to see in a simple scaffold index.html.erb:

I would even go as far as recommending the ci_lower_bound to be a float field in the table and to have an asynchronous ActiveJob to update it in some larger interval of time (every 5 minutes, for example) and then the PostsController#index action would perform a straightforward SELECT query ordering directly against a real indexed field ci_lower_bound DESC without performing the calculations on every query.

Now THIS is the correct way to implement a simple, naive popularity ranking system that actually works correctly.

And this is not the only way to do it. There are dozens of good discussions of algorithms online. Every service that depends on content popularity has been refining algorithms like this for years. Facebook used to have an algorithm called “EdgeRank” which relied on variables such as Affinity, Weight, Time Decay, and it seems to have evolved so much that it now calculates popularity against more than 100 thousand variables!!

But regardless of the online service, I can assure you that none sorts by simple visits count or simple average votes count. That would be downright wrong.

But you can imagine that it quickly becomes cumbersome when you have to deal with pluralization, inflection, and it starts to become something like this:

"There #{@users.size == 1 ? 'is' : 'are'} #{@users.size} user#{'s' unless @users.size == 1}."

Or use Rails I18n support like this:

I18n.backend.store_translations :en, :user_msg => {
  :one => 'There is 1 user',
  :other => 'There are %{count} users'
}
I18n.translate :user_msg, :count => 2
# => 'There are 2 users'

For simple transactional phrases (such as flash messages), this is more than enough.

But if you want to generate an entire article in plain English from data structures, then the logic becomes very convoluted very fast.

I looked around and found a few Ruby projects that could help, for example:

• “nameable” which can do useful stuff like this:

Nameable::Latin.new('Chris').gender
#=> :male
Nameable::Latin.new('Janine').female?
#=> true

• “calyx” which can be used to generate simple phrases like this:

class GreenBottle < Calyx::Grammar
  mapping :pluralize, /(.+)/ => '\\1s'
  start 'One green {bottle}.', 'Two green {bottle.pluralize}.'
  bottle 'bottle'
end

# => "One green bottle."
# => "Two green bottles."

Nice and dandy, but still useless for the more complex needs I have in mind.

So I decided to dig a bit deeper, into the dark world of NLG, or Natural Language Generation (not to be confused with NLP, which stands for Natural Language Processing, which is the opposite of what I want. NLP gets plain English text and returns a parsed data structure).

For NLP (parsing, tokenization, etc) I’d highly recommend “Stanford CoreNLP”. It seems to be one of the most robust and comprehensive out there (come on, it’s from Stanford). Again a Java project, and a big download (more than 300MB!). Those linguistics projects are super heavy because they have to download entire dictionaries and lexicon databases.

But focusing on my problem at hand, NLG, there are several options out there. In all honesty, I did not do very extensive research, so if you are aware of which is the most robust and also well maintained and with an easy to use interface, let me know in the comments section below.

My choice was SimpleNLG. From its GitHub page we can see that it seems to be quite well maintained to this day, it’s a simple Java library and it is one of the “simpler” alternatives. KPML is on the opposite spectrum: it seems to be one of the oldest (since the 80’s!) and most robust ones. But seriously, it feels like you need a Ph.D to even get started.

Reading the SimpleNLG Java source code was boring but easy enough. Give yourself one full day of study to get used to the code and you’re in business.

The main problem is that it’s written in Java and I don’t intend to write anything in Java (or any derivative) for now. For a short while I considered the endeavor of rewriting the damn thing in something more portable such as Rust, which I could load anywhere through FFI.

But even though SimpleNLG has “Simple” in its name it has a few hairy dependencies to load the lexicon database. And the database itself is an HSQLDB dump, which is a Java-written database. And then, there would be the issue of maintaining a fork.

I quickly gave up on that idea and instead I worked around this by wrapping the library under a simple Rails-API endpoint. I had a few issues because I had Git LFS tracking jar files in my system and Heroku doesn’t support it and I ended up with a corrupted deployment (beware of those quirks, by the way!)

Finally, I was able to deploy a working JRuby + Rails-API project embedding SimpleNLG at Heroku. You can deploy your own copy by cloning my nlg_service. It works fine with the latest JRuby 9.1.5.0. You should pay for at least a Hobby tier over Heroku. Java takes a ridiculous amount of time to start up and more time to warm up. Heroku’s free tier shuts down your dyno if it sits idle and a subsequent web request will definitely time out or take an absurd amount of time to return.

Once deployed it starts up Rails, then loads this initializer:

require 'java'
Java::JavaLang::System.set_property "file.encoding","UTF-8"

SIMPLE_NLG_DEFAULT_LEXICON_PATH = Rails.root.join("lib/SimpleNLG/resources/default-lexicon.xml").to_s.freeze
SIMPLE_NLG_PATH                 = Rails.root.join("lib/SimpleNLG").to_s.freeze

Dir["#{SIMPLE_NLG_PATH}/*.jar"].each { |jar| require jar }

And then I map the classes like this:

module SimpleNLG
  %w(
    simplenlg.aggregation
    simplenlg.features
    simplenlg.format.english
    simplenlg.framework
    simplenlg.lexicon
    simplenlg.morphology.english
    simplenlg.orthography.english
    simplenlg.phrasespec
    simplenlg.realiser.english
    simplenlg.syntax.english
    simplenlg.xmlrealiser
    simplenlg.xmlrealiser.wrapper
  ).each { |package| include_package package }
end

Finally, I have a simple endpoint mapped to a controller action:

class Api::RealisersController < ApplicationController
  def create
    reader = java::io::StringReader.new(params[:xml])
    begin
      records = SimpleNLG::XMLRealiser.getRecording(reader)
      output = records.getRecord.map do |record|
        SimpleNLG::XMLRealiser.realise(record&.getDocument)
      end
      @realisation = output.join("\n").strip
      render plain: @realisation
    ensure
      reader.close
    end
  end
end

The process of generating the final English text is called “realisation”. SimpleNLG has a comprehensive Java API but it also exposes it as a simpler XML format. The full XML Realiser Schema is available as an XSD.

If I want to write this sentence:

“There are some finished and delivered stories that may not have been tested.”

This is the XML that I need to assemble:

<?xml version="1.0"?>
<NLGSpec xmlns="http://simplenlg.googlecode.com/svn/trunk/res/xml" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
  <Recording>
    <Record>
      <Document cat="PARAGRAPH">
        <child xsi:type="SPhraseSpec">
          <subj xsi:type="NPPhraseSpec">
            <head cat="ADVERB">
              <base>there</base>
            </head>
          </subj>
          <vp xsi:type="VPPhraseSpec" PERSON="THIRD">
            <head cat="VERB">
              <base>be</base>
            </head>
            <compl xsi:type="NPPhraseSpec" NUMBER="PLURAL">
              <head cat="NOUN">
                <base>story</base>
              </head>
              <spec xsi:type="WordElement" cat="DETERMINER">
                <base>a</base>
              </spec>
              <preMod xsi:type="CoordinatedPhraseElement" conj="and">
                <coord xsi:type="VPPhraseSpec" TENSE="PAST">
                  <head cat="VERB">
                    <base>finish</base>
                  </head>
                </coord>
                <coord xsi:type="VPPhraseSpec" TENSE="PAST">
                  <head cat="VERB">
                    <base>deliver</base>
                  </head>
                </coord>
              </preMod>
              <compl xsi:type="SPhraseSpec" MODAL="may" PASSIVE="true" TENSE="PAST">
                <vp xsi:type="VPPhraseSpec" TENSE="PAST" NEGATED="true">
                  <head cat="VERB">
                    <base>test</base>
                  </head>
                </vp>
              </compl>
            </compl>
          </vp>
        </child>
      </Document>
    </Record>
  </Recording>
</NLGSpec>

Ok, this is preposterous, I know.

Which is why I decided to go ahead and use one of Ruby’s most recognized strengths: creating DSLs or Domain Specific Languages.

The result of my initial endeavor to simplify this process is the nlg_xml_realiser_builder ruby gem. Simply add the following to your Gemfile:

gem 'nlg_xml_realiser_builder'

And the humongous XML above becomes something more manageable like this:

dsl = NlgXmlRealiserBuilder::DSL.new
dsl.builder(true) do
  sp :child do
    subj :np, 'there', cat: 'ADVERB'
    verb 'be', PERSON: 'THIRD' do
      compl :np, ['a', 'story'], NUMBER: 'PLURAL'  do
        preMod :cp, conj: 'and' do
          coord :vp, 'finish', TENSE: 'PAST'
          coord :vp, 'deliver', TENSE: 'PAST'
        end
        compl :sp, MODAL: 'may', PASSIVE: true, TENSE: 'PAST' do
          verb 'test', TENSE: 'PAST', NEGATED: true
        end
      end
    end
  end
end.to_xml

Understanding the intricacies of an NPPhraseSpec vs a VPPhraseSpec or the difference between a WordElement or StringElement are beyond this blog post. But most of the original XSD has been mapped through this constants file.

I have a few acceptance specs that are generating XMLs like the above, posting to my live online NLG Web Service and fetching the resulting English sentences. I will change this process in the future but you can test it out yourself.

The advantages start here. Now let’s check out the previous example more closely. Again, it renders this phrase:

“There are some finished and delivered stories that may not have been tested.”

Now, it’s in plural form because I am talking about ‘stories’, but what if I want a singular version?

Below is the new version where I just wrap it around a method and make the attribute ‘NUMBER’ accept both ‘PLURAL’ or ‘SINGULAR’:

def example(plural = 'PLURAL')
  dsl = NlgXmlRealiserBuilder::DSL.new
  dsl.builder(true) do
    sp :child do
      subj :np, 'there', cat: 'ADVERB'
      verb 'be', PERSON: 'THIRD' do
        compl :np, ['a', 'story'], NUMBER: plural  do
          preMod :cp, conj: 'and' do
            coord :vp, 'finish', TENSE: 'PAST'
            coord :vp, 'deliver', TENSE: 'PAST'
          end
          compl :sp, MODAL: 'may', PASSIVE: true, TENSE: 'PAST' do
            verb 'test', TENSE: 'PAST', NEGATED: true
          end
        end
      end
    end
  end.to_xml
end

And I can run the singular version like this:

puts example('SINGULAR')

This is the resulting phrase:

“There is a finished and delivered story that may not have been tested.”

Check out how it changed the verb from “are” to “is” and the noun determiner from “some” to “a” on its own! And of course, this is a contrived example. Now imagine an entire customizable article, full of paragraphs and sentences that I can customize depending on several variables I have.

While I was studying and writing this DSL I got a good enough grasp of the SimpleNLG structure, but if you have more examples for more complex phrase structures, please let me know in the comments section down below.

Most of the specs were copied from the XML Realiser tests from the original Java project to make sure I am covering most cases.

It will be interesting to see if this DSL makes it easier for more people to experiment with NLG. As usual, send your Pull Requests, ideas and suggestions on my GitHub public repositories:

• nlg_service
• nlg_xml_realiser_builder

And if you’re interested in the subject of NLP and NLG I found this list of Ruby related open source projects as well.

As a disclaimer, I don’t know them, and I respect their points of view. This is not supposed to start a flame war, just to paint an alternative point of view.

Both articles represent the point of view of many veteran Ruby developers in this community. Some of whom already left for other technologies or stopped public appearances.

I already made my stand very clear in “Rails has Won: The Elephant in the Room”. But let me simplify here.

There is clearly a faction being formed. Apart from great developers like Solnic (dry-rb), Nick (Trailblazer), and Luca (Hanami), most people are complaining that Rails and some of the closest pieces of the ecosystem around it don’t conform to their newly found vision of what “good” should be.

In their view of the world, a “better” Rails should be way simpler, way more explicit, composed of super small libraries, super explicit external APIs, and super composability to take it apart back and forth.

And here lies my problem: if DHH complied and took that direction, that would be “Ruby OFF the Rails.” The whole point of “Ruby ON Rails” is precisely because it is defined by being a coherent, opinionated full stack with coarse-grained libraries that are meant to work together in tandem.

Let me present the true Elephant in the Room:

• Discussions like this are just inflammatory link baits. They serve no point but create a cloud of animosity against something without actually presenting an objective good alternative. Writing bullet points is easy, but writing complete solutions and maintaining them is not.

• There are indeed good alternatives popping up already, such as the aforementioned Trailblazer, Hanami, and the dry-rb collection of libraries, just to name a few. The complaint is that they don’t have as much traction right now. What’s the solution? Bad-mouthing Rails? Writing ranting articles of how life is unjust? Or, actually, are you writing more tutorials on how to use Trailblazer? Recording screencasts on how to use Hanami? Going to a conference and presenting more talks on importing dry-rb into your projects?

• Ruby on Rails was meant for programmers tired of super composability, super configuration, super explicit, super fine-grained libraries. It was meant for Java programmers coming out straight from the nightmare that was J2EE 1.x in 2004, remember that? Now, the “proposal” is to go back to that? Waste enormous amounts of time fine-tuning your tailor-made mini-stack? We have that already. It’s called Javascript. And let me tell you: configuring the current breed of package.json brings me back really bad memories from pom.xml, hibernate.cfg.xml to struts-config.xml.

• Beginners are not interested in fine-grained solutions. Most people forget how it was to be a beginner. Actually, I’d argue that 80% of the world’s developers benefit greatly from a Rails-like approach. It’s no coincidence that many platforms conformed to an opinionated, convention-over-configuration approach after it proved its point.

The fact that people are complaining against Rails makes it feel like it is Rails’ fault somehow. Actually not, it’s just a reflection of the frustration of the very people that failed (and are failing) to pitch the alternatives. Everybody wants a free lunch.

Let me tell you a short story.

Back in 2005 I was the only Rails developer I knew in my country (Brazil). I googled around and found perhaps half a dozen other hobbyists doing Ruby for fun, and a couple were doing it professionally already.

People would make fun of us everywhere I looked because they thought we were just crazy. “Of course, J2EE is the way to go.” “Of course, every project should conform to Eric Evan’s DDD approach.” “Of course, every project should conform to all of the Gang of Four’s Design Patterns, the more, the merrier.” “Of course, we should have very isolated deployment packages with very explicit API boundaries between them, no matter how it makes productivity decrease.”

We’ve been there before. In 2004, I already had more than ten years of experience in professional programming.

It took me another ten years of pilgrimage. 9 years organizing my own conference. More than 1,000 blog posts. Almost 200 talks, several of which I paid from my own pocket to buy bus and airplane tickets. I put my money where my mouth was.

The promise was to replace all the complexity and bureaucracy with an opinionated stack where most of the basic decisions were already made. It would be out of our way so we could focus on the most important part: the business.

Every talk I pitched to Rails since 2006 had the above section. And with every single person who ever watched any of my talks as my witness, I always said:

“Many of you have heard that Java is bad because it’s so complex and so bureaucratic. And many will try to pitch their alternatives at the expense of Java. Not us. We embrace what Java has to offer. Solr, Elasticsearch, and Hadoop are all made with Java. What will we do? Rewrite everything in Ruby? Of course not. For Rails to Win, Java doesn’t have to Lose.” (I put the “evil edition” image on fire, on stage).

I wrote an article (in pt-BR, sorry) in October of 2007 titled “For me to win, the other has to lose …” where I explained why this way of thinking is so lame. I also took inspiration from the famous 1998 Macworld talk by Steve Jobs, where he called Bill Gates on stage, on the big screen, to announce their collaboration. The Apple fans went nuts, they were horrified by that sight, it was like sleeping with Satan.

Almost 20 years later, Apple is one of the most valuable companies in the world. Because it didn’t matter. Jobs left pure ideology behind and became pragmatic. They needed Microsoft’s endorsement, got it, and proved their points by results: they spent the 90s masturbating over why classic OS 7 to 9 were better than Windows 3.1 up to NT 4. But it took them ten years to prove the worth of OS X over the first decade of the 21st century. One step at a time, one release after the other, frequently delivering instead of spending years closed in the labs until the “perfect” solution arose. Baby steps. It culminated when they squeezed OS X into iOS in 2007.

That’s how you prove a point: through sheer pain and sweat, by putting it out there in the streets, pitching it, selling it, and creating true value one small release at a time.

And then this happened:

Remember when Apple took over the world? One entire decade to come back from bleeding 1 billion a year up until becoming the most valuable company in the world, whose effects are still going on 5 years after Jobs’ passing.

Some of us understand that engineers want to masturbate over irrelevant things.

“- My algorithm runs 0.1 milliseconds faster than yours”.

“- Ha, but my design has fewer dependency points than yours!”

Rails did to Ruby what Apple did to Unix and Canonical did to Linux-based distros. People complaining about Rails remind me of people trying to argue why Archlinux is way superior to OS X or Ubuntu, but it’s just that people in general are too stupid to perceive that value.

And for some, it’s not exactly flattering to be in the same league as Apple or Canonical, but the results are undeniable.

Remember the legendary “15-minute blog”? Why did it surprise us so much in 2005? It was not what Rails could do; it was all the work we didn’t need to do. And Rails has stayed true to that until now.

If you’re a beginner programmer, Rails still has plenty to teach, and you can learn the details later.

If you’re the average programmer, Rails’ ecosystem will take you from point A to point B much faster, with better productivity and maintainability.

Why are you complaining if you’re the tech star programmer funded by a unicorn?

Programmers ranting are not following one of our own old favorite quotes: “Premature Optimization is the Root of All Evil.” And as the “UNIX way” already had:

The strategy is definitely:

Make it work,

Then make it right,

and, finally, make it fast.

There is nothing wrong with having alternatives to Rails, but the Rails bashing is getting very old very fast. It’s difficult to pitch alternatives, I know. I’ve spent many shoe soles in the last 10 years advocating Ruby, so I really know it.

I never believed in free lunch. I believe in targeted, hard work, and that’s been my philosophy for the last 25 years in programming.

The nature of the original Manga Downloader is no different from a web crawler: it fetches HTML pages, parses them in order to find collections of links and keeps fetching until a set of images is downloaded. Then I organize them in volumes, optimize them to fit the Kindle screen resolution, and compile them down into PDF files. This makes this project an interesting exercise in trying to make concurrent HTTP requests and process the results.

A year later I was learning Elixir. The Manga Downloadr was a nice candidate for me to figure out how to implement the same thing in a different language. You can follow my learning process in this series of posts.

Finally, I’ve been learning more about Crystal, a Ruby-inspired platform that can compile Ruby-like source code into LLVM-optimized native binaries. And as a bonus it features Go-like CSP channels and fibers to allow for concurrent code.

So I adapted my Elixir version to Crystal and the result is this code you can find over Github as akitaonrails/cr_manga_downloadr.

It works very well and performs really fast, limited mainly by how many requests MangaReader can respond to concurrently and the speed/reliability of the Internet connection. So, as my original Ruby version was terrible code, it was a good time to rewrite it. And as Crystal is surprisingly close to Ruby I decided to port it over.

The port was almost too trivial

It was mostly copying and pasting the Crystal code without the Type annotations. And I had to replace the lightweight Fibers and Channel implementation for concurrency with traditional Ruby Threads.

The new version 2.0 for the Ruby version of the tool can be found in this Github repository: akitaonrails/manga-downloadr.

Ruby, Ruby everywhere!

Moving between Ruby and Crystal is not so difficult. The Crystal team did a fantastic job implementing a very solid Standard Library (stdlib) that very closely resembles MRI Ruby’s.

For example, let’s compare a snippet from my Crystal version first:

def fetch(page_link : String)
  get page_link do |html|
    images = html.xpath("//img[contains(@id, 'img')]").as(XML::NodeSet)
    image_alt = images[0]["alt"]
    image_src = images[0]["src"]
    if image_alt && image_src
      extension      = image_src.split(".").last
      list           = image_alt.split(" ").reverse
      title_name     = list[4..-1].join(" ")
      chapter_number = list[3].rjust(5, '0')
      page_number    = list[0].rjust(5, '0')
      uri = URI.parse(image_src)
      CrMangaDownloadr::Image.new(uri.host as String, uri.path as String, "#{title_name}-Chap-#{chapter_number}-Pg-#{page_number}.#{extension}")
    else
      raise Exception.new("Couldn't find proper metadata alt in the image tag")
    end
  end
end

Now let’s check the ported Ruby version:

def fetch(page_link)
  get page_link do |html|
    images = html.css('#img')

    image_alt = images[0]["alt"]
    image_src = images[0]["src"]

    if image_alt && image_src
      extension      = image_src.split(".").last
      list           = image_alt.split(" ").reverse
      title_name     = list[4..-1].join(" ")
      chapter_number = list[3].rjust(5, '0')
      page_number    = list[0].rjust(5, '0')

      uri = URI.parse(image_src)
      Image.new(uri.host, uri.path, "#{title_name}-Chap-#{chapter_number}-Pg-#{page_number}.#{extension}")
    else
      raise Exception.new("Couldn't find proper metadata alt in the image tag")
    end
  end
end

It’s uncanny how similar they are, down to stdlib calls such as URI.parse or Array methods such as split.

Once you remove the Type annotations from the Crystal version, it’s 99% Ruby.

Ruby doesn’t care if you’re trying to call a method on a Nil object - at source code compiling time. Crystal does compile-time checks and if it feels like the method call will be over Nil, it won’t even compile. So this is one big win to avoid subtle bugs.

In Rails we are used to the #try method. Ruby 2.3 introduced the safe navigation operator.

So, in Ruby 2.3 with Rails, both of the following lines are valid:

obj.try(:something).try(:something2)
obj&.something&.something2

In Crystal we can do the following:

obj.try(&.something).try(&.something2)

So, it’s close. Use with care.

As I mentioned before, Crystal is close to Ruby but it’s not meant to be compatible, so we can’t just load Rubygems without porting. In this example I don’t have Nokogiri to parse the HTML. But this is where the stdlib shines: Crystal comes with good enough XML/HTML and JSON parsers. So we can parse HTML as XML and use plain XPath instead.

Instead of Ruby’s Net::HTTP we have HTTP::Client (but their methods and semantics are surprisingly similar).

There are other differences, for example, this is the main file that requires all the others in Ruby:

...
$LOAD_PATH.unshift File.join(File.dirname(__FILE__), "lib")

require "manga-downloadr/records.rb"
require "manga-downloadr/downloadr_client.rb"
require "manga-downloadr/concurrency.rb"
require "manga-downloadr/chapters.rb"
require "manga-downloadr/pages.rb"
require "manga-downloadr/page_image.rb"
require "manga-downloadr/image_downloader.rb"
require "manga-downloadr/workflow.rb"

And this is the Crystal version of the same manifest:

require "./cr_manga_downloadr/*"
...

On the other hand, we need to be a bit more explicit in each Crystal source code file, and declare the specific dependencies where needed. For example, in the pages.cr file it starts like this:

require "./downloadr_client"
require "xml"

module CrMangaDownloadr
  class Pages < DownloadrClient(Array(String))
  ...

Crystal has less room for “magic” but it’s able to maintain a high level of abstraction anyway.

A Word about Types

We can spend the next decade masturbating over all there is about Types, and it will be extremely boring.

The only thing you must understand: the compiler must know the method signature of classes before you can use them. There is no Runtime component that can introspect objects on the fly, like in Ruby and other dynamic languages (even Objective-C/Swift can do more dynamic stuff than Crystal).

Most of the time the Crystal compiler will be smart enough to infer the types for you, so you don’t need to be absolutely explicit. You should follow the compiler’s lead to know when to use Type Annotations.

What may really scare you at first is the need for Type Annotations, to understand Generics and so forth. The compiler will print out every scary error dump that you will need to get used to. Most scary errors will usually be a Type Annotation missing or you trying to call a method on a possibly Nil object.

For example, if I change the following line in the page_image_spec.cr test file:

# line 8:
image = CrMangaDownloadr::PageImage.new("www.mangareader.net").fetch("/naruto/662/2")

# line 10:
# image.try(&.host).should eq("i8.mangareader.net")
image.host.should eq("i8.mangareader.net")

The commented out line recognizes that the image instance might come as Nil, so we add an explicit #try call in the spec.

If we try to compile without this recognition, this is the error the compiler will dump on you:

$ crystal spec                                                        [
Error in ./spec/cr_manga_downloadr/page_image_spec.cr:10: undefined method 'host' for Nil (compile-time type is CrMangaDownloadr::Image?)

    image.host.should eq("i8.mangareader.net")
          ^~~~

=============================================================================

Nil trace:

  ./spec/cr_manga_downloadr/page_image_spec.cr:8

        image = CrMangaDownloadr::PageImage.new("www.mangareader.net").fetch("/naruto/662/2")
...

There is a large stack trace dump after that snippet above, but you only need to pay attention to the first few lines that already say what’s wrong: “undefined method ‘host’ for Nil (compile-time type is CrMangaDownloadr::Image?)”. If you know how to read, you shouldn’t have any problems most of the time.

Now, the Chapters, Pages, PageImage (all subclasses of DownloadrClient) are basically the same thing: they do HTTP::Client requests.

This is how the Pages class is implemented:

...
module CrMangaDownloadr
  class Pages < DownloadrClient(Array(String))
    def fetch(chapter_link : String)
      get chapter_link do |html|
        nodes = html.xpath_nodes("//div[@id='selectpage']//select[@id='pageMenu']//option")
        nodes.map { |node| [chapter_link, node.text as String].join("/") }
      end
    end
  end
end

#get is a method from the DownloadrClient superclass that receives a String chapter_link and a block. The block receives a parsed html collection of nodes and we can play with it, returning an Array of Strings.

That’s why we have the (Array(String)) when inheriting from DownloadrClient. Let’s see how the DownloadrClient superclass is implemented.

module CrMangaDownloadr
  class DownloadrClient(T)
    ...
    def get(uri : String, &block : XML::Node -> T)
      response = @http_client.get(uri)
      case response.status_code
      when 301
        get response.headers["Location"], &block
      when 200
        parsed = XML.parse_html(response.body)
        block.call(parsed)
      end
      ...
    end
  end
end

You can see that this class receives a Generic Type and it uses it as the return type for the yielded block in the #get method. The XML::Node -> T is the declaration of the signature for the block, sending XML::Node and receiving whatever the type T is. At compile time, imagine this T being replaced by Array(String). That’s how you can create classes that can deal with any number of different Types without having to overload for polymorphism.

If you come from Java, C#, Go or any other modern statically typed language, you probably already know what a Generic is.

You can go very far with Generics, check out how our Concurrency.cr begins:

class Concurrency(A, B, C)
  ...
  def fetch(collection : Array(A)?, &block : A, C? -> Array(B)?) : Array(B)?
    results = [] of B
    collection.try &.each_slice(@config.download_batch_size) do |batch|
      engine  = if @turn_on_engine
                  C.new(@config.domain)
                end
      channel = Channel(Array(B)?).new
      batch.each do |item|
      ...

And this is how we use it in the workflow.cr:

private def fetch_pages(chapters : Array(String)?)
  puts "Fetching pages from all chapters ..."
  reactor = Concurrency(String, String, Pages).new(@config)
  reactor.fetch(chapters) do |link, engine|
    engine.try( &.fetch(link) )
  end
end

In this example, imagine A being replaced by String, B also being replaced by String and C being replaced by Pages in the Concurrency class.

This is the “first-version-that-worked” so it’s probably not very idiomatic. Either this could be solved with less Generics exercising or maybe I could simplify it with the use of Macros. But it’s working quite ok as it is.

The pure Ruby version ends up just like this:

class Concurrency
  def initialize(engine_klass = nil, config = Config.new, turn_on_engine = true)
    ...
  end
  def fetch(collection, &block)
    results = []
    collection&.each_slice(@config.download_batch_size) do |batch|
      mutex   = Mutex.new
      threads = batch.map do |item|
      ...

This version is much “simpler” in terms of source code density. But on the other hand we would have to test this Ruby version a lot more because it has many different permutations (we even inject classes through engine_klass) that we must make sure respond correctly. In practice we should add tests for all combinations of the initializer’s arguments.

In the Crystal version, because all types have been checked at compile-time, it was more demanding in terms of annotations; but we can be pretty sure that if it compiles, it will run as expected.

I’m not saying that Crystal doesn’t need any specs.

Compilers can only go so far. But how much is “so far”? Whenever you’re “forced” to add Type Annotations, I will state that those parts are either trying to be too smart or they are intrinsically complex. Those are parts that would require extra levels of tests in Ruby and, if we can add the annotations properly, we can have fewer tests (we don’t need to cover most permutations) in the Crystal version (exponential complexity of permutations could go down to a linear complexity, I think).

A Word about Ruby Threads

The main concepts you must understand about concurrency in Ruby are these:

• MRI Ruby has a GIL, a Global Interpreter Lock, that forbids code from running concurrently.
• MRI Ruby does have access and exposes native Threads since version 1.9. But even if you fire up multiple Threads, they will run sequentially because only one thread can hold the Lock at a time.
• I/O operations are the exception: they do release the Lock for other threads to run while the operation is waiting to complete. The OS will signal the program through OS-level poll.

This means that if your app is I/O intensive (HTTP requests, File reads or writes, socket operations, etc), you will have some concurrency. A web server, such as Puma for example, can take some advantage of Threads because a big part of the operations involves receiving HTTP requests and sending HTTP responses over the wire, which would make the Ruby process idle while waiting.

If your app is CPU intensive (heavy algorithms, data crunching, stuff that really makes the CPU hot) then you can’t take advantage of native Threads, only one will run at a time. If you have multiple cores in your CPU, you can Fork your process to the number of cores available.

You should check out grosser/parallel to make it easy.

This is why Puma also has a “worker” mode. “Worker” is the name we usually give to forked children of processes.

In the case of this downloader process, it will perform thousands of HTTP requests to scrape the needed metadata from the MangaReader pages. So it’s definitely much more I/O intensive than CPU intensive (the CPU intensive parts are the HTML parsing and later the image resizing and PDF compiling).

A sequential version of what has to be done, in Ruby, looks like this:

def fetch_sequential(collection, &block)
  results = []
  engine  = @turn_on_engine ? @engine_klass.new(@config.domain) : nil
  collection&.each_slice(@config.download_batch_size) do |batch|
    batch.each do |item|
      batch_results = block.call(item, engine)&.flatten
      results += ( batch_results || [])
    end
  end
  results
end

If we have 10,000 links in the collection, we first slice it to what @config.download_batch_size defines and we iterate over those smaller slices, calling some block and accumulating the results. This is a naive algorithm, as you will find out in the next section, but bear with me.

In Elixir you can fire up micro-processes to make the HTTP requests in parallel. In Crystal you can fire up Fibers and wait for the HTTP requests to complete and signal the results through Channels.

Both are lightweight ways and you can have hundreds or even thousands running in parallel. Manga Reader will probably complain if you try that many at once, so the limit is not in the code, but in the external service.

To transform the sequential version into a concurrent one, this is what we can do in Crystal:

def fetch(collection : Array(A)?, &block : A, C? -> Array(B)?) : Array(B)?
  results = [] of B
  collection.try &.each_slice(@config.download_batch_size) do |batch|
    channel = Channel(Array(B)?).new
    batch.each do |item|
      spawn {
        engine  = if @turn_on_engine
                    C.new(@config.domain)
                  end
        reply = block.call(item, engine)
        channel.send(reply)
        engine.try &.close
      }
    end
    batch.size.times do
      reply = channel.receive
      if reply
        results.concat(reply.flatten)
      end
    end
    channel.close
    puts "Processed so far: #{results.try &.size}"
  end
  results
end

Fetching a huge collection and slicing it in smaller ‘batches’ is easy. Now we have a smaller batch collection. For each item (usually a URI) we spawn a Fiber and call a block that will request and process the results. Once it’s finished processing it sends the results over a channel.

Once we finish iterating over the batch and spawning that many Fibers we can “wait” for them by doing channel.receive, which will start receiving results as soon as the Fibers finish requesting/processing each URI.

We accumulate the results and go over the next batch of the collection until finished. The amount of concurrency is determined by the size of the batch (it’s like what I did with ‘poolboy’ over Elixir where we start a fixed number of processes to run in parallel and avoid doing a Denial of Service to Manga Reader).

By the way, this Crystal implementation is similar to what you would do if you were in Go, using Channels.

In the Ruby version you can fire up native Threads - which has a lot of overhead to spawn! - and assume the HTTP requests will run almost all in parallel. Because it’s I/O intensive, you can have them all in parallel. This is what it looks like:

def fetch(collection, &block)
  results = []
  collection&.each_slice(@config.download_batch_size) do |batch|
    mutex   = Mutex.new
    threads = batch.map do |item|
      Thread.new {
        engine  = @turn_on_engine ? @engine_klass.new(@config.domain) : nil
        Thread.current["results"] = block.call(item, engine)&.flatten
        mutex.synchronize do
          results += ( Thread.current["results"] || [] )
        end
      }
    end
    threads.each(&:join)
    puts "Processed so far: #{results&.size}"
  end
  results
end

Threads are all initialized in a “paused” state. Once we instantiate those many Threads, we can #join on each of them and wait for them all to finish.

Once each Thread finishes the same URI request/process, the results must be accumulated in a global storage, in this case a simple array called results. But because we might have the chance of 2 or more threads trying to update the same array, we might as well synchronize the access (I’m not sure if Ruby’s Array access is already synchronized, but I guess not). To synchronize access we use a Mutex, which is a fine-grained lock, to make sure only 1 thread can modify the global array at once.

To prove that Ruby can support concurrent I/O operations, I have added 2 methods to the Concurrent class, the first is just #fetch and it’s the Thread implementation above. The second is called #fetch_sequential and it is the sequential version also shown at the beginning of this section. And I added the following spec:

it "should check that the fetch implementation runs in less time than the sequential version" do
  reactor = MangaDownloadr::Concurrency.new(MangaDownloadr::Pages, config, true)
  collection = ["/onepunch-man/96"] * 10
  WebMock.allow_net_connect!
  begin
    concurrent_measurement = Benchmark.measure {
      results = reactor.fetch(collection) { |link, engine| engine&.fetch(link) }
    }
    sequential_measurement = Benchmark.measure {
      results = reactor.send(:fetch_sequential, collection) { |link, engine| engine&.fetch(link) }
    }
    /\((.*?)\)$/.match(concurrent_measurement.to_s) do |cm|
      /\((.*?)\)/.match(sequential_measurement.to_s) do |sm|
        # expected for the concurrent version to be close to 10 times faster than sequential
        expect(sm[1].to_f).to be > ( cm[1].to_f * 9 )
      end
    end
  ensure
    WebMock.disable_net_connect!
  end
end

Because it uses WebMock, I first disable it during this spec. I create a fake collection of 10 real links to MangaReader. And then I benchmark the Thread-based concurrent version and the plain sequential version. Because we have 10 links and they are all the same you can assume that the sequential version will be almost 10 times slower than the Thread-based version. And this is exactly what this spec compares and proves (the spec fails if the concurrent version is not at least 9x faster).

To compare all versions of the Manga Downloadrs I let it download and compile an entire manga collection, in this case “One-Punch Man”, which has almost 1,900 pages/images. I am just measuring the fetching and scraping processes and skipping the actual image downloading, image resizing and PDF generation as they take the majority of the time and the resizing and PDF part are all done by ImageMagick’s mogrify and convert tools.

This is how long this new Ruby version takes to fetch and scrape almost 1,900 pages (using MRI Ruby 2.3.1):

12,42s user 1,33s system 23% cpu 57,675 total

This is how long the Crystal version takes:

4,03s user 0,40s system 7% cpu 59,207 total

Just for fun I tried to run the Ruby version under JRuby 9.1.1.0. To run with JRuby just add the following line in the Gemfile:

ruby "2.3.0", :engine => 'jruby', :engine_version => '9.1.1.0'

Bundle install, run normally, and this is the result:

47,80s user 1,99s system 108% cpu 45,967 total

And this is how long the Elixir version takes:

11,38s user 1,04s system 85% cpu 14,590 total

Reality Check!

If you just look at the times above you might come to the wrong conclusions.

First of all, it’s an unfair comparison. The Elixir version uses a very different algorithm than the Ruby and Crystal versions.

In Elixir I boot up a pool of processes, around 50 of them. Then I start 50 HTTP requests at once. Once each process finishes, it releases itself back to the pool and I can fire up another HTTP request from the queue of links. So it’s a constant stream of at most 50 HTTP requests, constantly.

The Crystal and Ruby/JRuby versions slice the 1,900 links into batches of 40 links and then I fire up 40 requests at once. This implementation waits for all 40 to finish and they fire up another 40. So it’s never a constant stream, it’s bursts of 40 requests. So each batch is slowed down by the slowest request in the batch, not giving a chance for other requests to go through.

It’s a difference in architecture. Elixir makes it much easier to do streams and Crystal, with its CSP style, makes it easier to do bursts. A better approach would be to queue up the 1,900 links and use something like Sidekiq.cr to go over one link at a time (spawning 40 fibers to serve as a “pool”, for example).

The Elixir version has a more efficient architecture, which is why it takes no more than 15 seconds to fetch all the image links while the Crystal version takes almost a full minute to finish (the accumulation of the slowest requests in each batch).

Now, you will be surprised that the Crystal version is actually a bit slower than the Ruby version! And you won’t be too surprised to see JRuby being faster at 45 seconds!

This is more evidence that you should not dismiss Ruby (and that you should try JRuby more often). As I explained before, it does support concurrency in I/O operations and the applications I tested are all I/O heavy.

The difference is probably in the maturity of Ruby’s Net::HTTP library against Crystal’s HTTP::Client. I tried many tweaks in the Crystal version but I couldn’t get much faster. I tried to do larger batches, but for some reason the application just hangs, pauses, and never releases. I have to Ctrl-C out of it and retry until it finally goes through. If someone knows what I am doing wrong, please don’t forget to write in the comments section below.

Some of this is probably due to MangaReader’s unreliable servers, they probably have some kind of DoS protection, throttling connections or something like this.

Anyway, when they go through, because the Ruby and Crystal algorithms are essentially the same, they take roughly the same time to complete. So what’s missing is for me to evolve this algorithm to either use something like Sidekiq or implement an internal queue/pool of workers scheme.

Conclusion

The goal of this experiment was to learn more of Crystal’s ability and how easy it would be to go back and forth with Ruby.

As you can see, there are many differences, but it’s not so difficult. I may be missing something, but I stumbled upon some difficulties when pushing HTTP::Client + Fibers too hard, as I explained above. If you know what I am doing wrong, please let me know.

The difference between the Elixir vs Ruby/Crystal algorithms shows not just language performance differences, but also the importance of architecture and algorithms in the overall performance. This test was not conclusive, just a hint that my naive Fibers implementation needs some more work, and that Elixir’s natural handling of parallel processes makes it easier to achieve higher levels of parallelism.

This serves as a taste of what Crystal can do, and also that Ruby is still in the game. But also that Elixir surely is something to look at closely.

JRuby always pursued the goal of being a competent MRI replacement. If you have not followed what’s going on, JRuby changed version schemes when they switched from the 1.7 series to the 9.x series. You should read this blog post that explains it.

In summary, if you want MRI 2.2 compatibility you must use JRuby 9.0.x and if you want MRI 2.3 compatibility you must use the JRuby 9.1.x series. The most current release being 9.1.2.0. Anything prior to that you can refer to this table of versions from Heroku’s documentation.

There are important recommendations as well:

• Ideally you should use Rails 4.2. Try to be at least above 4.0, and you can turn on config.threadsafe! by default in the “config/environments/production.rb” file. To understand this subject, refer to Tenderlove’s excellent explanation.

• If you’re deploying to Heroku, don’t bother trying the free or 1X dyno, which only gives you 512Mb of RAM. While it is enough for most small Rails applications (even with 2 or 3 Puma concurrent workers), I found that even the smallest apps can easily go above that. So you must consider at least the 2X dyno. In any server configuration, always consider more than 1Gb of RAM.

Gems

There are several gems with C extensions that just won’t work. Some of them have drop-in replacements, some don’t. You should refer to their Wiki for a list of cases.

In my small sample application - which is nothing more than a content website backed by a Postgresql database, ActiveAdmin to manage content, RMagick + Paperclip (yes, it’s an old app) to handle image upload and resizing, there was not a lot to change. The important bits of my “Gemfile” end up looking like this:

source 'https://rubygems.org'

ruby '2.3.0', :engine => 'jruby', :engine_version => '9.1.1.0'
# ruby '2.3.1'

gem 'rails', '~> 4.2.6'

gem 'devise'
gem 'haml'
gem 'puma'
gem 'rack-attack'
gem 'rack-timeout'
gem 'rakismet'

# Database
gem 'pg', platforms: :ruby
gem 'activerecord-jdbcpostgresql-adapter', platforms: :jruby

# Cache
gem 'dalli'
gem "actionpack-action_caching", github: "rails/actionpack-action_caching"

# Admin
gem 'activeadmin', github: 'activeadmin'
gem 'active_skin'

# Assets
gem 'therubyracer', platforms: :ruby
gem 'therubyrhino', platforms: :jruby

gem 'asset_sync'
gem 'jquery-ui-rails'
gem 'sass-rails'
gem 'uglifier',     '>= 1.3.0'
gem 'coffee-rails', '~> 4.0.0'
gem 'compass-rails'
gem 'sprockets', '~>2.11.0'

# Image Processing
gem 'paperclip'
gem 'fog'
gem 'rmagick', platforms: :ruby
gem 'rmagick4j', platforms: :jruby

group :test do
  gem 'shoulda-matchers', require: false
end

group :test, :development do
  gem "sqlite3", platforms: :ruby
  gem "activerecord-jdbcsqlite3-adapter", platforms: :jruby

  # Pretty printed test output
  gem 'turn', require: false
  gem 'jasmine'

  gem 'pry-rails'

  gem 'rspec'
  gem 'rspec-rails'
  gem 'capybara'
  gem 'poltergeist'
  gem 'database_cleaner', '< 1.1.0'

  gem 'letter_opener'
  gem 'dotenv-rails'
end

group :production do
  gem 'rails_12factor'
  gem 'rack-cache', require: 'rack/cache'
  gem 'memcachier'
  gem 'newrelic_rpm'
end

Notice how I paired gems for the :ruby and :jruby platforms. After doing this change and bundle install everything, I ran my Rspec suite and - fortunately - they all passed on the first run without any further changes! Your mileage will vary depending on the complexity of your application, so have your tests ready.

Puma

In the case of Puma, the configuration is a bit trickier, mine looks like this:

web_concurrency = Integer(ENV['WEB_CONCURRENCY'] || 1)
if web_concurrency > 1
  workers web_concurrency
  preload_app!
end

threads_count = Integer(ENV['RAILS_MAX_THREADS'] || 5)
threads threads_count, threads_count

rackup      DefaultRackup
port        ENV['PORT']     || 3000
environment ENV['RACK_ENV'] || 'development'

on_worker_boot do
  # Worker specific setup for Rails 4.1+
  # See: https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server#on-worker-boot
  ActiveRecord::Base.establish_connection
end

It’s a bit different from what you might find in other documentation. The important part is to turn off workers and preload_app when loading over JRuby. Otherwise it will complain and crash. On my original MRI deploy I am using a small 1X dyno and I can leave WEB_CONCURRENCY=2 and RAILS_MAX_THREADS=5 but on the JRuby deploy I set it to WEB_CONCURRENCY=1 (to turn off workers) and RAILS_MAX_THREADS=16 (because I am assuming JRuby can handle more multithreading than MRI).

Another important bit: most people still assume that MRI can’t take advantage of native parallel threads at all because of the dreaded GIL (Global Interpreter Lock), but that’s only half true. MRI Ruby can parallelize threads on I/O waits. So, if a part of your app is waiting for the database to process and return rows, for example, another thread can take over and do something else, in parallel. There is some concurrency available, even if limited, so setting a small amount of threads for Puma might help a bit.

Do not forget to set the Pool size to at least the same number of RAILS_MAX_THREADS. You can either use the config/database.yml for Rails 4.1+ or add an initializer. Follow Heroku’s documentation on how to do so.

Initial Benchmarks

So, I was able to successfully deploy a secondary JRuby version of my original MRI-based Rails app.

The original app is still in a Hobby Dyno, sized at 512Mb of RAM. The secondary app needed a Standard 1X Dyno, sized at 1Gb of RAM.

The app itself is super simple and as I’m using caching - as you always should! -, the response time is very low, on the tens of milliseconds.

I tried to use the Boom tool to benchmark the requests. I did a lot of warming up on the JRuby version, running the benchmarks multiple times and even using Loader.io for added pressure.

I am running this test:

$ boom -n 200 -c 50 http://foo-my-site/

The MRI version performs like this:

Summary:
  Total:    16.4254 secs
  Slowest:  9.0785 secs
  Fastest:  0.8362 secs
  Average:  2.6551 secs
  Requests/sec: 12.1763
  Total data:   28837306 bytes
  Size/request: 144186 bytes

Status code distribution:
  [200] 200 responses

Response time histogram:
  0.836 [1] |
  1.660 [57]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  2.485 [57]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  3.309 [33]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  4.133 [22]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  4.957 [16]    |∎∎∎∎∎∎∎∎∎∎∎
  5.782 [6] |∎∎∎∎
  6.606 [3] |∎∎
  7.430 [1] |
  8.254 [3] |∎∎
  9.079 [1] |

Latency distribution:
  10% in 1.2391 secs
  25% in 1.5910 secs
  50% in 2.1974 secs
  75% in 3.4327 secs
  90% in 4.5580 secs
  95% in 5.6727 secs
  99% in 8.1567 secs

And the JRuby version performs like this:

Summary:
  Total:    15.5784 secs
  Slowest:  7.4106 secs
  Fastest:  0.5770 secs
  Average:  2.3224 secs
  Requests/sec: 12.8383
  Total data:   28848475 bytes
  Size/request: 144242 bytes

Status code distribution:
  [200] 200 responses

Response time histogram:
  0.577 [1] |
  1.260 [23]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  1.944 [62]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  2.627 [51]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  3.310 [24]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  3.994 [26]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  4.677 [8] |∎∎∎∎∎
  5.361 [1] |
  6.044 [0] |
  6.727 [1] |
  7.411 [3] |∎

Latency distribution:
  10% in 1.1599 secs
  25% in 1.5154 secs
  50% in 2.0781 secs
  75% in 2.8909 secs
  90% in 3.7409 secs
  95% in 4.2556 secs
  99% in 6.7685 secs

In general, I’d say that they are around the same. As this is not a particularly CPU-intensive processing, and most of the time is spent going through the Rails stack and hitting Memcachier to pull back the same content all the time, maybe it’s only fair to expect similar results.

On the other hand, I’m not sure I’m using the tool in the best way possible. The log says something like this for every request:

source=rack-timeout id=c8ad5f0c-b5c1-47ec-b88b-3fc597ab01dc wait=29ms timeout=20000ms state=ready
Started GET "/" for XXX.35.10.XXX at 2016-06-03 18:54:34 +0000
Processing by HomeController#home as HTML
Read fragment views/radiant-XXXX-XXXXX.herokuapp.com/en (6.0ms)
Completed 200 OK in 8ms (ActiveRecord: 0.0ms)
source=rack-timeout id=c8ad5f0c-b5c1-47ec-b88b-3fc597ab01dc wait=29ms timeout=20000ms service=19ms state=completed
source=rack-timeout id=a5389dc4-9a1a-46b7-a1e5-53f334ca0941 wait=35ms timeout=20000ms state=ready

Started GET "/" for XXX.35.10.XXX at 2016-06-03 18:54:36 +0000
Processing by HomeController#home as HTML
Read fragment views/radiant-XXXX-XXXXX.herokuapp.com/en (6.0ms)
Completed 200 OK in 9ms (ActiveRecord: 0.0ms)
source=rack-timeout id=a5389dc4-9a1a-46b7-a1e5-53f334ca0941 wait=35ms timeout=20000ms service=21ms state=completed
at=info method=GET path="/" host=radiant-XXXX-XXXXX.herokuapp.com request_id=a5389dc4-9a1a-46b7-a1e5-53f334ca0941 fwd="XXX.35.10.XXX" dyno=web.1 connect=1ms service=38ms status=200 bytes=144608

The times reported by the Boom tool are much larger (2 seconds?) than the processing times in the logs (10ms?). Even considering some overhead for the router and so on, it’s still a big difference. I wonder if requests are being queued for too long because the app isn’t able to respond to more of the concurrent requests.

The amount of requests divided by the number of concurrent connections will bring the overall performance and throughput down if you increase it too much, I wasn’t able to go much above 14 rpm with this configuration, though.

If you have more experience with http benchmarking and you can spot something wrong I am doing here, please let me know in the comments section below.

Conclusion

JRuby continues to evolve, and you might benefit if you already have a large set of Dynos or large servers around. I wouldn’t recommend it for small to medium applications.

I’ve seen many orders of magnitude improvements in specific use cases (I believe it was a very high traffic API endpoint). This particular case I tested is probably not its sweet spot and changing from MRI to JRuby didn’t give me too much advantage, so in this case I would recommend sticking to MRI.

Startup time is still an issue. There is an entry in their Wiki giving some recommendations, but even in the Heroku deploy I ended up having R10 errors (Boot Timeout) every once in a while for this small app.

I didn’t try increasing the dynos to the Performance tier introduced last year. I would bet that JRuby would be better at those and more able to leverage the extra power of having from 2.5GB up to 14GB if you have really big traffic (on the order of thousands or tens of thousands of requests per minute). With MRI the recommendation would be using small-ish dynos (2X or at most Performance-M dynos) and scale horizontally. With JRuby you could have fewer dynos with larger sizes (Performance-L, for example). Again, depends on your case.

Don’t take the benchmarks above as “facts” to generalize everywhere, they are just there to give you a notion of a specific use case of mine. Your mileage will vary, so you must test it out yourself.

Another use case (that I did not test) is, instead of just “porting” an MRI app to JRuby, leveraging JRuby’s unique strengths as this post from Heroku explains, in the case of using JRuby with Ratpack, for example.

All in all, JRuby is still a great project to experiment with. MRI itself has come a long way as well and 2.3.1 is quite decent. Most of the time it’s down to your overall architecture choices, and the language is just one piece of the puzzle. If you didn’t try it yet, you definitely should. It “just works”.

I’ll say that Erlang OTP and its internals intrigue me more than just the novelty of the language syntax. I believe that if you want to build highly distributed and also highly reliable systems you will either use Erlang (through Elixir) or you will end up replicating most of what Erlang already has with OTP.

This is true if you see that there are many Java/Scala-based distributed systems such as Hadoop, Spark, Cassandra, and they all could have benefitted from Erlang’s OTP architecture. On the other hand, if you follow Cassandra you will see that even Java still has a hard time competing with C++ if you compare the clone called ScyllaDB.

I believe Elixir (perhaps with HiPE) can compete in the same league as Java for distributed systems, avoiding people leaving the Erlang platform for C or Java as happened with CouchDB back in 2012 because of lack of interest and exotic syntax. And because Java already has a big ecosystem, Clojure is in the game primarily because it can interface directly with it.

I think Go is also a Java contender but for other use cases, particularly in the systems tools space where you have lots of C/C++ and also glue code in Perl and Python. An obvious example being Docker orchestration. Yes, you can do microservices, crawlers and other stuff but I am not so keen to build big “applications” in it, although there is nothing preventing it. Again, it’s just a personal opinion.

Rust is a big contender to low-level systems development. I think it can replace C in many use cases and be used to implement libraries and components to be used by other languages, with the added benefits of a modern language that tries to avoid security hazards such as the recent Heartbleed debacle. But because of its “Borrow system” I find it extremely bureaucratic, especially if you’re coming from highly dynamic languages such as Ruby, Python or even Swift or Elixir. I’m not saying it’s a bad thing, just something that takes way more time to become comfortable with than I expected.

Crystal is something between Rust and Go. LLVM is something you must have on your radar, because of Apple’s support towards Swift it’s better than ever. Crystal is similar to RubyMotion in terms of both being similar to Ruby but not fully compatible and both being front-end parsers to LLVM. RubyMotion, on the other hand, is closer to Ruby and can even use some Ruby libraries almost without changes.

You do have to take your hats off to Ary Borenszweig and his contributors. It’s very impressive how far they got in such a short period of time and without having deep pockets from Mozilla, Apple or Google.

Limitations

Crystal borrows heavily from Ruby but it’s not a goal to reach any level of compatibility. It’s a strong, statically typed language with Type Inference. It does not have any runtime component such as RubyMotion or Swift, so it has no notion of introspection or reflection over objects.

Mainly, you have neither “#eval” nor “#send”. The lack of runtime evaluation is not so bad as you do have compilation-time AST manipulation through a good enough system of Macros (more on that later).

But the lack of “#send” does hurt a bit. It’s the one thing that makes Crystal further away from Ruby’s dynamic object flexibility. But it’s also understandable why it’s not a priority to have it.

As of version 0.17.0, Crystal has one huge limitation: it’s using a Boehm-Demers-Weiser conservative garbage collector.

@asterite @headius I assumed as much... the Boehm GC, although handy for starting a new language, will only get you so far

— Jason Frey (@Fryguy9) May 29, 2016

The language is currently implemented as a single-threaded process. This means that you probably can’t max out all CPUs of your machine with just a single process. Although I may be wrong here, it’s just my first impressions.

Charles Nutter (from JRuby fame) gives a warning about this:

@fryguy9 From their site: https://t.co/uOjIT0c8ji It does say something about "when" parallelism happens. Risky to start single-threaded.

— Charles Nutter (@headius) May 29, 2016

It’s a double-edged sword. Using a “generic” plug-and-play GC such as Boehm - which is not a bad thing in itself, but it’s possibly not nearly as powerful as the JVM’s own set of high performance GC such as the brand new G1GC.

Making the language single-threaded by default also means that the entire standard library and the whole ecosystem are built assuming there is no parallelism. So there are no race-conditions and hence, no usage of proper synchronization anywhere. If one day they decide to add multi-threading, all existing code will not be thread-safe. This is probably the meaning of Charles’ warning.

I don’t think it’s a pressing problem though. You probably won’t have a highly parallel system built with Crystal but maybe it’s not the use case. In that case you should be using Elixir + OTP, Scala + Akka, for example.

Concurrency

Not having access to native Threads doesn’t mean that you can’t have Concurrency. Concurrency is not Parallelism. Since version 0.7.0 they have added support for non-blocking I/O and also Fibers and Channels.

In a nutshell, a Fiber is a special kind of coroutine. It’s a “piece of execution” that can “pause” itself and yield control of the execution back to its caller. It can pause itself but it can’t be paused from the outside like Erlang’s Scheduler can pause its running processes, for example.

In this case we can have a “Fiber Scheduler” of sorts, that can “resume” other Fibers. It is currently single-threaded, of course. And it works as an Event Loop. Whenever you have a non-blocking I/O operation waiting for file reading, network packet sending and stuff like that, it can yield control back to other fibers to resume work.

This makes it work more or less like Node.js. Javascript is also a single-threaded language and Node.js is basically an implementation of an event loop. Functions declare other anonymous functions as callbacks and it works based on the I/O triggers to callback those functions on every tick of the reactor.

On top of that you can add “Channels” as popularized by Google’s Go language. You can start as many channels as you like. Then you can spawn Fibers in the Scheduler. Fibers can execute and keep sending messages through the Channels. Execution control is yielded to whoever is expecting to receive from the same Channels. Once one of them receives and executes, control is sent back to the Scheduler to allow other spawned Fibers to execute, and they can keep “pinging” and “ponging” like this.

Speaking of ping-pong, you have this snippet in the “Go by Example” site:

package main
import "fmt"
func ping(pings chan<- string, msg string) {
    pings <- msg
}
func pong(pings <-chan string, pongs chan<- string) {
    msg := <-pings
    pongs <- msg
}
func main() {
    pings := make(chan string, 1)
    pongs := make(chan string, 1)
    ping(pings, "passed message")
    pong(pings, pongs)
    fmt.Println(<-pongs)
}

And this is mostly the same thing but implemented in Crystal:

def ping(pings, message)
  pings.send message
end

def pong(pings, pongs)
  message = pings.receive
  pongs.send message
end

pings = Channel(String).new
pongs = Channel(String).new
spawn ping pings, "passed message"
spawn pong pings, pongs
puts pongs.receive
# => "passed message"

Same thing, but more pleasant to my eyes, again a personal opinion - which is expected if you’re also a Rubyist.

So Crystal has Node.js/Javascript-like Event Loop in the form of a Fiber Scheduler and a Go-like Channel/CSP mechanism to coordinate concurrent processing. It’s a “cooperative multitasking” mechanism. It’s not good if you have CPU intensive processing, such as heavy number crunching, data processing. It works best for I/O intensive processing.

If you have a Fiber doing super heavy processing, it will block the Scheduler, as is also true for Node.js. The current implementation of the Scheduler is bound to a single native thread so it can’t use other available native threads of the system for now.

Because the Channel implementation is quite new, it’s not used throughout the standard library. But the standard implementation for HTTP::Server uses Fibers. And because it is compiled down to a native binary it’s way faster than Javascript, as this “fast-http-server” shows:

• fast-http-server (Crystal) 18348.47rpm at 8.67ms avg. response time
• http-server (Node.js) 2105.55rpm at 47.92ms avg. response time
• SimpleHTTPServer (Python) 785.14rpm at 1.91ms avg. response time

We’re talking about an order of magnitude of 8 times faster than Node.js and more than 20 times faster than Python.

As usual, you shouldn’t blindly trust synthetic benchmarks but depending on which kind of processing you decide to compare against, you will see Crystal being pretty on par with Rust, Go, Swift, D.

Type Inference

Crystal is a statically typed language. But it won’t require you to declare every single type beforehand. It’s quite smart in its Type Inference so if you just type:

a = 1
b = "HELO"

It will know that “a” is an “Int32” and that “b” is a “String”. By the way, contrary to Ruby, all Strings must be double-quoted. But it becomes particularly more complicated when you’re dealing with complicated data structures such as JSON.

In Ruby I can just parse a JSON String and immediately explore its structure like this:

parsed_json = JSON.parse(response.body)["files"].first["id"]

I can’t do this in Crystal, instead the recommended approach is to declare a schema-like structure like this:

class FilesResponse
  JSON.mapping({
    ok: {type: Bool},
    files: {type: Array(FilesItemResponse)}
  })
end
class FilesItemResponse
  JSON.mapping({
    id: {type: String},
    name: {type: String},
    filetype: {type: String},
    created: {type: Time, converter: Time::EpochConverter}
    timestamp: {type: Time, converter: Time::EpochConverter}
    permalink: {type: String}
  })
end
...
# parsed_json = JSON.parse(response.body)["files"].first["id"]
parsed_json = FilesResponse.from_json(@body).files.first.id

I am very used to Ruby’s duck typing and the ability to query objects in runtime. It’s just not the same in Crystal and it can be quite tedious to change your mindset to think about types beforehand. The compiler will scream a hell of a lot during your development cycle until you become acquainted with this concept.

I did an experiment with JSON parsing and the result is a project I called “cr_slack_cleanup” which showcases both this idea of JSON Schemas as well as Fibers and Channels as I explained in the previous section.

Update: after I posted this article @LuisLavena stepped in to correct me: you can do it like Ruby, without schemas:

@AkitaOnRails @CrystalLanguage re: JSON, you can parse it kinda-like Ruby, see JSON::Any: https://t.co/MD7PYy5AVH

— Luis Lavena (@luislavena) June 1, 2016

And it will be like this:

require "json"

data = <<-JSON
  {
    "files": [
        {
          "id": 1,
          "name": "photo.jpg"
        },
        {
          "id": 99,
          "name": "another.jpg"
        }
    ]
  }
  JSON

obj = JSON.parse(data)

obj["files"].each do |item|
  puts item["id"].as_i
end

So yeah, JSON parsing can be almost identical to what you would do with Ruby, although he also recommends Schemas as it’s a bit faster anyway.

Macros

The last important feature worth mentioning is the presence of Macros. As you don’t have control over your code in runtime, you don’t have “eval”, you don’t have “send”, then you would not have any way to do proper metaprogramming.

Macros bring back some of Ruby’s metaprogramming. For example, in Ruby, when you include a Module to a Class, there is the “included?” hook in the Module where you can add some metaprogramming such as “class_eval” stuff.

In Crystal, a Module does have an “included” hook but now it’s a Macro. You can do something like this:

module Foo
  macro included
    {% if @type.name == "Bar" %}
    def self.hello
      puts "Bar"
    end
    {% else %}
    def self.hello
      puts "HELLO"
    end
    {% end %}
  end
end

module Bar
  include Foo
end

module Something
  include Foo
end

Bar.hello # => "Bar"
Something.hello # => "HELLO"

It feels like having “ERB” templates but for code. A macro is code building code in compile time. In the resulting AST it’s as if you wrote the boring repetitive code yourself. The compiled native binary doesn’t care. If you’re from C it’s like pre-processing but having control of an AST tree of Nodes instead of just manipulating the source code. You can even have something akin to the venerable “#method_missing”.

class Foo
  macro method_missing(name, args, block)
    {{pp name}}
  end
end

Foo.new.hello_world
# => name = "hello_world"
Foo.new.bla_bla("bla")
# => name = "bla_bla"

Any macro function can receive an AST as the argument and you can manipulate this AST in any way as you see fit. For example:

module Foo
  macro teste(ast)
    puts {{ast.stringify}}
  end
end

Foo.teste "Hello World".split(" ").join(" - ")
# => ("Hello World".split(" ")).join(" - ")

In this example you just get a normalized version of the code you passed as argument. And the code won’t be “executed” at that location.

So, you can define new methods and call different versions of methods depending on the combination of elements that you get from the AST. One example of what is possible is this experiment from developer Oleksii that I bundled in this small project I called “cr_chainable_methods” and we can build code that is very different from either Crystal or Ruby:

result = pipe "Hello World"
  .>> Foo.split_words
  .>> Foo.append_message("Bar")
  .>> Bar.add_something
  .>> Foo.join
  .>> unwrap

(Yes, a subset of what a real Elixir Pipe can do.)

Crystal Shards

Finally, one aspect I like about Crystal is that it has task management and dependency management already figured out. No pointless discussions on which implementation of packages and dependencies we “should” be using.

You can just write a simple script into a file such as “test.cr” and run this file like this:

crystal test.cr

Or, you can build and compile to a native binary before executing like this:

crystal build test.cr --release
./test

And you can also start in a proper project structure and development lifecycle like this:

crystal init app test
cd test
crystal build src/test.cr --release
./test
git add .
git commit -m "initial commit"
git push -u origin master

It sets up a proper project directory structure, a Git repository is pre-initialized for you (why would you be programming without Git!?) with proper “.gitignore” file.

You will also find a “shard.yml” file where you can declare your application or library name, version, author, and also its dependencies like this:

name: kemal
version: 0.12.0

dependencies:
  radix:
    github: luislavena/radix
    version: 0.3.0
  kilt:
    github: jeromegn/kilt
    version: 0.3.3
author:
  - Serdar Dogruyol <dogruyolserdar@gmail.com>

This example is from the Kemal web framework by Serdar. It depends on the Radix and Kilt libraries. You must run crystal deps and it will fetch from Github before you can compile.

More than that: every project receives a proper “spec” directory where you can write Rspec-like tests and you can run them using crystal spec. And the spec runner results will look like this:

$ crystal spec
........................................................

Finished in 8.28 milliseconds
56 examples, 0 failures, 0 errors, 0 pending

All built-in, no external libraries needed.

And 8 milliseconds to run the entire test suite of a micro web framework? Not too shabby. If you add up start up time for the executable, it still is around 1 second of total running time. Pretty impressive.

Conclusion

Crystal is not meant to be a Ruby replacement like JRuby. So it will never run existing and complicated Ruby projects such as Ruby on Rails or even Sinatra for that matter.

To me Crystal fills a gap in some use cases where I would use Rust or Go. It has the raw performance of an LLVM-optimized native binary, a reasonably fast Go-like concurrency system (and much faster and less resource intensive than Node.js), and the added benefit of a much more enjoyable Ruby-like syntax set.

You can even start experimenting with Crystal to build Ruby native extensions without the need of FFI and without having to write C. There is an attempted port of ActiveSupport as a native extension to MRI Ruby, written as a proof of concept in Crystal.

If you’re a Rubyist you will find yourself pretty much at home with most of the standard library. You will miss a thing or two but most of the time it will feel like Ruby. The official Guide is good enough to get started and there is a comprehensive API documentation just to check if that favorite Ruby API is there or not, and if not what the replacements are. Most of the time you will think “hmm, in Ruby I would use Enumerator#each … and yes, sure enough, there is Enumerator#each as expected.” The API documentation could use some more love, so if you want to contribute this is a good place to start.

If you want to check out a curated list of Crystal libraries and applications, go straight to “Awesome Crystal”. You can also see the more dynamic “Trending Projects in Github”.

For the most part, your proficiency in Ruby will pay off to build small systems where raw performance and concurrency are really necessary. This is exactly the use case for Mike Perham’s port of Sidekiq.cr. Say you have gigabytes of files to crunch. Or you have thousands of external websites to crawl, parse and organize. Or you have petabytes of data in your database waiting to be number crunched. Those are tasks that MRI Ruby won’t help with but Crystal may well be the quick answer.

You can even get one of the many micro web frameworks such as Kemal and deploy your HTTP API providers directly to Heroku now. This will give you Go-levels of performance, and this is very compelling for me as I really also dislike Go’s archaic smelling syntax.

So, Rust, Go, Javascript, all very fast, but with very questionable syntaxes and not so enjoyable. They are way more mature, and their ecosystems are much larger. But unless I really need to, I’d rather choose Crystal for the use cases I described above, and I think you will find some good use for it too.

Update: after I posted, the creator of Crystal, @Asterite had a few things to add as well:

@AkitaOnRails @waj so it's likely that we'll have something similar. We aren't there yet because we are not Google, Mozilla nor Apple

— Ary Borenszweig (@asterite) May 31, 2016

First of all, it’s inevitable but I still hate dramatic headlines, even though I write like this myself sometimes. “My time with Rails is up” like it’s saying “And you should leave Rails now too if you’re smart”. I dismissed the article entirely because of that. I was about to post a counter-rant to that without reading it properly, but now that I have, I wrote this new article from scratch.

Solnic is right: Ruby, by itself, has little to no future alone. Rails is a real monopoly in this community and most OSS projects are targeting Rails. And yes, Rails does encourage some bad practices and anti-patterns. This can be very discouraging to many OSS contributors, especially because changing the direction of a huge Elephant takes humongous effort.

To make it clear, the dialectic technical arguments he makes are all true. But half the article - as he stated himself - is just rant, pure rhetoric. And I think it would be good to balance it out, which is what I will try to do in this post.

Accepting Reality

The reality is this: Rails is tailor-made for Basecamp.

We all know that, or we should. Basecamp-like apps are not too difficult to make, at least in terms of architecture. You don’t need fancy super performant languages with super duper highly concurrent and parallel primitives. Also a reality is that 80% of the web applications are Basecamp-like (disclosure: my feelings for years of experience in consulting). Which is why Rails has endured so far.

It’s like content management systems, or CMS. Most of them are blog-like systems. And for that you should go ahead and install Wordpress. The very same arguments made against Rails can be made against Wordpress. And you should never, ever, tweak Wordpress to be anything but a blog-system. Try to make it into an e-commerce for high traffic, and you will suffer.

To make it clear: Wordpress has one of the most offensive source codes I’ve ever seen. I would hate having to maintain that codebase. I’m sorry if anyone from the Wordpress base of contributors is reading this, I say this without malevolence. And you know what? Possibly half of all CMSs in the world are Wordpress. Over a million websites.

Then you have the case for Magento2. Big rewrite over the original Magento, written using all the dreaded Zend stuff that everybody else dislikes. But it’s huge. If you need a fast turn-key solution for e-commerce, look no further.

Do Wordpress plugins work with Magento? Nope. They are 2 fragmented, independent and isolated communities. But they both generate a lot of revenue, which is what covers the cost of redundancy between them. And this is not even counting Drupal, Joomla. PHP is one big ocean of disconnected islands. Countries with severe immigration laws.

Fragmentation is no stranger to the Javascript world. But it’s a different kind of value generation. Facebook, Google, Microsoft, they all want to be the thought-leaders in the fast evolving Millenials generation. It’s a long term strategy. And one of the elements of this game is the Browser. But not only in terms of Chrome vs Firefox vs IE, but also on how applications are implemented.

Facebook came up with React. Google came up with Polymer and Angular. The Node guys went through a power struggle with Joyent which almost resulted in further fragmentation but they settled for the Node Foundation.

Apple went all on war against Adobe’s Flash and then only now Google is turning them off in Chrome, but they are all looting on the consequences for all the attention it brings in the Web Development communities.

Apple wants native to succeed and Swift to be the one language to lead it all. Google has conflicting strategies because they want native Instant Apps to succeed but if it fails, plan B continues to be for them to dominate HTML5/CSS3 based web apps with Angular. Facebook doesn’t want to have its fate decided by the power struggle between Apple and Google.

It’s a complex power struggle unfolding, and you can see it has little to do with technical prowess or value generation. It’s about ego, influence and power. Very fitting for the YouTuber generation. And the web technologies are being held hostage in this siege, if you haven’t noticed.

Then there is the issue that Ruby’s future is now tightly coupled with Rails. This is a reality and if you’re a Rubyist who doesn’t like Rails, I feel bad for you. But not so much. For example, if Hanami is interesting I believe at least one company invested on it. If no one is using it, then it doesn’t matter how technically superior it is. If Rom.rb is great someone should be using it, otherwise what’s the point? Why create a technical marvel that no one wants? But if there is at least one company using it, it’s enough reason to keep going, regardless of what happens to Rails or what DHH says or does.

People think that because something is “technically superior” everybody else should blindly adopt. But this is not how the market works.

Of all the cosmic-size events going on out there, I really don’t sweat it that much if Ruby stays tied to Rails. What would it do without it?

All communities face fragmentation at some point. It’s very difficult and expensive to maintain cohesiveness for a long time. The only community that I think achieved that through sheer force of regulation is Microsoft’s .NET stack. It doesn’t mean that there was no pressure from the outside. Rails itself played a big role into influencing the move from old-ASP.NET to ASP.NET MVC. Now they finally acquired Xamarin before .NET could steer out of their control in open source platforms they don’t control.

Ruby on Rails is the only other “cohesive” community I’ve seen. With the upside that Basecamp doesn’t need hundreds of thousands of developers to exist. A niche market would suffice, enough for the framework to evolve gradually through OSS processes. Which is why I always question the history and origins of tools and technologies to make my decisions on where to use them, not just technical prowess.

Rails works because it doesn’t have to play politics with Apple, Facebook, Microsoft, Google or any other committees (by the way, by default, I never trust committees). Those who depend on Rails will do the house-keeping, directly. Heroku, Github, New Relic, Shopify, and many talented developers.

3 Laws of Market Reality

1. It’s easy to over-analyse something after the fact. 10 years down the road, I can easily trace back an optimal path, avoiding all booby traps and obstacles along the way. Doesn’t make me any genius, just shows that I can connect the - now clearly visible - dots.

2. No solution implementation is perfect. If it actually solves a real problem it’s bound to be imperfect. If it solves a real problem in a fast paced changing market, the more imperfect.

3. Either you build tools because your core business applications depend on it or you build tools to sell. The former will usually be better - in terms of market-fit - than the latter. So if you have to blindly choose, go with the former.

So, first of all, I will always prefer tools that solve a real problem made by those that actually depend on them. Otherwise, the shoemaker’s son will end up barefoot. Case in point: I will definitely use Angular, if I have to. But I would never, ever, begin a new business that depends solely on Angular to survive. Why? Because Google doesn’t need it. It didn’t blink to give up GWT, it didn’t have to think twice to decide to rewrite Angular 2 in an incompatible way to Angular 1, and so on.

Second of all, I will always see what other external factors will influence the fate of the technology. Imagine that I spent a whole lot of time writing scripts, libraries, tools for Grunt. Then people decide it’s bad. Now Gulp is the better choice - and you will find plenty of technical reasons. Now you invest a lot of time writing everything you had for Grunt to Gulp. Then, for plenty of other reasons people decide that Webpack is the best choice. And there you go again. NIH (Not-invented-here) syndrome galore.

This is clearly a small bubble. It’s tulips all over again. There are too many big players (Facebook, Google, Apple, Microsoft, Mozilla, etc) with big pockets, plenty of time and resources. This is how an experimental lab works in public. Lots and lots of experimental alternatives and several businesses blindly choosing depending on the best sales pitch of the week.

Sometimes this kind of situation makes the monopoly of ASP.NET on the Microsoft camp and the Rails monopoly on the Ruby camp seem innocuous. And yes, I compared Rails to .NET here. They are the 2 most comparable stacks. Possibly comparable to Spring faction in the Java camp. If you remember the history, Spring was like Rails in the Java community, rising up against the humongous complexity of the official J2EE stack back in 2002. And then Spring itself became the new J2EE-like behemoth to beat.

This is a millennia-old dilemma:

“You either die a hero, or live long enough to see yourself become the villain.”

Why is Rails a problem now?

As I said before, I agree to almost every technical problem that Solnic outlined.

Rails is indeed the brainchild of David Hansson (DHH). DHH is a person, with a very big ego, and a business to run. You can’t expect any person to be reasonable all the time, especially one who pushed something up from zero, both a business and a technology platform.

When it started, people defected from Java, .NET, PHP even Python, in droves and they all acknowledged how interesting Rails was compared to J2EE, ASP.NET, Plone. It offered not only productivity but technical enjoyment. We were discussing the wondrous world of dynamic languages, open classes, injecting behavior on the fly (aka monkey patching), we all stood up and applauded dropping all unnecessary abstractions.

We could not have enough of our Ruby fix, spitting out all the Perl-like magic we would accomplish in a language that didn’t feel ugly as PHP or bureaucratic like Java or C#. And they all laughed.

The Golden Age from 2004 to 2006 saw a never ending stream of celebratory masturbation of Perl-like coding prowess. We learned Ruby through Why, the Lucky Stiff most obscure black magic, remember that? It was everything but clean and modular architectures.

Then we entered the Silver Age, from 2007 to around 2011. Rails actually went too far, too fast. Suddenly we saw big companies popping up from everywhere! Twitter, Github, Engine Yard, Heroku, Zendesk, Airbnb, everybody drank the Rails Kool-Aid. The opportunity was there to offer something for the enterprise. Merb was ahead of its time and it was pitched the wrong way. I do think that confronting the almighty Rails upfront, at that point, was not smart. You should expect overreaction and it did come and it was a swift blow. I will be honest and say that I was very apprehensive in 2009 and 2010 to see if the Rails 3 pseudo-rewrite, pseudo-Merb-merge would actually come through.

2011 to 2016 was the Bronze Age, a bittersweet period. That’s because many new languages have emerged and finally reached “usable” state. From JS’s V8 and Node.js, to Rust, to Clojure, to Elixir, and even some gems from the past started to get attention, such as Scala and even Haskell. The most important change of all: 2010 saw the advent of the Walled Garden App Store, commercially available native applications for smartphone and tablets. Forget web development: mobile was getting it all.

And that’s when all the big companies started to show their deep pockets. Apple releases Swift. Google released Dart, Angular, Go. Microsoft released Typescript, ASP.NET MVC then vNext and had its hands full working on Windows 10. Facebook entered the game late by releasing React and then React Native.

Rails can now be considered a “problem” for the very same reasons that made it popular in the first place. And this is bound to happen to any technology.

But you can’t change the architecture of Rails too much, otherwise you risk breaking down very big chunks of the projects that are deployed in production now. And when someone has to rewrite big chunks of a project you might as well consider rewriting it in something else entirely.

Many people are doing exactly that, especially for APIs and web applications implemented as SPAs talking to APIs. The APIs can be written in Go, Elixir, Scala and avoid Ruby altogether. You lose the fast turn around of the Rails ecosystem, but if you can afford it (you’re a Unicorn startup with deep pockets), why not?

But again, for the 90% of small to medium projects out there, you can still get the best punch for the buck using Rails and all the libraries available for Rails. It’s like saying, if you want to build a blog, go for Wordpress and you will get the best benefit for the limited resources you have. Don’t try to be fancy and write an SPA blog using Go APIs with React from scratch. Feasible, but not worth it.

If you’re a medium company already using Rails for some time, first of all make sure you adhere to basic best practices. Add tests and specs if you haven’t already. Steadily refactor code, remove duplication, upgrade gems. Then you should consider adding an abstraction layer such as Trailblazer and possibly consider componentizing parts of your application as Rails Engines or removing those parts into separated Rails-API applications to be consumed, if possible. But do one step at a time, as needed.

One rarely benefits from big bang rewrites from scratch.

Conclusion

So yes, for developers such as Solnic the Rails community is probably a frustrating place to be. But it’s also an addiction that’s hard to drop because Rails is so much larger than any other competitor in any other new and fancy platform, you always feel bad for being the underdog.

Rails went from underdog to mainstream in 5 years. Possibly the fastest growth any web framework ever achieved. The life of a web developer from 1995 to 2003 was not particularly interesting. Rails did a lot to improve it. And if anyone thinks they can do better, just do it. What’s the point of writing about Rails? More than just code competing against code, results should compete against results.

Active Record’s architecture will indeed hurt hard maybe 10% of the cases out there. Active Support does not have a better alternative so far, and just removing it won’t bring anything of value for the end user. Replacing a big component such as Active Record with something “better” such as an improved version of DataMapper or Rom.rb as the default again won’t bring so much value, especially for the hundreds of applications out there. You’re telling everybody to just rewrite everything. And if I had to rewrite, I would definitely do a new application using Rails + Trailblazer or go straight to Hanami. But most people would decide in favor of ditching Ruby altogether.

Could Active Record be better? Sure! We have old Data Mapper, Sequel and ROM.rb to prove it. But the real question is: could it be done better back in 2004 when it was first created? I don’t think so. Now even the creator of DataMapper advocates for No-ORM. In 2004 “NoSQL” wasn’t even a thing. The best we had back then was Hibernate, way before JPA! And for all intents and purposes, Active Record still does much better than average. But if you’re big, you should be careful. That’s all.

The other communities will face the same predicaments we are now facing in the Rails community. It’s inevitable. Everything is so much easier when you have a small community that even if you break things it won’t be too bad. It’s much harder to maintain something that actually became big beyond your most optimistic scenarios.

I do understand the conservative approach DHH is taking by not making big disruptions. If this is something he is doing because he believes in conservative moves or because he doesn’t understand better architectural options is not up to me to judge, but it’s a valid move that will alienate advanced developers like Solnic but still allow for beginners to jump right into it without worrying too much right now about too many abstractions.

Update: DHH commented on this section later:

@akitaonrails It's not out of conservatism. But a difference of opinion and values. I love Active Record. Love callbacks.

— DHH (@dhh) May 24, 2016

@akitaonrails Will everyone love the same techniques and methods as me? Of course not (see RSpec for just one popular example!). That's fine

— DHH (@dhh) May 24, 2016

People forget that abstractions are very nice for advanced developers who have suffered the lack of them. But beginners will always suffer if presented with too many architectural choices upfront. Understanding GoF Design Patterns, Data-Driven Design, SOLID, Enterprise Architectures, etc is very overwhelming. Experienced people often forget the learning curve when they were themselves beginners, and at that time Rails was so attractive, so sexy. Remember that feeling? Of accomplishment in the face of the knowledge that some nice witch left super useful black magic behind?

Rails has won for its simplicity for beginners, having a “rails”-like guidance for experienced people as well, and somewhat acceptable flexibility for more advanced developers. Will it be able to last another 10 years in face of the many smaller alternatives out there trying to recreate everything from scratch? Time will tell.

I think it would be a good fit to finish with Bob Dylan:

Come gather ‘round people

Wherever you roam

And admit that the waters

Around you have grown

And accept it that soon

You’ll be drenched to the bone.

If your time to you

Is worth savin’

Then you better start swimmin'

Or you’ll sink like a stone

For the times they are a-changin'.

…

If you don’t know what the “Pipe Operator” is in Elixir, take the following code:

Enum.sum(Enum.filter(Enum.map(1..100_000, &(&1 * 3)), odd?))

It’s ugly, we all know that. In an Object Oriented language like Ruby we would do something like this:

(1..100_000).
  map { |i| i * 3 }.
  select(&:odd?).
  reduce(&:+)

But Elixir does not have objects, only functions, so how can we code more elegantly? The solution came up in the form of the so called “Pipe Operator” which takes the last returning value and passes it through as the first argument of the next function call, like this:

1..100_000
  |> Stream.map(&(&1 * 3))
  |> Stream.filter(odd?)
  |> Enum.sum

So Ruby and Elixir “feel” the same when we are able to “chain” methods. In the Ruby world we don’t have the “need” for an operator like that. But it would be nice to have a mechanism that we could use to make our code more expressive, or more testable, or more readable. For example, what if we could write something like this:

(1..100_000).
  multiple_each_element_by_three.
  filter_out_odd_elements.
  give_the_sum_of_all_elements

Of course, this is a very constrained example with really bad method naming. But if we look at Mo Lawson’s article I linked above, it becomes more interesting:

keywords
  .map { |kw| kw.gsub(/(^[^[:alpha:]]+|[^[:alpha:]]+$)/, '') }
  .map { |kw| LegacySpanishCorrector.new.correct(kw) }
  .map { |kw| kw.gsub(/[^[:alpha:]\d'_\-\/]/, '') }
  .reject { |kw| STOP_WORDS.include?(kw) }

Ruby allows us to chain Enumerable methods one after the other, transforming the initial keywords list into “something” that is very difficult to infer just by looking at this code.

What about this other version?

class KeywordNormalizer
  def call(keywords)
    Collection.new(keywords)
      .strip_outer_punctuation
      .fix_spanish
      .clean_inner_punctuation
      .remove_stop_words
      .to_a
  end

  # ...
end

This is where he ends up by the end of his article: much more readable and each new isolated method is unit testable, resulting in more robust code.

The whole idea of this post is to introduce you to my new little gem “Chainable Methods”. The source code is on Github, as usual, so please contribute.

My gem will allow you to write Lawson’s last code like this:

KeywordNormalizer
  .chain_from(keywords)
  .strip_outer_punctuation
  .fix_spanish
  .clean_inner_punctuation
  .remove_stop_words
  .to_a
  .unwrap

You add the chainable_methods to your Gemfile as usual (you know the drill), then you can write Lawson’s module like this:

module KeywordNormalizer
  extend ChainableMethods
  
  def self.strip_outer_punctuation(array)
    array.map { |el| el.gsub(/(^[^[:alpha:]]+|[^[:alpha:]]+$)/, '') }
  end
  def self.fix_spanish(array)
    array.map { |el| LegacySpanishCorrector.new.correct(el) }
  end
  def self.clean_inner_punctuation(array)
    array.map { |el| el.gsub(/[^[:alpha:]\d'_\-\/]/, '') }
  end
  def self.remove_stop_words(array)
    array.reject { |el| STOP_WORDS.include?(el) }
  end
end

And that’s it, now you can chain everything like I showed previously. The pattern is:

1. Write a Module with class-level methods that receive at least one argument and extend the ‘ChainableMethods’ module:

module MyModule
  extend ChainableMethods

  def self.method_a(argument1)
    # ...
  end

  def self.method_b(argument1, argument2)
    # ...
  end

  def self.method_c(argument1)
    # yield a block passing the received argument
  end
end

2. Wrap an initial state that will be passed to the first method as its first argument:

some_initial_state = "Hello World"
MyModule
  .chain_from(some_initial_state)
  # ...

3. Chain as many methods from the module or methods that the returning state recognizes:

MyModule
  .chain_from(some_initial_state)
  .upcase
  .method_a
  .method_b(argument2)
  .method_c { |foo| foo }
  .split(" ")
  .join(", ")
  .unwrap

Notice that we do not need to pass the first argument to the methods within the ‘MyModule’ module, it will get the result from the last call automatically.

4. Do not forget to call #unwrap as the last call to get the last result from the chain.

An Experiment

And that’s it! I isolated this behavior only into modules that explicitly extend the ‘ChainableMethods’ module instead of automagically enabling it at the BasicObject level as many would initially think, because we don’t want a global ‘method_missing’ dangling around unchecked.

This behavior makes use of ‘method_missing’ so it’s not going to be fast in a synthetic benchmark against a direct method call, for obvious reasons. The purpose is not to be fast, just expressive. So keep that in mind.

The use case is: whenever you have some kind of transformation, you will want a chain of unit testable, isolated functions, and this is how you can get it without too much hassle.

This is an experiment. Because I’m using ‘method_missing’ there may be side-effects I am not seeing right now, so please let me know in the Github Issues and send feedback if it helped you out in some project.

Pull Requests are most welcome!

If you’re a seasoned Ruby web developer you’re probably familiar and comfortable using the usual suspects to deploy your applications. You’re either deploying something simple through good old Thin, or you’re orchestrating several Ruby processes through Unicorn workers, or you’re trying out JRuby and want better concurrency and thread management, and therefore you’re using either Puma or Torquebox.

And even though it feels like we are already at the peak of what’s possible with Ruby, we do want more. Ruby 2.1.3 was just released, 3.0 is in development. But until then it should be possible to squeeze some more performance out of your boxes.

In fact, a new contender with some new approaches just showed up. I have few details so far, but it’s a brand new product - not derived from the others - from an unknown team. This product is named “Raptor” and they claim that it can squeeze the extra juice out of our boxes.

And this is their claim:

Fortunately, I had the chance to experiment with its beta release in a controlled Vagrant environment and check those claims! Bear in mind that those are synthetic benchmarks and real world throughput of real applications should give us different behavior. With that having been said, let’s see how the main Ruby servers perform a very simple Rack app returning a simple HTTP 200 with “Hello World”:

 ==> Benchmark parameters:
     Application         : hello_app
     Operating system    : Ubuntu 14.04 LTS (x86_64)
     Virtual CPU cores   : 4
     MRI Ruby            : ruby 2.1.3p242 (2014-09-19 revision 45877) [x86_64-linux-gnu]
     JRuby               : jruby 1.7.13 (1.9.3p392) 2014-06-24 43f133c on OpenJDK 64-Bit Server VM 1.7.0_65-b32 [linux-amd64]

     Unicorn workers     : 8
     Puma workers (MRI)  : 8 (16 threads each)
     Puma workers (JRuby): 1 (32 threads each)
     Torquebox threads   : 32
     Raptor workers      : 8

     Concurrent clients  : 16

 ==> Benchmark summary:
     Unicorn (MRI Ruby)         : 23015.36 req/sec
     Puma (MRI Ruby)            : 32538.62 req/sec
     Puma (JRuby)               : 399.14 req/sec
     Torquebox (JRuby)          : 29773.21 req/sec
     Raptor (MRI Ruby)          : 95309.35 req/sec
     Raptor (JRuby)             : 87523.65 req/sec

Now, this sounds very impressive indeed! And what about a more complex example: a simple Rails 4.1.6 application rendering a default scaffold index page with a model fetching 20 rows from MySQL:

==> Benchmark summary:
    Unicorn (MRI Ruby)         : 326.49 req/sec
    Puma (MRI Ruby)            : 327.36 req/sec
    Puma (JRuby)               : 221.78 req/sec
    Torquebox (JRuby)          : 226.57 req/sec
    Raptor (MRI Ruby)          : 79617.78 req/sec
    Raptor (JRuby)             : 73948.59 req/sec

Again, super impressive. My first impression when reading these numbers is that Raptor must have built-in response caching (which is great!). So I tweaked the example to make it much heavier than what’s considered “normal” for a small cache, rendering a table of 100 rows from the database in the index page, and the results are still competitive:

==> Benchmark summary:
    Unicorn (MRI Ruby)         : 85.98 req/sec
    Puma (MRI Ruby)            : 89.93 req/sec
    Puma (JRuby)               : 79.42 req/sec
    Torquebox (JRuby)          : 77.98 req/sec
    Raptor (MRI Ruby)          : 82.63 req/sec
    Raptor (JRuby)             : 88.92 req/sec

This could mean that in the best case scenario, you could get almost 4 times the throughput from your application, and in the worst case scenario, you’re still getting similar throughputs. This means that Raptor is adaptable and reasonably “smart”. So overall it’s a win-win situation!

The way they are able to achieve these superior numbers can be explained by the way they approached the implementation. This is the breakdown that they have released so far:

• Protection against slow clients: Slow clients on the Internet can block your app, like people standing still in front of your door. Unicorn only has a single small door, so it requires you to attach a buffering reverse proxy in front of it, like Nginx. Puma has as many small doors as you configure threads, so you still need to attach Nginx in front to be safe. But Raptor has a door that’s almost infinitely wide, and fully protects your app against slow clients without additional software.
• Optional integration with Nginx. Nginx is a great web server and provides useful features like compression, security, etc. But using Puma and Unicorn with Nginx requires you to configure Puma/Unicorn separately, to configure Nginx separately, to configure monitoring separately, and to manage each component separately. Doesn’t it make sense to consolidate all these moving parts into a single package? Enter Raptor’s optional Nginx integration mode: configuration, management and daemon monitoring all directly from Nginx.
• Multitenancy. If you run multiple apps on the same server, then managing all those Puma and Unicorn instances quickly becomes tedious. Raptor’s Nginx integration mode allows you to manage all your apps from a single Nginx instance.
• Security. Improve your server’s security by easily running multiple apps under multiple user accounts. This way, a vulnerability in one app won’t affect all your other apps.
• JRuby support. Like Puma (and unlike Unicorn), Raptor has great JRuby support.

So it’s a different implementation and also brings features for the future, transparently handling slow clients without causing bottlenecks for your application and avoiding extra layers of protection. And everything in an easy-to-use package (according to their release: you will be able to replace your current server in the Gemfile and bundle install it!)

Raptor is about to be released, so stay tuned for more news and for when it’s available for everybody to test their applications with. From what we have seen so far, this one looks like a winner!

Go to their website to know more about it and if you’re like me and feel like this is the real deal, Thunderclap it!

I recently wrote an article named _Why Documentary at Rubyconf 2012, Denver reminding people about this person that called himself “_Why”, who decided to vanish in August of 2009, taking all his work with him, essentially committing virtual suicide.

It has attracted lots of trolls and haters at the comments section, more discussion than I anticipated, questioning why we praise someone that was crazy enough to destroy all his work, that is obviously not a good role model, and that compared to other titans of programming has done little and with less quality?

Then, today I heard about this website about Ruby Dramas - that I’m not linking. I don’t know the author and I even think this is just for fun, but the reactions from the trolls and haters aren’t. Essentially it links a few of what we call “Ruby Dramas”, the discussions that trolls and haters call a waste of time, a demonstration of the immaturity and childish behavior of Ruby programmers.

In 2006, Kathy Sierra wrote an article called Dilbert and the zone of mediocrity where she asserts that if you want to be successful, then you don’t want to just please everybody in order to make all of them just like you. You do things in a way that people will love you, but knowing that the consequence is that many others will certainly hate you in the process. It’s an all or nothing proposition, if you just make efforts for everybody to like you, you are just being mediocre.

There’s a catch. It’s not that you make things in order to make people love you. It’s just that you should do things in a way that You Love It.

What are trolls and haters? In essence, they come back to you every time because they care about some of the same things that you do. But they are not producers, they are looters, plunderers. What they are actually saying is: “You’re so good, please me, I want the successful ones to please me, because I can’t do what you do, as the weaker one, I deserve to be pleased.”

Such is the mentality of altruism and collectivism: people that have less, that can achieve less, deserve what successful people produced, and the successful people must feel guilt for being so much better. The horrible Robin Hood philosophy.

I will allow myself for some philosophical liberty and the use of metaphors to illustrate my point. I have never talked to other Rubyists that knew _Why personally, so I hope they don’t get offended if my personal view differs, but again, this is just my personal opinion.

I learned recently that coincidentally the name “Why, the Lucky Stiff” is mentioned in Ayn Rand’s famous masterpiece The Fountainhead. – by the way, as a non-native English speaker, I was just told that this was a very common expression back in the day. But I couldn’t resist leaving this coincidence noted anyway :-)

This novel illustrates the ideal vision of Ayn Rand’s Objectivism philosophy. It tells the story of the ideal man: Howard Roark, a talented architect that lives for the passion of his creative skills. He bows to no one, even if it means his own destruction. He lives by his moral principles of voluntary trade and individual achievement.

Objectivism is the foundation of a more pure version of Capitalism (not the version we currently live in), where individual men make voluntary agreements only.

So, the trolls and haters, in The Fountainhead, are personified by the looters such as the Peter Keatings and Ellsworth Tooheys of the world.

Spoiler Alert!! (do not go forward if you prefer to read the book)

Summarizing, in the book Roark ends up in a project called Courtlandt, a huge complex for the poor and destitute, where no other architect was able to design a complex that would be within requirements and budget. Only Roark was able to solve the puzzle and he agrees to deliver if, and only if, his work is not modified by any means.

When he does deliver, the other architects think they have a say in his designs and they modify it according to a committee that thinks they know better for “the greater good”. When Roark knows about it, and sees his creation being built in a way that he did not want, he decides to dynamite and bring it down.

He goes to court and this is his defense:

Read the full novel version of this remarkable defense speech.

Again, as a metaphor, I can’t help picturing _Why in Howard’s shoes, talking to his detractors, the trolls and haters, the ones naming the so called “Ruby Dramas”, the looters, the ones that think they know about “the greater good”.

Some people question why “_Why” evokes such “irrational worship” when at the end of the day he didn’t even do so much coding, and his coding was not exactly good enough (according to their own views, of course).

My personal pet theory is exactly because _Why embodies Howard Roark’s ideals. He is an artist, just like Roark. He did things in ways that surprised people. He created, he produced, he remained an independent mind. But some contract was breached. I don’t know what it is, maybe the agreement was exactly that: “leave me alone”, “do not come after my personal identity”, “do not tell me how I should do my work”.

So when the Tooheys ran over him, he literally dynamited Courtlandt. It was his work, his terms, his choice. And his individual choice was to do what Roark defended:

“I am an architect. I know what is to come by the principle on which it is built. We are approaching a world in which I cannot permit myself to live.

“Now you know why I dynamited Cortlandt.

“I designed Cortlandt. I gave it to you. I destroyed it.

“I destroyed it because I did not choose to let it exist. It was a double monster. In form and in implication. I had to blast both. The form was mutilated by two second-handers who assumed the right to improve upon that which they had not made and could not equal. They were permitted to do it by the general implication that the altruistic purpose of the building superseded all rights and that I had no claim to stand against it.

“I agreed to design Cortlandt for the purpose of seeing it erected as I designed it and for no other reason. That was the price I set for my work. I was not paid.

So Kathy Sierra was right: if you do something remarkable, that you love, some will also love it, and others will denounce and hate it. Trolls are like leeches, they come crawling fast pursuing other people’s success in order to feed from it. They do it not by producing, but by evoking the altruistic principles, all against the individual. And this is the evil in our society: we should be able to understand that individual achievement is not to be shared. An individual that achieved success by his own merits deserves every penny of it without the need to feel guilty.

Trolls and Haters name things such as “Ruby Dramas” because they want your attention. They want the people they quote to be available to please them. And they can’t be pleased, so it’s an impossible proposition. Trolls exist just for the sake of trolling. That’s what Kathy may not have realized when she first wrote her essay: yes, success will attract the haters, but not because they care about what was achieved and want to protect it, but because - like leeches - they want to benefit without producing, on the principle that successful people must please those who cannot achieve.

Roark said: “We are approaching a world in which I cannot permit myself to live”, and so _Why deleted his work, even though many others contributed to it. Again, there was a breach of contract and again I quote Roark:

“I did not receive the payment I asked. But the owners of Cortlandt got what they needed from me. They wanted a scheme devised to build a structure as cheaply as possible. They found no one else who could do it to their satisfaction. I could and did. They took the benefit of my work and made me contribute it as a gift. But I am not an altruist. I do not contribute gifts of this nature.

“It is said that I have destroyed the home of the destitute. It is forgotten that but for me the destitute could not have had this particular home. Those who were concerned with the poor had to come to me, who have never been concerned, in order to help the poor. It is believed that the poverty of the future tenants gave them the right to my work. That their need constituted a claim on my life. That it was my duty to contribute anything demanded of me. This is the second-hander’s credo now swallowing the world.

“I came here to say that I do not recognize anyone’s right to one minute of my life. Nor to any part of my energy. Nor to any achievement of mine. No matter who makes the claim, how large their number or how great their need.

“I wished to come here and say that I am a man who does not exist for others.

“It had to be said. The world is perishing from an orgy of self-sacrificing.

“I wished to come here and say that the integrity of a man’s creative work is of greater importance than any charitable endeavor. Those of you who do not understand this are the men who’re destroying the world.

“I wished to come here and state my terms. I do not care to exist on any others.

And the point of this article is not to idealize _Why, just to state what it means to value individual achievement.

So, many people see value in _Why, enough to forgive him for destroying his Courtlandt and praise him as an inspiration. The trolls and haters are furious, of course, just like Ellsworth Toohey was. It doesn’t matter the volume or even the quality of his work. It could’ve been 6 lines of code, but if those 6 lines have meant plenty to a lot of people, he would still be remembered. Such is the nature of creation.

We can’t get rid of the trolls and haters. I myself know that for everything I do that people love, I know some of the haters by name and address. They are there, sneaking, crawling. But it doesn’t matter, because I don’t do what I do for their sake, I do for my own. And if you’re a person that produces, you should too.

And if anything, to me personally, that’s what _Why symbolizes. I don’t think we are praising just his person, but what he represented: being a creative individual producing for his own sake.

RubyConf Brazil 2012 has just finished but the US Rubyconf 2012 will begin on November 1-3, 2012, at Denver, Colorado. While I was finishing my post about the event, I came across this trailer for Why’s Documentary, to be released during the US Rubyconf, if I understood correctly.

Now, if you’re new to the Ruby ecosystem, you may have not heard about this character named “Why, The Lucky Stiff”. I think 2 of the best articles about him were written by a fellow Brazilian Rubyist, Diogo Terror to Smashing Magazine in 2010, titled _Why: A Tale Of A Post-Modern Genius and Annie Lowrey for Slate Where’s _why?.

He inspired an entire first generation of Rubyists around the turn of the century. He wasn’t just a programmer, he was a craftsman, literally. The only one I accept calling himself an “artist” within our programming world. His most famous work is the book Why’s Poignant Guide to Ruby used by many famous Rubyists you know when they first found out about Ruby.

Then on August 19, 2009 he vanished.

Some say that great men are recognized only after their passing (“You either die a hero, or live long enough to see yourself becoming the villain”). Maybe his last stroke of genius was to commit “virtual suicide”, by erasing every digital trace of his existence - he actually deleted all his work, his projects, his source codes that were available online. Thanks to technologies such as Git (by the way, this was a great showcase of the recovering capabilities of distributed versioning systems) almost all of the deleted material was recovered and you can find them in places such as here. But the fact that he disappeared and his whereabouts are still unknown kept the myth going and 3 years later he is still remembered.

In 2007 I was conducting a series of interviews with several famous Rubyists of our community. And of course, I did send an email to Why himself. And he responded in a very unexpected way. I think it’s time to disclose the unabridged content of the email so you understand his exotic personality.

Date: Thu, 19 Apr 2007 17:38:59 -0500
From: why the lucky stiff <why@whytheluckystiff.net>
To: fakita@bccl.com.br
Subject: SPAM: Re: Invitation for an Interview for Brazilian Blog

On Wed, Apr 18, 2007 at 10:28:25AM -0300, Fabio Akita wrote:
> First of all let me introduce myself, my name is Fabio Akita, I've written
> the first brazilian Rails book called "Repensando a Web com Rails". I have a
> very well known blog in Brazil called "Akita on Rails":

It is great to meet you, Fabio.  I love accomplished Brazilians and
try to fill my life with any kind of Brazilian I can find.  I also
try to drink milk from a bag as often as I can.

Do you like apricots?  I like mediterranean apricots only.  But I am
willing to entertain new forms of apricot, as they become available.

> I adore your work! I love creative stuff and you're an artist.

I appreciate your remarks, but I have a hard time believing that
anyone would like my art.  I will definitely die without recognition
and few will ever see the work I do.  But I like it that way a lot!!

One of the worst things a person can get in life is recognition.
But a scalp rash is very, very bad as well.  I have had some serious
scalp rashes and I also have thrown up blood quite a few times along
the way.

> So here goes the invitation: would you be available for an interview to my
> blog? We could do the usual way via Skype or Gtalk where I can record the
> chat.  Nothing formal or fancy, just a conversation between two tech geeks
> that anyone could enjoy.

I'm afraid I am currently in hiding, my friend Fabio.  As you may
know, I am operating as a freelance professor right now.  I am
unaccredited, unlicensed with an ill-advised curriculum.  There are
no international laws protecting freelance professors from
imprisonment and deportation, so am I forced into hiding.
Fortunately, I have found a rather remarkable hiding place which is
quite safe and has very colorful wallpaper and animal pelts.

I also cannot do interviews unless they are submitted in
handwriting.  I also cannot respond to interviews in anything but
handwriting.  There are several reasons for this rule, but the
foremost reason is that I like making people do unnecessary and
frivolous things.  And forcing interviews to be handwritten is way
too unnecessary and frivolous for me to pass up, oh Fabio can't you
see that!!

Anyway, it is a pleasure to know you.  I hope a free cart of food
and cheese and DVDs and satin robes and super-long sofas comes
crashing into your house today!!  It will all be free and courtesy
of a steep incline somewhere at a higher elevation.

_why

Unfortunately I didn’t pursue this enough, I was actually planning on sending the interview questions to him, handwritten. :-) In retrospect, I wish I had, it would’ve been fun.

And that was Why, an icon for the Ruby community, an original character that can’t be copied or replaced and we are very lucky for having had him here. Explore his material, learn his style, transmit his message of education, entertainment, friendliness, mastery and craftsmanship.

Several within the community have tried to define what “Beautiful Ruby Code” is. Most of them managed to show us really beautiful code, but I still haven’t seen anyone actually communicate the concept of “Beauty”.

Enter TED. If you know me, you know I’ve been completely addicted to it for years now, every theme inspires me. And tonight, while having dinner, I watched the great Richard Seymour (no, he is not a programmer) talking about this very subject and I was blown away.

Find the video below, but let me add a few words of my own. In our small world of programmers and technologists, we have lots of discussions around “Form” vs “Function”, as if we should choose one over the other, or balance one against the other.

I’m of the same opinion as Richard: it is not a matter of “Form” VS “Function”. The correct way of thinking is “Form” IS “Function”. And beauty is something you feel, before any analysis or rationalization.

Your reaction to beauty is directly tied to what you know and what you’ve experienced. Without living and without learning, you won’t find beauty. Back to programming, it’s impossible for someone that just started to write software, or even someone that already did a few small projects, to grasp the concept of “beauty”.

It’s only after some time, some experience, and some learning that they’ll start to grasp what beauty possibly feels like.

With that comment, please watch Richard Seymour (if you want subtitles for your native language, go to the original site so you can choose subtitles):

Knowledge and Experience are everything. Ignorance doesn’t know beauty.

AkitaOnRails: Ok. So, let’s start. First of all thank you for your time. This is very funny because we are like 5 hours apart. You’re in Paris right now, but I understand you were in Argentina. Can you tell us what you are doing now?

Luis Lavena: Yeah, funny indeed. Right now I’m Technical director of an interactive and design agency with offices in NY and Paris. My job is to bridge the design field with the technology one, mostly on web based application development.

I’m from Argentina, live there, family and friends. I moved a few months ago to boost some developments and get first steps of the new organization in place.

I’m from a small province named Tucuman, northwest area of Argentina (1200 kms from Buenos Aires to be exact)

AkitaOnRails: Awesome, have you ever been in Brazil? Do you have contact with other Latin America Railers?

Luis Lavena: Even though Brazil is so close from Argentina I have never been there, which is a shame. I have several friends that lived there over the years.

I keep in touch with several Rubists and Railers of Latin America, but most with the ones in Argentina due to several meetings we had during the years RubyArg mailing list has been working.

AkitaOnRails: I am very interested in understanding how the Railers are organizing themselves in Argentina as we, in Brazil, are a growing community as well. Do you see new programmers going straight to Rails or is it the usual move from Java/others to Rails? Are companies adopting it, or Rails (and Ruby in general) still has a long way to go there as well?

Luis Lavena: Well, there is a huge number of developers that came from Java and dotNET stuff (mostly Java) and they are playing with Rails, there is also lot of developers that do Ruby for administrative stuff or even creating games with it.
Companies that develop software still need to adopt Ruby/Rails for their business, the biggest problem with that is the lack of support shaped as companies like Sun or some consulting services for Java. (or even Microsoft).

AkitaOnRails: Going back to the past, how did you start in programming in general? Did you study computer science, or were you dragged in there in some other way? And, finally, how did you stumble upon Ruby and Rails?

Luis Lavena: It’s funny, I started programming back in 1989, playing with sprites and GOTO’s with BASIC on a Z80 computer.

I officially didn’t finish my degree on CS, since job and fun stuff distracted me away from it. I first met Ruby back in 2001, loved its syntax, enjoyed its verbosity but it was not mature enough for my needs, at least not in Windows.

Even though I was using Python for so many years, I internally used Ruby to manage several stuff, from simple scripts to later rake tasks that helped us build our tools.

With Rails it was kind of different, since we started to look for alternatives to Zope and we bet on Rails at that time (0.10 or 0.12 at that time, if memory plays nice with me).

I can say I don’t regret the bet on Rails at that time.

AkitaOnRails: It is very interesting to see professionals like you moving out from your zone of comfort toward something ‘unknown’ as Ruby and Rails. Most programmers are very defensive and even aggressive on trying to find justifications for not using neither. Did you feel like that back then? What drove you towards Ruby?

Luis Lavena: Well, I’m used to change languages to fit my needs.
Keep in consideration that our first choice was pre-J2EE era or dotNET stuff, we analyzed which would provide us better solutions in the long run and we decided to avoid them.

Java stuff sounded at that time, and until today, expensive for business. Argentina is not a country where you will find customers with huge budgets to invoice them the cost of using tools designed for other markets.

Banks and such can pay the luxury of have everything running with J2EE, but that doesn’t work when you try to go smaller and agile. (mostly based on experiences with the market flow in Argentina).

AkitaOnRails: It is even more daunting considering that you decided that you wanted Ruby on Windows, you probably didn’t find anything reasonable and decided to tackle the problem yourself. Is that how it happened?

Luis Lavena: I’ve been forced to use Windows for many years, mostly related to hardware development in the video broadcast field. Moreover, I’m a BSD user and Linux one, even played some years with Macs and lately with OSX.

So that restriction forced me to build most of my tools around that limitation.
Ruby allowed me to enjoy that limitation by being expressive, which made me more productive in that field.

Ruby is not capable of handling my main requirements for video processing, but it still powers 70% of my environment, along with Open Source Software and Freeware.

I’ve been using Ruby for so long, that I thought the community, now that it’s growing, would maybe value the experience and the willingness to help in this particular scenario.

So instead of just scratching my itch in the shadow, I decided to contribute back and help others that search for the same stuff. So you can say I’m trying to balance my karma.

AkitaOnRails: Americans probably don’t understand this but in emerging countries such as Brazil, people don’t actually have much choice, most are not educated enough (cof english cof) and so they find lots of barriers. The last resort for everybody is Windows. I think it is the same thing in Argentina. Railers here don’t simply migrate to Macs or Linux, I would say that at least 90% of all developers are locked in to Windows for one reason or another. Is it the same in Argentina, I presume?

Luis Lavena: Even though I don’t like it, I must agree. Thankfully the Universities started to change that some years ago, giving talks and been more open to Linux community, which grew a lot.

Still, there are plenty of users and developer that are locked in to some corporate / mid-size environments or they require to run Windows for some proprietary hardware or such.

So, we (as the community) can’t get to those developers without being more open minded or provide tools that ease the path between their current environment and alternative tools.

AkitaOnRails: But for some reason there seems to be near to zero interest from the community on supporting Ruby on Windows. I mean, there are people like you, some RubyGem developers who go through all the trouble of making available a binary for win32, but it is not enough it seems. How did the One-Click Ruby Installer begin, did you create it or you inherited it?

Luis Lavena: Yeah, I agree with you, but I cannot blame them, after all, everybody just scratches their own itch ;-)

I’ve inherited One-Click Installer from Curt Hibbs, who also inherited from Andy Hunt and other contributors.

The truth is that for many years I’ve been using my own version of Ruby, compiled in-house, so we had control over it. We didn’t patch it in any way, we like to have the whole process documented and automated, so we can take a look to the specific components updated that introduced conflicts.

One-Click Installer, on the other hand, relies on some builds made by the current ruby-core developers and maintainers, which make the process of finding the proper dependencies a bit hard. That build also imposed other problems, but that will require more time to explain

AkitaOnRails: Haha, I want to dive in to the more technical stuff so people can better understand the current situation on Windows. How different is Ruby on Windows compared to Ruby on Linux, for instance? The most obvious thing is that any Gem with C Extensions without proper binaries for Windows will fail. Trying to execute shell commands will fail and RubyInline as well. What else?

Luis Lavena: Hehe, that’s just the tip of the iceberg, let me show you one example. Let’s say that you have a package ABC that was built for your current distribution of Linux. That specific version of Linux links to a common runtime that handles the basic commands like file handling, console, etc. That is often called CRT (C Runtime) and on Linux, glibc.

So, your distro of Linux is linked to this glibc version A.B.C. If your distribution upgrades glibc version (like from 2.2 to 2.5) then you’re forced to:

1. upgrade all your installed packages, or
2. build everything from scratch.

I’ve even heard of some of these upgrades go so bad that users are forced to start from scratch. Windows, on the other hand, cannot break 2 billion applications just because of a change of CRT, so they keep old versions and maintain compatibility in the MSVCRT.dll file (where the base version is 6.0).

A plus for Windows is that you can have several CRT co-existing in your OS, the bad thing is that you cannot safely link to one CRT and use components that link to another version without worrying about segfaults and such stuff.

I tried to condense this information in a post some time ago. So you can see, the lack of development tools is not the worst problem dealing with Windows development.

AkitaOnRails: Interesting. And another thing is the Visual Studio compiler vs MinGW. Can you give us a glimpse on why you chose MinGW? Is it just because VS is commercial and developers would have to pay?

Luis Lavena: Visual Studio is great, but only when you can pay for it. The free versions lack some stuff that is very useful, like Profile-Guided Optimization (PGO) which is being used to build Windows binaries of Python, as an example.

The thing with VS, even the free versions, is that you’re not allowed to distribute them, I think it is even illegal to link to the download URL/page of it. So even for our automated sandbox project, Visual Studio would be required to be manually downloaded, installed and configured.

Thankfully in the latest versions you don’t need to install Platform SDK kit, which was 1GB. VS versions link only to the latest version of MSVCRT, which then force us to relink every library that ruby depends on with it (and Ruby source code depends on lots of externals).

Also, Ruby doesn’t use the safe CRT version of string copy functions, so you need to use a compiler flags just to avoid them… So after weeks of hard work getting it built with VC8 and every dependency, you end with something that performs almost the same, since there is no performance gain (but now you have less hair).

MinGW, on the other hand, was closer to a Linux environment, so most of the tools worked out of the box. The good part was that we didn’t need to build all the dependencies with it, since by default everything links with MSVCRT (the default CRT on Windows). MinGW also provides some cross-compilation tools that lets you build, using Linux environment, shared libraries (dll) and executables for Windows.
I can consider that a plus, dunno what others think about it

AkitaOnRails: Wow, this is really overwhelming and I recommend anyone interested in the details to take a look at your blog. But all that said, you now have everything in place, a proper process and such, I assume. If I am a Windows C developer and I want to contribute, where should I go first, I mean, so I can know what tools to use, how to build stuff and so on?

Luis Lavena: That’s the good part! Gordon Thiesfeld and I have been working on two packages for new One-Click Installer: Runtime and Developer Kit.

Runtime ships the minimum Ruby+RubyGems so you get started, and it can also be used as modules for other installations. The DevKit provides the MinGW environment so you create Ruby extensions or even install the ones that are not pre-built for Windows but they are easy to get it.

This DevKit will not only let you get easily working with Ruby on Windows (and have access to some great and cool Rubygems) but also lets you contribute back to Ruby project. How? Easily, the Ruby Installer project (on Github) is self-hosted. What does it mean? It is possible for you to replicate our development environment anywhere, hack your changes or even debug Ruby’s own C code and contribute back to the community. We have been doing that for several months, and it’s working, how cool is that?

AkitaOnRails: That sounds awesome, I didn’t know that, and I bet lots of developers didn’t also. So I hope C developers reading this can get up to speed with this now. Another detail: RubyGems. How difficult is it to port a Gem with C Extensions to provide a Windows binary when you install it?

Luis Lavena: Oh, that requires that some developers don’t fall into platform specific tricks to make their tools work. One example, that is also a bad practice, is to use hardcoded paths…

Another one is to rely on the existence of some external tools, which sometimes are not available or developers/users didn’t install it. And I’m not talking about Linux-Windows platform issues, even between Linux distros that is a problem.

Installing Gems on RubyInstaller from Luis Lavena on Vimeo.

AkitaOnRails: So it is pretty much dependent on the quality of the original Gem, right? The cleaner the code the easier to port it, of course. But does your DevKit provide some shortcuts to achieve that? And what about RubyInline, does it simply break down?

Luis Lavena: Yeah, I’ve learned over the years that you shouldn’t assume anything about the environment your application is supposed to run, in that way, you avoid those mistakes.

Thankfully RubyInline is playing nice with Ruby Installer (One-Click Installer that uses MinGW) for a few versions back (Thanks goes to Ryan Davis to include those patches).

The current, VC6 build of One-Click Installer is not safe, since you need VS6 to work, which is no longer available. I’ve heard of users enjoying Sequel, Ambition and ImageScience on Windows thanks to those patches.

AkitaOnRails: Correct me if I am wrong, but isn’t it a way to embed C code snippets mixed in Ruby code? This means it will natively compile this snippet to run? How does it achieve this in a Windows environment that doesn’t have compilers by default?

Luis Lavena: We managed to sneak some patches to Hoe and RubyInline that “bundles” the pre-built extensions into the gems, so users don’t need a compiler in those cases.

Anyway, right now using VC6 build of Ruby requires man power to maintain those gems and keep up with new releases, which takes time. Up until now I was the only man in the show (since we own some VC6 licenses). Building everything for everyone is a burden I’m looking forward to drop into DevKit.

AkitaOnRails: Are you the ones maintaining the Windows versions of Gems or each Gem developer takes care of his own project? Is there a central repository for Windows-ready Gems? Is the DevKit already available, by the way?

Luis Lavena: For some projects I’m the only one maintaining gems for Windows, for others I only had contributed patches and other users manage the builds. There is no central repository for Windows gems since I want RubyForge to be the central one.
In any case, I’m pushing MinGW specific ones based on github forks here:

gem list --remote --source http://gems.rubyinstaller.org

The devkit is already available, but we are wrapping it into a good Windows Installer package. You can get your hands dirty and grab the sandbox project from Github to get your environment, or simply download the packages.

AkitaOnRails: So, as far as tools and processes are concerned you're doing a great job. But then there is the Ruby code that, by itself, is a big challenge from what I hear. You mentioned to me that it doesn't even look like C. What are the biggest problems? Technical or getting help from the Ruby-Core mailing list, or both? And what about test suites, how are you squashing compatibility bugs?

Luis Lavena: To be honest, sometimes peeking into Ruby C code gives me headaches. There are lot of macros and definitions and macro conditions that happens in the middle of a if .. else block… The hardest part, besides tracing the bug is getting feedback from ruby-core. Several times I've expressed my concerns about it on mailing lists, #irc and on my blog.

Several times I asked if it was OK to get some errors in the unit tests of Ruby itself and sometimes got answers, but sometimes didn't. Thankfully we no longer need to base our testing quality in bundled unit tests of Ruby. We started to find issues with the Ruby implementation on Window using the Ruby Specs created by the guys at the Rubinius project and that most of the Implementations (IronRuby and jRuby) actively contribute and use as foundation of their interpreters.

AkitaOnRails: So, speaking of which, RubySpecs how far is Ruby on Windows from passing it through? Or is it already?

Luis Lavena: Well, that's a big tasks. First we started making MSpec to work properly on Windows, since RubySpec require some audit to remove some hardcoded paths in it. So we started to add guards around things that are not supported or don't even work on Windows in MRI.

That takes longer since we need to review the results on each platform, dig into the Ruby documentation (that is sometimes lacking) or take a deep breath and look at the C code to find what is actually doing.

If we had more man power, maybe we could finish guarding all the code and get a clear running specs on Windows. Right now there are many that just break or segfault.

AkitaOnRails: You should document this process so people can help, if you didn't already :-) Now speaking of more practical stuff: Ruby on Windows, today, is it a viable runtime to run Rails apps on Windows? Even production apps? You worked with Zed Shaw on Mongrel for Windows, right? Did you try the newest FastCGI support from Microsoft for IIS?

Luis Lavena: Hehee, documentation, documentation, it's totally overrated. We planned to have several screencasts in the RubyInstaller website, but it just remains as 'coming soon' for now.

Not only you can develop Rails applications in Windows (as I do too), but I know several companies that run it side-to-side with IIS or even Tomcat.

Even 2 years after Zed created it, Mongrel still rules the scene of Ruby web servers. I didn't do too much to Mongrel itself, since Zed created something that worked out of the box on Windows with small tweaks.

I created something to run Mongrel as a service (mongrel_service), which helped several people to sneak in Ruby and Rails into their corporate environments, even talking to MSSQL Servers.

I personally didn't test the FCGI support on IIS, but I was the one fixing it to be bundled with One-Click Installer so some users could enjoy it. We could say it works.

AkitaOnRails: About performance, Windows seems to have an inherent condition: in the same machine when I start, let's say, a simple IRB in Windows, then I dual boot to Linux and run it, it seems to open up much faster (not exactly a scientific experiment :-). What is your experience around Ruby performance on Windows?

Luis Lavena: There are performance issues related to the C code being generated by the compiler, and issues related to the OS and I/O handled by it. Loading IRB on Windows could be requiring RubyGems in the background, which in turn needs to scan and load all the gems specifications. I try to keep RUBYOPT=rubygems out of my environment variables.

AkitaOnRails: By the way, why I have the impression that I see RUBYOPT=rubygems being recommended in every windows tutorial I see?

Luis Lavena: RUBYOPT=rubygems is the part of the magic we must avoid.
On Linux, if you start creating a script and you need a gem, you start with:

require 'rubygems'
require 'some_gem'

The same should be done for Windows, it is not painful, it's good practice. RUBYOPT is a shortcut that I dislike, I haven't seen a Linux distro that enables that by default, and I dislike that being set on Windows. (of course, that's my personal point of view)

So to avoid falling into standard testing procedures, I keep using a simple but good sudoku solver that exercise recursion, array and conditions of the VM instead of being IO bound.

You can see that, with funny results on my blog (see "Rubinius, RubySpecs, and speed." section of the post).

The main thing is that VC6 make it slow. Then you have I/O bound stuff (I/O stuff on Windows is not as good as Linux). I consider the default VC6 build of Ruby on Windows painfully slow. On the other hand, the MinGW one is snappier but I still think that there is a huge path for improvement.

AkitaOnRails: So, overall, Ruby on Windows is always evolving which is great, runs great on Windows, which is remarkable, but it could use some more help, right? Do you think more Windows C developers would help? I mean, adding more hands would help improve its quality?

Luis Lavena: Definitely!

AkitaOnRails: What kind of developers, I mean what one should know to help?

Luis Lavena: Dunno if Ruby "core" will accept that, I lately only hear the fork word around MRI ruby. Moreover, we can squash bugs in the Ruby for Windows implementation, but we cannot improve the quality too much since those changes would affect other platforms too.

The bad thing is that there is no clear separation by platform of specific code, everything is around a series of macros and compilation conditions that makes it hard to track (for the non-sadistic developers).

AkitaOnRails: Which reminds me of the Phusion guys with their Enterprise Edition which is a fork of MRI. Did you try to take a look on the copy-on-write GC patches from them? Maybe have it on the Windows version? Ruby on Windows itself is a 'fork'. You end up having to synchronize your trunk with theirs, is that how it works?

Luis Lavena: Honestly I didn't have time to look at Enterprise Edition, even though I followed all the posts and the discussion of those patches in ruby-core. The funny thing is that the latest releases of the Ruby 1.8.6 branch are giving me headaches (more than looking at the C code). They cannot even complete the self-tests and segfault with some extensions like the mysql gem.

AkitaOnRails: Yes, it has been causing a lot of fuzz around the community, and it should. Hope they can get around it soon. Well, I think I already took a lot of your time. Is there any subject you would like to tackle, or at least some message for your younger brazilian community?

Luis Lavena: Yeah, don't be afraid to ask questions, don't feel ashamed for using Windows as Platform. You can accomplish great applications using Windows, and you can infiltrate more easily into corporate environments with it.

AkitaOnRails: Oh, which reminds me of one last question that everybody asks me all the time – and I shall forward it to you, as you're a Windows developer: what are your tools of choice to edit Ruby/Rails projects?

Luis Lavena: I mostly use Free or Open Source stuff, so I've been using Komodo Edit (not Pro) and Programmer Notepad, since sometimes I use them to work on C, FreeBASIC and Assembler code for embedded hardware.

Lately I've been using a lightweight Editor called Intype that ships with some bundles to mimic TextMate, but I really don't take advantage of them. I used NetBeans for a while, but after 500MB I just closed it :-)

AkitaOnRails: Haha, and what would you recommend for a beginner Rails programmer?

Luis Lavena: To get started NetBeans is good, but with practice they will learn that even notepad is enough to code.

(PS: We are already including Enterprise patches for latest issues on MinGW build! :-)

AkitaOnRails: Haha, right. Well, I think this is it. I think this will reach a broader audience and call the attention of Windows developers out there. Thanks a lot!

Luis Lavena: Thanks to you Fabio.

It was Avi Bryant that evangelized the neat idea of “turtles all the way”, meaning that for a language to be called ‘complete’ it should be able to extend itself. So, the ideal world would have Ruby being extended in Ruby, not in C. JRuby goes as far as it can building up a sandbox for Ruby code to run under the JVM. As cool as it is, we still rely on Java to fully extend it.

Enter Rubinius and its author Evan Phoenix , currently a full-time employee for EngineYard. Rubinius borrows heavily from Smalltalk’s concepts of a virtual machine and does as little as possible in C just for the bootstrap and all the rest is developed over pure Ruby.

Rubinius answers lots of questions about going forward over the current Ruby MRI but also raises several other questions that I hope we can nail down today in this interview with Evan himself.

So let’s get started.

AkitaOnRails: It is a tradition now in my interviews to ask about the background of the programmer. What was your path? I mean, what made you come in to the programming world and how did you get into Ruby?

Evan Phoenix: I started programming in high-school with a friend of mine. We started a small computer company doing odds and ends and I really got into it. I had gotten into Linux a few years before that, so programming became a natural extension of using Linux.

When I applied to college, I declared myself as a Computer Science major on the application, so I’ve been doing this since about then I guess.

As for Ruby, I got into it in about 2002, when I first moved to Seattle. A friend had just found it and was writing some fun stuff in it and I picked it up on his suggestion. Eventually, I found the seattle.rb, and have been working in Ruby since.

AkitaOnRails: EngineYard seems to have great faith in the future of Rubinius. How did this relationship start? Can you tell us what are EngineYard’s expectations regarding your work for them?

Evan Phoenix: The relationship started mainly with them approaching me about working for them, on Rubinius. So they saw promise in Rubinius on their own, from my presentations and such, and approached me.

Their expectations are that I make Rubinius the best Ruby VM it can be. That’s my basic mandate with them. They’re pretty much the ideal stewards of the project. They allow me to work at the pace I want, they understand the rigors of the work I’m doing, and give me a lot of freedom.

AkitaOnRails: Now let’s get into Rubinius. For non-starters how would you define Rubinius? We all know that this is an alternative Ruby implementation, so maybe it would be good to point out what makes it different than the MRI. What are the project main goals?

Evan Phoenix: One of the primary goals is to have as much of the system written in Ruby itself. With MRI, none of the core classes are implemented in Ruby, all are in C. So we wanted a system that was easier to work with, and thus written in Ruby itself.

Another thing we’ve done is enrich the number of things which are objects, which adds considerable power. An easy example is the CompiledMethod class which Rubinius has. It contains the bytecode representation of a method and can be inspected, manipulated, etc, just like any other object. This opens a lot of new doors with regard to how problems can be solved.

AkitaOnRails: It’s not so obvious to the brazilian audience when we speak of “Squeak”, “Garnet”. Smalltalk is probably less known here than Ruby. The point is that it is well known that you were inspired by the way Squeak is implemented. This led to features like “Cuby”. Can you explain what is so great about the way Squeak is implemented and how it is helping you in the Rubinius project?

Evan Phoenix: Well, we’re currently in the process of developing Garnet/Cuby, so those tools aren’t in use really yet. But almost the entire system architecture of Rubinius is modeled off the original Smalltalk-80 Virtual Machine. It defined things like CompiledMethod, etc, which I took name for name into Rubinius.

In addition, the execution model of Rubinius is almost identical to Squeak. A good example is the way in which the system calls methods and figures out how to keep track of where to return control to when a method call returns. All that information is kept in first class objects called MethodContexts. In a language such as C, that information is stored on a process’s memory stack. By keeping the data in first class objects, we’re able to query them directly to find out information about the running system. It also vastly simplifies how the garbage collector works.

AkitaOnRails: Today you have shotgun. I read somewhere that you consider it ‘cheating’ because this is not an interpreter for Ruby written in Ruby, which will lead to Cuby. It’s more kind of a bootstrapping to ease development. What’s the current state of shotgun, can we use it outside its purpose of developing Rubinius?

Evan Phoenix: Shotgun is a virtual machine written in C that provides instructions and primitive operations for running a Ruby-like language. Shotgun itself actually has no knowledge of parsing, compiling etc. It has a very simple way to load code in, and a way to execute it. You could easily write a new language which targets shotgun, in fact, Ola Bini and I have talked a few times about writing a simple lisp-style language that would run directly on shotgun.

The primitive operations it provides are kinda like syscalls in a unix system. They provide very low level operations which are built on. For example, there is an add primitive which adds two Fixnum objects together and puts the result on the stack.

AkitaOnRails: This is a personal curiosity. The MRI today uses a simple mark and sweep garbage collector (GC) whereas Java uses a highly customizable generational GC. What did you choose for memory management on the Rubinius VM?

Evan Phoenix: Rubinius uses a generational GC, combining a copy collector for the young objects, with a mark/sweep collector for the old. It’s proved to work quite well for us. The trick has been tuning how long an object exists before it is promoted to the old object space. We’ve only tuned it a little. There is a lot more work that could be put into the GC, and it’s architected to only interact with the rest of the system at a few places, so its logic can be completely replaced if need be without disturbing anything else.

AkitaOnRails: We have parties that favors green threads, we have parties that are all go native threads. How are you dealing with threading issues in Rubinius? Will we still have the same kind of global locking that MRI has today?

Evan Phoenix: At the moment, Rubinius only has green threads, built on the first class MethodContext objects. It currently uses libevent to allow threads to wait on IO objects quickly.

The plan (likely for 2.0) is to support native threads as well as the current green threads. You’ll be able to allocate a pool of native threads that the green threads can all share. So perhaps you have 100 green threads, you could allocate 10 native threads for them to split time between. Because of the architecture, it’s trivial to migrate a green thread between native threads.

As for the global lock on native threads, no, we don’t consider a global lock a solution for using native threads. If there is a global lock, there is no reason to use them over green threads. Because so much of Rubinius is written in Ruby, we have an easier time of figuring out how to properly lock things to keep all the objects in the system safe. We haven’t really even started this work, but it’s been talked about quite a bit.

AkitaOnRails: The current goal is probably to make Rubinius as compatible with the MRI 1.8 as possible. Did you already start any work toward YARV compatibility already? And speaking of YARV, it is the official virtual machine and bytecode compiler for the next release of Ruby. Is there any chance to make Rubinius and YARV byte-code compliant or this is not a short-term goal?

Evan Phoenix: We’re actively working toward being 1.8.6 compliant. This a big goal because there is a lot of functionality in there. We’re going to do a 1.0 release of Rubinius once it’s able to run all the specs that 1.8.6 runs and properly runs Rails 1.2, which is a good benchmark.

No, we haven’t done any work yet toward YARV compatibility, but because of how things are architected, we’ll likely just be able to have —yarv flag that you pass to Rubinius to have it use YARV compatible versions of things.

We aren’t currently working toward a common bytecode format between Rubinius and YARV, mainly because there is nothing to be gained by doing that. YARV’s bytecode format is largely internal to YARV only, last I looked, you couldn’t easily save it out to a file. Rubinius operates primarily off .rbc files, which contain a serialized CompiledMethod, and thus contain bytecodes. That being said, they are actually similar and could be unified if we saw the need to.

AkitaOnRails: Performance is always subject for a lot of discussion and it is still premature to talk about performance under Rubinius, as it is still work in progress. But I read in your blog that it is quite fast already, even surpassing MRI in many tests. How is performance evolving?

Evan Phoenix: Currently, our performance is evolving slowly. This is mainly because it’s not a priority for people writing the kernel of Rubinius (the majority of the Ruby code which makes it up). As we get closer and closer to being compliant, we’ll begin tuning things. That being said, I do write code to make Rubinius faster here and there. We went ahead and implement fast math operations to speed up the simple math that people commonly do. We have a simple profiler which can easily be turned on to give developers feedback about how their code is run.

AkitaOnRails: One of the main problems for any alternative implementation to Ruby is that it has a lot of C-based extensions, including a handful of core libraries. So if you want to stay “pure-Ruby” you have to re-implement all of them in Ruby. The JRuby guys had to go through something similar. Are there any cross-project collaborations already in this regard?

Evan Phoenix: Actually, that’s not entirely true. We also saw that the landscape of Ruby includes a lot of C extensions. To make migration to Rubinius faster, we’ve written a layer into Rubinius which allows all those C extensions to run (nearly) unmodified, only requiring a simple recompile. These are so that things such as ImageScience, RMagick, postgres bindings, etc can all be used under Rubinius without having to be rewritten.

AkitaOnRails: Still about C code, I know it is difficult to deliver a bytecode virtual machine and still be dependent on non-managed C code. Java has some jury rigging toward JNA, .NET has pinvoke. Do you plan something similar for Rubinius so that it can more or less easily speak with C libraries?

Evan Phoenix: We actually directly support a mechanism called FFI (Foreign Function Interface), which allows a developer to bind a C function directly as a method call. Here is a simple example:

module LibC
 attach_function nil, :puts, [:string], :void
end

LibC.puts "hello!"

The attach_function line is the primary interface into FFI. You simply indicate which library the function is in (in this case, nil is used because it’s included in the existing process), the name of the function (puts), the types of arguments it takes (just 1, a string), and finally, the type it returns (void, ie, nothing).

Using this, you can tie directly to C functions without having to write C wrapper code.

AkitaOnRails: You are very right when you say that formal language specs too early in the game can make it difficult to evolve the language. Ruby as been growing for 10 years now, and it feels that this is the right time to start thinking of something like this, so that we can have a baseline from where every other alternative implementation could start. Do you see any efforts towards this end?

Evan Phoenix: We’ve been actively working to develop a set of specs (in rSpec format) for 1.8. These form the primary mechanism with which we test how Rubinius is doing. These specs are being actively used by JRuby and IronRuby to test their respective Ruby environments. We’re currently in the process of moving these specs into a generic Ruby spec project to try and begin moving things toward a more formal standard.

AkitaOnRails: Another recurring question is about Rails, of course. Any new Ruby implementation today will have to face the inevitable question: does it run Rails already? :-) Rails is serving kinda like a ‘mini test suite’ and baseline for most implementations. How is Rubinius performing on this matter?

Evan Phoenix: When we run Rails, we know we’re close to a 1.0. :)

AkitaOnRails: You have a very interesting approach toward open source management: everyone that sent you a good patch gains commit access to the repository. How is this working for you? Isn’t it a bit ad hoc or people tend to follow your lead and your list of priorities?

Evan Phoenix: It’s actually working out great so far. By being spec/test oriented, new people have a place to start, either writing missing specs or making existing ones pass. Those become great intros into the whole system for them. They then begin to explore a bit more and commonly find more and more they’d love to help with. Because we have a lot of work to do in a lot of different places, people don’t have to worry about my priority list, since it’s usually involving things which others don’t need to be actively concerned with.

I really can’t say enough about the people who contribute. The one patch barrier has proved to work great because while we’ve now got over 60 committers, not once have I had to pull commit rights from someone. Sure, people mess up, but in every case, the person has corrected the behavior and/or code.

AkitaOnRails: One byproduct of this commit access approach of yours is the integration of rSpec into the Rubinius project so that you have a nice requirements tool and test suite. Do you practice Behavior Driven Development? Or this is covering just a small sub-set of the project?

Evan Phoenix: I’m largely a testing newbie, simply following the guidelines set by Brian Ford, the primary spec keeper of the project. It has proved to be a great tool for viewing how the project is progressing.

AkitaOnRails: As many other Ruby projects, Rubinius is also managed over GIT. There’s been a lot of fuzz around Git and Ruby these days. Do you think there is a new trend here? Why did you decide to go with GIT?

Evan Phoenix: Yeah, I think that GIT is picking up as a trend, because of the features it offers over subversion mainly.

The big reason we decided was it’s feature set. One big feature I love is local branches, allowing a person to manage a number of outstanding changes without trampling on each other.

We haven’t yet, but we plan to exercise the great merge / public branch capabilities.

AkitaOnRails: We heard from Ezra’s recent interviews about your new experiment on multi-VM. That would be a blessing for deployment without we having to deal with several processes in a mongrel cluster. Can you describe this feature and pin point its advantages?

Evan Phoenix: It’s pretty experimental currently, but it will only get more stable over time. The idea is that you can spin up a whole new VM in the current process. This new VM is completely separate from other VMs, living in its own native thread and having its own garbage collector, etc.

There is a mechanism for VMs to communicate with each other though, and this allows them to coordinate work. This would mean that a primary thread could accept new connections, then pass them off to a new VM to actually process. This allows you to process connections in actual parallel, in addition, because the VMs are completely separated, this even works for non-thread safe applications like Rails.

AkitaOnRails: Ezra also mentioned about a possible mod_rubinius in the making. How is it going? This definitely makes the Rubinius stack even stronger. Together with Merb, Rubinius plus multi-VM capabilities plus mod_rubinius would be a killer deployment package.

Evan Phoenix: Yes, a mod_rubinius project is just beginning. The multi-VM code and mod_rubinius overlap a bunch, so you’ll be able to have VM pools for sites, all managed through apache. We’re hoping that mod_rubinius can really simplify the Rails/Ruby deployment picture. It will operate similarly to mod_python, allowing a VM to remain running between requests, and likely even run background tasks.

AkitaOnRails: How long do you guess we are from a fully MRI-1.8 compatible release?

Evan Phoenix: We still don’t have a firm date, mainly because what fully compliant MRI-1.8 is is still unknown. We are still adding specs, working to properly define 1.8. That being said, we’re really hitting a stride and progress is advancing rapidly. We’re very close to properly running Rubygems. Ezra recently was able to run Merb on top of Rubinius, running under webrick.

I’m going to be doing a 0.9 release this week, because we’ve made so much progress since 0.8. My big hope is that by RailsConf in May, we’ll be running Rails. But don’t hold me to that. This is open source after all. :)

AkitaOnRails: Thanks, I think this is it.

Evan Phoenix: Thanks for your patience!

The Ruby Way is the undisputed must-have book in any Rubyist bookshelf. Rather than being a ‘reference’ book it explains what it takes to really dive into the intricacies and marvels of the Ruby programming style.

Today I am very happy being able to engage in a conversation with one of my favorite authors, Hal Fulton. This was a great chat and I know people will be delighted as well. He is one of the Ruby veterans and certainly has a lot of experience to share. So, let’s start:

AkitaOnRails: First of all, it is a tradition at my blog to ask for the guest’s background. How long have you been in the programming career? How did you first get there? What inspires you about the computer world?

Hal Fulton: I started college as a physics major, but I found that I was taking computer courses for fun. I switched to computer science and the rest was history.

Unlike most younger people now, I never was really exposed to computers until I was sixteen, because personal computers were much less common then. I was hooked right away. I saw the computer as a “magic box” that could do anything I was smart enough to instruct it to do. Really I still feel that way about it.

AkitaOnRails: This is almost a cliché already but I have to ask: you were one of the ‘first generation’ Rubyists. How did you come to find Ruby and what was it that ‘clicked’ for you about the language?

Hal: I was on a contract at IBM in Austin in the fall of ‘99. In a conversation with a friend across the hall, I complained that I was never on the “ground floor” of any new technology — I was always a late adopter. And he said, “Well, you should learn Ruby then.” And I said: “What’s that?” So I got on the English mailing list and started to learn Ruby (version 1.4).

My experience before was with very static languages. I had started (like many people in the earlier days) with BASIC, FORTRAN, and Pascal. Then I learned C, C++, Java, and various other things along the way. But I was never exposed much to LISP, and I never knew Smalltalk. So the whole concept of a dynamic language was a little foreign to me. I had always known that I wanted more power, but I wasn’t sure exactly what I wanted. I tried to envision macros that would give me the kind of flexibility I wanted, but it seemed like the wrong solution.

Then I learned Ruby, and I felt that I had taken not just a single leap forward, but three leaps. It was clear to me that this was very similar to what I had been looking for subconsciously.

AkitaOnRails: The Pickaxe is another must-have book, but it serves the role of a complete ‘Reference’ book whereas your book is about inner underpinnings and foundations of the language – heck, you spend 40 pages just talking about String, if this isn’t detailed enough I don’t know what would be. What was your intent when you wrote the 1st edition, how it came to be?

Hal: I met Dave Thomas and Andy Hunt online on that mailing list. At that time, their excellent book The Pragmatic Programmer had just come out, and they were working on the Ruby book. I was one of the reviewers of that book – helping to iron out little inaccuracies and problems.

A little later, Sams Publishing went looking for someone to write a book. At that time, the English mailing list had very few native speakers of English, so I had a good chance of being picked. I submitted a proposal, and they liked it.

The Pickaxe wasn’t on the shelves yet, but I had intimate knowledge of its contents. So when I put together the proposal for the book, I tried to make sure it was not in direct competition; rather, I wanted it to be complementary. If you look on the “praise page” of the first edition, there is a quote from me, saying “This is the first Ruby book you should buy.” But privately I was already thinking: “And I know what the second one should be, too.”

AkitaOnRails: As detailed and complete as you try to be Ruby has several subtleties and nuances that are hard to capture. Was there anything that you didn’t cover at that time that you would’ve like to spend more time with?

Hal: There are certainly some subtleties with classes and modules, reflection, metaprogramming, and that kind of thing, that are not mentioned or not stressed. But in general a programmer shouldn’t be “too clever” when writing code anyhow.

The most glaring omission, I think, is that there is no coverage of writing C extensions. In the first edition, I simply ran out of time, energy, and space – I couldn’t cover that. In the second edition, I omitted it because I thought the C API was going to undergo radical
changes as we approached Ruby 2.0 – more radical changes than the core classes, for example. I think this is turning out not to be true, however.

AkitaOnRails: You probably stumbled upon many Ruby students and enthusiasts that were learning Ruby through your book. By your experience, what do you think people get most amazed about the language, and what are the features that bite them initially?

Hal: I think the conciseness is one really attractive feature. Take a look at this single line of Ruby, which creates two accessors for a class:

attr_accessor :alpha, :beta

How would you do that in C++? Well, you would declare two attributes; then you would write a pair of “reader” functions and a pair of “writer” functions. What’s that, at least seven lines of code right there? But then what if you want to be able to assign different
types to these? Now you have to get into overloading. It quickly becomes a nightmare. Meanwhile, in Ruby — it’s still just one line of code.

AkitaOnRails: What do you like the most about Ruby that sets it apart from other similar languages?

Hal: One of the best features about Ruby is dynamic typing. Variables don’t have types; only objects have types (or should I say, classes). Static typing was so ingrained in me that I assumed it was the right way. But when I gave it up, I felt as if I had removed weights that I didn’t know I was wearing.

Its regularity is also an appealing feature. For example, a class in Ruby is in a sense a “special” thing – but not nearly so special as in (for example) C++. A class is “just an object,” so you can do things to classes that you can do to other objects – such as pass them around, store them in arrays, test their values, and so on.

AkitaOnRails: And what would you improve if you were the guardian of it?

First of all, I’m glad I’m not the guardian of it. Matz does an excellent job, and he is much smarter than I am. I think there are some issues with reflection and dynamic programming that need to be cleaned up — exactly how, I’m not sure.

I look forward to a truly usable virtual machine, and I look forward to some form of method combination (pre, post, wrap). I admit, though, there are small syntax and core changes I would like to make – mostly pretty minor.

For example, I’ve always wanted an “in” operator that would be syntax sugar for calling the include? method – for example, “x in y” would mean “y.include?(x)” – this operator would be essentially the same as the set membership operator in mathematics (instead of the reverse) and would sometimes make code more readable and even make parentheses unnecessary sometimes. I’ve used the “in” operator in Pascal since I was 18; Python has it, even SQL to an extent. I wish Ruby had it, too.

Many times I’ve felt a need for a data structure that would be accessed like a hash, but would preserve the order specified in the code. For example, imagine a kind of “dynamic case statement” — we pass in possible matches and code to execute (as procs) for each of those matches. (It would have the advantage over a case statement that we can control the number of case limbs and their associated code at runtime.) Let’s implement it as a method called “choose” that we call in this way:

choose regex1 => proc { handle_case1 }, regex2 => proc { handle_case2}

Now, what’s the problem here? The syntax fools us into thinking that regex1 somehow precedes regex2 – but when we iterate over a hash, the order is not guaranteed to be the same as in the code. (That is a property of hashes, of course, not a bug.) So we can’t control or predict the order in which these are applied. And I have found several other cases where I wanted such a thing – an associative array, a set of tuples, that had a convenient syntax for literals and had an order.

There are two usual responses to my desire for an “ordered hash.” Some people say that I’m crazy, that a hash isn’t ordered and that’s that. Some people say I could always make my own class that behaved that way – which is true, except that I wouldn’t have the convenience of representing it as a “first-class citizen” in Ruby syntax. After all, a hash “could” be represented as a second-class citizen as well – I could say

Hash["a",2,"b",4,"c",6]

instead of saying:

{"a" => 2, "b" => 4, "c" => 6}

But I am glad for the expressiveness of the latter syntax.

AkitaOnRails: Your second edition covers Ruby 1.8. Are you researching into Ruby 1.9 (YARV) already? Being a Ruby veteran, what do you think of the directions Matz and Koichi have been choosing for the next version? Is it satisfying for you? For you, what are the good, the bad and the ugly about Ruby right now and the next version?

Hal: There have been minor points where I disagreed with Matz from time to time. But as I said, I am glad he is in charge. I like YARV from what I have seen so far. I am anxious for it to be 100% stable so that I can really get a feel for how it works.

AkitaOnRails: Your book is in the second edition, and it is a huge book, covering not only the language itself but a few 3rd party components as pdf_writer. Are you working on a 3rd edition? Maybe we will have “The YARV Way”?

Hal: I am not working on a third edition, and I think it would be very hard to write one. The second one was harder than I expected – although much content was re-used, it still had to be re-examined line by line – and more than 50 errors still crept into those 800+ pages.

I once joked that if there was a 3rd edition, it might have to be two volumes. But really, that might not be such a bad idea – basic concepts and core classes in one volume, standard and third-party libraries in the second. But that’s idle speculation. There is a kind of add-on for the book being planned – but I won’t mention that until we know more details. Ask me in a few months.

As for YARV – I think Ruby will always be Ruby. We may change the inside, but the name will remain the same for the foreseeable future.

AkitaOnRails: Around 2002 Ruby was not widely adopted and only a handful of hobbyists and enthusiasts were using and helping to improve the language. Yourself, David Black, Nathaniel Talbott, and a lot of big guns from what I call ’the first Ruby generation’. But in 2004 came this full fledged web framework called Rails that made Ruby publicly known and acknowledged, attracting many great minds but also a lot of trolls and amateurs. I hear that some in the first generation regret it because what was once a peaceful and insightful community became very noisy. What are your thoughts about this sudden celebrity status that Ruby gained and its side-effects?

Hal: Well, Ruby isn’t a private club. I wonder if the Japanese felt any offense when Europeans and others started to be interested in it? I think that we’ve certainly lost a little bit from the old days, but I don’t waste any time thinking about it. We’ve gained quite a bit also. And it’s interesting that the “friendliness” of the community has scaled better than many of us expected. Ruby is still Ruby, and Rubyists are still Rubyists. It’s just that there are more of us now – also more projects, more activity, more opportunities. That can only be a good thing.

AkitaOnRails: Do you work with systems development in pure Ruby or do you use Ruby on Rails for web development today? What’s your occupation or day job?

Hal: I’ve done a little work with Rails here and there, but I’m more of an old-fashioned generalist. I’m not a web guy. Many younger programmers, of course, seem unaware that there is any other kind of programming to be done. But if you were writing a compiler or interpreter, a custom web server, any kind of daemon or background networking app… would you reach for your Rails book? Of course not.

I’ve done system-level work in the past, but I can’t imagine Ruby ever being appropriate for the kinds of things I did back then. For those things, we used C (the universal assembly language). In my day job, I’m happily coding Ruby every single day – telecommuting for a Dallas
company called Personifi. I’d describe my work as internal applications and tools – a lot of text analysis and indexing, a little number-crunching and a lot of word-crunching.

AkitaOnRails: Here in Brazil we have a lot of people that were initially attracted by the Rails hype machine, now they are struggling to learn the Ruby language. Metaprogramming, Closures, Mixins, make them go nuts. Do you have recommendations for new comers?

Hal: What would be a good way to start experimenting with the language? Well, you can keep a copy of The Ruby Way nearby. :-)

Really, I only have three recommendations. First of all, the mailing list or newsgroup is a powerful source of information, and there are still friendly and knowledgeable people there who will answer your questions. Second, there are millions of lines of Ruby source
out there. Study it and learn from it. Third, and most important: Just play around! You don’t really learn a language by reading about it, but by using it.

AkitaOnRails: By the way, are you already writing a new edition? :-)

Hal: I’m not, and I can’t see ahead that far. But wait six or seven months, and I do hope to have a little surprise for you…

AkitaOnRails: In the light of the current Zed episode, he said a lot of thing specifically against the Pickaxe. One of the things is his claim that the Pickaxe didn’t mention clearly all the meta-programming stuff. Then I was talking to ‘arton’ – a Ruby book author from Japan – and he reminded me: americans and japanese seem to do Ruby differently. He said that they don’t usually do all that meta-programming in the level that Rails does, for example. As you said, they don’t try to be “too clever”.

Now, I don’t want to be apologetic because I don’t think Dave needs it, but this explains a lot given that the Pickaxe mostly only had japanese sources to rely on back in 2001. I personally learned most of my Ruby reading the Rails source code at first (and books like yours later). What do you think about this? (meaning, about the Zed episode in general, and about the different american/japanese programming styles?)

Hal: As for Zed, I won’t take any sides in that discussion. Zed can say whatever he wants, and usually does. I’m sure there is some truth on both sides of any controversy.

As for metaprogramming – well, I think those capabilities are in Ruby for a reason. They are meant to be used. But any tool has both appropriate and inappropriate uses. It’s impossible to generalize and say “this is always bad,” but certainly metaprogramming can be misused.

As for Rails itself, I have no doubt there is both good and bad code in it. I suspect the developers working on it would admit as much. And it is possible that they have used “too much magic” here and there, though I couldn’t say specifically where. But could Rails exist without metaprogramming features? Not in its current sense — not in any meaningful sense, I would argue.

AkitaOnRails: I interviewed Avi Bryant, from Seaside fame, and one of his mottos is that ‘a language is not finished until it can extend itself’ or ’turtles all the way down’ as he says. Meaning, that Ruby still relies a lot on C extensions to this day. Of course, the initial motivations for Ruby are different than Smalltalk. On the other hand people like Evan Phoenix are pursuing this very goal: make Ruby be extensible using pure Ruby. What do you think about this new direction?

Hal: I like the idea of “Ruby in Ruby” in general, though Matz does not. And I am impressed with what I have seen of Rubinius so far – I think it is an important project and will grow more important.

If it’s possible to move toward an elimination of C extensions without sacrificing speed, I suppose that’s a good thing. I am not sure if it is possible. We might at least create a “Ruby in Ruby” that does not rely directly on MRI (Matz’s Ruby Interpreter). But I don’t expect to see the “original Ruby” replaced any time soon.

AkitaOnRails: This was a very insightful conversation, thank you!

Peter speaks about Ruby on Rails, business, novices and, as a last-minute exclusive, he comments on the recent Nuclear Zed episode that shocked a lot of people in the community. Just to clarify, Peter answered my questions before New Years Eve, it’s only the last question that was added today.

Once again, I deeply apologize to the brazilian audience because I didn’t have time to translate this into Portuguese today, but I will do very soon. Stay tuned.

AkitaOnRails: For the Ruby community Peter Cooper stands for Ruby Inside. I personally enjoy reading your website as it accumulates most of the important Ruby and Rails news out there. Let me thank you again for adding my blog to your sidebar list. But how did it start? What do you think were the most important steps you made to build its good reputation?

Peter Cooper: Ruby Inside began as a vehicle for promoting my forthcoming (now released) book, “Beginning Ruby.” The name itself came from a comment made in an interview Geoffrey Grosenbach did for the Rails podcast where someone adapted Ruby into the old “Intel Inside” slogan, which sounded like a great name for a blog covering Ruby issues. Initially, the blog was simply to promote the book, but it quickly turned into a digest of news from the whole Ruby community. I think the key to its success was that the larger Ruby blogs at the time, the O’Reilly Ruby blog being a good example, were posting less and less as their writers were getting more and more Ruby work. Ruby Inside filled a gap of providing heavily digested news from the whole Ruby community, with only one or two posts per day, allowing Ruby developers to keep up to date with things, but without having to subscribe to hundreds of blogs.

AkitaOnRails: Looking through your LinkedIn profile I see that you’ve been quite involved with internet-based business for the last 10 years. You also describe yourself as a ‘serial entrepreneur’. Entrepreneurship is something very difficult here in Brazil, mainly because of economics and overwhelming bureaucracy involved. How difficult is it there in England to open and run your own business?

Peter: It’s very easy to start up in business in the UK. Depending on what you want to do, you can just say you’re a business right away, without filing any papers (although you do have to call the tax office to let them know!), and operate as a “sole trader.” Registering a business officially, however, is quite easy too and gives you a number of legal protections and a more professional image. Getting funding and so forth isn’t particularly easy here, compared to the UK, but if you’re willing to bootstrap, then the UK’s very good for easy banking, fast communication, and trading with the rest of the world. With the natural advantage of the English language, the US market is easy to target from the UK.

AkitaOnRails: Your other most well known products should be Code Snippets and Feed Digest. Can you introduce both of them to our Brazilian uninitiated? How did the idea occur to you? It was something like ‘scratching your own itch’ or it was something totally unrelated and maybe following some trend that you identified? What tools did you use to build them? Many people are interested to know what does it take to build a successful product. What do you think it takes to be a good internet entrepreneur?

Peter: Code Snippets was the first del.icio.us-style “tagged” public code snippet repository. In fact, I believe it was the first Rails application to use tagging at all, as I wrote the earliest code which was appropriated into a number of following tagging libraries. The site was launched in early 2005 and I sold it in February 2007 to DZone. Feed Digest is a service for mashing up, processing, and re-publishing RSS and Atom feeds into other formats, such as HTML, JavaScript, Flash, images, and so on. I sold it to a Russian company in the summer of 2007.

Both businesses / sites were a result of “scratching my own itch” as you say. I needed a system to store code snippets in a tagged way, and del.icio.us was not suitable for this, so Code Snippets came out of a 24 hour coding marathon, was publicly released, and then grew to thousands of users over time. Feed Digest evolved from an earlier service called RSS Digest that I created to scratch another itch, republishing my del.icio.us links onto the header of my blog. It grew far beyond that though, and now has over 25,000 users and serves perhaps about half a billion requests each month. I no longer own or have a direct interest in either of those projects now though, but I must be doing okay on the chart of people selling Rails apps!

AkitaOnRails: I congratulate you on your recently published book, from Apress, Beginning Ruby, from Novice to Professional. As Larry Wright states in his review, he is right that the Pickaxe is a good Reference book, but you take a different approach in being more careful about actually ‘teaching’ the environment. It does fill a gap as there are more and more people interested in learning Rails but lacking an introductory book to learn Ruby itself. Please, describe your book for our audience as I think many Brazilians don’t know about your book yet. What’s its target audience? What do you talk about?

Peter: The book is currently only available in English, unfortunately, so the US has definitely been the biggest target market so far. Content-wise, the book is very reminiscent of the “classic” programming tuition books I was brought up on in the 1980s. It takes a very “walkthrough” approach, with the installation of Ruby coming first, followed by a walkthrough of all the main concepts regarding object orientation, Ruby’s syntax, and the basic data types. Eventually, by the end of the book, the reader has gone through file operations, database usage (there’s even an SQL primer), developed a text adventure game, built a simple Rails application, and so forth. It’s very much a book you’d go through, as a beginner, from beginning to end, and then move on to more advanced books like the Pickaxe or The Ruby Way. Hopefully we will see it in Portuguese one day!

AkitaOnRails: And speaking of beginners, again I think it amusing to read at your LinkedIn profile: Does “left school at 16” count? If so, I’m in. Education is great for most, but overrated if you’re a grafter. I dropped college to work in the Internet bubble as well (though not as an entrepreneur, which I regret). I don’t recommend people to just drop college as I did, of course, this is a personal choice and everybody should take an educated decision about their own future. But there are a lot of people – at least here in Brazil – that praise certifications and diplomas much more than hands-on experience. Sure, there are very good certified developers, but there are more bad developers that actually ‘believe’ in their own certifications as a graduation for competency! I don’t see any correlation between diplomas and good developers. What is your opinion about this matter?

Peter: I have a lot of respect for people who take the academic route, and feel that in general they turn out to be better at their jobs than the less well educated. One of the downsides to being indoctrinated in academia, however, is that people with many degrees, certifications and diplomas often act like technological automatons, only understanding one course of action, not able to think dynamically on the fly, and unwilling to try radical new directions. There is a lot of room in the world for people like that, but on the other foot we also need a lot of “crazy” people who try things that initially seem idiotic or radical.

The other problem with people who brag about their certifications is that in many cases they are just proving that they managed to pass a test. Passing a test and having a lot of knowledge doesn’t necessarily mean you develop good problem solving skills or can work as a true developer, architecting solutions from scratch. For example, a builder might be excellent at building houses, but he’s likely to be very poor at actually designing buildings, which is an architect’s job. In IT, especially within smaller companies, it’s often necessary for developers to fill both architectural and development roles, and many people with certifications, but little real world experience, struggle with the design of systems.

AkitaOnRails: We are both Ruby and Rails evangelists, of course, but as myself I don’t see you as a ‘blind’ developer, meaning, the kind that needs to make everything else look bad just to justify putting Ruby up on a pedestal. I actually believe in the ‘right tool for the job’ approach. Not even DHH states that Rails is the solution for every use case. That said, I think Rails is the best approach for a niche of web development. Recently I involved myself – and I regret it – against a troll rant from a Brazilian .net advocate. I don’t dislike .net, far from that. Anyway, do you see yourself involved in rants like this? Or you do try to avoid them altogether? What is your opinion about ‘X vs Y’ type of discussions?

Peter: I think it’s possible to play interesting political games in any situation. As a regular human being I do have, of course, biases and preferences, but it’s important to recognize these for what they are and still be able to see the big picture. I believe in trying to use the right tool for the job (or, more accurately, the most efficient tool for the job – I’m not going to learn a whole new language just to solve one problem), so Ruby Inside runs on WordPress , a PHP-based weblog platform. In terms of bias, I’m certainly rather anti-Microsoft, although I do acknowledge that their approach to the development of GUI applications is light years ahead of everyone else.

I don’t try to avoid or become embroiled in X vs Y types of discussions, although I do tend to play devil’s advocate when they come up. It is important to remain as scientific and objective as possible in these situations, but I’ll certainly let a little bias show from time to time, as any human would. For example, I (and I’m far from alone on this, although a lot of people seem not to want to rock the boat publicly) am not particularly keen on David Heinemeier Hansson’s overprotection of the terms Ruby and Ruby on Rails, and especially the Rails logo to the point where he rejects the Rails logo’s usage on books that he has not personally worked on. I’m not afraid to show some bias and reference issues like these from time to time.

AkitaOnRails: This leads to another question: everyday I see people here asking us “Why should I choose Ruby on Rails over X?” Personally, I think there is no clear answer for that. Being experienced in many platforms like ASP.NET, Java, PHP, Perl, I did my choice based not only on raw performance (which Ruby lacks, for now), not even a mature library set (which is evolving fast, though). Some reasons were based solely on me being happier with this platform. I do think Ruby enables me to write more ‘beautiful’ code as Marcel Molina Jr. explains. But “Beauty is in the eye of the beholder”. What’s your take on this matter? Being an experienced programmer in other platforms, why did you choose Rails?

Peter: I was a Perl developer before I came across Ruby and Rails, and when I did see Rails, the first thing I did was try to clone it in Perl. Unfortunately, I’m definitely not in the top 10% of programmers, by any means, so my attempt was messy. I tried using Rails directly, reluctantly, with a self promise not to get suckered in to learning a whole new language, but merely to develop what I needed to develop at the time. Of course, the rest is history and I’ve been converted over to Ruby entirely, due to its consistency, ease of use, and the way it just feels like a “natural” language.

Back in 2004, Ruby’s library set was poor, RubyGems wasn’t popular, and there weren’t many books available, so it seemed like a poor choice, but these things have all changed very rapidly, and I’d say Ruby’s library set in 2007 competes very strongly with, say, Python, and while the raw numbers are behind those of Perl’s CPAN, the quality and modernity of Ruby’s libraries exceed it. Ruby is now a strong choice.

AkitaOnRails: I have to extend this theme because I think it is relevant as we see more and more switchers to Rails. I see 2 kinds of developers coming on board: the first being experienced programmers that were not satisfied with their current day-jobs and tools and wanted to try something new and refreshing. The second being novice developers that think Java or .NET have a higher learning curve than Rails. The latter worries me a little bit because some tend to think that Rails is too easy to be considered professionally and some even get frustrated in finding out that Ruby is not Basic, and you do have to make an effort to learn its way. You do training as well, what is your experience dealing with newcomers?

Peter: I have not dealt with anyone who’s completely new to Web application development, so I cannot give a strong answer. However, I certainly don’t feel that people are choosing to learn Ruby or Rails because it’s perceived as “easy.” Indeed, I think it’s the more experienced developers who consider Rails to be easy, and that’s why they’re giving it a go. Beginners tend to just go with whatever is in fashion at the time or what’s the most popular. There’s no shortage of newcomers to all of the major languages right now, it’s just that Ruby is now getting its fair share at last.

AkitaOnRails: 3 years ago the majority of developers predicted the “demise” of Rails as a viable alternative. Today we just saw Rails 2.0 out of the door and the Rails community growing faster and stronger each day. Now people do think of Rails as the contender to beat. I think this is a good thing because we tend to move faster (develop more gems, innovate more) when we are under pressure. Do you think people are accepting Rails more or do you still see barriers for Rails adoption, at least there in England or Europe in general?

Peter: I disagree with your assertion that three years ago a majority of developers predicted the demise of Rails. I think there were just a lot of defensive programmers choosing to disregard it. I also don’t think the majority of developers on other platforms see Rails as something to beat. There’s actually a lot of blindness in the branches of the programming community to the efforts being made in other branches. The majority of .Net developers don’t care about, and most likely haven’t heard of, Ruby or Rails. Developers on dying platforms, such as Cold Fusion, are more familiar with Rails, simply because they’ve been looking for alternatives, but Java and .Net are still taking on new developers at a far higher rate than Ruby and Rails. There is a large difference in what the blogs and other online media want to report and what’s actually happening out there.

AkitaOnRails: There were many things that Ruby was lacking and now we just saw Ruby 1.9 released. It is still no stable 2.0 but this is a clear sign that the language is evolving fast as a mature platform. Free performance is always welcome, but this release will break many things. Are you already using 1.9 or do you feel it’s still not ready for prime time?

Peter: I’ve played with Ruby 1.9 a bit, but I don’t routinely use it. I don’t consider myself a programmer in the academic sense. I like to be aware of, and dabble briefly with, technologies on the fringes, but I’m not one of these super-programmers who downloads, say, Erlang or Haskell and tries to learn it in a weekend. Ruby 1.8 will definitely remain my main Ruby interpreter for at least a few months yet.

AkitaOnRails: Our Brazilian community is still very young and small, I usually describe it as the US community from 2004 ~ 2005. We still didn’t reach critical mass. How are things going there at England or Europe? Is your community big enough that the market started to demand Rails related professionals already?

Peter: The market in the UK is still very small. There was literally nothing in 2005 but things have grown quickly in the last couple of years, although it’s still perhaps 1000th the size of the market for Java or .Net services. There are usually a handful of Rails related jobs floating around, but so far I’ve found that Rails is a technology that agencies are using for their clients behind the scenes, rather than something being actively demanded. I am not directly involved in the UK Ruby or Rails communities, however, so I cannot provide much more detail than this.

AkitaOnRails: This is just for amusement: if you were DHH or Matz what would you like to change in Rails or Ruby? Or maybe, where do you see both of them leading to?

Peter: If I were DHH, then Rails probably wouldn’t exist, simply because I think a lot of his excellent evangelism of Rails is driven by some craving of fame. At a more practical level, though, I would make Rails less opinionated and more customizable, simply because a lot of the opinions are too rigid to be practical in the real world. The initial opinion of relying on the database to provide information about model structure was a good one, but then along came migrations.. and suddenly we’re defining table layouts in code after all, often in many tens of different files! Why not just put the attributes into the models directly and let Rails take care of the schema changes transparently? DataMapper does this, and ActiveRecord is starting to look backwards in comparison with its bizarro approach.

On the Ruby side of things, I think Matz has done an excellent job technically, although whereas DHH scores high on the PR scale, Matz seems reluctant to promote Ruby at all. I think this is both because this sort of modesty is a honorable trait in Japanese culture, but also because Ruby is something he developed to scratch an itch, and another million or so developers using the language won’t benefit him significantly. He’s a great guy, but he lacks the craziness of Larry Wall (Perl) and the authority of Guido van Rossum (Python) and is incredibly lucky that he can rely on a very strong and vocal community to promote Ruby. One benefit of Matz’s hands-off approach is that people feel that they can take authority over various areas of the language and community for themselves without having to answer to a higher authority.

In terms of the technical side, the only thing I’d perhaps do if I were in control of the development of the Ruby language would be to make heavy pushes towards more functional constructs in Ruby 2.0. JavaScript and Perl 6 are taking strides into the world of functional programming, and Ruby needs to follow. I’d also be considering concurrency very strongly and making this a major focus for new language feature developments.

AkitaOnRails: What would be your recommendation for novice Railers here in Brazil? I mean, for those that wants to learn both Ruby and Rails but may need some guidance to start?

Peter: Well, if you can speak English, then naturally I’m going to suggest my book “Beginning Ruby” published by Apress! Unfortunately I’m not familiar with Portuguese resources for learning Ruby or Rails, although I’m sure you are! One free resource I would definitely recommend to newer developers is Chris Pine’s “Learn to Program.” It’s available as a print book, but a more raw version is available to read free online.

AkitaOnRails: Now, speaking of information, we had a very explosive new year beginning. Zed Shaw literally exploded in rants, cursing and every form of aggressive communication. As I stated in my previous article I was personally shocked at first, but I read it through – trying to filter down all the hatred, and uncivilized manners – and there’s actually some information buried there. What is your take about this episode?

Peter: Personally, I think Zed is spot on. I don’t like the way he has specifically attacked a few people personally (Kevin Clark, for example) for minor infractions. His language is harsh, and he clearly has a lot of vitriol. Behind all the swear words and chest beating, however, is a lot of truth. I sympathize with his main arguments, especially those raised in the “round two” update published on January 3rd.

Zed touches on the way some people form into elitist, ego driven gangs, such as Rails Core and the caboo.se. I was involved with the caboo.se for a while myself, and I have a great respect for many of its members (Courtenay Gasking and Amy Hoy, for example), but the whole thing stank too much of elitism to me, and I’ve distanced myself from all of this Rails related online socialization. I could recall quite a few pathetic condescending incidents from members of these communities myself, but unlike Zed I leave quietly. I think one of the biggest causes of the tension, gang-forming, and general childishness is that many of the people involved with these gangs have no social life outside of this community. This probably explains why parlor games like Werewolf and general goofery are so popular at the conferences, as opposed to, say, actually revolutionizing the field.

Lastly, Zed is spot on with his comments about the “Programming Ruby” book, more commonly known as the “Pickaxe.” I tend to remain politically correct on this subject, especially as I have a competing book, but the Pickaxe really is stunning in its mediocrity. It sells well because it has first mover advantage and name recognition, and no-one else has bothered writing a better reference book (The Ruby Way is excellent, but it’s not a reference book). For having the only printed Ruby reference book, Dave Thomas deserves credit, but it’s still mediocre.

AkitaOnRails: For the good or the bad, I have mixed feelings about this whole episode. I won’t judge anyone yet. Let’s see how it unfolds. And of course, I really appreciate Peter Cooper’s patience with me, kindly answering all my questions. Thanks a lot!

Taking clues from the venerable Apple WebObjects he set his way to implement Seaside and his very successful web product, Dabble DB. Check out who the man is, what are his opinions and why he is so relevant to the Ruby and Rails community even though he advocates another language and another framework. Sounds strange, but when Avi speaks, you listen.

He was very kind to provide me a very long interview. It is so long I divided it into 2 parts. This is the first Part. I will release the second one in a few days. Hope you all enjoy it.

AkitaOnRails: Great. I am kind of nervous. Up until now I only interviewed Ruby and Rails advocates including Matz and DHH, this is the first time I am looking for an outside point of view.

And I know you’re very opinionated about Smalltalk and Seaside, but at the same time you have very strong ties to the Ruby community. May I start asking you more about Avi Bryant, the person? Meaning, what’s your history in the programming field?

Avi Bryant: I started programming quite young; it was something I did with my brother and father for fun, mostly writing games in C on the early Mac. As I got closer to college age, my interests turned more to theatre and film, and so when I started university it was essentially as a film major.

But in the long run computer science sucked me in, and I ended up graduating with a CS degree and working in a software engineering lab on campus. That was when I first started programming seriously, and particularly where I got especially interested in software tools.

This was around the time that the very first builds of Mac OS X – when it was essentially still just NeXTStep, without Aqua or Carbon or anything – started to become available, and since I had always programmed for the Mac I set out to learn Objective-C and Cocoa. That was my first exposure to “real” OO, and it was intoxicating.

At the same time, I was doing some web consulting on the side for some extra money, and so I was looking for a language that reminded me of Objective-C but was more dynamic and better suited for the web, and so eventually stumbled across Ruby, which fit the bill nicely.

I was still enamored with all things NeXT and Objective-C, and so when I started to work on web tools for Ruby I took NeXT’s WebObjects framework as my model, and ended up with IOWA, which is very close to being a port of WebObjects to Ruby. I presented that at the first RubyConf, which was, I think, in 2000.

RubyConf was collocated with OOPSLA that year, which I was also attending to give a talk on my research from the software engineering lab. Now, when OOPSLA started out, it was full of Smalltalkers, and even though the mainstream of OO software development – and thus the main focus of OOPSLA – moved on to Java and C#, the Smalltalkers kept coming. But instead of participating in the rest of the conference, they’d just set up a semi-official “Camp Smalltalk” with their own projector and their own schedule and hack and demo Smalltalk stuff all day.

I had of course been aware of Smalltalk, as the source of a lot of the ideas behind things I was interested in – Objective-C and Ruby, obviously, but also TDD, wikis, even the Mac to some extent – but I had never gotten anywhere with actually using it; the environments were too different, the documentation too sparse, etc.

Sitting for a day at Camp Smalltalk was the boost I needed to get past that. And once I got a taste of Smalltalk, I couldn’t let go – I started right away on a port of IOWA to Smalltalk, and convinced my next couple of clients on web projects that I should use Smalltalk instead of Ruby. This port became Seaside, and over the next several years I built up a consulting business around Seaside training and development.

Smalltalk web development may seem like a small niche to build a business around, but the fact is that nobody else was doing it at the time (for that matter, nobody else was doing web development in Ruby at the time either – that situation is a little different now).

Some people to mention in this history – Julian Fitzell was my early partner on the web projects, and we worked together on the first version of Seaside. Colin Putney was one of my early clients, who had come to me based on my work in Ruby and I had to work hard to convince to use Smalltalk; he’s now a confirmed Smalltalker and works with me on Dabble DB. Andrew Catton worked with me at the software engineering lab, and was a frequent collaborator on various other projects over the years.

A few years in, a little fed up with consulting, Andrew and I started to work on a product on the side, and that project became Dabble DB. Last summer, we were lucky enough to get funded to work on that full time, and that’s what I’ve been doing since.

AkitaOnRails: I see that you like the fact that Smalltalk is not “mainstream” in the Java-sense of the word. I can see the advantages and I agree with it. It’s like “we are a small group of very smart people”. Much like Apple’s own “Think Different” campaign. On the other hand the masses tend to like big bang marketing and support. What do you think of this culture clash? The most important part is: here in Brazil the market is very closed off to non-widely-commercially-supported tech. Meaning most of them don’t like anything different than Java or .NET. How do you convince your customers about using Smalltalk – if this is actually a matter there in the US?

Avi Bryant: Yes, that’s a tricky balance. Some of the people I knew in the early days of Ruby’s popularity – post-Pickaxe, but pre-Rails – have complained about how much more popular Ruby has gotten lately, because inevitably the signal-to-noise ratio goes down on the mailing lists and at the conferences. Smalltalk doesn’t have that problem – in particular, Smalltalk has a nice mix of very experienced hackers – people like Dan Ingalls, who has been writing language implementations for more than 30 years, and has just an amazing depth of knowledge to share – and young troublemakers like myself. When you get too popular too quickly, you get a huge influx of young, inexperienced, energetic people, and the voice of experience tends to get drowned out. Not that new ideas aren’t valuable – they’re essential – but I think in this field we tend to overlook history far too much.

As for using non-standard tech, I think the best way to look at it is as a competitive advantage. Paul Graham points out that on average, startups fail, and the same could probably be said about software projects in general. So if you want not to fail, you have to do something different from the average, and using better technology is a great way to do that.

But you clearly have to be in a situation where results matter more than policy – it’s almost impossible to sell an enterprise IT department on a Smalltalk project. On the other hand, a small business owner couldn’t care less what technology you use when building a custom app for him – it’s not like he has an army of Java developers who are going to maintain it.

In general, the closer the alignment between the person who will be using the app, and the person who is writing the checks, the more they’ll care about having something that actually works, and the less about the buzzwords it involves.

That was my strategy when doing Smalltalk consulting, and that’s also our strategy with Dabble DB – we sell to users, not to CIOs.

AkitaOnRails: That’s exactly how I think. Which boils down to you having a startup. I can understand being fed up by consulting work, but what led you to choose to open your own company to sell your own products? I heard that Tim Bray is also an investor for your company? Many people wouldn’t like to lose their comfortable job positions for something as risky as this. What was the reason for this direction in your career?

Avi Bryant: When we were consulting, it became clear to Andrew and me that a) most companies, especially small businesses, couldn’t afford to hire us, and b) even those who could, they could only hire us for the very most important projects. The only way we were going to scale our business, and help as many people as possible, would be to effectively replace ourselves with software: build a product that would let them do the simple projects themselves.

We didn’t abandon everything to go and build this, however – we kept our existing clients (in fact, we took on some new ones to get a little extra capital to hire a web designer, etc), and we worked on the product on evenings and weekends, where we could.

As the product got further along, and it became more and more clear that there might be something worthwhile there, we started to work on it more in earnest, but still keeping some cash coming in with consulting.

Eventually, when the product was ready to launch, we knew we couldn’t handle Dabble DB customers at the same time as our consulting clients, and so we had to drop them entirely. That was when we took some investment from Ventures West and from Tim Bray.

So it was a fairly gradual transition, which made it a lot easier to do.

AkitaOnRails: But even then, were you doing Smalltalk/Seaside consulting only? Or were you working on different languages/frameworks for customers and Smalltalk afterwork? And for non-starters how would you briefly describe Dabble DB? I hear that you’re about to release a companion product for it, right? Can you describe them?

Avi Bryant: During that period, the consulting was all around Seaside, yes – there were some dark times earlier when I had to take some Java work :-)

Dabble DB is a hosted web app for small businesses and teams who need some kind of custom data management – maybe they need custom project management, or a custom contact database, or a custom HR application, or event planning, or just their DVD collection, but for whatever reason the off the shelf software doesn’t do what they need. Dabble lets them interactively build very rich reports on their data, with structured grouping and filtering, simple calculations and subtotals, and display it as a table, or a chart, or a map, or a calendar. You can then share these reports with people, or invite them in to enter or modify the data itself.

The companion product you’re referring to is, I think Dabble Pages. This is really just a new feature set on top of Dabble DB, which all Dabble customers get. What we’ve found is that in any organization there are usually a small number of people who want the full power of the Dabble DB user interface, and want to build new reports or add fields or restructure their data, etc. But there are also a much larger number of people who just want a simple web page where they can enter new data or view a report. So Pages lets that small core group build something for the larger group. It’s sort of like a souped up version of WuFoo or other form builders, but with Dabble DB as the administrative back-end.

It makes Dabble DB almost into a web development tool, in the sense that the administrative users really are building simple applications for the other people in their organizations – but without any whiff of programming, so of course the applications are very limited.

AkitaOnRails: Changing subjects a little bit, one thing that I do admire about you is your relationship with the Ruby/Rails community. Those are different paradigms and technologies, but you never dive in flame wars like ‘Smalltalk is better than Ruby’ just for the sake of it. On the contrary, to me it feels like you do like Ruby, as you presented at RailsConf about the possibility of running Ruby empowered by a modern VM like GemStone’s. But, you did switch from Ruby to Smalltalk. What was it that ‘clicked’ for you about Smalltalk? Continuations? The whole coding-in-runtime? The tighter environment integration?

Avi Bryant: So, first of all, I very much do like Ruby, and Dabble DB includes about 3000 lines of Ruby code (I just checked) along with however much Smalltalk (and a sprinkling of Python).

What clicked about Smalltalk was a few things: one was simply the maturity, both of the community and of the technology. Smalltalk implementations have been refined over the last 25 years, and they’ve really benefited from it.

One of the major benefits, in my opinion, is that Smalltalk VMs are fast enough that it’s reasonable to implement all of the standard libraries – Array, Hash, Thread, and so on – in Smalltalk itself. So there’s no barrier where you switch from your Ruby code to the underlying C implementation; it’s “turtles all the way down”. This may not seem like a big deal but once you have it, it’s really hard to give it up.

Rubinius, of course, is moving in this direction for Ruby, and mad props to Evan and everyone else working on that project. For me, the Smalltalk environment is also key.

In some ways it’s the same things I talk about with Dabble DB: you don’t have to choose one way to organize your data, you can have one view that’s grouped by date and another filtered by customer and another where a map is the only way to understand. In Smalltalk, you don’t have to choose one file layout or order of methods and classes in your code; you’re constantly switching views on what’s basically a database of code.

My friend Brian Marick hates this, because you lose any “narrative” that might have existed by putting the code in a certain order in the file, and it’s hard to get used to, but for me, it’s the ultimate power tool for hacking on a large code-base.

That’s it for Part 1. Stay tuned for more Avi, talking about Smalltalk and Ruby on Part 2 of this interview, to be released in a few days.

It’s been a while since my last international interview, and I am back with none other than one of the people responsible for bringing Ruby to the .NET platform. That’s correct, I’ve been covering a lot about JRuby and Rubinius but we can’t neglect that one of the biggest platforms out there in the market is receiving the Ruby treatment as well. So I invited John Lam , who kindly answered several questions regarding this endeavor.

Remembering that IronRuby – named after IronPython, the first of the main open source dynamic languages built on top of .NET – is a true open source project, and also has a 3rd party addon for Visual Studio.NET, so programmers used to the VS.NET workflow can get onboard with a lower learning curve ahead of them.

Despite opinions against Microsoft just for the sake of arguing, the fact remains that Java and .NET represent the biggest corporate development market today. And this is also a fact that the Ruby meme is spreading in a very fast way. Being built to run on top of both the JVM and the CLR represents Ruby being enabled for market niches that it wouldn’t reach otherwise, and this is a huge win. I talked a little bit about this in my article (in portuguese): For myself to win, the other one has to lose. There are very intelligent people at Microsoft, John Lam being one of them.

That said, let’s get started:

AkitaOnRails: First of all, I always ask my guests about their careers: what was your path up until now? Meaning, how did you get yourself involved in computing, what drives you into it?

John Lam: I wrote a lot of software before I graduated from High School in 1986. I shipped a bunch of commercial software for the Commodore ‘platform’ (aka PET, VIC-20, C-64/128) back in the day. In 1986 I decided that there was no future in computing (which shows why folks shouldn’t take my opinions seriously :)) and decided to study the physical sciences in University. I graduated in 1995 with my doctorate in Organic Chemistry and decided that there was no future in chemistry (which, if you look at how the reverse engineering of life today is based a lot on cross-discipline interaction with other branches of life sciences, shows why folks shouldn’t take my opinions seriously :)).

Once I decided to work in computing again, I took up an obscure new development platform called Delphi which shipped in 1.0 form around the time that I re-entered the field. I spent a bunch of time working on integrating Delphi with COM, which is where I originally hung my hat. The dark side of COM pulled me into its orbit and I’ve been working on Microsoft related stuff since then.

AkitaOnRails: I see that you’re at Microsoft since January 2007, is that correct? How did you get in there, were you hired with the IronRuby project in mind? Is this your main assignment today?

John Lam: Yes, I started in January. Before joining the company, I worked on an open source bridge between Ruby and the CLR called RubyCLR. When I decided to start promoting that work at conferences, I crossed paths with the DLR team at Microsoft. Mahesh Prakriya graciously donated half of his talk at Tech Ed in 2006 so that I could talk about RubyCLR. After much hand-wringing/mind-conditioning I wound up accepting their offer to work full time on a Ruby implementation on top of the DLR and we (my wife, two kids and dog) arrived in Redmond in January.

AkitaOnRails: What were the motivations behind starting IronRuby? How did it start? Were you the original and sole developer of a .NET
implementation of Ruby?

John Lam: We want to demonstrate that the DLR is truly a multi-language dynamic language runtime, so we had to build a number of languages on top of it. Ruby certainly fit the criteria: dynamic, strong user community, lots of attention, so it made sense to implement it. IronRuby has definitely forced a lot of design changes to the DLR, making it less ‘Pythonic’ which is a good thing for other folks who are interested in building a dynamic language on top of our platform.

I’m definitely not the sole developer on IronRuby. We have two great devs working on the project: Tomas Matousek and John Messerly and they’re doing awesome work. Our tester, Haibo Luo, is doing great work to make sure that we ship a high quality implementation. Haibo’s work will be released along with IronRuby, which means that the community can also benefit from his test suite.

AkitaOnRails: Not many people here are familiar with the DLR and Silverlight, can you briefly describe them for non-starters? What makes a ‘DLR’ different from a ‘CLR’ besides jumping from “C” to “D”? :-)

John Lam: The DLR is a platform that layers on top of the CLR. This means that we can run anywhere that the CLR can run, which of course includes Silverlight. Silverlight is our cross-platform browser-based platform that runs on Windows, Mac and Linux (via the Moonlight project). We ship a cross-platform version of the CLR which means that you’ll be able to run DLR-based languages with just a 5MB initial download into your web browser.

AkitaOnRails: This is delicate: I once read Ola Bini and Charles Nutter criticizing the “Microsoft Way”, meaning that it would be extremely hard – if not impossible – for you to write a 100% complete Ruby 1.8 implementation on top of .NET without being able to peek into the source code. The reason being the company policy toward open source licenses. Was that the case?

John Lam: I think that IronPython is the existence proof that you can build a compatible implementation of a dynamic language without looking at the original source code. In our work with IronRuby, I can tell you that not being able to look at the source code hasn’t prevented us from figuring out how Ruby accomplishes some feature – it’s all about designing the right experiments.

AkitaOnRails: I also saw that IronRuby is going to be released under open source-compatible license from Microsoft, making it probably the first true open source project ever from Microsoft, meaning that you would not only allow people to take a look at the source code but also to contribute back to it. How far is this?

John Lam: The Microsoft Public License was officially sanctioned by the OSI in early October. IronRuby, IronPython and the DLR are all released under this license.

We’ve already accepted contributions back in from the community, and are continuing to accept more contributions from the community.

We’re actually not the first Open Source project at Microsoft that accepts contributions back from the community. That honor belongs to the ASP.NET AJAX control toolkit which is an enormously successful project with > 1M downloads to date.

AkitaOnRails: The DLR was originally conceived as an experimental way to run Python on top of .NET. Did this make it easier for a Ruby implementation or do you feel that Python and Ruby were different enough to guarantee lots of tweaks into the DLR to make it feasible?

John Lam: It’s fair to call the original version of the DLR the “Python Language Runtime”. Adding Ruby to the mix forced a lot of additional changes to the DLR, the most significant being the refactoring of the runtime to migrate all of the Python specific idioms (which were thought to be general but turned out not to be the case) into IronPython.dll. What remains is a core which our the 4 languages developed by MSFT (IronRuby, IronPython, Managed JScript and VB) continue to use effectively today.

AkitaOnRails: Is it a goal to be 100% compatible with MRI 1.8? Or are you going toward a sub-set of the Ruby implementation? This would make it possible to write .NET programs in Ruby, using the full set of .NET libraries and frameworks as ASP.NET, Windows.Form, LINQ, etc. Or are you planning on a full blown MRI compatible implementation, enough to be able to run an entire Rails application?

John Lam: Our #1 priority is being as compatible as possible with MRI 1.8.×. The one feature that we will not support is continuations, which is consistent with JRuby. Other than that (and in the absence of a Ruby spec), compatibility will be judged by whether we can run real Ruby applications and community developed Ruby test suites. Obviously Rails is an extremely important Ruby application, which we will absolutely run on top of IronRuby.

.NET interop is our #2 priority. We will integrate with the .NET platform, and allow you to write Ruby applications using the rich set of .NET libraries. Obviously things like Silverlight, WPF, LINQ, WinForms, and ASP.NET will be supported.

AkitaOnRails: Ruby grew as a collective and low profile effort that really exploded 3 years ago because of Rails. That’s one reason no one ever bothered to write a formal specification and test suite. You have Alioth, you have the current YARV test suite. Did you have a hard time because of this lack of formal specs? Are you in contact with people like Charles from JRuby, Koichi Sasada from YARV or Evan Phoenix from Rubinius? What do you think of all these efforts going on in parallel? Is there any way you can contribute with one another?

John Lam: We’re using the Rubinius spec suite today, and plan on shipping it along with our sources to ensure that we catch compat bugs early in the process. I’m guilty of not participating as much as I would like to in talking more with Charlie, Evan and Koichi about collaborating on specs. We’re focused on getting the core language up and running today, and, like a lot of other fast-moving projects, have been neglecting to document the things we have learned. Once the pace of the project slows down a bit, I’ll be able to spend more time engaging with the other implementers.

AkitaOnRails: Ruby has a lot of known deficiencies. Koichi is trying to improve on some of them for Ruby 1.9. How are you tackling problems like lack of native threading, simple mark and sweep garbage collector, no virtual machine and thus no byte-code compilation, performance issues, lack of native internationalization (Unicode), lots of C-based extensions.

John Lam: We get a lot of things for ‘free’ by running on top of the CLR. Native threading is supported in IronRuby. We use the CLR garbage collector, which is a world-class GC. We compile natively to IL, and our IL can be GC’d as well which is goodness for long-running server based applications. We will obviously support CLR strings, which are Unicode strings and we will also support Ruby mutable strings which are based on byte arrays. We will not support C-based extensions at all, instead we will be porting the existing C-based libraries in the Ruby Standard Library to C# (or Ruby).

AkitaOnRails: Are there any benchmarks comparing IronRuby with the MRI? Do you think you’re already outperforming it? Are there any issues today around Ruby performance over the DLR? Or more interestingly: is there any comparison between Ruby on DLR and C# or VB.NET over the CLR?

John Lam: We are faster than MRI in the YARV benchmarks, but that number changes daily. One great thing about our team is that we have a performance regression test suite that we regularly run across all of our code. This makes it trivial to pinpoint a build that causes a regression in performance, even if we postpone that work for a while.

We have done zero performance tuning so far; we’re just using features that exist in the DLR today, like polymorphic inline caches for efficient call-site method caching. As our implementation matures, we will tune IronRuby to perform well on application-level benchmarks (not just microbenchmarks which are useful to us but not so interesting to real customers). The DLR (and CLR) is becoming faster all the time, and any dynamic language that is built on top of the DLR will benefit from these improvements.

Obviously, languages like C# and VB.NET are faster than Ruby today. But we are confident that we can significantly narrow that gap over time.

AkitaOnRails: What about tooling? I saw you tweaked your Visual Studio.NET to have the Vibrant Ink that we – TextMate users – are used to, and it looks nice. I even saw a screenshot of code folding of Ruby inside of Visual Studio. How is its support for Ruby?

John Lam: Right now our team isn’t involved in VS integration. However, the good folks at Sapphire in Steel are doing great work at integrating IronRuby into VS.

AkitaOnRails: You mentioned in an interview the possibility of Ruby running on top of Silverlight. How feasible is this today?

John Lam: We demonstrated IronRuby running on top of Silverlight at RubyConf. The talk should be posted soon. We’re planning on rendezvousing with Silverlight in time for their next CTP (late this year / early next year). At that time, you will be able to code in Ruby that runs in your browser.

AkitaOnRails: Microsoft is learning on how to deal with the open source community. Do you think the adoption of technologies as Python and Ruby are helping the company to understand the open source way? What do you think still has to improve? Meaning, if you could, what would you change?

John Lam: Microsoft is a big company and we’re working very hard on working together with the Open Source community. IronRuby is one of the most visible Open Source projects in the company, and we’re helping to pave the way for other teams that want to engage with the Open Source community. Change happens incrementally at big companies and it’s been fun being a part of that change!

AkitaOnRails: Silverlight was released over an open source license as well, so that the Mono guys were able to run it very quickly over Mono. Are you close to Miguel De Icaza? What do you think about Mono? An IronRuby running on top of it means a multi-platform Ruby that can run over Linux. If you ever get 100% compatibility with MRI and Rails, this means running over Apache+mod_mono on Linux and over IIS on Windows, everything developed under Orcas. Are there any plans for this kind of integration and end-to-end solution?

John Lam: Actually, Silverlight is not released under an Open Source license. The DLR, IronPython and IronRuby are all released under the Microsoft Public License, which makes it possible for Miguel and his band of merry men to just ship it on top of their Moonlight implementation.

I love how pragmatic Miguel is. We showed IronRuby running on top of Mono at RubyConf and that was largely due to the Mono team fixing a bunch of bugs at the last minute so that we could demo (thanks!). DLR + Iron* tends to push the C# and the CLR pretty hard. We’ve uncovered bugs in the Microsoft C# compiler, and we’ve uncovered bugs in gmcs under Mono as well. But thankfully, both the C# team and the Mono team have been great at fixing those bugs to unblock us.

AkitaOnRails: I think, that’s it for today. Any closing remarks for the Brazilian audience?

John Lam: Thanks for sending along a great set of questions! Good luck with your Ruby adventures and I hope you’ll have a chance to check out IronRuby in the future.

This is another big interview. This time with Jamis Buck, the programmer that helped David Hansson himself right at the beginning of Rails, at 37signals.

Today most renowned for his achievements in Capistrano and a lot of other Ruby libraries as sqlite-ruby bindings and Net::SSH. Jamis was very kind to give us the opportunity to know more about his career and the beginnings of the Ruby on Rails story.

AkitaOnRails: Ok, so let’s get started. first thing I always ask my guests: what’s your background? I mean, when, how, why did you start on computer programming? Did it begin the classic way, as a hobby, or was it a more recent thing because of career or something else?

Jamis Buck: I started programming when I was in the 10th grade, when my mom got a brand-spanking-new Tandy personal computer. It had all of 20 megs of disk space, and came with GW-BASIC. I’d done some programming before, turtle-graphics and a bit of BASIC, but that Tandy was when I got serious about it. I taught myself GW-BASIC from the little reference book the computer came with and wrote a few simple games and utilities. Then I taught myself Turbo Pascal and Turbo C++ in 11th and 12th grades, and it’s gone from there :)

AkitaOnRails: Then you decided to go for CS in college? I read at your profile in WorkingWithRails that you’re a repentant Java programmer? How long have you been programming in Java?

Jamis Buck: Yeah, I then jumped into Computer Science when I got to college. I graduated a bit late, in 1999, after working full-time for the university (BYU) as a programmer for a couple of years. That was where I did all my Java work. We were actually doing web-apps in C, which was not as insane as it sounds! But then we had a change of leadership when a new CIO came in, and he basically issued a blind mandate: “learn Java, ditch C!” So we had to massively retool and get certified and so forth. I was on the team that led the research into what Java tools we should use, and so got a lot of experience that way. I did Java work full-time there for about 2 years, until 37signals came along in Feb 2005 and made me a better offer, and I’ve never looked back!

AkitaOnRails: Wow, so you jumped right in from Java in the university to Rails at 37signals? This is a little earlier than I expected, but as you already mentioned it: how did you get in touch with the 37signals guys? How did you start on Ruby being so overwhelmed with Java before?

Jamis Buck: Well, I’d been involved with Ruby (primarily as a hobby) since about 2001. I’d written a few libraries, like the sqlite and sqlite3 bindings, and at RailsConf 2004, I met DHH. I did a bit of work at the conference there to make Rails speak to sqlite databases. After that, around November 2004, I think, David asked if I would be interested in doing some consulting for 37signals. In December and January I did a few odd jobs for them like adding SFTP and time zone support to Basecamp. And then in February they flew me to Seattle to attend the Building of Basecamp workshop there and they made me an offer.

AkitaOnRails: I understand that 37signals is a small company in terms of number of employees and most of its inner power comes from this very fact. Did it change much since you began, I mean, in the last 2 ~ 3 years? What did you like the most back then and what do you like the most right now?

Jamis Buck: Well, when I came on, I was the 5th employee, and the second programmer. Previous to my coming, David did all the programming and sysadmin work, so I think it was a big relief to him to have someone to share the load! About a year later, we hired two more programmers, and we’ve since hired two more people, including a full-time sysadmin. The changes have been for the better :) I like having a real sysadmin. I can’t tell you how much I hated being a sysadmin, since I had little idea what I was doing!

AkitaOnRails: You seem to be the kind of guy that like the kitchen-sink because you’ve been involved with ‘low-level’ stuff like sqlite-bindings, Net::SSH and so on. Is this the kind of thing that excites you the most? Or it’s just a coincidence and you actually rather do GUI work as well?

Jamis Buck: I actually swore, some years ago, that I would never be a web-app developer :) I’ve always preferred writing the tools as opposed to writing the apps. But I’ve really enjoyed writing web-apps at 37signals. The team is great and the toolset (Rails!) is amazing. I still do enjoy writing libraries (working on Net::SSH and Net::SFTP v2 right now, in fact) but apps aren’t so bad either :)

AkitaOnRails: But back then when you were literally second-in-command you didn’t have much choice, did you? I bet you did some of the Basecamp interfaces? And you mentioned that you didn’t like much of the sysadmin work because you didn’t know what you were doing, but now you probably are an expert because Capistrano rocks in this field. I remember it being called SwitchTower before. Can you tell us its story?

Jamis Buck: I’ve done quite a bit of the work on all of the apps, some more than others. I don’t actually write the UI’s (that’s the domain of Jason Fried and Ryan Singer), but I’ve plugged them in and made them work.

I learned a ton about sysadmin work from when I had to do it. I’m still a major newbie at it but I learned more than I ever thought I’d learn about how mysql really works, for instance :)

When I first released Capistrano, a year and a half ago or so, it was called SwitchTower, but around March 2006 I got a C&D from a company that had the name trademarked, and so I had to scramble for a new name. The Rails Core Team helped immensely at that time, they offered a TON of suggestions for names. Marcel eventually IM’d me and suggested “Capistrano” which just clicked for me and I went with it. I got some flack from some people, who thought the name was lousy, but I laugh now because most people that use cap today probably came to it after the name change and never knew any other name, and they don’t even think twice about it.

AkitaOnRails: You are probably very overwhelmed now because of the Capistrano 2.0 release. It is a major departure from the 1.4.1 if I understand correctly. You rewrote a lot of stuff, had to break backward compatibility. Do you feel you got what you wanted with this release or are you aiming for more in a 3.0 release?

Jamis Buck: Cap2 is pretty much just what I wanted it to be. There are few minor tweaks in the pipeline, but nothing significant. It is about 90% backwards compatible with Cap1, so most people should have no problems switching. And since the 2.0 release, the maintenance load for cap has been really small. I’m very pleased with it. I still need to get busy with documentation though :) That’s the thing that still holds a lot of people back.

AkitaOnRails: What is it that you like the most about cap? You obviously use it at your own deployments at 37signals. I fantasize you having a bunch of shell scripts more than a year ago and then weaving them together to build what was the first release of Capistrano.

Jamis Buck: Prior to SwitchTower/Capistrano, we had a tool that David had written for deploying Basecamp. Back then, Basecamp ran on a single server, so deployment was little more than “svn up”. Later, we moved Basecamp to multiple servers and launched Backpack, and we realized we needed something more powerful for deployments. So I was tasked with writing something and Capistrano was the result. The thing I like most about it is how it takes the tedium out of being a sysadmin. The “cap shell”, in particular, lets you immediately and easily execute arbitrary commands on multiple servers, simultaneously, which is really handy. Of course, “cap deploy” is nice too :)

AkitaOnRails: I remember one episode where David reported about a 72 hours glitch that let Basecamp out. Were you involved in that episode? :-) Headaches like this seems funny when you remember about them now, but obviously not so funny at the time.

Jamis Buck: ha, yeah, that actually happened a few days after I was hired :) We were adding UTF-8 support to Basecamp and the database migration did not go well. We pulled an all-nighter. Quite the introduction to 37signals’ culture :) Fortunately, that kind of thing hasn’t happened since, though we’ve had our share of other emergencies, of course.

AkitaOnRails: Lessons learned? I think that’s the most important outcome of these situations :-)

Jamis Buck: The main lesson I learned that time: when you’re mucking with charsets, make sure you test it on as many non-Latin1 alphabets as possible :) Naturally, we tested it a lot before we deployed, but we didn’t really think to check any Russian or Japanese data that was in our database after the test, which seems obvious in hindsight. We did try some non-Latin1 alphabets during testing but the ones we tried just happened to work in spite of the bugs. Another lesson learned: no matter how hard you prepare, something’s going to fall through the cracks!

AkitaOnRails: I headed a group of very committed volunteers and got your Getting Real book translated into Brazilian Portuguese a while back. I believe you were involved in it too? This book is iconic to picture the 37signals spirit I guess. Do you really move the way it’s written? :-)

Jamis Buck: I was actually not too involved in the composition of the book, though I gave some feedback during the writing process. But yes, that book describes our own processes. There really isn’t anything in that book that we don’t actually do.

AkitaOnRails: How do you operate? I believe that not everybody work in the same physical location? Do you do a lot of remote work? How many programmers/sysadmins/designers are there now?

Jamis Buck: Chicago is definitely the “heart” of 37signals: Jason, Ryan, David, and Sam all live and work there. We have an office there as well, and the Chicagoites go there pretty regularly to work (though they also work from home some, too) Matt Linderman is our writer, he lives and works in NYC. Mark Imbriaco, our sysadmin, lives and works in Chesapeake, Virginia and Jeremy Kemper is another programmer, living in Pasadena, California now. The distributed nature of it works very well for us. I really love working from home, it lets me be more involved in my kids’ lives, for instance, than I would otherwise be.

AkitaOnRails: I hear you’re from Provo, Utah? It is a coincidence as my current boss, Carl Youngblood, is also from there and also studied at BYU :-) You don’t possibly know each other, do you?

Jamis Buck: I actually live in Caldwell, Idaho now, but I was in Provo for about 8 years and yes, I do know Carl! We were both in the Utah Ruby Users Group. He’s a great guy.

AkitaOnRails: You attended many conference for the Ruby community. I don’t remember if you wrote any books. Are you writing some now?

Jamis Buck: I was originally going to write a book about Capistrano for Apress, but that’s fizzled. I’m not actively involved in any book writing at the moment, and don’t really have much interest in it, honestly. I much prefer writing software! And I try to post to my blog once in awhile, too.

AkitaOnRails: Yes, the buckblog, it has lots of great tips that I like to read. Did you start it because of Rails or you already had a Java blog back then?

Jamis Buck: I started it pre-Rails. If you read the early articles, you’ll see they were pretty eclectic. I was still trying to figure out this whole “blog” thing :) I’ve since come to focus on software-development and use my family blog for more personal posts these days.

AkitaOnRails: I was a Java programmer myself and I still try to maintain myself informed about what’s going on in other communities as Java, some PHP, Python. What do you think or feel about other frameworks and technologies as CakePHP, Avi Bryant’s Seaside or even Django?

Jamis Buck: Honestly, I’m a bit ashamed to admit that I don’t keep up on other frameworks as much as I should. Seaside has some neat ideas and my desire to improve my Smalltalk programming skills has made me want to poke into that further. But other than that, I haven’t shopped around much (to my own personal detriment, I’m sure).

AkitaOnRails: And still about Java, I am very excited nowadays about JRuby. I interviewed Ola Bini and I’m also kind of stalking Charles Nutter. Did you know the guys? Did you got interested in trying it?

Jamis Buck: I know who they are, and I’ve met Charles at a few conferences, but I’ve never really had a chance to talk to them in depth. JRuby definitely sounds exciting. I’m very curious, in fact, to know whether it can support Net::SSH or not since that would mean you could run Capistrano under it. But I’ve not had a chance to actually try it out.

AkitaOnRails: Insisting on the Java subject, I posted a while back about some discussions about Design Patterns pointing to flaws in languages, and I just read a post you wrote about Dependency Injection and your Net::SSH library. What do you think about Design Patterns now that you’re a full-time Ruby programmer?

Jamis Buck: Design Patterns are wonderful. They give us a language to talk more concisely about programming. It is so much easier to say “use a factory method” than to circumlocate and confuse each other. As for specific design patterns, some are more useful than others in different environments. As I wrote in my post, Dependency Injection is just not something you need in Ruby, but in environments like Java, it can be a life-saver.

AkitaOnRails: About 37signals again, you are very committed to Apple products. I saw that video on Apple Education with Jason and David. And now we see that Leopard will bring not only Ruby but also Rails, Mongrel and even Capistrano. How is your relationship with Apple?

Jamis Buck: I wouldn’t say there is any “relationship” with Apple. We’re big fans of their designs and interfaces and I think that reflects in our own work. Apple itself is including Rails, Mongrel, and Capistrano in the wake of those tools’ successes. Mac OS X really is a powerful environment for programmers and adding those tools “out of the box” makes it even more so. It makes sense to include the popular Ruby tools, too, since Ruby is going to be a first class citizen when it comes to developing OS X applications, thanks to the adoption by Apple of the Cocoa bindings for Ruby.

AkitaOnRails: You obviously use a Mac for your work. What are the tools you use the most besides Textmate?

Jamis Buck: Well, Capistrano :) Firefox w/ Firebug, Parallels (for IE testing), iTerm (though I’m looking forward to the new Terminal.app in Leopard), Campfire is indispensable, AdiumX, NetNewsWire, Knox (for simpler encrypted filesystem access). I still fall back and use vim pretty often too, for quick edits. There are probably others :) But I can’t think of them off the top of my head.

AkitaOnRails: There are a lot of talk about GTD nowadays. I think I saw your profile at 43folders. Are you into this kind of organization philosophy as well? How do you maintain your routine organized? Many people (myself included) have a hard time differentiating work from hobby and keeping ourselves within the project’s deadlines and such.

Jamis Buck: I’m not a GTD fanatic. I guess I try to be pretty pragmatic about it. I’m terrible about procrastinating, though. Mostly, from 8-5 I try to focus on the tasks that have been assigned to me and outside of that, I work on my hobbies… which happen to overlap work sometimes :) I’m really the wrong person to ask about being organized! If it weren’t for my wife, I think I’d get very little done, actually.

AkitaOnRails: What are you most interested nowadays besides Rails, Capistrano? Not only technology-wise. Any hobbies?

Jamis Buck: I’ve lately become very interested in military history, which has lead me to read about the American Civil War among other things. My wife and I are really enjoying Harry Potter #7 right now, too :) And I’ve become quite attached to my Nintendo Wii lately, as well! I used to be quite into role playing games, but haven’t done that for a few years. I’ve still got a ton of books for D&D, though, and maybe someday I’ll dust them off :) I enjoy spending time with my kids (Nathaniel is 5, and Katie is 3). We’ll be going camping for a few days in a week or two, actually. Always something to keep me busy!

AkitaOnRails: Haha, I see. And do you travel a lot? Outside of the US, I mean.

Jamis Buck: Not really. Last year I went to London for the first time, for RailsConf there, and this year I went to the Czech Republic. But aside from that, I’m really a homebody. I’d kind of like to do more traveling, but I don’t like to travel alone and having kids puts a damper on the spontaneous “get up and go” aspect of travel. We do occasional road trips and stuff, though, which is fun.

AkitaOnRails: Do you hear what’s going on the Ruby/Rails communities around the world? What do you feel about the meteoric rise of Rails in the US? 37signals, of course, was the fuel for all this and the reason we – both Rubyists now – are talking today.

Jamis Buck: I hear some news, but I’ve trimmed my newsfeed quite a bit. I think I only subscribe to 30 or 40 feeds these days. It was really exciting to go to the Czech Republic and see the excitement there about Rails. It reminded me a lot of how things were in the US a couple of years ago.

The rise of Rails has been incredible. There have been growing pains, of course. With wider adoption you begin to accumulate people who aren’t passionate about the tool, and only want to learn it for the resumé, or because a client has asked for it, and I think that dilutes the waters quite a bit. But there will always be a core of passionate users and that’s what makes the community so wonderful. It’s been a really amazing experience to have been a part of that core, almost since the beginning of Rails.

AkitaOnRails: You have the rare opportunity of seeing all this from a VIP perspective near Jason and David. And of course, yourself, being one the propellers of the community. Many people think of David as an arrogant :-) I can perfectly understand his opinionated point of view. What do you think of it, from the perspective of someone who actually know the man?

Jamis Buck: David is one of the most confident people I have ever met. My biggest frustration with him is that he’s RIGHT so often. He’s a very clear thinker and jumps right to the core of the issue, which can be very disconcerting if you aren’t used to that. All those attributes combine, I think, to make people who don’t know him consider him “arrogant”. But he’s not arrogant. He doesn’t put people down in order to raise himself up. He’s just extremely gifted, very smart, and doesn’t beat around the bush. I’m very honored to be able to know him. I’ve learned a lot from both him and Jason.

AkitaOnRails: I don’t know if I got it correctly but Jason Fried was the one that started 37signals? Can you give us a small glimpse on the origins of the company? I know that David is not from the US, how did they get to meet each other?

Jamis Buck: Jason co-founded 37signals, in the late 90’s. It was originally a design consultancy. In 2003, Jason started to work on a project in PHP, and asked around online whenever he got stuck. David responded to his queries several times and Jason finally just asked David if he would write the project. David came on part-time for awhile, then, since he was still finishing school. But eventually, Jason wanted to write Basecamp, and David did all the work on that, part-time, in Ruby. His experiences with that, of course, led to Rails. And eventually David become a partner in the company. That was in 2004, if I remember correctly.

AkitaOnRails: Great. And coming back about you. Now that Capistrano 2.0 is finally released, are you involved in some new open source tool?

Jamis Buck: With Cap2 out the door, I’m really focused on rewriting Net::SSH and Net::SFTP. Some frustrations with their limitations in writing cap2 has made this necessary :) The only “new” thing to come of this is Net::SCP, which is an implementation of an SCP client in Ruby and which I will release when I release v2 of Net::SSH and Net::SFTP.

AkitaOnRails: Oh, and speaking of which, I remember that the former Cap 1.4.1 didn’t play very well on Windows boxes. Is that correct? Any advances in this field or do you feel that supporting Windows is just too much?

Jamis Buck: Well, as far as I know, cap 1.4.1 worked fine on Windows machines. The limitation was that you can not deploy TO Windows. Deploying FROM Windows works fine. And as for making cap deploy TO Windows, I have no plans. The difficulty is that Windows is not a POSIX environment, which is one of the fundamental assumptions of Capistrano, regarding the target machines.

AkitaOnRails: Ah, yes. I never tried that, therefore my confusion. You’re right. Well, I think I already got a lot from you. Anything that you would want to say to the Brazilian community? :-)

Jamis Buck: “Stay passionate” :) Keep the passion for Rails burning, it’s what makes the community worth belonging to.

AkitaOnRails: Couldn’t be better. Any plans for coming here?

Jamis Buck: Ha, none in the immediate future :) I’ve got a baby coming in September, which puts a damper on travel plans :) I’d love to visit Brazil someday, though.

AkitaOnRails: Didn’t know that. Congratulations!

Jamis Buck: Thanks.

AkitaOnRails: Well, thanks a lot for your time. It was a very insightful conversation. I know people will enjoy this.

Jamis Buck: Thank you very much, Fabio. I’m grateful for the opportunity!

This will be a fun interview. My guest this time is my boss. He’s not only an employee of Surgeworks but an active Ruby contributor and participates mainly in the Utah Rails User Groups (URUG). Many of his values and ideas resemble my own.

Carl Youngblood has worked professionally as a software engineer for over ten years, and is currently the Rails Practice Manager for Surgeworks, a software engineering consulting firm (where I happen to be as well :-). In addition to his work at Surgeworks, Carl is an owner of Confreaks, a company that records conferences, seminars and workshops, and is a director of the Mormon Transhumanist Association. He received a bachelors degree in Portuguese from Brigham Young University and a masters degree in Computer Science from the University of Washington.

AkitaOnRails: So, let’s get started. This is very strange to interview someone you already know. But let me try. Before talking about Ruby. The first thing most of my readers get intrigued is about your Portuguese skills. Can you tell them what led you to it?

Carl Youngblood: Yes. My first introduction to Portuguese was as a Mormon missionary in Brazil. I arrived in Sao Paulo having only received a two-month crash course in Portuguese. When I got there, it seemed like I had to learn everything over again – this wasn’t the language they taught me! :-) But after spending two years in Brazil in a context in which it was imperative to converse with other people, I learned Portuguese well. However, I would say that my skills were primarily in conversational Portuguese. Later on I discovered Portuguese literature and developed my written skills further.

Upon returning to the States, I decided to do a double-major in Portuguese and Computer Science. I decided on Portuguese because I enjoyed the language so much, and felt like I had a real talent for it. I chose CS because I have been in love with computers ever since I was 12 and got my first Commodore 64. Or maybe it was 10 years old. I can’t exactly remember.

AkitaOnRails: Where did you stay during these 2 years here in Brazil? What were your responsibilities here?

Carl Youngblood: I spent time in Monte Mor (a small town just outside of Campinas), Campinas, Marilia, Umuarama, Maringa and Campo Grande.

AkitaOnRails: But did you do anything computer related here during this period? People here would be interested to know what do you think of our country, being an outsider that really spent a serious amount of time around us. What’re your personal impressions about the country, the people, the culture, our local habits? Maybe a few words about the good and the bad :-)

Carl Youngblood: My first 4 months were spent mostly meeting people in the streets and in their homes and trying to teach them about our church’s doctrines. Just to clarify, Mormon is a nick-name given to my church because they believe in a book of scripture called The Book of Mormon, along with the Bible. But the full name of the church is The Church of Jesus Christ of Latter-day Saints. Anyway, I spent the first four months teaching people and helping to conduct church meetings, and walking A LOT – we walked about 15 miles a day, and I frequently remember waking up in the middle of the night with pains in my legs :-) After the first four months, I was called to work in the mission office in Campinas as Historiador da Missão – a secretary of sorts. Here my computer skills came into play.

After spending four months in Campinas as mission historian, I was called to help open up a new mission in Marilia, and spent another four months working in that mission office on the computer.

I didn’t really develop any amazing software for the mission, although I did help design an Access database that promptly fell into disuse after my departure :-)

Overall I really enjoyed my time in Brazil. I discovered how much I enjoyed learning another language, and this newfound fascination led me to study this language later at the university level. The Brazilian people I met were for the most part very friendly. It was not uncommon to be invited into peoples’ homes and offered food before we even knew them, and they were usually willing to give you the shirt off their backs. I also really like Brazilian music, especially Brazilian Popular Music (MPB). Tom Jobim is my hero, along with other legends like João Gilberto, Luiz Bonfá, Vinícius de Moraes, and other more modern artists like Caetano Veloso, Gilberto Gil and Djavan. It was wonderful to discover a whole world of music and literature that opened up to me when I learned Portuguese. The food was good too, although I missed a few items from home, like corn chips and peanut butter, strange as it sounds.

On the other hand, I frequently noticed a profound lack of ambition. Most Brazilians we met were content to have a roof over their heads, food in the fridge and a TV. Of course, my observations are bound to be skewed a little by the fact that I mostly interacted with the poorer class of people who did not live in closed highrise apartment complexes, which were difficult for us to make house calls in, unless we had a prior appointment. One of Brazil’s most difficult problems is the large disparity between rich and poor and the consequently small middle-class. When Brazil abolished slavery (one of the last countries to do so), little or no effort was made to assist emancipated slaves in transitioning towards gainful employment in the new economic system, which led to the development of many autonomous shanty towns that were nearly completely isolated from the more “respectable” centers of commerce. This isolation continues today. It is not uncommon to see a wealthy suburb right next to a sprawling slum, as if each were ignoring the existence of the other. Overall though, Brazil is really making headway and the standard of living is improving dramatically.

One event that I distinctly remember was the advent of public internet access and the WWW in 1995 while I was in Maringa, I believe. I had often commented to my brother back in the days of text-based bulletin-board systems that it would be cool to create a markup language that allowed computers to render graphics over a modem connection. When I saw the WWW in 1995, I said, “Somebody stole my idea!” :-)

AkitaOnRails: Haha, so you were involved in online activities since before the advent of the Web per se? And, you went to college after this 2 year period here?

Carl Youngblood: I was involved in modem-based BBSes in our local town of Provo, UT since about 1990 or so. My brother ran a BBS called London after Midnight. He was sort of into the goth/cyberpunk scene (or at least wanted to be :-) ). We mostly used them to pirate games, before I grew up and realized that other people were trying to earn a living off this stuff.

I had already done one year of college as a music major (Clarinet performance) before I went on a mission. After the mission I returned to college and changed majors.

By the way, we were also connected to a worldwide network called Fidonet, where we participated in newsgroups and had delayed discussions. Every night our computer dialed up a system in Nevada to download the new messages.

AkitaOnRails: Oh, that’s cool. Going back to this subject, I was into BBSes as well because of my neighbor. He hosted a hacking BBS called Medusa and we played a lot brute-forcing against passwd files, using Blue boxes and so on. (Damn, we’re confessing a lot of crimes here, hope we’re not going to litigation for that ;-)) Getting back on track. So were you also caught in the period of the Internet Bubble? Doing PHP websites and so on?

Carl Youngblood: Haha, good point. Those BBS days were fun. We felt like we were on the cutting edge. Yes, I was involved in the Internet bubble. Omniture (called MyComputer.com at the time) was in the middle of an acquisition by NetObjects. However, before the acquisition took place NetObjects’ share price had plummeted to the point where they were valued at half of the acquisition cost, which basically meant it wasn’t going to happen. MyComputer.com laid off a bunch of people, but I left a couple months before that happened. My boss, speaking after the fact, told me that I would have been able to keep my job (I was one of the first employees), but I saw signs of trouble and had another opportunity, so I took a job at Legato Systems developing mirroring drivers in Windows of all things :-)

AkitaOnRails: Same thing for me here in Brazil. I was involved with a dotcom company that went bankrupt but I was able to get out a few months earlier. Anyway, so you went straight to low-level programming. How many programming skills do you have? From low-level assembler all the way to Ruby not to mention Pascal, VB, PHP, etc?

Carl Youngblood: Legato eventually closed down their office in Orem, Utah, at which point I went back to school to get a masters degree. I began thinking I was going to do a masters in Portuguese and eventually work as a professor at a University after getting my PhD. However, I found myself constantly working on software projects, even while I was teaching Portuguese classes and doing my graduate studies. I eventually decided I should face the obvious and continue my passion of software development. So I switched over to the Computer Science department and began working towards a masters degree in CS. All this was at Brigham Young University in Provo, Utah.

I first learned Pascal and C++ in high school Computer Science classes. My first professional experience was as a C++ programmer developing Windows applications using MFC. I then got a job doing PHP/MySQL development for Omniture. After that I was doing C-based windows drivers but also doing Ruby, Perl and PHP work on the side. Drivers are really a totally different beast because you don’t even have the regular ANSI C libraries available to you. So the C syntax is the same but that’s about it. I later got a job at Boeing doing more C++ development on Linux platforms, all the while nurturing my Ruby/Rails skills with side projects. I also learned functional programming with
Scheme, as well as a few other languages like OCaml, Prolog and ZPL during my masters degree. I wouldn’t call myself anything but a beginner in most of those other languages, but they definitely opened up my mind to new development paradigms.

AkitaOnRails: You’ve pretty much followed the paths described by Pragmatic Programmers as David Thomas and Chad Fowler: learn new languages, specially ones with very different paradigms. Did you already know their work around these recommendations? Or was it just a natural curiosity thing that led you to learn so many languages?

Carl Youngblood: Well, the CS programs I’ve studied in (BYU and the University of Washington) did a good job of giving us at least some exposure to a wide range of concepts and languages. I really am grateful for my education because it introduced me to ideas and concepts that I probably would not otherwise have studied. I had also read the Pragmatic Programmer and tried to follow the advice of always learning new things and exposing myself to new languages. I am also somewhat of an information junkie. I am nearly addicted to my RSS feeds, and I feel an incredible anxiety if I start to get behind on my reading—almost as if I’m being left behind while the world reinvents itself. The truth is that I usually don’t miss anything extremely revolutionary by being away from the computer for a few days, but sometimes it feels like it.

Or rather, I worry that it might happen. :-) It can be a problem. It’s hard to strike a balance, especially with a wife and kids to pay attention to.

AkitaOnRails: You’re right. That’s another thing that I always shout at my blog posts: being well informed, all the time. There’s no excuse with tools like RSS aggregators nowadays. And if an American went all the way to learn Portuguese and still being a technology specialist like yourself, there’s even less excuse for Brazilians to avoid improving their English and read more.

Carl Youngblood: I’m not sure if you’ve heard the theory of maximizers vs. satisficers. I read about it on somebody’s blog, taken from a book (can’t remember which). Sometimes I … worry so much that there must be a more elegant solution out there that I can’t ever decide when it’s good enough and get down to the actual business of coding.

AkitaOnRails: hm, yes I believe I read something about this.

Carl Youngblood: That is a problem in this era of rapid innovation. It’s always a moving target, and sometimes you just have to say something is good enough for now and move on.

Joel Spolsky talks about this and compares it to the military tactic of laying cover fire and firing while advancing. Rarely is a new technology or buzzword all it promises, but it keeps the competition struggling to keep up.

In general, it seems like society is advancing toward ever-increasing levels of abstraction. Rails is a good example of this. Its whole mentality is to predict as much as possible the things that the developer will be wanting to do all the time and make convenient abstractions for them. But I’m sure this is only the beginning. We can expect future frameworks to do more and more of the heavy lifting for us, which will in turn enable us to accomplish more and more with less and less effort.

AkitaOnRails: Yes, this is a good very point. Because people tend to ‘fall in love’ with one particular super-new technology and forget about all the rest. This leads to pure obsession. And after a while, this ‘new trend’ is forgotten altogether and you just wasted your time.

Carl Youngblood: Yes, we need to stay current on new technologies but try not to get too dogmatic about them. Chad Fowler’s recent address at the MountainWest Ruby Conference talked about this.

AkitaOnRails: Hey, sorry for the pun, but coming from someone with such a formal dogmatic religious formation like yourself, this is a very refreshing thing to listen: not being too dogmatic. In the end its all a matter of common sense, but even common sense has to be trained. Hence the need to be always informed, constantly studying and keeping an open mind toward new stuff while not closing the eyes to all the rest.

Carl Youngblood: I believe that one’s belief structure should never be so rigid as to not allow for modification. I should always be willing to re-examine my beliefs in light of new evidence and be willing to reject or modify beliefs that seem wrong or misinformed upon closer scrutiny.

In defense of my Mormon upbringing, although it is a fairly conservative religion, it does place a high value on education, which inevitably leads to enlightenment and often to a modification of one’s prior beliefs. Fortunately my fellow parishioners still welcome me, though some of my beliefs may be a bit unorthodox and are different from those I had when I was younger.

AkitaOnRails: We do need more of this way of thinking here in Brazil: pragmatics. Anyhow, coming back again, you were very busy for a long time. Learning Portuguese, finishing your major in CS, working in Internet companies, traditional companies. And all the time Ruby was sneaking somewhere, waiting. When did you finally realized you wanted to do more Ruby? Was it during some project at your day-job, or something you decided by yourself?

Carl Youngblood: I think the real awakening was the meteoric rise of rails and 37signals evangelization of the whole getting real mentality. I attended the Building of Basecamp seminar in Seattle and went to the first RailsConf. I was really inspired by Nathaniel Talbott’s speech about homesteading.

One thing that really struck me about my experience at RailsConf was that I saw colleagues there who started using Ruby at the same time I did and whose names appeared on the ruby-talk mailing list alongside mine. The main difference between them and me was that they decided to get actively involved, to develop open source projects of their own, and to build a reputation for themselves in the community, while I dabbled in Ruby as a mere hobby. Fast forward four or five years and they were running their own companies and really making a career for themselves in Ruby. Railsconf was a wake up call for me in which I realized that I needed to be more involved in the Ruby community.

So I decided to release some prior work of mine as a gem and give a presentation at our local MountainWest Ruby conference. I released the SBN (Simple Bayesian Networks) gem. Preparing my presentation and code was stressful but worth it.

Some colleagues and I are also working on a conference recording business called Confreaks. We are hoping to do a better job of recording and disseminating technical conferences for posterity’s sake. It’s a shame that all these great conferences are going on and nobody is recording them.

Anyway, my increased involvement in the community has really helped to boost my career and allowed me to code in Ruby as a day job.

AkitaOnRails: But where were you at the time Rails started to call attention to itself? Did your job allow you to have enough time to dedicate to Ruby or you switched jobs by then? I mean, were you not satisfied by the current state of your career and the technologies you were using? What was your ‘inflection point’? Just seeing RailsConf and the news around Rails?

Carl Youngblood: I was working for Boeing when Rails first came out. It was a comfortable low-pressure job on a DoD project with a long drawn-out timeline that allowed me time for study. I was already in the habit of staying current on technologies (Ruby in particular) and reading a wide range of blogs regularly. I would have accepted a Ruby position at any time but they were still fairly hard to come by. At my request, my boss paid for me to go to RailsConf, so I was already really interested. I had also been working on a freelance project that I had written earlier in PHP, and offered to rebuild it in Rails at a heavily discounted rate just to gain experience with Rails. So I guess you could say that I have been really interested in all things Ruby ever since 2000 but only recently has the job market begun supporting full-time Ruby positions.

AkitaOnRails: And your first endeavor towards Ruby was your SBN open source project and the Mountain West keynote? Can you describe what it was like? At least for me, when I want to ‘say’ something, I get very excited and particularly anxious to have it written (in my case, my blog posts). I can imagine that you hardly slept before the conference. :-)

Carl Youngblood: Yes, I was up many late nights preparing for my presentation. In fact, most of my late nights were spent getting the code ready for human consumption. Knowing that the world at large is going to use your code makes you want to spruce it up a little more, write more unit tests, etc. So I spent the month or two before the conference working on the code and only had about a week to prepare my actual slides. My presentation material was fairly technical, so I ended up writing down my presentation in the slide notes, just to make sure that I didn’t explain something wrong during the pressure of the actual presentation. Had I had more time, I would have liked to have memorized the key points so that the presentation could be a little more natural, but overall I was pleased with the way things turned out. I was also working on the Confreaks project to record the conference, so I was busy on two fronts, not to mention my day job.

AkitaOnRails: And by that time were you still at Boeing? Or were you already at Surgeworks? How did you get into Surgeworks to work on Rails full-time?

Carl Youngblood: Around the time I was finishing my CS Masters degree at the University of Washington while working for Boeing, I received an offer from a friend of mine to return to Utah and work for him as CTO of a Construction Lending Company called Construction Capital Source. One of my main responsibilities would be to develop a construction lending system in my technology of choice, which just so happened to be Rails. I took the bait and had a great time there for about a year and a half. But the company went through some difficulties with the recent crash of the sub-prime lending market. Although their customers are only A-paper borrowers with excellent credit ratings, the sub-prime market has cast a pall on the entire industry, and our investors got scared. I ended up leaving my position and accepting a job at Surgeworks as their “Rails Practice Manager,” which means I supervise all the company’s Rails projects, hopefully while still getting to spend at least half my time coding, although at the moment it hasn’t exactly worked out that way. We have been getting a lot more work coming in since I started, so a lot of my time has been spent writing proposals and project estimates so far, but I’m starting to settle into a groove.

AkitaOnRails: But how did you know about Surgeworks? I remember that you mentioned you already knew Dane Falkner from another occasion?

Carl Youngblood: Dane and I both attend the local Ruby users group meetings in Salt Lake City, so I knew him from there. Upon leaving CCS, I posted a message on the URUG list and was happy to see that there are a lot of Ruby positions available. I don’t think I’ll have trouble finding Ruby work in the future for as long as I’m interested in coding in Ruby :-)

AkitaOnRails: That’s very nice. There are a lot of local user groups there. We still are starting local groups here in Brazil. What can you say about the growth and enthusiasm of the community? What do you see for Rails in the time being?

Carl Youngblood: In my experience, the key to a successful users group is having one or two dedicated individuals with the passion to organize and promote the meetings and line up interesting presentations. If you do that, people will attend. Our local groups have gone through ups and downs depending on the level of the organizers’ enthusiasm. Another thing that seems to help is providing content that is geared towards various skill levels. Making sure that you welcome and help beginners is important. The community is still growing, as evidenced by the many regional Ruby conferences that are cropping up around the US.

I was excited to see DHH’s planned features for Rails 2.0 and his more measured rhetoric as opposed to the first Railsconf, which some folks commented seemed more like Nuremburg rallies :-) I think Rails will continue to mature and has at least 2-3 more years of great success before being eclipsed by some new paradigm breaker.

AkitaOnRails: This is important and maybe you can lay some ideas. Many of the core Ruby and Rails contributors in Brazil, from our local forums, user groups, blog and websites still don’t have the opportunity to work full time on Rails. So they have little to no time to devote do Ruby yet. I believe that there was a time like that there in the US. What do you think was the turning point for the community to flourish?

Carl Youngblood: I think more than anything else, the incredible success (especially the marketing success) of Rails has been the largest contributor to this change. But that didn’t stop passionate Ruby developers from blazing their own trails before then. I think the thing that needs to change first in Brazil is the 9-5 just-a-job mentality. People need to start taking some of the recommendations from the Pragmatic Programmer to heart and making time for extracurricular skills-honing activities, including ruby meetings. This is what folks in the US did and I think something similar will be required in Brazil. I think there is also a cultural difference that is partly to blame. I don’t want to claim that either system is superior, but I think Americans (at least the ambitious types who are in the Ruby community) spend more than 40 hours a week focusing on their craft, whereas I think many folks in Brazil look at software development as something that just pays the bills.

Of course there are many programmers here with the same mentality.

AkitaOnRails: Speaking of passion, David always states about the sense of aesthetics of the community, their inherent sense of beauty towards coding. Did you sense that when you first started? Many say that’s one of the reasons for the Ruby and Rails community being so Mac geared. What do you think about this?

Carl Youngblood: I agree. Matz often quotes a Chinese proverb that says something like “the beginning of wisdom is naming things correctly.” Ruby people tend to spend a lot of time focusing on semantics and making sure that code is not only useful but elegant. I think when you are seeking to maximize your level of abstraction and hence your programming power, these kinds of issues are very important and can provide a huge productivity increase.

Apple seems to reflect a similar mentality in their designs.

AkitaOnRails: I think its nice to mention that I was pretty surprised to see that almost everybody at Surgeworks are Apple-fans. I never expected that. I guess Apple is gaining into the developers workspace as a valuable productive tool. Well, and I know I already pushed you for too long :-) Any other remarks for the Brazilian audience?

Carl Youngblood: I think a lot of Surgeworks’ passion for Apple comes from Dane, who is a big fan. My main advice to my friends in Brazil is to recognize the importance that your individual effort can make. You have a relatively strong IT industry for South America and a burgeoning open source community. I have been very impressed with some of the more vocal Brazilian techies like yourself, Fabio, who have shared their opinions on the Internet. Change is not brought about by the masses, but by thoughtful courageous individuals who break free from the status quo. The more Brazilians take that to heart, the sooner Ruby jobs will start to appear. I am reminded of a quote along these lines:

“Never doubt that a small dedicated group of people can change the world; indeed it is the only thing that ever has.” (Margaret Mead)

AkitaOnRails: Very nice. Well, I think we can wrap it up for today. Thanks a lot! I think people will like this.

Carl Youngblood: Great! Thanks for taking the time, Fabio.