Now, this is a fair question. If you know anything about the blockchain architecture, you instinctively conclude that “no”, it’s quite improbable that anyone will break it. In practice, it’s basically impossible.

It’s quite amazing that most of my peer programmers have a very difficult time overcoming the prejudice against cryptocurrencies. I have no idea where this prejudice comes from, but I know very smart people that can solve the most difficult web scalability problems, but that never once glanced over Satoshi Nakamoto’s extremely short original paper describing the blockchain.

Seriously, the Bitcoin: A Peer-to-Peer Electronic Cash System paper is so ridiculously small and easy to understand that most computer science students should be able to understand it. So all the smart programmers I know should be able to grasp it in a coffee break. Any average programmer should be able to read and understand this paper in 30 minutes or so.

You can simplify a mental model of it as a Linked List, each node of the List is what we call a Block. Each Block is a stupid struct with the usual previous/next pointers and a body comprised of a tree-like structure (a Merkle Tree, to be more exact).

The catch is that each block has the hash signature of the previous block, thus creating a secure “chain”. Hence “block-chain”.

Yes, in computer science terms, we’re dealing with undergraduate levels of data structures here. If you understand a Linked List and a stupid Binary Tree, plus the easiest crypto thing to understand, a simple digest hash such as SHA256, and boom, you understand the basic backbone of the blockchain database.

Yes, it is just a database. A distributed-database to be more exact. Or a very crude and simple distributed database for that matter. It is not very efficient, and it pales in comparison to more serious NoSQL distributed databases such as Redis or Cassandra. So the query-abilities are basically non-existent beyond finding a block by its identity.

Of course, the Bitcoin source-code is more sophisticated than that but the basics are really so ridiculous that you don’t need more than 20 lines of Ruby code to replicate it. Check out this example implementation from Gerald Bauer.

require "digest"    # for hash checksum digest function SHA256

class Block

  attr_reader :index
  attr_reader :timestamp
  attr_reader :data
  attr_reader :previous_hash
  attr_reader :hash

  def initialize(index, data, previous_hash)
    @index         = index
    @timestamp     = Time.now
    @data          = data
    @previous_hash = previous_hash
    @hash          = calc_hash
  end

  def calc_hash
    sha = Digest::SHA256.new
    sha.update( @index.to_s + @timestamp.to_s + @data + @previous_hash )
    sha.hexdigest
  end


  def self.first( data="Genesis" )    # create genesis (big bang! first) block
    ## uses index zero (0) and arbitrary previous_hash ("0")
    Block.new( 0, data, "0" )
  end

  def self.next( previous, data="Transaction Data..." )
    Block.new( previous.index+1, data, previous.hash )
  end

end

I know, right!?

One question remains: how does this simple structure become a “distributed” database?

Now, either you need to have a centralized “master-copy” out of which all other copies replicate from. Or you need some form of “consensus” between the different copies.

How do you reach consensus between rogue, random nodes spread across the globe? This is the problem that is called “Byzantine Fault Tolerance”, masterfully explained and solved by Barbara Liskov and Miguel Castro, from MIT, in 1999.

In a nutshell, imagine that you have Byzantine generals, each with their own armies, surrounding a hostile city. Now, you can either attack or retreat. But all generals must either do one or the other, in consensus. How do you reach consensus when you don’t have direct communication with all the generals and, worse, when some of the generals may be traitors or double-agents?

That’s the kind of problem we face here. Anyone on the internet can download a copy of the blockchain, and they can check that the blocks are valid and unadulterated by recomputing the digest hashes for each block.

But how do you add new blocks and make the other nodes accept your new block?

That’s why Satoshi added the so-called “Proof of Work” to the equation. Remember that I said that each block is chained together to the previous by containing the hash of the previous block? Computing a digest hash is quite trivial these days.

In Ruby if you do:

Digest::SHA256.hexdigest("abcd")
# => "88d4266fd4e6338d13b845fcf289579d209c897823b9217da3e161936f031589"
Digest::SHA256.hexdigest("123")
# => "a665a45920422f9d417e4867efdc4fb8a04a1f3fff1fa07e998e86f7f7a27ae3"

It takes a fraction of a millisecond to run.

Now, what if I ask you to find the hash that starts with a certain amount of “zeroes” in the beginning of the hash?

For example:

# I want to find 4 zeros ("0000") in the hash:
Digest::SHA256.hexdigest("79026" + "123")
# => "0000559fb4a55f135c7db3d83405b86b4b63cd035993873a5b676bae08b64334"

How do I know that I had to prepend “79026”? I don’t, I have to start from 0 and incrementing one by one until I find the hash with the format I want.

If we check Gerald’s example we would implement this lookup like this:

def compute_hash_with_proof_of_work( difficulty="00" )
  nonce = 0
  loop do
    hash = calc_hash_with_nonce( nonce )
    if hash.start_with?( difficulty )
      return [nonce,hash]    ## bingo! proof of work if hash starts with leading zeros (00)
    else
      nonce += 1             ## keep trying (and trying and trying)
    end
  end
end

def calc_hash_with_nonce( nonce=0 )
  sha = Digest::SHA256.new
  sha.update( nonce.to_s + "123" )
  sha.hexdigest
end

Just a simple SHA256 takes somewhere between 0.000010 to 0.000020 seconds (remember: fractions of milliseconds). Now how long does it take to find that “79026” (which we call a “nonce”)?

> puts Benchmark.measure { compute_hash_with_proof_of_work("0000") }
  0.190000   0.000000   0.190000 (  0.189615)

Yep, considerably more, now it takes 0.18 seconds instead of 0.000020. We can increase the “difficult” variable to make it even more laborious to find the nonce. And that’s exactly how Bitcoin is implemented: each block adjusts the difficulty so that the fastest time to find the hash for the next block is around 10 minutes.

And this, my friends, is what we call “MINING”. What a miner does is compute a loop, incrementing nonces, over the block digest to find the correct nonce.

Once a nonce is found, the miner can add the block to the blockchain and broadcast it to other nodes. The other nodes can then double-check (and now it’s just the 0.000020 seconds procedure again, super fast).

When the nodes double-check and confirm the nonce, they all add the block to the top of the blockchain. And usually, when the other miners keep adding other blocks on top of that, that block becomes “solidified”. The most recent block on the top of the blockchain is usually unstable, but once you have more blocks on top of it, it is said to be more “guaranteed”. Which is why most exchanges and other services that accept bitcoin wait for the so-called “6 blocks” confirmation.

And because the difficulty is such that the fastest node takes around “10 minutes” to find that nonce, a block is said to be “secure” when around 1 hour passes and 6 blocks are added after it.

Can we break this?

Now, you will understand why we talk about “hash power” when we talk about mining.

Mining is the act of signing and confirming blocks to the blockchain. It’s a maintenance service, which is why you reward miners with “transaction fees” and a couple of “satoshis” (fractions of 1 Bitcoin), for their work. And also why you call this “Proof of Work” because when someone finds a nonce, we know it had to go through a lot of hash computation to reach it.

And also why we talk about CPUs or GPUs that miners use to have “hash rates”. You need to have an absurd capacity to be able to mine Bitcoins nowadays. No one will use a home-built rig to do it. One must build special hardware, such as the famous AntMiners. A USD 1,500 AntMiner S9 is able to compute around 14 TH.

Each crypto-currency different from Bitcoin calculates hashes differently so the hashrate differs from coin to coin.

The current Hash Power of the entire Bitcoin consensus network is almost reaching 14 EH (exa-hashes or millions of tera-hashes).

So, let’s say that I am a billionaire and I want to troll the Bitcoin community by adding enough hash power to surpass the entire hash power of the network. I’d have to buy 1 million AntMiner s9, or an investment of around USD 1.5 billion! And this is without adding the energy required to boot and run those machines, of course.

But even then, do you know what happens? Remember that difficulty variable I mentioned above? It will adjust again, to make sure the next block takes 10 minutes to compute again!

Then, no, even if you’re willing to put USD 1.5 billion to waste, you won’t break it. And that’s how Bitcoin deals with Byzantine generals in this consensus network.

To begin, I believe by now most people already watched the movie “The Social Network”. It’s interesting that I heard many people saying how this movie influenced them to begin their own startups. So I was thinking, “what does this movie actually teach?”

Well, from the movie we learn that David Fincher is a fantastic director, that Aaron Sorkin writes very sharp dialogue that is very compelling, that Justin Timberlake does a better Sean Parker than the real deal, that Andrew Garfield does an ok Eduardo Saverin, and that Jesse Eisenberg will forever be Mark Zuckerberg.

And that’s it, we can’t learn anything else from the movie.

Or can we?

Facemesh

One of my favorite scenes from the movie is when Zuckerberg/Eisenberg is pissed by the Erica Albright split up and he starts scraping photos of women from Harvard websites and organizing them into a troll website called “Facemesh” where he puts them to compete. People can vote on which photo they prefer and it sorts them out in a ranking of popularity.

When Eduardo Saverin/Andrew Garfield shows up, Zuckerberg asks him:

“Wardo, I need you (…) I need the algorithm used to rank chess players”

And he goes ahead and writes the following on the dorm room window:

Here most people would just think:

“pff, another gibberish formula just to show off that they are little geniuses, but most certainly this formula doesn’t even exist”

Except it does. And this is the one scene in the movie that stuck in my head as I had seen it before. To help you out, let’s un-mirror the mirrored image:

And this is the “algorithm”.

As I said in my previous article, most developers would create a Facemesh-like website adding integer fields in the table of contestants with the count of upvotes, downvotes and they would do something silly such as:

score = upvotes - downvotes

Or even sillier:

score = upvotes / (upvotes + downvotes)

And it doesn’t work that way, you will get very wrong rankings.

The Demonstration

To show you how wrong, I created a simple demonstration project called elo_demo which you can git clone and run yourself.

It will create 2,000 random matches against 10 players. This will be the sorted results if we use the wrong methods of subtracting losses from wins and order through that result:

   Name        Games  Wins  Losses Points (wins - losses)
 1 Kong          217   117    100     17
 2 Samus         211   110    101      9
 3 Wario         197   102     95      7
 4 Luigi         186    95     91      4
 5 Zelda         160    81     79      2
 6 Pikachu       209   105    104      1
 7 Yoshi         223   112    111      1
 8 Mario         203   101    102     -1
 9 Fox           208    95    113    -18
10 Bowser        186    82    104    -22

Now, let’s make the 2nd place Samus win 10 times in a row against the 3rd place Wario. This is the new ranking:

   Name        Games  Wins  Losses Points
 1 Samus         221   120    101     19
 2 Kong          217   117    100     17
 3 Luigi         186    95     91      4
 4 Zelda         160    81     79      2
 5 Pikachu       209   105    104      1
 6 Yoshi         223   112    111      1
 7 Mario         203   101    102     -1
 8 Wario         207   102    105     -3
 9 Fox           208    95    113    -18
10 Bowser        186    82    104    -22

Sounds fair, Samus jumps to 1st place and Wario goes down to 8th place.

Now, what if we make the weaker 10th place Bowser win 10 times against the current 2nd place Kong?

   Name        Games  Wins  Losses Points
 1 Samus         221   120    101     19
 2 Kong          227   117    110      7
 3 Luigi         186    95     91      4
 4 Zelda         160    81     79      2
 5 Pikachu       209   105    104      1
 6 Yoshi         223   112    111      1
 7 Mario         203   101    102     -1
 8 Wario         207   102    105     -3
 9 Bowser        196    92    104    -12
10 Fox           208    95    113    -18

This is where you see how wrong this method is. Even though he lost 10 times against the weakest player, Kong still reigns supreme at 2nd place. And poor Bowser, in spite of all his hard work and effort, levels up just 1 meager position from 10th to 9th.

This is very frustrating and if it feels unfair, it’s because it is. This kind of calculation is wrong.

From Chess to League of Legends

If you ever played League of Legends you are probably familiar with something called “ELO Boosts”.

This is a way to level up your account for money. I’d strongly recommend against it if you intend to compete professionally as it’s against the rules stipulated by Riot.

Anyway, have you ever wondered why it’s called “ELO”?

This is for Austro-Hungarian professor Arpad Emmerich Elo. He is best known for his system of rating chess players. Quoting from Wikipedia:

The original chess rating system was developed in 1950 by Kenneth Harkness (…). By 1960, using the data developed through the Harkness Rating System, Elo developed his own formula which had a sound statistical basis and constituted an improvement on the Harkness System. The new rating system was approved and passed at a meeting of the United States Chess Federation in St. Louis in 1960.

In 1970, FIDE, the World Chess Federation, agreed to adopt the Elo Rating System. From then on until the mid-1980s, Elo himself made the rating calculations. At the time, the computational task was relatively easy because fewer than 2000 players were rated by FIDE.

His system has been refined and evolved to make tournament leaderboards actually fair and competitive. One such evolution is in the form of Microsoft’s TrueSkill Ranking System used in all Xbox Live games.

That “algorithm” that Eduardo Saverin writes on the window of Harvard’s dorm room? It’s the ELO Rating System!!

I don’t know if Zuckerberg actually implemented the ELO rating system equations. If he did, it was the correct choice. But the whole Eduardo writing the equations on the window probably didn’t happen that way as it would be way easier to Google for it :-)

ELO Rating System Demonstration

My pet demonstration project also calculates that exact ELO score. The calculations are done by the elo rubygem.

The idea is to calculate the probability that one player has to win over the other player. So if a strong player plays against a weak player, he is expected to win, and if this is the outcome, he will not score a lot and the losing player will not fall a lot either. But if the unexpected happens and the strong one loses then it’s expected for him to fall down a lot and for the “weaker” player to jump up a lot.

That will make the tournaments more competitive and make the new players more motivated to play against the strongest and also make the strongest play harder to hold their positions.

From the elo gem documentation, this is how you use it:

kong  = Elo::Player.new
bowser = Elo::Player.new(:rating => 1500)

game1 = kong.wins_from(bowser)
game2 = kong.loses_from(bowser)
game3 = kong.plays_draw(bowser)

game4 = kong.versus(bowser)
game4.winner = bowser

game5 = kong.versus(bowser)
game5.loser = bowser

game6 = kong.versus(bowser)
game6.draw

game7 = kong.versus(bowser)
game7.result = 1 # result is in perspective of kong, so kong wins

game8 = kong.versus(bowser, :result => 0) # bowser wins

And this is how you assess the results:

kong.rating       # => 1080
kong.pro?         # => false
kong.starter?     # => true
kong.games_played # => 8
kong.games        # => [ game1, game2, ... game8 ]

The gem has more tuning besides that original algorithm, such as the K-factor to reward new players. Those kinds of tunings are what makes matches more competitive today and how you evolve it to TrueSkill levels, but it’s beside the point of this article.

Let’s see the wrong ranking again:

   Name        Games  Wins  Losses Points (wins - losses)
 1 Kong          217   117    100     17
 2 Samus         211   110    101      9
 3 Wario         197   102     95      7
 4 Luigi         186    95     91      4
 5 Zelda         160    81     79      2
 6 Pikachu       209   105    104      1
 7 Yoshi         223   112    111      1
 8 Mario         203   101    102     -1
 9 Fox           208    95    113    -18
10 Bowser        186    82    104    -22

Now let’s see the correct ranking by calculating the Elo score using the exact same 2,000 matches:

   Name        Games  Wins  Losses Points  Elo Rating
 1 Pikachu       209   105    104      1         851
 2 Zelda         160    81     79      2         847
 3 Samus         211   110    101      9         842
 4 Luigi         186    95     91      4         841
 5 Wario         197   102     95      7         824
 6 Mario         203   101    102     -1         820
 7 Yoshi         223   112    111      1         803
 8 Kong          217   117    100     17         802
 9 Bowser        186    82    104    -22         785
10 Fox           208    95    113    -18         754

See how different it is? In the wrong ranking, Kong is considered the strongest, but in the Elo ranking he is just 8th place. And the reason is that even though he is the one that won most matches (217) he also lost a heck of a lot (117). Someone with fewer wins such as Zelda in 2nd place (160 wins) lost a heck of a lot less (81), which is why she is higher in the ranking.

Now, if we make her win 10 matches in a row against 3rd place Samus, this is the new ranking:

   Name        Games  Wins  Loses  Points  Elo Rating
 1 Zelda         170    91     79     12         904
 2 Pikachu       209   105    104      1         851
 3 Luigi         186    95     91      4         841
 4 Wario         197   102     95      7         824
 5 Mario         203   101    102     -1         820
 6 Yoshi         223   112    111      1         803
 7 Kong          217   117    100     17         802
 8 Bowser        186    82    104    -22         785
 9 Samus         221   110    111     -1         775
10 Fox           208    95    113    -18         754

Again, Zelda jumps up from 2nd to 1st place and Samus falls down from 3rd to 9th. So far so good. But what about the scenario where we make strong 2nd place Pikachu play against a much weaker 10th place Fox McCloud?

   Name        Games  Wins  Loses  Points  Elo Rating
 1 Zelda         170    91     79     12         904
 2 Luigi         186    95     91      4         841
 3 Fox           218   105    113     -8         829
 4 Wario         197   102     95      7         824
 5 Mario         203   101    102     -1         820
 6 Yoshi         223   112    111      1         803
 7 Kong          217   117    100     17         802
 8 Bowser        186    82    104    -22         785
 9 Samus         221   110    111     -1         775
10 Pikachu       219   105    114     -9         766

Now, this is fairness: Pikachu should have won, but losing 10 times in a row against someone considered much weaker makes him fall down from 2nd place all the way to the last place. And noobie Fox, having won 10 times against a much stronger opponent, deserves jumping up all the way to 3rd place.

This is the kind of dynamic that can make matches and games competitive, which is exactly why every online leaderboard and professional tournament uses those kinds of algorithms.

And it all began in chess, using math that has been known since the late 40’s!!

This is the point of this post and my previous one: the math is not new.

Developers waste a great amount of time in stupid pissing contests over which language or tool is “shinier”, but they ignore the proper math and deliver wrong results. But the math has been around for decades, in some cases more than a full century already!

We should strive to earn the title of Computer Scientists. There is a lot of science lacking from computing nowadays. Pissing contests do not make a programmer any good.

But in the process, the author made the same mistake I’ve seen time and time again.

Let’s assume you have a project with resources that you want to list by “popularity”. It can be a Reddit-like site where people like or dislike posts or comments. It can be an e-commerce where people like or dislike products.

It can be anything where people like or dislike something.

The biggest error people make is to consider a simple score like this:

popularity = positive_votes - negative_votes

There is an old article titled “How Not To Sort By Average Rating” and I quote:

“Why it is wrong: Suppose one item has 600 positive ratings and 400 negative ratings: 60% positive. Suppose item two has 5,500 positive ratings and 4,500 negative ratings: 55% positive. This algorithm puts item two (score = 1000, but only 55% positive) above item one (score = 200, and 60% positive). WRONG.”

Then you may think, I know how to fix it:

Score = average_rating = positive_votes / total_votes

Again, this is wrong, and again I quote:

“Why it is wrong: Average rating works fine if you always have a ton of ratings, but suppose item 1 has 2 positive ratings and 0 negative ratings. Suppose item 2 has 100 positive ratings and 1 negative rating. This algorithm puts item two (tons of positive ratings) below item one (very few positive ratings). WRONG.”

Correct Solution: Lower Bound of Wilson Score Confidence Interval for a Bernoulli

And I quote again:

“Say what: We need to balance the proportion of positive ratings with the uncertainty of a small number of observations. Fortunately, the math for this was worked out in 1927 by Edwin B. Wilson. What we want to ask is: Given the ratings I have, there is a 95% chance that the “real” fraction of positive ratings is at least what? Wilson gives the answer. Considering only positive and negative ratings (i.e. not a 5-star scale), the lower bound on the proportion of positive ratings is given by:”

I recommend you read the original article but let’s cut to the chase. If we follow the original post I linked in the beginning, I have a simple blog post Rails app, but instead of a visits_count field I need to add a positive:integer and negative:integer fields and user interface to post votes.

And I will replace the PostWithPopularityQuery class with the following:

class PostWithPopularityQuery
  def self.call
    Post.find_by_sql ['SELECT id, title, body, positive, negative,
        ((positive + 1.9208) / (positive + negative) -
        1.96 * SQRT((positive * negative) / (positive + negative) + 0.9604) /
        (positive + negative)) / (1 + 3.8416 / (positive + negative))
        AS ci_lower_bound
      FROM posts 
      WHERE positive + negative > 0
      ORDER BY ci_lower_bound DESC']
  end
end

And this is what I expect to see in a simple scaffold index.html.erb:

I would even go as far as recommending the ci_lower_bound to be a float field in the table and to have an asynchronous ActiveJob to update it in some larger interval of time (every 5 minutes, for example) and then the PostsController#index action would perform a straightforward SELECT query ordering directly against a real indexed field ci_lower_bound DESC without performing the calculations on every query.

Now THIS is the correct way to implement a simple, naive popularity ranking system that actually works correctly.

And this is not the only way to do it. There are dozens of good discussions of algorithms online. Every service that depends on content popularity has been refining algorithms like this for years. Facebook used to have an algorithm called “EdgeRank” which relied on variables such as Affinity, Weight, Time Decay, and it seems to have evolved so much that it now calculates popularity against more than 100 thousand variables!!

But regardless of the online service, I can assure you that none sorts by simple visits count or simple average votes count. That would be downright wrong.

But there are many opportunities to accelerate Rails applications as well. For example, we just saw the release of the gem “faster_path”, this time written in Rust and bridged through FFI (Foreign Function Interface). The author’s claim is that Sprockets has to compute lots of paths and making this library, natively compiled and optimized with Rust, added a huge improvement in the asset pipeline task.

Sam Saffron, from Discourse, also built a very small gem called “fast_blank” which is a tiny library written in C that reimplements ActiveSupport’s String#blank? method to be up to 9x faster. Because Rails digests volumes of strings, checking if they are blank every time, this adds some performance (depends on your app, of course).

The Holy Grail to native-level performance is to be able to write close-to-Ruby code instead of having to hack low-level C or having the high learning curve of a language such as Rust. More than that, I’d like to avoid having to use FFI. I am not an expert in FFI but I remember understanding that it adds overhead to make the bindings.

By the way, it’s important to disclose right now: I am not a C expert by any means of the imagination, far from that. So I have very little experience dealing with hardcore C development. Which is again, why this possibility of writing in Crystal is even more appealing to me. So if you are a C expert and you spot something silly I am saying about it, please let me know in the comments section below.

My exercise is to rewrite the C-based Fast Blank gem in Crystal, add it to the same Gem to compile under Crystal if it’s available or fall back to C, and make the specs pass so it’s a seamless transition for the user.

To achieve that I had to:

• Extend the Gem’s extconf.rb to generate different Makefiles (for C and Crystal) that are able to compile under OS X or Linux (Ubuntu at least) - OK
• Make the specs pass in the Crystal version - Almost (it’s ok for all intents and purposes but an edge case)
• Make the performance be faster than Ruby and close to C - Not so much yet (under OS X the performance is quite good, but under Ubuntu it doesn’t scale so well for large strings)

You can check out the results so far on my fork over Github and follow the Pull Request discussion as well.

Comparing C and Crystal

Just to get us started, let’s check out a snippet of Sam’s original C version:

static VALUE
rb_str_blank(VALUE str)
{
  rb_encoding *enc;
  char *s, *e;

  enc = STR_ENC_GET(str);
  s = RSTRING_PTR(str);
  if (!s || RSTRING_LEN(str) == 0) return Qtrue;

  e = RSTRING_END(str);
  while (s < e) {
    int n;
    unsigned int cc = rb_enc_codepoint_len(s, e, &n, enc);

    if (!rb_isspace(cc) && cc != 0) return Qfalse;
    s += n;
  }
  return Qtrue;
}

Yeah, quite scary, I know. Now let’s see the Crystal version:

struct Char
  ...
  # same way C Ruby implements it
  def is_blank
    self == ' ' || ('\t' <= self <= '\r')
  end
end

class String
  ...
  def blank?
    return true if self.nil? || self.size == 0
    each_char { |char| return false if !char.is_blank }
    return true
  end
end

Hell yeah! If you’re a rubyist I bet you can understand 100% of the snippet above. It’s not “exactly” the same thing (as the specs are not fully passing yet), but it’s damn close.

The Quest for a Makefile to Crystal

I’ve researched many experimental Github repos and Gists out there. But I didn’t find one that has it all so I decided to tweak what I found until I got to this version:

Obs: again, I am not a C expert. If you have experience with Makefiles I know this one can be refactored to something nicer than this. Let me know in the comments below.

ifeq "$(PLATFORM)" ""
PLATFORM := $(shell uname)
endif

ifeq "$(PLATFORM)" "Linux"
UNAME = "$(shell llvm-config --host-target)"
CRYSTAL_BIN = $(shell readlink -f `which crystal`)
LIBRARY_PATH = $(shell dirname $(CRYSTAL_BIN))/../embedded/lib
LIBCRYSTAL = $(shell dirname $(CRYSTAL_BIN) )/../src/ext/libcrystal.a
LIBRUBY = $(shell ruby -e "puts RbConfig::CONFIG['libdir']")
LIBS = -lpcre -lgc -lpthread -levent -lrt -ldl
LDFLAGS = -rdynamic

install: all

all: fast_blank.so

fast_blank.so: fast_blank.o
  $(CC) -shared $^ -o $@ $(LIBCRYSTAL) $(LDFLAGS) $(LIBS) -L$(LIBRARY_PATH) -L$(LIBRUBY)

fast_blank.o: ../../../../ext/src/fast_blank.cr
  crystal build --cross-compile --release --target $(UNAME) $<

.PHONY: clean
clean:
  rm -f bc_flags
  rm -f *.o
  rm -f *.so
endif

ifeq "$(PLATFORM)" "Darwin"
CRYSTAL_FLAGS = -dynamic -bundle -Wl,-undefined,dynamic_lookup

install: all

all: fast_blank.bundle

fast_blank.bundle: ../../../../ext/src/fast_blank.cr
  crystal $^ --release --link-flags "$(CRYSTAL_FLAGS)" -o $@

clean:
  rm -f *.log
  rm -f *.o
  rm -f *.bundle
endif

Most people using Crystal are on OS X, including the creators of Crystal. LLVM is under Apple’s umbrella and their entire ecosystem relies heavily on LLVM. They spent many years migrating the C front-end first, then the C back-end away from GNU’s standard GCC to Clang. And they were able to make both Objective-C and Swift compile down to LLVM’s IR and that’s how both can interact back and forth natively.

Then, they improved the ARM backend support and that’s how they can have an entire iOS “Simulator” (not a dog slow emulator like Android) where the iOS apps are natively compiled to run over Intel’s x86_64 processor while in development and then quickly recompile to ARM when ready to package to the App Store.

This way you can run natively, test quickly, without the slowness of an emulated environment. By the way, I will say this once: Google’s biggest mistake is not supporting LLVM as they should, and reinventing the wheel instead. If they had, Go could already be used to implement for Android and Chromebooks as well as x86 based servers and they could put away all the Java/Oracle debacle.

But I digress.

In OS X you can pass a “-bundle” link-flag to crystal and it will probably use clang underneath to generate the binary bundle.

On Ubuntu crystal just compiles down to an object file (.o) and you have to manually invoke GCC with the “-shared” option to create a shared-object. To do that we have to use the "–cross-compile" and pass an LLVM target triplet so it generates the .o (this requires the llvm-config tool).

Shared Libraries (.so) and Loadable Modules (.bundle) are different beasts, check this documentation out for more details.

Keep in mind that benchmarking binaries built with different compilers can make a difference. I am not an expert but out of pure anecdote I believe Ruby under RVM on OS X is compiled using OS X’s default Clang. On Ubuntu it’s compiled under GCC. This seems to make Ruby on OS X “so slightly” inefficient in synthetic benchmarks.

On the other hand, Crystal binaries linked with GCC feel “so slightly” inefficient on Ubuntu, while Ruby on Ubuntu feels a bit faster, having been compiled and linked with GCC.

So when we compare Fast Blank/OS X/bit faster with Ruby/OS X/slower against Fast Blank/Ubuntu/bit slower with Ruby/Ubuntu/bit faster, it seems to give a wider advantage to the OS X benchmark comparison against the Ubuntu benchmark, even though individual computation times are not so far from each other.

I will come back to this point in the benchmarks section.

Finally, every time you have a rubygem with a native extension, you will find this bit in their gemspec files:

Gem::Specification.new do |s|
  s.name = 'fast_blank'
  ...
  s.extensions = ['ext/fast_blank/extconf.rb']
  ...

When the gem is installed through gem install or bundle install it will run this script to generate a proper Makefile. In a pure C extension it will use the built-in “mkmf” library to generate it.

In our case, if we have Crystal installed, we want to use the Crystal version, so I tweaked the extconf.rb to be like this:

require 'mkmf'

if ENV['VERSION'] != "C" && find_executable('crystal') && find_executable('llvm-config')
  # Very dirty patching
  def create_makefile(target, srcprefix = nil)
    mfile = open("Makefile", "wb")
    cr_makefile = File.join(File.dirname(__FILE__), "../src/Makefile")
    mfile.print File.read(cr_makefile)
  ensure
    mfile.close if mfile
    puts "Crystal version of the Makefile copied"
  end
end

create_makefile 'fast_blank'

So, if it finds crystal and llvm-config (which in OS X you have to add the proper path like this: export PATH=$(brew --prefix llvm)/bin:$PATH).

The Rakefile in this project declares the standard :compile task as the first one to run, and it will execute the extconf.rb, which will generate the proper Makefile and run the make command to compile and link the proper library in the proper lib/ path.

So we will end up with lib/fast_blank.bundle on OS X and lib/fast_blank.so on Ubuntu. From there we can just have require "fast_blank" from any Ruby file in the gem and it will have access to the publicly exported C function mappings from the Crystal library.

Mapping C-Ruby to Crystal

Now, any direct C extension - without FFI, fiddle or other “bridges” - will ALWAYS have a much better advantage.

The reason is that you literally have to “copy” data from C-Ruby to Crystal/Rust/Go or whatever other language you’re binding. While with a C-based extension you can operate directly in the memory space with the data without having to move it or copy it away.

For example. First, you have to bind the C functions from C-Ruby to Crystal. And we accomplish that with Paul Hoffer’s Crystalized Ruby mappings. It’s an experimental repository that I helped a bit to clean up in order for him to later extract this mapping library into its own Shard (shards are the same as gems for Crystal). For now I had to simply copy the file over to my Fast Blank.

Some of the relevant bits are like this:

lib LibRuby
  type VALUE = Void*
  type METHOD_FUNC = VALUE -> VALUE
  type ID = Void*
  ...

  # strings
  fun rb_str_to_str(value : VALUE) : VALUE
  fun rb_string_value_cstr(value_ptr : VALUE*) : UInt8*
  fun rb_str_new_cstr(str : UInt8*) : VALUE
  fun rb_utf8_encoding() : VALUE
  fun rb_enc_str_new_cstr(str : UInt8*, enc : VALUE) : VALUE
  ...
  # exception handling
  fun rb_rescue(func : VALUE -> UInt8*, args : VALUE, callback: VALUE -> UInt8*, value: VALUE) : UInt8*
end
...
class String
  RUBY_UTF = LibRuby.rb_utf8_encoding
  def to_ruby
    LibRuby.rb_enc_str_new_cstr(self, RUBY_UTF)
  end

  def self.from_ruby(str : LibRuby::VALUE)
    c_str = LibRuby.rb_rescue(->String.cr_str_from_rb_cstr, str, ->String.return_empty_string, 0.to_ruby)
    # FIXME there is still an unhandled problem: then we receive \u0000 from Ruby it raises "string contains null bytes"
    # so we catch it with rb_rescue, but then we can't generate a Pointer(UInt8) that represents the unicode 0, instead we return a plain blank string
    # but then the specs fail
    new(c_str)
  ensure
    ""
  end

  def self.cr_str_from_rb_cstr(str : LibRuby::VALUE)
    rb_str = LibRuby.rb_str_to_str(str)
    c_str  = LibRuby.rb_string_value_cstr(pointerof(rb_str))
  end

  def self.return_empty_string(arg : LibRuby::VALUE)
    a = 0_u8
    pointerof(a)
  end
end

Then I can use these mappings and helpers to build a “Wrapper” class in Crystal:

require "./lib_ruby"
require "./string_extension"

module StringExtensionWrapper
  def self.blank?(self : LibRuby::VALUE)
    return true.to_ruby if LibRuby.rb_str_length(self) == 0
    str = String.from_ruby(self)
    str.blank?.to_ruby
  rescue
    true.to_ruby
  end

  def self.blank_as?(self : LibRuby::VALUE)
    return true.to_ruby if LibRuby.rb_str_length(self) == 0
    str = String.from_ruby(self)
    str.blank_as?.to_ruby
  rescue
    true.to_ruby
  end

  def self.crystal_value(self : LibRuby::VALUE)
    str = String.from_ruby(self)
    str.to_ruby
  end
end

And this “Wrapper” depends on the “pure” Crystal library itself like with the snippets for the Char struct and String class extensions I showed in the first section of the article above.

Finally, I have a main “fast_blank.cr” file that externs those Wrapper functions so C-Ruby can see them as plain String methods:

require "./string_extension_wrapper.cr"

fun init = Init_fast_blank
  GC.init
  LibCrystalMain.__crystal_main(0, Pointer(Pointer(UInt8)).null)

  string = LibRuby.rb_define_class("String", LibRuby.rb_cObject)
  LibRuby.rb_define_method(string, "blank?", ->StringExtensionWrapper.blank?, 0)
  LibRuby.rb_define_method(string, "blank_as?", ->StringExtensionWrapper.blank_as?, 0)
  ...
end

This is mostly boilerplate. But now check out what I am having to do in the wrapper, in this particular snippet:

def self.blank?(self : LibRuby::VALUE)
  return true.to_ruby if LibRuby.rb_str_length(self) == 0
  str = String.from_ruby(self)
  str.blank?.to_ruby
rescue
  true.to_ruby
end

I am receiving a C-Ruby String casted as a pointer (VALUE) then I go through the lib_ruby.cr mappings to get the C-Ruby string data and copy it over into a new instance of Crystal’s internal String representation. So at any given time I have 2 copies of the same string, one in the C-Ruby memory space and another in the Crystal memory space.

This happens with all FFI-like extensions but it doesn’t happen to the pure C implementation. In Sam Saffron’s C implementation it directly works with the same address in C-Ruby’s memory space:

static VALUE
rb_str_blank(VALUE str)
{
  rb_encoding *enc;
  char *s, *e;

  enc = STR_ENC_GET(str);
  s = RSTRING_PTR(str);
  ...

It receives a pointer (direct memory address) and goes. And this is a huge advantage for the C version. When you have a big volume of medium to large sized strings being copied over from C-Ruby to Crystal, it adds a noticeable overhead that can’t be removed.

String mapping Caveat

I still have a problem though. There is one edge case I was not able to overcome yet (help is most welcome). When C-Ruby passes a unicode "\u0000" I am unable to create the same character in Crystal and I end up passing just an empty string ("") which is not the same thing.

The way to deal with it is to receive a Ruby String (VALUE) and get the C-String from it this way:

rb_str = LibRuby.rb_str_to_str(str)
c_str  = LibRuby.rb_string_value_cstr(pointerof(rb_str))

If the “str” is the “\u0000” (under Ruby 2.2.5 at least) C-Ruby raises a “string contains null bytes” exception. Which is why I rescue from this exception like this:

c_str = LibRuby.rb_rescue(->String.cr_str_from_rb_cstr, str, ->String.return_empty_string, 0.to_ruby)

When an exception is triggered I have to pass the pointer to another function to rescue from it:

def self.return_empty_string(arg : LibRuby::VALUE)
  a = 0_u8
  pointerof(a)
end

But this is not correct, I am just passing the pointer to a “0” character, which is “empty”. Therefore, specs are not passing correctly:

Failures:

  1) String provides a parity with active support function
     Failure/Error: expect("#{i.to_s(16)} #{c.blank_as?}").to eq("#{i.to_s(16)} #{c.blank2?}")

       expected: "0 false"
            got: "0 true"

       (compared using ==)
     # ./spec/fast_blank_spec.rb:22:in `block (3 levels) in <top (required)>'
     # ./spec/fast_blank_spec.rb:19:in `times'
     # ./spec/fast_blank_spec.rb:19:in `block (2 levels) in <top (required)>'

  2) String treats  correctly
     Failure/Error: expect("\u0000".blank_as?).to be_falsey
       expected: falsey value
            got: true
     # ./spec/fast_blank_spec.rb:47:in `block (2 levels) in <top (required)>'

Ary gave a simple tip later, I will add it to the conclusion below.

The Synthetic Benchmarks (careful on how you interpret them!)

The original Rails ActiveSupport implementation of String#blank? looks like this:

class String
  # 0x3000: fullwidth whitespace
  NON_WHITESPACE_REGEXP = %r![^\s#{[0x3000].pack("U")}]!

  # A string is blank if it's empty or contains whitespaces only:
  #
  #   "".blank?                 # => true
  #   "   ".blank?              # => true
  #   "　".blank?               # => true
  #   " something here ".blank? # => false
  #
  def blank?
    # 1.8 does not takes [:space:] properly
    if encoding_aware?
      self !~ /[^[:space:]]/
    else
      self !~ NON_WHITESPACE_REGEXP
    end
  end
end

It’s mainly a regular expression comparison, which can be a bit slow. Sam’s version is a more straightforward loop through the string to compare each character with what’s considered “blank”. There are many unicode codepoints that are considered blank, some are not, which is why the C and Crystal versions are similar, but they are different from Rails’ version.

In the Fast Blank gem there is a benchmark Ruby script to compare the C-extension against Rails’ Regex based implementation.

The Regex implementation is called “Slow Blank”. It’s particularly slow if you pass a real empty String, so in the benchmark Sam added a “New Slow Blank” that checks through String#empty? first, and this version is faster in this edge case.

The fast C version is called “Fast Blank” but although you can consider it “correct” it’s not compatible with all the edge cases from Rails. So he implemented a String#blank_as? which is compatible with Rails. Sam calls it “Fast Activesupport”.

In my Crystal version I did the same, having both String#blank? and String#blank_as?.

So, without further ado, here is the C Version over OS X benchmark for empty strings, and we exercise each function many times within a few seconds to have more accurate results (check out Evan Phoenix’s “benchmark/ips” to understand the “iteration per second” methodology).

================== Test String Length: 0 ==================
Warming up _______________________________________
          Fast Blank   191.708k i/100ms
  Fast ActiveSupport   209.628k i/100ms
          Slow Blank    61.487k i/100ms
      New Slow Blank   203.165k i/100ms
Calculating _______________________________________
          Fast Blank     20.479M (± 9.3%) i/s -    101.414M in   5.001177s
  Fast ActiveSupport     21.883M (± 9.4%) i/s -    108.378M in   5.004350s
          Slow Blank      1.060M (± 4.7%) i/s -      5.288M in   5.001365s
      New Slow Blank     18.883M (± 6.9%) i/s -     94.065M in   5.008899s

Comparison:
  Fast ActiveSupport: 21882711.5 i/s
          Fast Blank: 20478961.5 i/s - same-ish: difference falls within error
      New Slow Blank: 18883442.2 i/s - same-ish: difference falls within error
          Slow Blank:  1059692.6 i/s - 20.65x slower

It’s super fast. Rails’ version is 20x slower on my machine.

Now, Crystal version over OS X

================== Test String Length: 0 ==================
Warming up _______________________________________
          Fast Blank   174.349k i/100ms
  Fast ActiveSupport   174.035k i/100ms
          Slow Blank    64.684k i/100ms
      New Slow Blank   215.164k i/100ms
Calculating _______________________________________
          Fast Blank      8.647M (± 1.6%) i/s -     43.239M in   5.001530s
  Fast ActiveSupport      8.580M (± 1.3%) i/s -     42.987M in   5.010759s
          Slow Blank      1.047M (± 3.7%) i/s -      5.239M in   5.008907s
      New Slow Blank     19.090M (± 9.3%) i/s -     94.672M in   5.009057s

Comparison:
      New Slow Blank: 19090034.8 i/s
          Fast Blank:  8647459.7 i/s - 2.21x slower
  Fast ActiveSupport:  8580487.9 i/s - 2.22x slower
          Slow Blank:  1047465.3 i/s - 18.22x slower

As I explained before, even checking empty strings, the Crystal version is slower than the Ruby check for String#empty? (New Slow Blank) because I have the string copying routine of the Wrapper mappings. This adds overhead that is perceptible over many iterations. It’s still 18x faster than Rails, but it loses to C-Ruby.

Finally, Crystal version over Ubuntu

================== Test String Length: 0 ==================
Warming up _______________________________________
          Fast Blank   255.883k i/100ms
  Fast ActiveSupport   260.915k i/100ms
          Slow Blank   105.424k i/100ms
      New Slow Blank   284.670k i/100ms
Calculating _______________________________________
          Fast Blank      8.895M (± 9.8%) i/s -     44.268M in   5.037761s
  Fast ActiveSupport      8.647M (± 8.2%) i/s -     43.051M in   5.020125s
          Slow Blank      1.736M (± 3.9%) i/s -      8.750M in   5.048253s
      New Slow Blank     22.170M (± 6.2%) i/s -    110.452M in   5.004909s

Comparison:
      New Slow Blank: 22170031.0 i/s
          Fast Blank:  8895113.3 i/s - 2.49x slower
  Fast ActiveSupport:  8646940.8 i/s - 2.56x slower
          Slow Blank:  1736071.0 i/s - 12.77x slower

Notice that it’s around the same ballpark, but the Rails version on Ubuntu runs almost twice as fast compared to its counterpart in OS X, which makes the comparison against the Crystal library go down from 18x to 12x.

The benchmark keeps comparing against strings of larger and larger sizes, from 6, to 14, to 24, up to 136 characters in length.

Let’s get just the last test case of 136 characters. First with C version on OS X:

================== Test String Length: 136 ==================
Warming up _______________________________________
          Fast Blank   177.521k i/100ms
  Fast ActiveSupport   193.559k i/100ms
          Slow Blank    89.378k i/100ms
      New Slow Blank    60.639k i/100ms
Calculating _______________________________________
          Fast Blank     10.727M (± 8.7%) i/s -     53.256M in   5.006538s
  Fast ActiveSupport     11.600M (± 8.3%) i/s -     57.681M in   5.009692s
          Slow Blank      1.872M (± 5.7%) i/s -      9.385M in   5.029243s
      New Slow Blank      1.017M (± 5.3%) i/s -      5.094M in   5.022994s

Comparison:
  Fast ActiveSupport: 11600112.2 i/s
          Fast Blank: 10726792.8 i/s - same-ish: difference falls within error
          Slow Blank:  1872262.5 i/s - 6.20x slower
      New Slow Blank:  1016926.7 i/s - 11.41x slower

The C-version is consistently much faster in all test cases and in the 136 characters it’s still 11x faster than Rails in pure Ruby.

Now the Crystal version over OS X:

================== Test String Length: 136 ==================
Warming up _______________________________________
          Fast Blank   127.749k i/100ms
  Fast ActiveSupport   126.538k i/100ms
          Slow Blank    94.390k i/100ms
      New Slow Blank    60.594k i/100ms
Calculating _______________________________________
          Fast Blank      3.283M (± 1.8%) i/s -     16.480M in   5.021364s
  Fast ActiveSupport      3.235M (± 1.3%) i/s -     16.197M in   5.008315s
          Slow Blank      1.888M (± 4.4%) i/s -      9.439M in   5.009458s
      New Slow Blank    967.950k (± 4.7%) i/s -      4.848M in   5.018946s

Comparison:
          Fast Blank:  3283025.1 i/s
  Fast ActiveSupport:  3234586.5 i/s - same-ish: difference falls within error
          Slow Blank:  1887800.5 i/s - 1.74x slower
      New Slow Blank:   967950.2 i/s - 3.39x slower

It’s also faster, but just by 2 to 3 times compared to pure Ruby, a far cry from 11x. But my hypothesis is that the mapping and copying of so many strings over adds a large overhead that the C version does not have.

And the Crystal version over Ubuntu:

================== Test String Length: 136 ==================
Warming up _______________________________________
          Fast Blank   186.810k i/100ms
  Fast ActiveSupport   187.306k i/100ms
          Slow Blank   143.439k i/100ms
      New Slow Blank    98.308k i/100ms
Calculating _______________________________________
          Fast Blank      3.517M (± 3.9%) i/s -     17.560M in   5.000791s
  Fast ActiveSupport      3.485M (± 3.8%) i/s -     17.419M in   5.006427s
          Slow Blank      2.755M (± 4.2%) i/s -     13.770M in   5.008490s
      New Slow Blank      1.551M (± 4.3%) i/s -      7.766M in   5.017853s

Comparison:
          Fast Blank:  3516960.7 i/s
  Fast ActiveSupport:  3484575.5 i/s - same-ish: difference falls within error
          Slow Blank:  2754669.0 i/s - 1.28x slower
      New Slow Blank:  1550815.2 i/s - 2.27x slower

Again, the Ubuntu versions of both the Crystal library and the Ruby binary run faster and the comparison shows no more than twice as much faster. And pure Ruby’s String#empty? is in the same ballpark as Crystal’s version.

Conclusion

The most obvious conclusion is that I probably made a mistake in choosing Fast Blank as my first proof of concept. The algorithm is too trivial and a simple check for String#empty? in pure Ruby is orders of magnitude faster than the added overhead of mapping and string copying to Crystal.

Also, any use case where you have a huge amount of small bits of data being transferred from C-Ruby to Crystal or any FFI-based extension will have the overhead of data copying, which a pure C-version will not have. Which is why Fast Blank is better done in C.

Any other use case where you have smaller amounts of data, or data that can be transferred in bulk (less calls from C-Ruby to the extension, with arguments with a larger size, and with more costly processing) is a better candidate to benefit from extensions.

Again, not everything gets automatically faster, we always have to figure out the use case scenarios first. But because it’s so much easier to write in Crystal and benchmark, we can make faster proofs of concept and scrap the idea if the measurements prove that we won’t benefit as much.

The Crystal documentation recently received a “Performance Guide”. It’s very useful for you to avoid common pitfalls that harm overall performance. Even though LLVM is quite competent in heavy optimization, it can’t do everything. So read it through to improve your general Crystal skills.

That being said, I still believe that this exercise was well worth it. I will probably do some more. I’d really want to thank Ary (Crystal creator) and Paul Hoffer for the patience in helping me out through many of the quirks I found along the way.

While I was finishing this post, Ary pointed out that I could probably ditch Strings altogether and work directly with an array of bytes, which is a good idea and I will probably try that. I think I made it clear by now that the whole String copying adds a very perceptible overhead as we saw in the benchmarks above. Let me know if someone is interested in contributing as well. With a few more tweaks I believe we can have a Crystal version that can at least compete against the C version while also being more readable and maintainable for most Rubyists, which is my goal.

I hope the code I published here will serve as boilerplate examples for more Crystal-based Ruby extensions in the future!