The problem: subtitles

Anyone who maintains a movie and TV collection knows the pain. You download the MKV, but the embedded subtitle is out of sync. Or worse: there’s no subtitle at all in the language you want. So you go to OpenSubtitles, download a subtitle, and it’s 3 seconds ahead because it was made for a different release. The manual flow is:

1. Extract subtitle tracks from the MKV with mkvextract
2. Go to OpenSubtitles, look for a subtitle for the movie
3. Download, test, see it’s out of sync
4. Run some sync tool (ffsubsync, alass)
5. Rename the file, move it to the right place
6. Repeat for every language, for every movie

For someone with 10 movies, it’s tolerable. For someone with hundreds, it’s insanity. It’s exactly the kind of thing that should be automated.

Subservient: the Python solution

Looking at what already existed, I found Subservient. It’s a Python project that automates exactly this flow: it extracts subtitles from MKVs, downloads them from OpenSubtitles via REST API, syncs them with ffsubsync, and cleans ads out of SRT files.

The project is complete. It has movie mode and series mode, smart sync (tests every candidate in parallel and picks the best one) and first-match (stops at the first one that works). It uses the OpenSubtitles hash for exact matching and cleans watermarks and ads with more than 30 regex patterns.

But it has the typical Python distribution problems:

• 7 pip dependencies: colorama, requests, langdetect, ffsubsync, platformdirs, pycountry, tqdm
• ffsubsync as the sync engine: which in turn depends on numpy, auditok, and a bunch more Python packages
• Interactive menu UI: good for manual use, terrible for scriptability
• Config in INI format: not the end of the world, but YAML is more ergonomic
• 10,220 lines across 6 Python files: 2,700-line files with hundreds-of-lines functions each

The point isn’t that Python is bad for this. Subservient works. But installing and maintaining it in production is another story. If you want to run it on a headless server, you need Python 3.8+, pip, virtualenv (or you’re going to pollute the system), and pray no dependency breaks with the next OS update.

The experiment: porting it to Crystal with Claude

Here’s where it gets interesting. I wanted to test a hypothesis: can Claude take a large open source project, understand the architecture, and do a complete port to another language?

I’m not talking about translating it file by file. I’m talking about understanding what the project does, redesigning the architecture where it makes sense, and generating idiomatic code in Crystal.

What I did:

1. Asked Claude to clone and analyze the Subservient repo
2. Explained the design decisions: use alass (a Rust binary, no Python dependencies) instead of ffsubsync, CLI subcommands instead of interactive menus, YAML instead of INI
3. Asked for a feature-parity port, with tests

alass is an important detail. ffsubsync works fine, but it’s a Python package that pulls in numpy and does audio analysis. alass does the same thing (subtitle synchronization through timing analysis), but it’s a standalone Rust binary. Swapping one for the other eliminates the biggest Python dependency in the stack.

The result: easy-subtitle

Five commits. Less than 40 minutes from the first to the last.

The first commit already delivers a working project: 8 CLI commands, OpenSubtitles client with rate limiting, 76 passing tests, GitHub Actions with CI and release for Linux and macOS.

Numbers

The LOC difference is loud: 10,220 vs 2,516. But that’s not all Crystal’s doing. The original Python has monolithic files of thousands of lines, with a lot of duplication and UI logic mixed with business logic. The port separates the responsibilities into small, focused modules.

Architecture of the port

easy-subtitle/
  src/easy_subtitle/
    cli/           # Router + 9 commands (init, extract, download, sync, run, clean, scan, hash, doctor)
    core/          # Language map, SRT parser/writer/cleaner, video scanner
    acquisition/   # OpenSubtitles API client, auth, search, download, movie hash
    extraction/    # MKV track parsing, extraction, remuxing
    synchronization/  # alass runner, offset computation, smart/first-match strategies
    models/        # VideoFile, SubtitleCandidate, CoverageEntry

Every module has a clear responsibility. The biggest file is 144 lines (config). In the original Python, acquisition.py alone has 2,726 lines.

What each command does

# Generate the default config
easy-subtitle init

# Extract subtitles from inside MKVs
easy-subtitle extract /path/to/movies

# Download subtitles from OpenSubtitles
easy-subtitle download -l en,pt /path/to/movies

# Sync downloaded subtitles with the video
easy-subtitle sync /path/to/movies

# Full pipeline: extract → download → sync
easy-subtitle run /path/to/movies

# Clean ads/watermarks from SRTs
easy-subtitle clean /path/to/subtitles

# See subtitle coverage by language
easy-subtitle scan --json /path/to/movies

# Compute the OpenSubtitles hash (debug)
easy-subtitle hash /path/to/movie.mkv

# Validate the setup: config, credentials, dependencies
easy-subtitle doctor

doctor is a command I added later. It checks whether the config exists, whether the API key is configured, tests login against the API, and checks whether mkvmerge, mkvextract and alass are on the PATH. It shows OS-specific install instructions when something is missing.

Smart sync with fibers

Smart sync is the part I most enjoyed seeing in the port. In the original Python, it uses ThreadPoolExecutor to run multiple candidates in parallel. In Crystal, the same logic is more natural with fibers and channels:

def execute(candidates : Array(Path), video : VideoFile) : SyncResult?
  channel = Channel(SyncResult).new(candidates.size)

  candidates.each do |candidate|
    spawn do
      result = sync_one(candidate, video)
      channel.send(result)
    end
  end

  results = Array(SyncResult).new(candidates.size)
  candidates.size.times do
    results << channel.receive
  end

  accepted = results.select(&.accepted?)
  accepted.min_by(&.offset)
end

Each subtitle candidate gets synced in a separate fiber (via spawn). The results come back through the Channel. At the end, it picks the accepted one with the smallest offset. No ThreadPoolExecutor, no futures, no callbacks.

API rate limiting

OpenSubtitles requires throttling of 500ms between requests. The Crystal client implements that with a Mutex:

RATE_LIMIT_MS = 500

private def throttle! : Nil
  @mutex.synchronize do
    elapsed = Time.utc - @last_request_at
    remaining = RATE_LIMIT_MS - elapsed.total_milliseconds
    sleep(remaining.milliseconds) if remaining > 0
    @last_request_at = Time.utc
  end
end

Simple, thread-safe, no external library.

Installation

The static binary comes out of GitHub Actions and can be installed three ways:

# Homebrew (macOS / Linux)
brew install akitaonrails/tap/easy-subtitle

# Install script
curl -fsSL https://raw.githubusercontent.com/akitaonrails/easy-subtitle/master/install.sh | bash

# Or grab the binary directly from Releases

One binary. No Python, no pip, nothing.

On porting things “just because”

I’ve always argued that porting software from one language to another just for the language fetish is a waste of time. How many projects have been rewritten in Rust “just because”? How much effort spent on rewrites that delivered no new value?

But I have to admit this experiment made me reconsider.

When the cost of porting drops from weeks/months to less than 40 minutes, the equation changes. Porting Subservient to Crystal with Claude wasn’t an exercise in linguistic vanity. I wanted a static binary I could drop on a server and forget. No managing a Python runtime, no pip install breaking on the next system update.

And the result isn’t a mechanical port. It’s 2,516 lines across 42 files, against 10,220 in 6 monolithic ones. The port came with 76 tests the original didn’t have, CI with automatic release for Linux and macOS, a Homebrew formula and an install script with checksum verification.

The point isn’t that Python is bad. It’s that the bar for “is it worth porting?” got ridiculously low. Feature-parity port with tests in less than an hour. Hard to argue against that.

The repository is here. GPL-3.0, like the original.

ffmpeg -i input.mkv -c:v libx264 -crf 23 -preset medium \
  -profile:v high -level 4.1 -c:a aac -b:a 128k \
  -movflags +faststart output.mp4

What does that do? Convert to H.264 with reasonable quality for the web, using AAC for audio and moving the moov atom to the start of the file to allow progressive streaming. If you already knew that, congratulations. If you didn’t, welcome to the club of 99% of people who use FFmpeg by copying commands from Stack Overflow.

And that’s the easy case. Want to make an animated GIF from a sequence of PNGs? You need a two-pass pipeline with palette generation:

ffmpeg -framerate 10 -i frame_%04d.png \
  -vf "split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse" output.gif

Want to cut a clip, resize to 720p and force a 16:9 aspect ratio? Good luck assembling the filter chain.

I got tired of memorizing flags. I wanted a CLI where I’d say “convert this to MP4 optimized for the web” and it would figure it out. So I built easy-ffmpeg.

What easy-ffmpeg does

It’s a smart wrapper. You give it the input file and the output format, and it analyzes the video with ffprobe, finds out which video, audio and subtitle streams exist, checks codec compatibility against the destination container, and decides on its own what can be copied directly (no re-encoding, instant) and what needs to be transcoded.

The most basic use:

# Convert MKV to MP4 (copies compatible streams, no unnecessary re-encoding)
easy-ffmpeg input.mkv mp4

# Optimized for the web (H.264 + AAC, faststart)
easy-ffmpeg input.mkv mp4 --web

# Optimized for mobile (720p, AAC stereo, smaller file)
easy-ffmpeg input.mkv mp4 --mobile

# Maximum compression (H.265, CRF 28)
easy-ffmpeg input.mkv mp4 --compress

# High quality for streaming (H.265, CRF 18)
easy-ffmpeg input.mkv mp4 --streaming

Examples of what you can do

Cut a clip from the video:

# From minute 1:30 to 3:00
easy-ffmpeg video.mp4 mp4 --start 1:30 --end 3:00

# The first 90 seconds
easy-ffmpeg video.mp4 mp4 --duration 90

# Accepts several time formats: 90, 1:30, 01:30.5, 1:02:30

Resize:

# To 720p
easy-ffmpeg video.mp4 mp4 --scale hd

# To 1080p
easy-ffmpeg video.mp4 mp4 --scale fullhd

# Presets: 2k (1440p), fullhd (1080p), hd (720p), retro (480p), icon (240p)

Change aspect ratio:

# Force 16:9 (adds black bars if needed)
easy-ffmpeg video.mp4 mp4 --aspect wide

# TikTok/Stories format (9:16 vertical)
easy-ffmpeg video.mp4 mp4 --aspect tiktok

# Square for Instagram
easy-ffmpeg video.mp4 mp4 --aspect square

# Crop instead of adding bars
easy-ffmpeg video.mp4 mp4 --aspect wide --crop

Image sequence to video:

# Auto-detects the numbering pattern (frame_0001.png, frame_0002.png...)
easy-ffmpeg /folder/with/frames/ mp4

# Animated GIF with optimized palette
easy-ffmpeg /folder/with/frames/ gif

# Image sequence at 720p, 30fps
easy-ffmpeg /folder/with/frames/ mp4 --fps 30 --scale hd

See what it’ll do without running it:

easy-ffmpeg video.mkv mp4 --web --dry-run
# Shows the exact ffmpeg command that would be executed

Combine everything:

# Cut, resize and compress to send over WhatsApp
easy-ffmpeg video.mkv mp4 --start 0:30 --end 2:00 --scale hd --compress

And if you run easy-ffmpeg with no arguments in an interactive terminal, it opens a TUI mode with file selection through fuzzy search, preset choice through a menu, and time input with validation — all without having to remember any flags.

Why Crystal

I hadn’t touched Crystal in years. The last time was for fun, before version 1.0. And I wanted to revisit it.

Crystal occupies an interesting niche. Go and Crystal compete for the same space: compiled languages for applications, generating static binaries, with garbage collection and no runtime dependency. But the approach is very different.

Go is famously, deliberately simple. No generics for years (they only landed in 1.18), no exceptions (error returns), no expressiveness. The argument is that this makes it easier to read and maintain in big teams. In practice, it produces verbose, repetitive code, with if err != nil on every line.

Crystal has static typing with inference, compile-time macros, blocks as closures (just like Ruby), exceptions, generics from day one, and a syntax any Rubyist will recognize:

# Crystal: read a JSON and extract data
streams = json["streams"].as_a.map do |s|
  StreamInfo.new(
    codec: s["codec_name"]?.try(&.as_s) || "unknown",
    width:  s["width"]?.try(&.as_i) || 0,
    height: s["height"]?.try(&.as_i) || 0,
  )
end

# Go equivalent: would be 3x more lines with type assertions and error checks

Crystal’s stdlib has an HTTP server, JSON parsing, YAML, regex, fibers (green threads with a cooperative scheduler), channels (just like Go), and even IO::FileDescriptor with raw mode for the terminal — which I used for the interactive mode. Concurrency works with spawn (the equivalent of Go’s go) and Channel (identical to Go’s chan). The difference is that everything comes with Ruby’s ergonomics.

For a CLI that needs to compile into a static binary with no dependencies, run on Linux and macOS, and be distributed as a direct download — Crystal is perfect. The compiler generates native binaries, and with Docker Alpine you can do static linking with musl for Linux. The final easy-ffmpeg binary is around 6MB.

I think Rust is better for systems code (kernels, drivers, databases, things where ownership and lifetime matter). But for a video conversion CLI? Rust would be overkill. Crystal gives you the same end result (fast static binary) with a third of the code and without fighting the borrow checker.

The numbers

The whole project was built in one afternoon. 10 commits between 18:32 and 21:33 on March 7, 2026. Three hours.

Language     Files       Lines     Code
Crystal      13          2,823     2,342
Shell        1           109       91
Markdown     1           266       210
YAML         2           46        34
─────────────────────────────────────────
Total        20          3,326     2,419

2,342 lines of Crystal do: a CLI with 5 presets, media analysis through ffprobe, smart conversion planning with a codec compatibility matrix, a progress bar with ETA, an interactive mode with fuzzy search, support for image sequences and GIFs, and trimming/scaling/aspect ratio with validation.

The tool already has CI/CD on GitHub Actions, compiling static binaries for Linux (x86_64 and arm64) and macOS (arm64), with installation through a one-line curl:

curl -fsSL https://raw.githubusercontent.com/akitaonrails/easy-ffmpeg/master/install.sh | sh

Three hours of one afternoon, from the first line of code to a release on GitHub with binaries for three platforms. FFmpeg keeps doing the heavy lifting — I just put a decent interface in front of it.

The repository is here. MIT license.

But there are many opportunities to accelerate Rails applications as well. For example, we just saw the release of the gem “faster_path”, this time written in Rust and bridged through FFI (Foreign Function Interface). The author’s claim is that Sprockets has to compute lots of paths and making this library, natively compiled and optimized with Rust, added a huge improvement in the asset pipeline task.

Sam Saffron, from Discourse, also built a very small gem called “fast_blank” which is a tiny library written in C that reimplements ActiveSupport’s String#blank? method to be up to 9x faster. Because Rails digests volumes of strings, checking if they are blank every time, this adds some performance (depends on your app, of course).

The Holy Grail to native-level performance is to be able to write close-to-Ruby code instead of having to hack low-level C or having the high learning curve of a language such as Rust. More than that, I’d like to avoid having to use FFI. I am not an expert in FFI but I remember understanding that it adds overhead to make the bindings.

By the way, it’s important to disclose right now: I am not a C expert by any means of the imagination, far from that. So I have very little experience dealing with hardcore C development. Which is again, why this possibility of writing in Crystal is even more appealing to me. So if you are a C expert and you spot something silly I am saying about it, please let me know in the comments section below.

My exercise is to rewrite the C-based Fast Blank gem in Crystal, add it to the same Gem to compile under Crystal if it’s available or fall back to C, and make the specs pass so it’s a seamless transition for the user.

To achieve that I had to:

• Extend the Gem’s extconf.rb to generate different Makefiles (for C and Crystal) that are able to compile under OS X or Linux (Ubuntu at least) - OK
• Make the specs pass in the Crystal version - Almost (it’s ok for all intents and purposes but an edge case)
• Make the performance be faster than Ruby and close to C - Not so much yet (under OS X the performance is quite good, but under Ubuntu it doesn’t scale so well for large strings)

You can check out the results so far on my fork over Github and follow the Pull Request discussion as well.

Comparing C and Crystal

Just to get us started, let’s check out a snippet of Sam’s original C version:

static VALUE
rb_str_blank(VALUE str)
{
  rb_encoding *enc;
  char *s, *e;

  enc = STR_ENC_GET(str);
  s = RSTRING_PTR(str);
  if (!s || RSTRING_LEN(str) == 0) return Qtrue;

  e = RSTRING_END(str);
  while (s < e) {
    int n;
    unsigned int cc = rb_enc_codepoint_len(s, e, &n, enc);

    if (!rb_isspace(cc) && cc != 0) return Qfalse;
    s += n;
  }
  return Qtrue;
}

Yeah, quite scary, I know. Now let’s see the Crystal version:

struct Char
  ...
  # same way C Ruby implements it
  def is_blank
    self == ' ' || ('\t' <= self <= '\r')
  end
end

class String
  ...
  def blank?
    return true if self.nil? || self.size == 0
    each_char { |char| return false if !char.is_blank }
    return true
  end
end

Hell yeah! If you’re a rubyist I bet you can understand 100% of the snippet above. It’s not “exactly” the same thing (as the specs are not fully passing yet), but it’s damn close.

The Quest for a Makefile to Crystal

I’ve researched many experimental Github repos and Gists out there. But I didn’t find one that has it all so I decided to tweak what I found until I got to this version:

Obs: again, I am not a C expert. If you have experience with Makefiles I know this one can be refactored to something nicer than this. Let me know in the comments below.

ifeq "$(PLATFORM)" ""
PLATFORM := $(shell uname)
endif

ifeq "$(PLATFORM)" "Linux"
UNAME = "$(shell llvm-config --host-target)"
CRYSTAL_BIN = $(shell readlink -f `which crystal`)
LIBRARY_PATH = $(shell dirname $(CRYSTAL_BIN))/../embedded/lib
LIBCRYSTAL = $(shell dirname $(CRYSTAL_BIN) )/../src/ext/libcrystal.a
LIBRUBY = $(shell ruby -e "puts RbConfig::CONFIG['libdir']")
LIBS = -lpcre -lgc -lpthread -levent -lrt -ldl
LDFLAGS = -rdynamic

install: all

all: fast_blank.so

fast_blank.so: fast_blank.o
  $(CC) -shared $^ -o $@ $(LIBCRYSTAL) $(LDFLAGS) $(LIBS) -L$(LIBRARY_PATH) -L$(LIBRUBY)

fast_blank.o: ../../../../ext/src/fast_blank.cr
  crystal build --cross-compile --release --target $(UNAME) $<

.PHONY: clean
clean:
  rm -f bc_flags
  rm -f *.o
  rm -f *.so
endif

ifeq "$(PLATFORM)" "Darwin"
CRYSTAL_FLAGS = -dynamic -bundle -Wl,-undefined,dynamic_lookup

install: all

all: fast_blank.bundle

fast_blank.bundle: ../../../../ext/src/fast_blank.cr
  crystal $^ --release --link-flags "$(CRYSTAL_FLAGS)" -o $@

clean:
  rm -f *.log
  rm -f *.o
  rm -f *.bundle
endif

Most people using Crystal are on OS X, including the creators of Crystal. LLVM is under Apple’s umbrella and their entire ecosystem relies heavily on LLVM. They spent many years migrating the C front-end first, then the C back-end away from GNU’s standard GCC to Clang. And they were able to make both Objective-C and Swift compile down to LLVM’s IR and that’s how both can interact back and forth natively.

Then, they improved the ARM backend support and that’s how they can have an entire iOS “Simulator” (not a dog slow emulator like Android) where the iOS apps are natively compiled to run over Intel’s x86_64 processor while in development and then quickly recompile to ARM when ready to package to the App Store.

This way you can run natively, test quickly, without the slowness of an emulated environment. By the way, I will say this once: Google’s biggest mistake is not supporting LLVM as they should, and reinventing the wheel instead. If they had, Go could already be used to implement for Android and Chromebooks as well as x86 based servers and they could put away all the Java/Oracle debacle.

But I digress.

In OS X you can pass a “-bundle” link-flag to crystal and it will probably use clang underneath to generate the binary bundle.

On Ubuntu crystal just compiles down to an object file (.o) and you have to manually invoke GCC with the “-shared” option to create a shared-object. To do that we have to use the "–cross-compile" and pass an LLVM target triplet so it generates the .o (this requires the llvm-config tool).

Shared Libraries (.so) and Loadable Modules (.bundle) are different beasts, check this documentation out for more details.

Keep in mind that benchmarking binaries built with different compilers can make a difference. I am not an expert but out of pure anecdote I believe Ruby under RVM on OS X is compiled using OS X’s default Clang. On Ubuntu it’s compiled under GCC. This seems to make Ruby on OS X “so slightly” inefficient in synthetic benchmarks.

On the other hand, Crystal binaries linked with GCC feel “so slightly” inefficient on Ubuntu, while Ruby on Ubuntu feels a bit faster, having been compiled and linked with GCC.

So when we compare Fast Blank/OS X/bit faster with Ruby/OS X/slower against Fast Blank/Ubuntu/bit slower with Ruby/Ubuntu/bit faster, it seems to give a wider advantage to the OS X benchmark comparison against the Ubuntu benchmark, even though individual computation times are not so far from each other.

I will come back to this point in the benchmarks section.

Finally, every time you have a rubygem with a native extension, you will find this bit in their gemspec files:

Gem::Specification.new do |s|
  s.name = 'fast_blank'
  ...
  s.extensions = ['ext/fast_blank/extconf.rb']
  ...

When the gem is installed through gem install or bundle install it will run this script to generate a proper Makefile. In a pure C extension it will use the built-in “mkmf” library to generate it.

In our case, if we have Crystal installed, we want to use the Crystal version, so I tweaked the extconf.rb to be like this:

require 'mkmf'

if ENV['VERSION'] != "C" && find_executable('crystal') && find_executable('llvm-config')
  # Very dirty patching
  def create_makefile(target, srcprefix = nil)
    mfile = open("Makefile", "wb")
    cr_makefile = File.join(File.dirname(__FILE__), "../src/Makefile")
    mfile.print File.read(cr_makefile)
  ensure
    mfile.close if mfile
    puts "Crystal version of the Makefile copied"
  end
end

create_makefile 'fast_blank'

So, if it finds crystal and llvm-config (which in OS X you have to add the proper path like this: export PATH=$(brew --prefix llvm)/bin:$PATH).

The Rakefile in this project declares the standard :compile task as the first one to run, and it will execute the extconf.rb, which will generate the proper Makefile and run the make command to compile and link the proper library in the proper lib/ path.

So we will end up with lib/fast_blank.bundle on OS X and lib/fast_blank.so on Ubuntu. From there we can just have require "fast_blank" from any Ruby file in the gem and it will have access to the publicly exported C function mappings from the Crystal library.

Mapping C-Ruby to Crystal

Now, any direct C extension - without FFI, fiddle or other “bridges” - will ALWAYS have a much better advantage.

The reason is that you literally have to “copy” data from C-Ruby to Crystal/Rust/Go or whatever other language you’re binding. While with a C-based extension you can operate directly in the memory space with the data without having to move it or copy it away.

For example. First, you have to bind the C functions from C-Ruby to Crystal. And we accomplish that with Paul Hoffer’s Crystalized Ruby mappings. It’s an experimental repository that I helped a bit to clean up in order for him to later extract this mapping library into its own Shard (shards are the same as gems for Crystal). For now I had to simply copy the file over to my Fast Blank.

Some of the relevant bits are like this:

lib LibRuby
  type VALUE = Void*
  type METHOD_FUNC = VALUE -> VALUE
  type ID = Void*
  ...

  # strings
  fun rb_str_to_str(value : VALUE) : VALUE
  fun rb_string_value_cstr(value_ptr : VALUE*) : UInt8*
  fun rb_str_new_cstr(str : UInt8*) : VALUE
  fun rb_utf8_encoding() : VALUE
  fun rb_enc_str_new_cstr(str : UInt8*, enc : VALUE) : VALUE
  ...
  # exception handling
  fun rb_rescue(func : VALUE -> UInt8*, args : VALUE, callback: VALUE -> UInt8*, value: VALUE) : UInt8*
end
...
class String
  RUBY_UTF = LibRuby.rb_utf8_encoding
  def to_ruby
    LibRuby.rb_enc_str_new_cstr(self, RUBY_UTF)
  end

  def self.from_ruby(str : LibRuby::VALUE)
    c_str = LibRuby.rb_rescue(->String.cr_str_from_rb_cstr, str, ->String.return_empty_string, 0.to_ruby)
    # FIXME there is still an unhandled problem: then we receive \u0000 from Ruby it raises "string contains null bytes"
    # so we catch it with rb_rescue, but then we can't generate a Pointer(UInt8) that represents the unicode 0, instead we return a plain blank string
    # but then the specs fail
    new(c_str)
  ensure
    ""
  end

  def self.cr_str_from_rb_cstr(str : LibRuby::VALUE)
    rb_str = LibRuby.rb_str_to_str(str)
    c_str  = LibRuby.rb_string_value_cstr(pointerof(rb_str))
  end

  def self.return_empty_string(arg : LibRuby::VALUE)
    a = 0_u8
    pointerof(a)
  end
end

Then I can use these mappings and helpers to build a “Wrapper” class in Crystal:

require "./lib_ruby"
require "./string_extension"

module StringExtensionWrapper
  def self.blank?(self : LibRuby::VALUE)
    return true.to_ruby if LibRuby.rb_str_length(self) == 0
    str = String.from_ruby(self)
    str.blank?.to_ruby
  rescue
    true.to_ruby
  end

  def self.blank_as?(self : LibRuby::VALUE)
    return true.to_ruby if LibRuby.rb_str_length(self) == 0
    str = String.from_ruby(self)
    str.blank_as?.to_ruby
  rescue
    true.to_ruby
  end

  def self.crystal_value(self : LibRuby::VALUE)
    str = String.from_ruby(self)
    str.to_ruby
  end
end

And this “Wrapper” depends on the “pure” Crystal library itself like with the snippets for the Char struct and String class extensions I showed in the first section of the article above.

Finally, I have a main “fast_blank.cr” file that externs those Wrapper functions so C-Ruby can see them as plain String methods:

require "./string_extension_wrapper.cr"

fun init = Init_fast_blank
  GC.init
  LibCrystalMain.__crystal_main(0, Pointer(Pointer(UInt8)).null)

  string = LibRuby.rb_define_class("String", LibRuby.rb_cObject)
  LibRuby.rb_define_method(string, "blank?", ->StringExtensionWrapper.blank?, 0)
  LibRuby.rb_define_method(string, "blank_as?", ->StringExtensionWrapper.blank_as?, 0)
  ...
end

This is mostly boilerplate. But now check out what I am having to do in the wrapper, in this particular snippet:

def self.blank?(self : LibRuby::VALUE)
  return true.to_ruby if LibRuby.rb_str_length(self) == 0
  str = String.from_ruby(self)
  str.blank?.to_ruby
rescue
  true.to_ruby
end

I am receiving a C-Ruby String casted as a pointer (VALUE) then I go through the lib_ruby.cr mappings to get the C-Ruby string data and copy it over into a new instance of Crystal’s internal String representation. So at any given time I have 2 copies of the same string, one in the C-Ruby memory space and another in the Crystal memory space.

This happens with all FFI-like extensions but it doesn’t happen to the pure C implementation. In Sam Saffron’s C implementation it directly works with the same address in C-Ruby’s memory space:

static VALUE
rb_str_blank(VALUE str)
{
  rb_encoding *enc;
  char *s, *e;

  enc = STR_ENC_GET(str);
  s = RSTRING_PTR(str);
  ...

It receives a pointer (direct memory address) and goes. And this is a huge advantage for the C version. When you have a big volume of medium to large sized strings being copied over from C-Ruby to Crystal, it adds a noticeable overhead that can’t be removed.

String mapping Caveat

I still have a problem though. There is one edge case I was not able to overcome yet (help is most welcome). When C-Ruby passes a unicode "\u0000" I am unable to create the same character in Crystal and I end up passing just an empty string ("") which is not the same thing.

The way to deal with it is to receive a Ruby String (VALUE) and get the C-String from it this way:

rb_str = LibRuby.rb_str_to_str(str)
c_str  = LibRuby.rb_string_value_cstr(pointerof(rb_str))

If the “str” is the “\u0000” (under Ruby 2.2.5 at least) C-Ruby raises a “string contains null bytes” exception. Which is why I rescue from this exception like this:

c_str = LibRuby.rb_rescue(->String.cr_str_from_rb_cstr, str, ->String.return_empty_string, 0.to_ruby)

When an exception is triggered I have to pass the pointer to another function to rescue from it:

def self.return_empty_string(arg : LibRuby::VALUE)
  a = 0_u8
  pointerof(a)
end

But this is not correct, I am just passing the pointer to a “0” character, which is “empty”. Therefore, specs are not passing correctly:

Failures:

  1) String provides a parity with active support function
     Failure/Error: expect("#{i.to_s(16)} #{c.blank_as?}").to eq("#{i.to_s(16)} #{c.blank2?}")

       expected: "0 false"
            got: "0 true"

       (compared using ==)
     # ./spec/fast_blank_spec.rb:22:in `block (3 levels) in <top (required)>'
     # ./spec/fast_blank_spec.rb:19:in `times'
     # ./spec/fast_blank_spec.rb:19:in `block (2 levels) in <top (required)>'

  2) String treats  correctly
     Failure/Error: expect("\u0000".blank_as?).to be_falsey
       expected: falsey value
            got: true
     # ./spec/fast_blank_spec.rb:47:in `block (2 levels) in <top (required)>'

Ary gave a simple tip later, I will add it to the conclusion below.

The Synthetic Benchmarks (careful on how you interpret them!)

The original Rails ActiveSupport implementation of String#blank? looks like this:

class String
  # 0x3000: fullwidth whitespace
  NON_WHITESPACE_REGEXP = %r![^\s#{[0x3000].pack("U")}]!

  # A string is blank if it's empty or contains whitespaces only:
  #
  #   "".blank?                 # => true
  #   "   ".blank?              # => true
  #   "　".blank?               # => true
  #   " something here ".blank? # => false
  #
  def blank?
    # 1.8 does not takes [:space:] properly
    if encoding_aware?
      self !~ /[^[:space:]]/
    else
      self !~ NON_WHITESPACE_REGEXP
    end
  end
end

It’s mainly a regular expression comparison, which can be a bit slow. Sam’s version is a more straightforward loop through the string to compare each character with what’s considered “blank”. There are many unicode codepoints that are considered blank, some are not, which is why the C and Crystal versions are similar, but they are different from Rails’ version.

In the Fast Blank gem there is a benchmark Ruby script to compare the C-extension against Rails’ Regex based implementation.

The Regex implementation is called “Slow Blank”. It’s particularly slow if you pass a real empty String, so in the benchmark Sam added a “New Slow Blank” that checks through String#empty? first, and this version is faster in this edge case.

The fast C version is called “Fast Blank” but although you can consider it “correct” it’s not compatible with all the edge cases from Rails. So he implemented a String#blank_as? which is compatible with Rails. Sam calls it “Fast Activesupport”.

In my Crystal version I did the same, having both String#blank? and String#blank_as?.

So, without further ado, here is the C Version over OS X benchmark for empty strings, and we exercise each function many times within a few seconds to have more accurate results (check out Evan Phoenix’s “benchmark/ips” to understand the “iteration per second” methodology).

================== Test String Length: 0 ==================
Warming up _______________________________________
          Fast Blank   191.708k i/100ms
  Fast ActiveSupport   209.628k i/100ms
          Slow Blank    61.487k i/100ms
      New Slow Blank   203.165k i/100ms
Calculating _______________________________________
          Fast Blank     20.479M (± 9.3%) i/s -    101.414M in   5.001177s
  Fast ActiveSupport     21.883M (± 9.4%) i/s -    108.378M in   5.004350s
          Slow Blank      1.060M (± 4.7%) i/s -      5.288M in   5.001365s
      New Slow Blank     18.883M (± 6.9%) i/s -     94.065M in   5.008899s

Comparison:
  Fast ActiveSupport: 21882711.5 i/s
          Fast Blank: 20478961.5 i/s - same-ish: difference falls within error
      New Slow Blank: 18883442.2 i/s - same-ish: difference falls within error
          Slow Blank:  1059692.6 i/s - 20.65x slower

It’s super fast. Rails’ version is 20x slower on my machine.

Now, Crystal version over OS X

================== Test String Length: 0 ==================
Warming up _______________________________________
          Fast Blank   174.349k i/100ms
  Fast ActiveSupport   174.035k i/100ms
          Slow Blank    64.684k i/100ms
      New Slow Blank   215.164k i/100ms
Calculating _______________________________________
          Fast Blank      8.647M (± 1.6%) i/s -     43.239M in   5.001530s
  Fast ActiveSupport      8.580M (± 1.3%) i/s -     42.987M in   5.010759s
          Slow Blank      1.047M (± 3.7%) i/s -      5.239M in   5.008907s
      New Slow Blank     19.090M (± 9.3%) i/s -     94.672M in   5.009057s

Comparison:
      New Slow Blank: 19090034.8 i/s
          Fast Blank:  8647459.7 i/s - 2.21x slower
  Fast ActiveSupport:  8580487.9 i/s - 2.22x slower
          Slow Blank:  1047465.3 i/s - 18.22x slower

As I explained before, even checking empty strings, the Crystal version is slower than the Ruby check for String#empty? (New Slow Blank) because I have the string copying routine of the Wrapper mappings. This adds overhead that is perceptible over many iterations. It’s still 18x faster than Rails, but it loses to C-Ruby.

Finally, Crystal version over Ubuntu

================== Test String Length: 0 ==================
Warming up _______________________________________
          Fast Blank   255.883k i/100ms
  Fast ActiveSupport   260.915k i/100ms
          Slow Blank   105.424k i/100ms
      New Slow Blank   284.670k i/100ms
Calculating _______________________________________
          Fast Blank      8.895M (± 9.8%) i/s -     44.268M in   5.037761s
  Fast ActiveSupport      8.647M (± 8.2%) i/s -     43.051M in   5.020125s
          Slow Blank      1.736M (± 3.9%) i/s -      8.750M in   5.048253s
      New Slow Blank     22.170M (± 6.2%) i/s -    110.452M in   5.004909s

Comparison:
      New Slow Blank: 22170031.0 i/s
          Fast Blank:  8895113.3 i/s - 2.49x slower
  Fast ActiveSupport:  8646940.8 i/s - 2.56x slower
          Slow Blank:  1736071.0 i/s - 12.77x slower

Notice that it’s around the same ballpark, but the Rails version on Ubuntu runs almost twice as fast compared to its counterpart in OS X, which makes the comparison against the Crystal library go down from 18x to 12x.

The benchmark keeps comparing against strings of larger and larger sizes, from 6, to 14, to 24, up to 136 characters in length.

Let’s get just the last test case of 136 characters. First with C version on OS X:

================== Test String Length: 136 ==================
Warming up _______________________________________
          Fast Blank   177.521k i/100ms
  Fast ActiveSupport   193.559k i/100ms
          Slow Blank    89.378k i/100ms
      New Slow Blank    60.639k i/100ms
Calculating _______________________________________
          Fast Blank     10.727M (± 8.7%) i/s -     53.256M in   5.006538s
  Fast ActiveSupport     11.600M (± 8.3%) i/s -     57.681M in   5.009692s
          Slow Blank      1.872M (± 5.7%) i/s -      9.385M in   5.029243s
      New Slow Blank      1.017M (± 5.3%) i/s -      5.094M in   5.022994s

Comparison:
  Fast ActiveSupport: 11600112.2 i/s
          Fast Blank: 10726792.8 i/s - same-ish: difference falls within error
          Slow Blank:  1872262.5 i/s - 6.20x slower
      New Slow Blank:  1016926.7 i/s - 11.41x slower

The C-version is consistently much faster in all test cases and in the 136 characters it’s still 11x faster than Rails in pure Ruby.

Now the Crystal version over OS X:

================== Test String Length: 136 ==================
Warming up _______________________________________
          Fast Blank   127.749k i/100ms
  Fast ActiveSupport   126.538k i/100ms
          Slow Blank    94.390k i/100ms
      New Slow Blank    60.594k i/100ms
Calculating _______________________________________
          Fast Blank      3.283M (± 1.8%) i/s -     16.480M in   5.021364s
  Fast ActiveSupport      3.235M (± 1.3%) i/s -     16.197M in   5.008315s
          Slow Blank      1.888M (± 4.4%) i/s -      9.439M in   5.009458s
      New Slow Blank    967.950k (± 4.7%) i/s -      4.848M in   5.018946s

Comparison:
          Fast Blank:  3283025.1 i/s
  Fast ActiveSupport:  3234586.5 i/s - same-ish: difference falls within error
          Slow Blank:  1887800.5 i/s - 1.74x slower
      New Slow Blank:   967950.2 i/s - 3.39x slower

It’s also faster, but just by 2 to 3 times compared to pure Ruby, a far cry from 11x. But my hypothesis is that the mapping and copying of so many strings over adds a large overhead that the C version does not have.

And the Crystal version over Ubuntu:

================== Test String Length: 136 ==================
Warming up _______________________________________
          Fast Blank   186.810k i/100ms
  Fast ActiveSupport   187.306k i/100ms
          Slow Blank   143.439k i/100ms
      New Slow Blank    98.308k i/100ms
Calculating _______________________________________
          Fast Blank      3.517M (± 3.9%) i/s -     17.560M in   5.000791s
  Fast ActiveSupport      3.485M (± 3.8%) i/s -     17.419M in   5.006427s
          Slow Blank      2.755M (± 4.2%) i/s -     13.770M in   5.008490s
      New Slow Blank      1.551M (± 4.3%) i/s -      7.766M in   5.017853s

Comparison:
          Fast Blank:  3516960.7 i/s
  Fast ActiveSupport:  3484575.5 i/s - same-ish: difference falls within error
          Slow Blank:  2754669.0 i/s - 1.28x slower
      New Slow Blank:  1550815.2 i/s - 2.27x slower

Again, the Ubuntu versions of both the Crystal library and the Ruby binary run faster and the comparison shows no more than twice as much faster. And pure Ruby’s String#empty? is in the same ballpark as Crystal’s version.

Conclusion

The most obvious conclusion is that I probably made a mistake in choosing Fast Blank as my first proof of concept. The algorithm is too trivial and a simple check for String#empty? in pure Ruby is orders of magnitude faster than the added overhead of mapping and string copying to Crystal.

Also, any use case where you have a huge amount of small bits of data being transferred from C-Ruby to Crystal or any FFI-based extension will have the overhead of data copying, which a pure C-version will not have. Which is why Fast Blank is better done in C.

Any other use case where you have smaller amounts of data, or data that can be transferred in bulk (less calls from C-Ruby to the extension, with arguments with a larger size, and with more costly processing) is a better candidate to benefit from extensions.

Again, not everything gets automatically faster, we always have to figure out the use case scenarios first. But because it’s so much easier to write in Crystal and benchmark, we can make faster proofs of concept and scrap the idea if the measurements prove that we won’t benefit as much.

The Crystal documentation recently received a “Performance Guide”. It’s very useful for you to avoid common pitfalls that harm overall performance. Even though LLVM is quite competent in heavy optimization, it can’t do everything. So read it through to improve your general Crystal skills.

That being said, I still believe that this exercise was well worth it. I will probably do some more. I’d really want to thank Ary (Crystal creator) and Paul Hoffer for the patience in helping me out through many of the quirks I found along the way.

While I was finishing this post, Ary pointed out that I could probably ditch Strings altogether and work directly with an array of bytes, which is a good idea and I will probably try that. I think I made it clear by now that the whole String copying adds a very perceptible overhead as we saw in the benchmarks above. Let me know if someone is interested in contributing as well. With a few more tweaks I believe we can have a Crystal version that can at least compete against the C version while also being more readable and maintainable for most Rubyists, which is my goal.

I hope the code I published here will serve as boilerplate examples for more Crystal-based Ruby extensions in the future!

I’ll say that Erlang OTP and its internals intrigue me more than just the novelty of the language syntax. I believe that if you want to build highly distributed and also highly reliable systems you will either use Erlang (through Elixir) or you will end up replicating most of what Erlang already has with OTP.

This is true if you see that there are many Java/Scala-based distributed systems such as Hadoop, Spark, Cassandra, and they all could have benefitted from Erlang’s OTP architecture. On the other hand, if you follow Cassandra you will see that even Java still has a hard time competing with C++ if you compare the clone called ScyllaDB.

I believe Elixir (perhaps with HiPE) can compete in the same league as Java for distributed systems, avoiding people leaving the Erlang platform for C or Java as happened with CouchDB back in 2012 because of lack of interest and exotic syntax. And because Java already has a big ecosystem, Clojure is in the game primarily because it can interface directly with it.

I think Go is also a Java contender but for other use cases, particularly in the systems tools space where you have lots of C/C++ and also glue code in Perl and Python. An obvious example being Docker orchestration. Yes, you can do microservices, crawlers and other stuff but I am not so keen to build big “applications” in it, although there is nothing preventing it. Again, it’s just a personal opinion.

Rust is a big contender to low-level systems development. I think it can replace C in many use cases and be used to implement libraries and components to be used by other languages, with the added benefits of a modern language that tries to avoid security hazards such as the recent Heartbleed debacle. But because of its “Borrow system” I find it extremely bureaucratic, especially if you’re coming from highly dynamic languages such as Ruby, Python or even Swift or Elixir. I’m not saying it’s a bad thing, just something that takes way more time to become comfortable with than I expected.

Crystal is something between Rust and Go. LLVM is something you must have on your radar, because of Apple’s support towards Swift it’s better than ever. Crystal is similar to RubyMotion in terms of both being similar to Ruby but not fully compatible and both being front-end parsers to LLVM. RubyMotion, on the other hand, is closer to Ruby and can even use some Ruby libraries almost without changes.

You do have to take your hats off to Ary Borenszweig and his contributors. It’s very impressive how far they got in such a short period of time and without having deep pockets from Mozilla, Apple or Google.

Limitations

Crystal borrows heavily from Ruby but it’s not a goal to reach any level of compatibility. It’s a strong, statically typed language with Type Inference. It does not have any runtime component such as RubyMotion or Swift, so it has no notion of introspection or reflection over objects.

Mainly, you have neither “#eval” nor “#send”. The lack of runtime evaluation is not so bad as you do have compilation-time AST manipulation through a good enough system of Macros (more on that later).

But the lack of “#send” does hurt a bit. It’s the one thing that makes Crystal further away from Ruby’s dynamic object flexibility. But it’s also understandable why it’s not a priority to have it.

As of version 0.17.0, Crystal has one huge limitation: it’s using a Boehm-Demers-Weiser conservative garbage collector.

@asterite @headius I assumed as much... the Boehm GC, although handy for starting a new language, will only get you so far

— Jason Frey (@Fryguy9) May 29, 2016

The language is currently implemented as a single-threaded process. This means that you probably can’t max out all CPUs of your machine with just a single process. Although I may be wrong here, it’s just my first impressions.

Charles Nutter (from JRuby fame) gives a warning about this:

@fryguy9 From their site: https://t.co/uOjIT0c8ji It does say something about "when" parallelism happens. Risky to start single-threaded.

— Charles Nutter (@headius) May 29, 2016

It’s a double-edged sword. Using a “generic” plug-and-play GC such as Boehm - which is not a bad thing in itself, but it’s possibly not nearly as powerful as the JVM’s own set of high performance GC such as the brand new G1GC.

Making the language single-threaded by default also means that the entire standard library and the whole ecosystem are built assuming there is no parallelism. So there are no race-conditions and hence, no usage of proper synchronization anywhere. If one day they decide to add multi-threading, all existing code will not be thread-safe. This is probably the meaning of Charles’ warning.

I don’t think it’s a pressing problem though. You probably won’t have a highly parallel system built with Crystal but maybe it’s not the use case. In that case you should be using Elixir + OTP, Scala + Akka, for example.

Concurrency

Not having access to native Threads doesn’t mean that you can’t have Concurrency. Concurrency is not Parallelism. Since version 0.7.0 they have added support for non-blocking I/O and also Fibers and Channels.

In a nutshell, a Fiber is a special kind of coroutine. It’s a “piece of execution” that can “pause” itself and yield control of the execution back to its caller. It can pause itself but it can’t be paused from the outside like Erlang’s Scheduler can pause its running processes, for example.

In this case we can have a “Fiber Scheduler” of sorts, that can “resume” other Fibers. It is currently single-threaded, of course. And it works as an Event Loop. Whenever you have a non-blocking I/O operation waiting for file reading, network packet sending and stuff like that, it can yield control back to other fibers to resume work.

This makes it work more or less like Node.js. Javascript is also a single-threaded language and Node.js is basically an implementation of an event loop. Functions declare other anonymous functions as callbacks and it works based on the I/O triggers to callback those functions on every tick of the reactor.

On top of that you can add “Channels” as popularized by Google’s Go language. You can start as many channels as you like. Then you can spawn Fibers in the Scheduler. Fibers can execute and keep sending messages through the Channels. Execution control is yielded to whoever is expecting to receive from the same Channels. Once one of them receives and executes, control is sent back to the Scheduler to allow other spawned Fibers to execute, and they can keep “pinging” and “ponging” like this.

Speaking of ping-pong, you have this snippet in the “Go by Example” site:

package main
import "fmt"
func ping(pings chan<- string, msg string) {
    pings <- msg
}
func pong(pings <-chan string, pongs chan<- string) {
    msg := <-pings
    pongs <- msg
}
func main() {
    pings := make(chan string, 1)
    pongs := make(chan string, 1)
    ping(pings, "passed message")
    pong(pings, pongs)
    fmt.Println(<-pongs)
}

And this is mostly the same thing but implemented in Crystal:

def ping(pings, message)
  pings.send message
end

def pong(pings, pongs)
  message = pings.receive
  pongs.send message
end

pings = Channel(String).new
pongs = Channel(String).new
spawn ping pings, "passed message"
spawn pong pings, pongs
puts pongs.receive
# => "passed message"

Same thing, but more pleasant to my eyes, again a personal opinion - which is expected if you’re also a Rubyist.

So Crystal has Node.js/Javascript-like Event Loop in the form of a Fiber Scheduler and a Go-like Channel/CSP mechanism to coordinate concurrent processing. It’s a “cooperative multitasking” mechanism. It’s not good if you have CPU intensive processing, such as heavy number crunching, data processing. It works best for I/O intensive processing.

If you have a Fiber doing super heavy processing, it will block the Scheduler, as is also true for Node.js. The current implementation of the Scheduler is bound to a single native thread so it can’t use other available native threads of the system for now.

Because the Channel implementation is quite new, it’s not used throughout the standard library. But the standard implementation for HTTP::Server uses Fibers. And because it is compiled down to a native binary it’s way faster than Javascript, as this “fast-http-server” shows:

• fast-http-server (Crystal) 18348.47rpm at 8.67ms avg. response time
• http-server (Node.js) 2105.55rpm at 47.92ms avg. response time
• SimpleHTTPServer (Python) 785.14rpm at 1.91ms avg. response time

We’re talking about an order of magnitude of 8 times faster than Node.js and more than 20 times faster than Python.

As usual, you shouldn’t blindly trust synthetic benchmarks but depending on which kind of processing you decide to compare against, you will see Crystal being pretty on par with Rust, Go, Swift, D.

Type Inference

Crystal is a statically typed language. But it won’t require you to declare every single type beforehand. It’s quite smart in its Type Inference so if you just type:

a = 1
b = "HELO"

It will know that “a” is an “Int32” and that “b” is a “String”. By the way, contrary to Ruby, all Strings must be double-quoted. But it becomes particularly more complicated when you’re dealing with complicated data structures such as JSON.

In Ruby I can just parse a JSON String and immediately explore its structure like this:

parsed_json = JSON.parse(response.body)["files"].first["id"]

I can’t do this in Crystal, instead the recommended approach is to declare a schema-like structure like this:

class FilesResponse
  JSON.mapping({
    ok: {type: Bool},
    files: {type: Array(FilesItemResponse)}
  })
end
class FilesItemResponse
  JSON.mapping({
    id: {type: String},
    name: {type: String},
    filetype: {type: String},
    created: {type: Time, converter: Time::EpochConverter}
    timestamp: {type: Time, converter: Time::EpochConverter}
    permalink: {type: String}
  })
end
...
# parsed_json = JSON.parse(response.body)["files"].first["id"]
parsed_json = FilesResponse.from_json(@body).files.first.id

I am very used to Ruby’s duck typing and the ability to query objects in runtime. It’s just not the same in Crystal and it can be quite tedious to change your mindset to think about types beforehand. The compiler will scream a hell of a lot during your development cycle until you become acquainted with this concept.

I did an experiment with JSON parsing and the result is a project I called “cr_slack_cleanup” which showcases both this idea of JSON Schemas as well as Fibers and Channels as I explained in the previous section.

Update: after I posted this article @LuisLavena stepped in to correct me: you can do it like Ruby, without schemas:

@AkitaOnRails @CrystalLanguage re: JSON, you can parse it kinda-like Ruby, see JSON::Any: https://t.co/MD7PYy5AVH

— Luis Lavena (@luislavena) June 1, 2016

And it will be like this:

require "json"

data = <<-JSON
  {
    "files": [
        {
          "id": 1,
          "name": "photo.jpg"
        },
        {
          "id": 99,
          "name": "another.jpg"
        }
    ]
  }
  JSON

obj = JSON.parse(data)

obj["files"].each do |item|
  puts item["id"].as_i
end

So yeah, JSON parsing can be almost identical to what you would do with Ruby, although he also recommends Schemas as it’s a bit faster anyway.

Macros

The last important feature worth mentioning is the presence of Macros. As you don’t have control over your code in runtime, you don’t have “eval”, you don’t have “send”, then you would not have any way to do proper metaprogramming.

Macros bring back some of Ruby’s metaprogramming. For example, in Ruby, when you include a Module to a Class, there is the “included?” hook in the Module where you can add some metaprogramming such as “class_eval” stuff.

In Crystal, a Module does have an “included” hook but now it’s a Macro. You can do something like this:

module Foo
  macro included
    {% if @type.name == "Bar" %}
    def self.hello
      puts "Bar"
    end
    {% else %}
    def self.hello
      puts "HELLO"
    end
    {% end %}
  end
end

module Bar
  include Foo
end

module Something
  include Foo
end

Bar.hello # => "Bar"
Something.hello # => "HELLO"

It feels like having “ERB” templates but for code. A macro is code building code in compile time. In the resulting AST it’s as if you wrote the boring repetitive code yourself. The compiled native binary doesn’t care. If you’re from C it’s like pre-processing but having control of an AST tree of Nodes instead of just manipulating the source code. You can even have something akin to the venerable “#method_missing”.

class Foo
  macro method_missing(name, args, block)
    {{pp name}}
  end
end

Foo.new.hello_world
# => name = "hello_world"
Foo.new.bla_bla("bla")
# => name = "bla_bla"

Any macro function can receive an AST as the argument and you can manipulate this AST in any way as you see fit. For example:

module Foo
  macro teste(ast)
    puts {{ast.stringify}}
  end
end

Foo.teste "Hello World".split(" ").join(" - ")
# => ("Hello World".split(" ")).join(" - ")

In this example you just get a normalized version of the code you passed as argument. And the code won’t be “executed” at that location.

So, you can define new methods and call different versions of methods depending on the combination of elements that you get from the AST. One example of what is possible is this experiment from developer Oleksii that I bundled in this small project I called “cr_chainable_methods” and we can build code that is very different from either Crystal or Ruby:

result = pipe "Hello World"
  .>> Foo.split_words
  .>> Foo.append_message("Bar")
  .>> Bar.add_something
  .>> Foo.join
  .>> unwrap

(Yes, a subset of what a real Elixir Pipe can do.)

Crystal Shards

Finally, one aspect I like about Crystal is that it has task management and dependency management already figured out. No pointless discussions on which implementation of packages and dependencies we “should” be using.

You can just write a simple script into a file such as “test.cr” and run this file like this:

crystal test.cr

Or, you can build and compile to a native binary before executing like this:

crystal build test.cr --release
./test

And you can also start in a proper project structure and development lifecycle like this:

crystal init app test
cd test
crystal build src/test.cr --release
./test
git add .
git commit -m "initial commit"
git push -u origin master

It sets up a proper project directory structure, a Git repository is pre-initialized for you (why would you be programming without Git!?) with proper “.gitignore” file.

You will also find a “shard.yml” file where you can declare your application or library name, version, author, and also its dependencies like this:

name: kemal
version: 0.12.0

dependencies:
  radix:
    github: luislavena/radix
    version: 0.3.0
  kilt:
    github: jeromegn/kilt
    version: 0.3.3
author:
  - Serdar Dogruyol <dogruyolserdar@gmail.com>

This example is from the Kemal web framework by Serdar. It depends on the Radix and Kilt libraries. You must run crystal deps and it will fetch from Github before you can compile.

More than that: every project receives a proper “spec” directory where you can write Rspec-like tests and you can run them using crystal spec. And the spec runner results will look like this:

$ crystal spec
........................................................

Finished in 8.28 milliseconds
56 examples, 0 failures, 0 errors, 0 pending

All built-in, no external libraries needed.

And 8 milliseconds to run the entire test suite of a micro web framework? Not too shabby. If you add up start up time for the executable, it still is around 1 second of total running time. Pretty impressive.

Conclusion

Crystal is not meant to be a Ruby replacement like JRuby. So it will never run existing and complicated Ruby projects such as Ruby on Rails or even Sinatra for that matter.

To me Crystal fills a gap in some use cases where I would use Rust or Go. It has the raw performance of an LLVM-optimized native binary, a reasonably fast Go-like concurrency system (and much faster and less resource intensive than Node.js), and the added benefit of a much more enjoyable Ruby-like syntax set.

You can even start experimenting with Crystal to build Ruby native extensions without the need of FFI and without having to write C. There is an attempted port of ActiveSupport as a native extension to MRI Ruby, written as a proof of concept in Crystal.

If you’re a Rubyist you will find yourself pretty much at home with most of the standard library. You will miss a thing or two but most of the time it will feel like Ruby. The official Guide is good enough to get started and there is a comprehensive API documentation just to check if that favorite Ruby API is there or not, and if not what the replacements are. Most of the time you will think “hmm, in Ruby I would use Enumerator#each … and yes, sure enough, there is Enumerator#each as expected.” The API documentation could use some more love, so if you want to contribute this is a good place to start.

If you want to check out a curated list of Crystal libraries and applications, go straight to “Awesome Crystal”. You can also see the more dynamic “Trending Projects in Github”.

For the most part, your proficiency in Ruby will pay off to build small systems where raw performance and concurrency are really necessary. This is exactly the use case for Mike Perham’s port of Sidekiq.cr. Say you have gigabytes of files to crunch. Or you have thousands of external websites to crawl, parse and organize. Or you have petabytes of data in your database waiting to be number crunched. Those are tasks that MRI Ruby won’t help with but Crystal may well be the quick answer.

You can even get one of the many micro web frameworks such as Kemal and deploy your HTTP API providers directly to Heroku now. This will give you Go-levels of performance, and this is very compelling for me as I really also dislike Go’s archaic smelling syntax.

So, Rust, Go, Javascript, all very fast, but with very questionable syntaxes and not so enjoyable. They are way more mature, and their ecosystems are much larger. But unless I really need to, I’d rather choose Crystal for the use cases I described above, and I think you will find some good use for it too.

Update: after I posted, the creator of Crystal, @Asterite had a few things to add as well:

@AkitaOnRails @waj so it's likely that we'll have something similar. We aren't there yet because we are not Google, Mozilla nor Apple

— Ary Borenszweig (@asterite) May 31, 2016