• MRI Ruby 2.3.1: 57 seconds
• JRuby 9.1.1.0: 45 seconds
• Crystal 0.17.0: 59 seconds
• Elixir: 14 seconds

I also said how it was an unfair comparison as the Elixir version uses a different - and obviously more efficient - algorithm.

It was the “first-version-that-worked” so I decided to go ahead and improve the implementations. In the Ruby/JRuby version I added the thread gem to have a good enough implementation of a Thread pool that works for Ruby and JRuby. I probably should have used Concurrent-Ruby but I was having some trouble making FixedThreadPool work.

Anyway, now all versions will have a constant pool of requests running and we can make a better comparison.

Another thing that may have skewed the results against Crystal is that it seems to have a faulty DNS resolver implementation, so for now I just added the Manga Reader IP address directly to my /etc/hosts file to avoid getaddrinfo: Name or service not known exceptions.

To begin, let’s test the same Elixir implementation, and as expected the result is still the same:

11,49s user 0,82s system 77% cpu 15,827 total

Now, the MRI Ruby with the “batch burst” algorithm was taking 57 seconds and this new implementation using a ThreadPool runs much better:

12,67s user 0,92s system 50% cpu 27,149 total

In Crystal, it’s a bit more complicated as there is no way to implement the equivalent of a “Fiber Pool”. What we have to do is run an infinite loop until the last process signals a loop break. Within the loop we create a maximum number of fibers, wait for each one to signal that it finished through an individual channel and loop again to create a new fiber, and so on. Compared to yesterday’s 59 seconds, this is much better:

5,29s user 0,33s system 26% cpu 21,166 total

The JRuby version is not so fast though. Still better than yesterday’s 45 seconds but now it’s losing even to MRI:

49,24s user 1,41s system 146% cpu 34,602 total

I tried to use the --dev flag for faster start up time and it does improve a bit, getting it closer to MRI:

22,26s user 0,99s system 76% cpu 30,320 total

Not sure if it can be improved more though, any tips are welcome - don’t forget to comment below.

So, Elixir is still at least twice as fast as Crystal at this point. But this also demonstrates how a different algorithm does make a huge difference where it matters. I can probably tweak it a bit more but this should suffice for now.

Changing the Ruby implementation to use ThreadPool

require "thread/pool"

module MangaDownloadr
  class Concurrency
    ...
    def fetch(collection, &block)
      pool    = Thread.pool(@config.download_batch_size)
      mutex   = Mutex.new
      results = []

      collection.each do |item|
        pool.process {
          engine  = @turn_on_engine ? @engine_klass.new(@config.domain) : nil
          reply = block.call(item, engine)&.flatten
          mutex.synchronize do
            results += ( reply || [] )
          end
        }
      end
      pool.shutdown

      results
    end
    ...

The idea is that we will have a fixed number of spawned native threads (in this case, determined by @config.download_batch_size). As one thread finishes it will pop a new link from the collection Array, essentially working as a “queue” to be depleted.

The results are accumulated in the results Array. Because many threads may want to modify it at once we have to synchronize access through a Mutex.

This way we always have a fixed amount of workers performing requests constantly instead of slicing the collection Array and doing bursts as in the previous version.

Changing the Crystal implementation to simulate a Fibers Pool

The Crystal version became a bit more complicated as I didn’t find a pool library to pull in. I found a rough implementation of a pool in this stackoverflow post and I was able to implement an improved version into a new shard so you can take advantage of it in your projects. Check out the source code at akitaonrails/fiberpool. This is how I added it to my project:

dependencies:
  ...
  fiberpool:
    github: akitaonrails/fiberpool

And this is how I used it. Notice that the logic itself is very close to the MRI Ruby version, but using a “Fiber Pool” instead of a Thread Pool.

require "fiberpool"

module CrMangaDownloadr
  struct Concurrency
    def initialize(@config : Config, @turn_on_engine = true); end

    def fetch(collection : Array(A)?, engine_class : E.class, &block : A, E? -> Array(B)?) : Array(B)
      results = [] of B
      if collection
        pool = Fiberpool.new(collection, @config.download_batch_size)
        pool.run do |item|
          engine = if @turn_on_engine
                     engine_class.new(@config.domain, @config.cache_http)
                   end
          if reply = block.call(item, engine)
            results.concat(reply)
          end
        end
      end
      results
    end
  end
end

But again, this is very I/O intensive and both the Ruby and Crystal versions take advantage of the fact that they can do more work while waiting for one HTTP request to finish.

Reproducing the Tests

I implemented a “Test Mode” in all 3 implementations. You can clone from my repositories:

• Crystal version
• Ruby/JRuby version
• Elixir version

And you can run the test mode like this:

# crystal:
time ./cr_manga_downloadr --test

# MRI:
time bin/manga-downloadr --test

# JRuby (you have to edit Gemfile to uncomment the JRuby engine):
time jruby --dev -S bin/manga-downloadr --test

# Elixir:
time ./ex_manga_downloadr --test

This will run only the fetching of chapters, pages and image links, skipping the actual downloading of the images, optimization through mogrify and PDF compilation. Those skipped parts take too long and don’t say anything about the tested languages.

And if you want to test just the CPU intensive parts and avoid all networking interference altogether, you can turn on HTTP Cache mode and run the tests twice so the first run will cache everything first, like this:

# crystal:
time ./cr_manga_downloadr --test --cache

# MRI:
time bin/manga-downloadr --test --cache

# JRuby (you have to edit Gemfile to uncomment the JRuby engine):
time jruby --dev -S bin/manga-downloadr --test --cache

# Elixir:
time CACHE_HTTP=true ./ex_manga_downloadr --test

So, with all requests already cached these are the results:

Elixir:

7,13s user 0,24s system 331% cpu 2,227 total

MRI Ruby:

5.590000   0.180000   5.770000 (  5.678714)
5,87s user 0,21s system 101% cpu 5,996 total

JRuby:

10.580000   0.180000  10.760000 (  3.184472)
14,54s user 0,44s system 262% cpu 5,716 total

Crystal:

1.610000   0.050000   1.660000 (  1.344123)
1,62s user 0,06s system 124% cpu 1,350 total

The Ruby/JRuby/Crystal versions have internal benchmarks to take away startup time (this is why they have 2 lines of times).

So, Elixir is very fast. It takes roughly 2 seconds to parse all the 1,900 HTML files to find the links.

Ruby is the slowest, obviously. It takes almost 6 seconds.

The JRuby version also takes almost 6 seconds but internally it processes in 3 seconds, the rest is startup time and warming up of the JVM underneath.

And Crystal is the fastest, as you would expect because it’s a super optimized binary doing CPU bound operations, clocking in at a bit more than 1 second.

Conclusion

Even though the algorithms are roughly similar, Elixir is still winning by a very large margin in the total process (with the external HTTP requests).

There is more than just interpreter/compiler speeds, there is more than single-thread, multi-thread, fibers infrastructure.

We also have the maturity of the respective standard libraries (including TCP stack, HTTP client libraries, String/Array/Regex operations, etc) and 3rd party libraries (libXML, Nokogiri, etc). So there is a lot that can interfere with the tests. I’d guess that Crystal’s standard library, especially the networking parts, are not battle-tested enough at this point (pre 1.0!).

So, the summary with the new results is this:

• MRI Ruby 2.3.1: 27 seconds
• JRuby 9.1.1.0: 30 seconds
• Crystal 0.17.0: 21 seconds
• Elixir: 15 seconds

Let me know if you have ideas to make them even faster!

The nature of the original Manga Downloader is no different from a web crawler: it fetches HTML pages, parses them in order to find collections of links and keeps fetching until a set of images is downloaded. Then I organize them in volumes, optimize them to fit the Kindle screen resolution, and compile them down into PDF files. This makes this project an interesting exercise in trying to make concurrent HTTP requests and process the results.

A year later I was learning Elixir. The Manga Downloadr was a nice candidate for me to figure out how to implement the same thing in a different language. You can follow my learning process in this series of posts.

Finally, I’ve been learning more about Crystal, a Ruby-inspired platform that can compile Ruby-like source code into LLVM-optimized native binaries. And as a bonus it features Go-like CSP channels and fibers to allow for concurrent code.

So I adapted my Elixir version to Crystal and the result is this code you can find over Github as akitaonrails/cr_manga_downloadr.

It works very well and performs really fast, limited mainly by how many requests MangaReader can respond to concurrently and the speed/reliability of the Internet connection. So, as my original Ruby version was terrible code, it was a good time to rewrite it. And as Crystal is surprisingly close to Ruby I decided to port it over.

The port was almost too trivial

It was mostly copying and pasting the Crystal code without the Type annotations. And I had to replace the lightweight Fibers and Channel implementation for concurrency with traditional Ruby Threads.

The new version 2.0 for the Ruby version of the tool can be found in this Github repository: akitaonrails/manga-downloadr.

Ruby, Ruby everywhere!

Moving between Ruby and Crystal is not so difficult. The Crystal team did a fantastic job implementing a very solid Standard Library (stdlib) that very closely resembles MRI Ruby’s.

For example, let’s compare a snippet from my Crystal version first:

def fetch(page_link : String)
  get page_link do |html|
    images = html.xpath("//img[contains(@id, 'img')]").as(XML::NodeSet)
    image_alt = images[0]["alt"]
    image_src = images[0]["src"]
    if image_alt && image_src
      extension      = image_src.split(".").last
      list           = image_alt.split(" ").reverse
      title_name     = list[4..-1].join(" ")
      chapter_number = list[3].rjust(5, '0')
      page_number    = list[0].rjust(5, '0')
      uri = URI.parse(image_src)
      CrMangaDownloadr::Image.new(uri.host as String, uri.path as String, "#{title_name}-Chap-#{chapter_number}-Pg-#{page_number}.#{extension}")
    else
      raise Exception.new("Couldn't find proper metadata alt in the image tag")
    end
  end
end

Now let’s check the ported Ruby version:

def fetch(page_link)
  get page_link do |html|
    images = html.css('#img')

    image_alt = images[0]["alt"]
    image_src = images[0]["src"]

    if image_alt && image_src
      extension      = image_src.split(".").last
      list           = image_alt.split(" ").reverse
      title_name     = list[4..-1].join(" ")
      chapter_number = list[3].rjust(5, '0')
      page_number    = list[0].rjust(5, '0')

      uri = URI.parse(image_src)
      Image.new(uri.host, uri.path, "#{title_name}-Chap-#{chapter_number}-Pg-#{page_number}.#{extension}")
    else
      raise Exception.new("Couldn't find proper metadata alt in the image tag")
    end
  end
end

It’s uncanny how similar they are, down to stdlib calls such as URI.parse or Array methods such as split.

Once you remove the Type annotations from the Crystal version, it’s 99% Ruby.

Ruby doesn’t care if you’re trying to call a method on a Nil object - at source code compiling time. Crystal does compile-time checks and if it feels like the method call will be over Nil, it won’t even compile. So this is one big win to avoid subtle bugs.

In Rails we are used to the #try method. Ruby 2.3 introduced the safe navigation operator.

So, in Ruby 2.3 with Rails, both of the following lines are valid:

obj.try(:something).try(:something2)
obj&.something&.something2

In Crystal we can do the following:

obj.try(&.something).try(&.something2)

So, it’s close. Use with care.

As I mentioned before, Crystal is close to Ruby but it’s not meant to be compatible, so we can’t just load Rubygems without porting. In this example I don’t have Nokogiri to parse the HTML. But this is where the stdlib shines: Crystal comes with good enough XML/HTML and JSON parsers. So we can parse HTML as XML and use plain XPath instead.

Instead of Ruby’s Net::HTTP we have HTTP::Client (but their methods and semantics are surprisingly similar).

There are other differences, for example, this is the main file that requires all the others in Ruby:

...
$LOAD_PATH.unshift File.join(File.dirname(__FILE__), "lib")

require "manga-downloadr/records.rb"
require "manga-downloadr/downloadr_client.rb"
require "manga-downloadr/concurrency.rb"
require "manga-downloadr/chapters.rb"
require "manga-downloadr/pages.rb"
require "manga-downloadr/page_image.rb"
require "manga-downloadr/image_downloader.rb"
require "manga-downloadr/workflow.rb"

And this is the Crystal version of the same manifest:

require "./cr_manga_downloadr/*"
...

On the other hand, we need to be a bit more explicit in each Crystal source code file, and declare the specific dependencies where needed. For example, in the pages.cr file it starts like this:

require "./downloadr_client"
require "xml"

module CrMangaDownloadr
  class Pages < DownloadrClient(Array(String))
  ...

Crystal has less room for “magic” but it’s able to maintain a high level of abstraction anyway.

A Word about Types

We can spend the next decade masturbating over all there is about Types, and it will be extremely boring.

The only thing you must understand: the compiler must know the method signature of classes before you can use them. There is no Runtime component that can introspect objects on the fly, like in Ruby and other dynamic languages (even Objective-C/Swift can do more dynamic stuff than Crystal).

Most of the time the Crystal compiler will be smart enough to infer the types for you, so you don’t need to be absolutely explicit. You should follow the compiler’s lead to know when to use Type Annotations.

What may really scare you at first is the need for Type Annotations, to understand Generics and so forth. The compiler will print out every scary error dump that you will need to get used to. Most scary errors will usually be a Type Annotation missing or you trying to call a method on a possibly Nil object.

For example, if I change the following line in the page_image_spec.cr test file:

# line 8:
image = CrMangaDownloadr::PageImage.new("www.mangareader.net").fetch("/naruto/662/2")

# line 10:
# image.try(&.host).should eq("i8.mangareader.net")
image.host.should eq("i8.mangareader.net")

The commented out line recognizes that the image instance might come as Nil, so we add an explicit #try call in the spec.

If we try to compile without this recognition, this is the error the compiler will dump on you:

$ crystal spec                                                        [
Error in ./spec/cr_manga_downloadr/page_image_spec.cr:10: undefined method 'host' for Nil (compile-time type is CrMangaDownloadr::Image?)

    image.host.should eq("i8.mangareader.net")
          ^~~~

=============================================================================

Nil trace:

  ./spec/cr_manga_downloadr/page_image_spec.cr:8

        image = CrMangaDownloadr::PageImage.new("www.mangareader.net").fetch("/naruto/662/2")
...

There is a large stack trace dump after that snippet above, but you only need to pay attention to the first few lines that already say what’s wrong: “undefined method ‘host’ for Nil (compile-time type is CrMangaDownloadr::Image?)”. If you know how to read, you shouldn’t have any problems most of the time.

Now, the Chapters, Pages, PageImage (all subclasses of DownloadrClient) are basically the same thing: they do HTTP::Client requests.

This is how the Pages class is implemented:

...
module CrMangaDownloadr
  class Pages < DownloadrClient(Array(String))
    def fetch(chapter_link : String)
      get chapter_link do |html|
        nodes = html.xpath_nodes("//div[@id='selectpage']//select[@id='pageMenu']//option")
        nodes.map { |node| [chapter_link, node.text as String].join("/") }
      end
    end
  end
end

#get is a method from the DownloadrClient superclass that receives a String chapter_link and a block. The block receives a parsed html collection of nodes and we can play with it, returning an Array of Strings.

That’s why we have the (Array(String)) when inheriting from DownloadrClient. Let’s see how the DownloadrClient superclass is implemented.

module CrMangaDownloadr
  class DownloadrClient(T)
    ...
    def get(uri : String, &block : XML::Node -> T)
      response = @http_client.get(uri)
      case response.status_code
      when 301
        get response.headers["Location"], &block
      when 200
        parsed = XML.parse_html(response.body)
        block.call(parsed)
      end
      ...
    end
  end
end

You can see that this class receives a Generic Type and it uses it as the return type for the yielded block in the #get method. The XML::Node -> T is the declaration of the signature for the block, sending XML::Node and receiving whatever the type T is. At compile time, imagine this T being replaced by Array(String). That’s how you can create classes that can deal with any number of different Types without having to overload for polymorphism.

If you come from Java, C#, Go or any other modern statically typed language, you probably already know what a Generic is.

You can go very far with Generics, check out how our Concurrency.cr begins:

class Concurrency(A, B, C)
  ...
  def fetch(collection : Array(A)?, &block : A, C? -> Array(B)?) : Array(B)?
    results = [] of B
    collection.try &.each_slice(@config.download_batch_size) do |batch|
      engine  = if @turn_on_engine
                  C.new(@config.domain)
                end
      channel = Channel(Array(B)?).new
      batch.each do |item|
      ...

And this is how we use it in the workflow.cr:

private def fetch_pages(chapters : Array(String)?)
  puts "Fetching pages from all chapters ..."
  reactor = Concurrency(String, String, Pages).new(@config)
  reactor.fetch(chapters) do |link, engine|
    engine.try( &.fetch(link) )
  end
end

In this example, imagine A being replaced by String, B also being replaced by String and C being replaced by Pages in the Concurrency class.

This is the “first-version-that-worked” so it’s probably not very idiomatic. Either this could be solved with less Generics exercising or maybe I could simplify it with the use of Macros. But it’s working quite ok as it is.

The pure Ruby version ends up just like this:

class Concurrency
  def initialize(engine_klass = nil, config = Config.new, turn_on_engine = true)
    ...
  end
  def fetch(collection, &block)
    results = []
    collection&.each_slice(@config.download_batch_size) do |batch|
      mutex   = Mutex.new
      threads = batch.map do |item|
      ...

This version is much “simpler” in terms of source code density. But on the other hand we would have to test this Ruby version a lot more because it has many different permutations (we even inject classes through engine_klass) that we must make sure respond correctly. In practice we should add tests for all combinations of the initializer’s arguments.

In the Crystal version, because all types have been checked at compile-time, it was more demanding in terms of annotations; but we can be pretty sure that if it compiles, it will run as expected.

I’m not saying that Crystal doesn’t need any specs.

Compilers can only go so far. But how much is “so far”? Whenever you’re “forced” to add Type Annotations, I will state that those parts are either trying to be too smart or they are intrinsically complex. Those are parts that would require extra levels of tests in Ruby and, if we can add the annotations properly, we can have fewer tests (we don’t need to cover most permutations) in the Crystal version (exponential complexity of permutations could go down to a linear complexity, I think).

A Word about Ruby Threads

The main concepts you must understand about concurrency in Ruby are these:

• MRI Ruby has a GIL, a Global Interpreter Lock, that forbids code from running concurrently.
• MRI Ruby does have access and exposes native Threads since version 1.9. But even if you fire up multiple Threads, they will run sequentially because only one thread can hold the Lock at a time.
• I/O operations are the exception: they do release the Lock for other threads to run while the operation is waiting to complete. The OS will signal the program through OS-level poll.

This means that if your app is I/O intensive (HTTP requests, File reads or writes, socket operations, etc), you will have some concurrency. A web server, such as Puma for example, can take some advantage of Threads because a big part of the operations involves receiving HTTP requests and sending HTTP responses over the wire, which would make the Ruby process idle while waiting.

If your app is CPU intensive (heavy algorithms, data crunching, stuff that really makes the CPU hot) then you can’t take advantage of native Threads, only one will run at a time. If you have multiple cores in your CPU, you can Fork your process to the number of cores available.

You should check out grosser/parallel to make it easy.

This is why Puma also has a “worker” mode. “Worker” is the name we usually give to forked children of processes.

In the case of this downloader process, it will perform thousands of HTTP requests to scrape the needed metadata from the MangaReader pages. So it’s definitely much more I/O intensive than CPU intensive (the CPU intensive parts are the HTML parsing and later the image resizing and PDF compiling).

A sequential version of what has to be done, in Ruby, looks like this:

def fetch_sequential(collection, &block)
  results = []
  engine  = @turn_on_engine ? @engine_klass.new(@config.domain) : nil
  collection&.each_slice(@config.download_batch_size) do |batch|
    batch.each do |item|
      batch_results = block.call(item, engine)&.flatten
      results += ( batch_results || [])
    end
  end
  results
end

If we have 10,000 links in the collection, we first slice it to what @config.download_batch_size defines and we iterate over those smaller slices, calling some block and accumulating the results. This is a naive algorithm, as you will find out in the next section, but bear with me.

In Elixir you can fire up micro-processes to make the HTTP requests in parallel. In Crystal you can fire up Fibers and wait for the HTTP requests to complete and signal the results through Channels.

Both are lightweight ways and you can have hundreds or even thousands running in parallel. Manga Reader will probably complain if you try that many at once, so the limit is not in the code, but in the external service.

To transform the sequential version into a concurrent one, this is what we can do in Crystal:

def fetch(collection : Array(A)?, &block : A, C? -> Array(B)?) : Array(B)?
  results = [] of B
  collection.try &.each_slice(@config.download_batch_size) do |batch|
    channel = Channel(Array(B)?).new
    batch.each do |item|
      spawn {
        engine  = if @turn_on_engine
                    C.new(@config.domain)
                  end
        reply = block.call(item, engine)
        channel.send(reply)
        engine.try &.close
      }
    end
    batch.size.times do
      reply = channel.receive
      if reply
        results.concat(reply.flatten)
      end
    end
    channel.close
    puts "Processed so far: #{results.try &.size}"
  end
  results
end

Fetching a huge collection and slicing it in smaller ‘batches’ is easy. Now we have a smaller batch collection. For each item (usually a URI) we spawn a Fiber and call a block that will request and process the results. Once it’s finished processing it sends the results over a channel.

Once we finish iterating over the batch and spawning that many Fibers we can “wait” for them by doing channel.receive, which will start receiving results as soon as the Fibers finish requesting/processing each URI.

We accumulate the results and go over the next batch of the collection until finished. The amount of concurrency is determined by the size of the batch (it’s like what I did with ‘poolboy’ over Elixir where we start a fixed number of processes to run in parallel and avoid doing a Denial of Service to Manga Reader).

By the way, this Crystal implementation is similar to what you would do if you were in Go, using Channels.

In the Ruby version you can fire up native Threads - which has a lot of overhead to spawn! - and assume the HTTP requests will run almost all in parallel. Because it’s I/O intensive, you can have them all in parallel. This is what it looks like:

def fetch(collection, &block)
  results = []
  collection&.each_slice(@config.download_batch_size) do |batch|
    mutex   = Mutex.new
    threads = batch.map do |item|
      Thread.new {
        engine  = @turn_on_engine ? @engine_klass.new(@config.domain) : nil
        Thread.current["results"] = block.call(item, engine)&.flatten
        mutex.synchronize do
          results += ( Thread.current["results"] || [] )
        end
      }
    end
    threads.each(&:join)
    puts "Processed so far: #{results&.size}"
  end
  results
end

Threads are all initialized in a “paused” state. Once we instantiate those many Threads, we can #join on each of them and wait for them all to finish.

Once each Thread finishes the same URI request/process, the results must be accumulated in a global storage, in this case a simple array called results. But because we might have the chance of 2 or more threads trying to update the same array, we might as well synchronize the access (I’m not sure if Ruby’s Array access is already synchronized, but I guess not). To synchronize access we use a Mutex, which is a fine-grained lock, to make sure only 1 thread can modify the global array at once.

To prove that Ruby can support concurrent I/O operations, I have added 2 methods to the Concurrent class, the first is just #fetch and it’s the Thread implementation above. The second is called #fetch_sequential and it is the sequential version also shown at the beginning of this section. And I added the following spec:

it "should check that the fetch implementation runs in less time than the sequential version" do
  reactor = MangaDownloadr::Concurrency.new(MangaDownloadr::Pages, config, true)
  collection = ["/onepunch-man/96"] * 10
  WebMock.allow_net_connect!
  begin
    concurrent_measurement = Benchmark.measure {
      results = reactor.fetch(collection) { |link, engine| engine&.fetch(link) }
    }
    sequential_measurement = Benchmark.measure {
      results = reactor.send(:fetch_sequential, collection) { |link, engine| engine&.fetch(link) }
    }
    /\((.*?)\)$/.match(concurrent_measurement.to_s) do |cm|
      /\((.*?)\)/.match(sequential_measurement.to_s) do |sm|
        # expected for the concurrent version to be close to 10 times faster than sequential
        expect(sm[1].to_f).to be > ( cm[1].to_f * 9 )
      end
    end
  ensure
    WebMock.disable_net_connect!
  end
end

Because it uses WebMock, I first disable it during this spec. I create a fake collection of 10 real links to MangaReader. And then I benchmark the Thread-based concurrent version and the plain sequential version. Because we have 10 links and they are all the same you can assume that the sequential version will be almost 10 times slower than the Thread-based version. And this is exactly what this spec compares and proves (the spec fails if the concurrent version is not at least 9x faster).

To compare all versions of the Manga Downloadrs I let it download and compile an entire manga collection, in this case “One-Punch Man”, which has almost 1,900 pages/images. I am just measuring the fetching and scraping processes and skipping the actual image downloading, image resizing and PDF generation as they take the majority of the time and the resizing and PDF part are all done by ImageMagick’s mogrify and convert tools.

This is how long this new Ruby version takes to fetch and scrape almost 1,900 pages (using MRI Ruby 2.3.1):

12,42s user 1,33s system 23% cpu 57,675 total

This is how long the Crystal version takes:

4,03s user 0,40s system 7% cpu 59,207 total

Just for fun I tried to run the Ruby version under JRuby 9.1.1.0. To run with JRuby just add the following line in the Gemfile:

ruby "2.3.0", :engine => 'jruby', :engine_version => '9.1.1.0'

Bundle install, run normally, and this is the result:

47,80s user 1,99s system 108% cpu 45,967 total

And this is how long the Elixir version takes:

11,38s user 1,04s system 85% cpu 14,590 total

Reality Check!

If you just look at the times above you might come to the wrong conclusions.

First of all, it’s an unfair comparison. The Elixir version uses a very different algorithm than the Ruby and Crystal versions.

In Elixir I boot up a pool of processes, around 50 of them. Then I start 50 HTTP requests at once. Once each process finishes, it releases itself back to the pool and I can fire up another HTTP request from the queue of links. So it’s a constant stream of at most 50 HTTP requests, constantly.

The Crystal and Ruby/JRuby versions slice the 1,900 links into batches of 40 links and then I fire up 40 requests at once. This implementation waits for all 40 to finish and they fire up another 40. So it’s never a constant stream, it’s bursts of 40 requests. So each batch is slowed down by the slowest request in the batch, not giving a chance for other requests to go through.

It’s a difference in architecture. Elixir makes it much easier to do streams and Crystal, with its CSP style, makes it easier to do bursts. A better approach would be to queue up the 1,900 links and use something like Sidekiq.cr to go over one link at a time (spawning 40 fibers to serve as a “pool”, for example).

The Elixir version has a more efficient architecture, which is why it takes no more than 15 seconds to fetch all the image links while the Crystal version takes almost a full minute to finish (the accumulation of the slowest requests in each batch).

Now, you will be surprised that the Crystal version is actually a bit slower than the Ruby version! And you won’t be too surprised to see JRuby being faster at 45 seconds!

This is more evidence that you should not dismiss Ruby (and that you should try JRuby more often). As I explained before, it does support concurrency in I/O operations and the applications I tested are all I/O heavy.

The difference is probably in the maturity of Ruby’s Net::HTTP library against Crystal’s HTTP::Client. I tried many tweaks in the Crystal version but I couldn’t get much faster. I tried to do larger batches, but for some reason the application just hangs, pauses, and never releases. I have to Ctrl-C out of it and retry until it finally goes through. If someone knows what I am doing wrong, please don’t forget to write in the comments section below.

Some of this is probably due to MangaReader’s unreliable servers, they probably have some kind of DoS protection, throttling connections or something like this.

Anyway, when they go through, because the Ruby and Crystal algorithms are essentially the same, they take roughly the same time to complete. So what’s missing is for me to evolve this algorithm to either use something like Sidekiq or implement an internal queue/pool of workers scheme.

Conclusion

The goal of this experiment was to learn more of Crystal’s ability and how easy it would be to go back and forth with Ruby.

As you can see, there are many differences, but it’s not so difficult. I may be missing something, but I stumbled upon some difficulties when pushing HTTP::Client + Fibers too hard, as I explained above. If you know what I am doing wrong, please let me know.

The difference between the Elixir vs Ruby/Crystal algorithms shows not just language performance differences, but also the importance of architecture and algorithms in the overall performance. This test was not conclusive, just a hint that my naive Fibers implementation needs some more work, and that Elixir’s natural handling of parallel processes makes it easier to achieve higher levels of parallelism.

This serves as a taste of what Crystal can do, and also that Ruby is still in the game. But also that Elixir surely is something to look at closely.

git pull
mix escript.build
./ex_manga_downloadr -n onepunch -u http://mangafox.me/manga/onepunch_man/ -d /tmp/onepunch -s mangafox

And there you go: download from Mangafox built-in! \o/

It starts when I wanted to download manga that is not available at MangaReader but exists under Mangafox.

So, the initial endeavor was to copy the MangaReader parser modules (IndexPage, ChapterPage, and Page) and paste them at a specific “lib/ex_manga_downloadr/mangafox” folder. Same thing done to the unit tests folder. Just a matter of copying and pasting the files and change the “MangaReader” module name to “Mangafox”.

Of course the URL formats are different, the Floki CSS selectors are a bit different, so that’s what had to change in the parser. For example, this is how I parse the chapter links from the main page at MangaReader:

defp fetch_chapters(html) do
  Floki.find(html, "#listing a")
  |> Enum.map fn {"a", [{"href", url}], _} -> url end
end

And this is the same thing but for Mangafox:

defp fetch_chapters(html) do
  html
  |> Floki.find(".chlist a[class='tips']")
  |> Enum.map fn {"a", [{"href", url}, {"title", _}, {"class", "tips"}], _} -> url end
end

Exactly the same logic but the pattern matching structure is different because the returning HTML DOM nodes are different.

Another difference is that MangaReader returns everything in plain text by default, but Mangafox returns everything Gzipped regardless of whether I send the “Accept-Encoding” HTTP header (curiously, if I retry several times it changes behavior and sometimes sends plain text).

What I did differently was to check if the returned %HTTPotion.Response{} structure had a “Content-Encoding” header set to “gzip” and if so, gunzip it using the built-in Erlang “zlib” package (nothing to import!):

def gunzip(body, headers) do
  if headers[:"Content-Encoding"] == "gzip" do
    :zlib.gunzip(body)
  else
    body
  end
end

I would’ve preferred if HTTPotion did that out of the box for me (#OpportunityToContribute!), but this was easy enough.

Once the unit tests were passing correctly after tuning the scrapper (HTTPotion requests) and parser (Floki selectors) it was time to make my Worker aware of the existence of this new set of modules.

The Workflow module just calls the Worker, which in turn does the heavy lifting of fetching pages and downloading images. The Worker called the MangaReader module directly, like this:

defmodule PoolManagement.Worker do
  use GenServer
  use ExMangaDownloadr.MangaReader
  require Logger
  ...
  def chapter_page(chapter_link) do
    Task.async fn -> 
      :poolboy.transaction :worker_pool, fn(server) ->
        GenServer.call(server, {:chapter_page, chapter_link}, @timeout_ms)
      end, @transaction_timeout_ms
    end
  end
  ...
  def handle_call({:chapter_page, chapter_link}, _from, state) do
    {:reply, ChapterPages.pages(chapter_link), state}
  end
  ...
end

That “use ExMangaDownloadr.MangaReader” statement up above is just a macro that will alias the corresponding modules:

defmodule ExMangaDownloadr.MangaReader do
  defmacro __using__(_opts) do
    quote do
      alias ExMangaDownloadr.MangaReader.IndexPage
      alias ExMangaDownloadr.MangaReader.ChapterPage
      alias ExMangaDownloadr.MangaReader.Page
    end
  end
end

So when I call “ChapterPages.pages(chapter_link)” it’s a shortcut to use the fully qualified module name like this: “ExMangaDownloadr.MangaReader.ChapterPages.pages(chapter_link)”.

An Elixir module namespace is just an Atom. Nested module names have the full, dot-separated name, prefixed with its parent. For example:

defmodule Foo do
  defmodule Bar do
    defmodule Xyz do
       def teste do
       end
    end
  end
end

You can just call “Foo.Bar.Xyz.teste()” and that’s it. But there is a small trick. Elixir also transparently prefixes the full module name with “Elixir”. So in reality, the full module name is “Elixir.Foo.Bar.Xyz”, in order to make sure no Elixir module ever conflicts with an existing Erlang module.

This is important because of this new function I added to the Worker module first:

def manga_source(source, module) do
  case source do
    "mangareader" -> String.to_atom("Elixir.ExMangaDownloadr.MangaReader.#{module}")
    "mangafox"    -> String.to_atom("Elixir.ExMangaDownloadr.Mangafox.#{module}")
  end
end

This is how I map from “mangafox” to the new “ExMangaDownloadr.Mangafox.” namespace. And because of the dynamic, message passing nature of Elixir, I can replace this code:

def handle_call({:chapter_page, chapter_link}, _from, state) do
  {:reply, ChapterPages.pages(chapter_link), state}
end

With this:

def handle_call({:chapter_page, chapter_link, source}, _from, state) do
  links = source
    |> manga_source("ChapterPage")
    |> apply(:pages, [chapter_link])
  {:reply, links, state}
end

I can now choose between the “Elixir.ExMangaDownloadr.Mangafox.ChapterPage” or “Elixir.ExMangaDownloadr.MangaReader.ChapterPage” modules, call the pages/1 function and send the same argument as before. I just have to make sure I can receive a “source” string from the command line now, so I change the CLI module like this:

defp parse_args(args) do
  parse = OptionParser.parse(args,
    switches: [name: :string, url: :string, directory: :string, source: :string],
    aliases: [n: :name, u: :url, d: :directory, s: :source]
  )
  case parse do
    {[name: manga_name, url: url, directory: directory, source: source], _, _} -> process(manga_name, url, directory, source)
    {[name: manga_name, directory: directory], _, _} -> process(manga_name, directory)
    {_, _, _ } -> process(:help)
  end
end

Compared to the previous version I just added the “:source” string argument to the OptionParser and passed the captured value to process/4. I should add some validation here to avoid strings different than “mangareader” or “mangafox”, but I will leave that to another time.

And in the Workflow module, instead of starting from just the manga URL, now I have to start with both the URL and the manga source:

[url, source]
  |> Workflow.chapters
  |> Workflow.pages
  |> Workflow.images_sources

Which means that each of the above functions has to not only return the new URL lists but also pass through the source:

def chapters([url, source]) do
  {:ok, _manga_title, chapter_list} = source
    |> Worker.manga_source("IndexPage")
    |> apply(:chapters, [url])
  [chapter_list, source]
end

This was the only function in the Workflow module hardcoded to MangaReader so I also made it dynamic using the same manga_source/2 function from the Worker, and notice the return value being “[chapter_list, source]” instead of just “chapter_list”.

And now, I can finally test with “mix test” and create the new executable command line binary with “mix escript.build” and run the new version like this:

./ex_manga_downloadr -n onepunch -u http://mangafox.me/manga/onepunch_man/ -d /tmp/onepunch -s mangafox

The Mangafox site is very unreliable with several concurrent connections and it quickly times out sometimes, dumping ugly errors like this:

15:58:46.637 [error] Task #PID<0.2367.0> started from #PID<0.124.0> terminating
** (stop) exited in: GenServer.call(#PID<0.90.0>, {:page_download_image, {"http://z.mfcdn.net/store/manga/11362/TBD-053.2/compressed/h006.jpg", "Onepunch-Man 53.2: 53rd Punch [Fighting Spirit] (2) at MangaFox.me-h006.jpg"}, "/tmp/onepunch"}, 1000000)
    ** (EXIT) an exception was raised:
        ** (HTTPotion.HTTPError) connection_closing
            (httpotion) lib/httpotion.ex:209: HTTPotion.handle_response/1

I haven’t figured out how to retry HTTPotion requests properly yet. But one small thing I did was add an availability check in the Worker module. So you can just re-run the same command line and it will resume downloading only the remaining files:

defp download_image({image_src, image_filename}, directory) do
  filename = "#{directory}/#{image_filename}"
  if File.exists?(filename) do
    Logger.debug("Image #{filename} already downloaded, skipping.")
    {:ok, image_src, filename}
  else
    Logger.debug("Downloading image #{image_src} to #{filename}")
    case HTTPotion.get(image_src,
      [headers: ["User-Agent": @user_agent], timeout: @http_timeout]) do
      %HTTPotion.Response{ body: body, headers: _headers, status_code: 200 } ->
        File.write!(filename, body)
        {:ok, image_src, filename}
      _ ->
        {:err, image_src}
    end
  end
end

This should at least reduce rework. Another thing I am still working on is this other bit at the main “CLI.process” function:

defp process(manga_name, url, directory, source) do
  File.mkdir_p!(directory)
  dump_file = "#{directory}/images_list.dump"
  images_list = if File.exists?(dump_file) do
                  :erlang.binary_to_term(File.read(dump_file))
                else
                  list = [url, source]
                    |> Workflow.chapters
                    |> Workflow.pages
                    |> Workflow.images_sources
                  File.write(dump_file, :erlang.term_to_binary(list))
                  list
                end
  images_list
    |> Workflow.process_downloads(directory)
    |> Workflow.optimize_images
    |> Workflow.compile_pdfs(manga_name)
    |> finish_process
end

As you can see the idea is to serialize the final images URL links to a file using the built-in serializer “:erlang.binary_to_term/1” and check if that dump file exists, and deserialize with “:erlang.term_to_binary/1” before fetching all pages all over again. Now the process can resume directly from the process_downloads/2 function after.

Mangafox is terribly unreliable and I will need to figure out a better way to retry timed out connections without having to crash and manually restart from the command line. It’s either a bad site or a clever one that shuts down scrapers like me, although I am guessing it’s just bad infrastructure on their side.

If I downgrade from 50 processes to 5 in the pool, it seems to be able to handle it better (but the process slows down, of course):

    pool_options = [
      name: {:local, :worker_pool},
      worker_module: PoolManagement.Worker,
      size: 5,
      max_overflow: 0
    ]

If you see time out errors, change this parameter. MangaReader still supports 50 or more for concurrency.

And now you know how to add support for more manga sources. Feel free to send me a Pull Request! :-)

As an exercise (and also because I’m obviously an Otaku) I implemented a simple Elixir-based scraper for the great MangaReader website. One can argue whether it’s OK to scrape their website, and one might also argue whether their providing those mangas is OK in the first place, so let’s not go down this path.

I had an older version written in Ruby. It still works but it’s in sore need of a good refactoring (sorry about that). The purpose of that version was to see if I could actually do parallel fetching and retry connections using Typhoeus.

As I’ve been progressing in my studies of Elixir, that tool felt like a great candidate to test my current knowledge of the platform. It would make me test:

1. Fetch and parse ad hoc content through HTTP (HTTPotion and Floki).
2. Test parallel/asynchronous downloads (Elixir’s built-in Task module).
3. The built-in command line and option parser support.
4. Basic testing through ExUnit and mocking the workflow (Mock).

The exercise was very interesting, and a scraper is also an ideal candidate for TDD. The initial steps had to go like this:

1. Parse the main manga page to fetch all the chapter links.
2. Parse each chapter page to fetch all the individual pages.
3. Parse each page to parse the main embedded image (the actual manga page).

For each of those initial steps I did a simple unit test and the IndexPage, ChapterPage and Page modules. They have roughly the same structure, this is one example:

defmodule ExMangaDownloadr.MangaReader.IndexPage do
  def chapters(manga_root_url) do
    case HTTPotion.get(manga_root_url, [timeout: 30_000]) do
      %HTTPotion.Response{ body: body, headers: _headers, status_code: 200 } ->
        {:ok, fetch_manga_title(body), fetch_chapters(body) }
      _ ->
        {:err, "not found"}
    end
  end

  defp fetch_manga_title(html) do
    Floki.find(html, "#mangaproperties h1")
    |> Enum.map(fn {"h1", [], [title]} -> title end)
    |> Enum.at(0)
  end

  defp fetch_chapters(html) do
    Floki.find(html, "#listing a")
    |> Enum.map fn {"a", [{"href", url}], _} -> url end
  end
end

Here I am already exercising some of Elixir’s awesome features such as pattern matching the result of the HTTPotion.get/2 function to extract the body from the returning record.

Then I pass the HTML body to 2 different functions: fetch_manga_title and fetch_chapters. They both use the Floki package which can use CSS selectors to return a List. Then I need to walk through the list (using Enum.map/2 for example) and pattern match on it to extract the values I need.

Pattern Matching is one of the most important concepts to learn about Elixir/Erlang. It’s different from simply assigning a value to a variable, it can be used to destructure a value into its components and get their individual parts.

Then I just went through building the skeleton for the command line interface. This is already explained in other tutorials such as this and this, so I won’t waste time explaining it again. At the core I needed to have the following workflow:

1. Starting from the Manga main URL at MangaReader, extract the chapters
2. Then loop through the chapters and fetch all the pages
3. Then loop through the pages and fetch all the image sources
4. Then loop through the images and download them all to a temporary directory
5. Then sort through files in the directory and move them to sub-directories of 250 images each (I think it is a good size for each volume)
6. Finally, resize and convert all the images of each sub-directory into a PDF file for my Kindle to consume.

This workflow is defined like this:

defp process(manga_name, url, directory) do
  File.mkdir_p!(directory)

  url
    |> Workflow.chapters
    |> Workflow.pages
    |> Workflow.images_sources
    |> Workflow.process_downloads(directory)
    |> Workflow.optimize_images
    |> Workflow.compile_pdfs(manga_name)
    |> finish_process
end

This is one place where the pipeline notation from Elixir really shines. It’s much better than having to write this equivalent:

Workflow.compile_pdfs(Workflow.optimize_images(directory))

This notation is just syntactic sugar where the returning value of the previous statement is used as the first argument of the following function. Combine that with other syntactic sugars such as parentheses being optional (just like beloved Ruby) and you have a clear exposure of “transforming from a URL into compiled PDFs”.

I separated the Workflow into its own module and each step is very similar, each taking a list and walking through it. This is the simplest of them:

def pages(chapter_list) do
  chapter_list
    |> Enum.map(&(Task.async(fn -> ChapterPage.pages(&1) end)))
    |> Enum.map(&(Task.await(&1, @http_timeout)))
    |> Enum.reduce([], fn {:ok, list}, acc -> acc ++ list end)
end

If you’re new to Elixir here you will find another oddity, this "&(x(&1))", this is just a shortcut macro to this other similar statement:

Enum.map(fn (list) ->
  Task.async(fn ->
    ChapterPage.pages(list)
  end)
end)

Enum is one of the most useful modules you need to master. If you come from Ruby it feels like home, you must learn all of its functions. You’re usually having to transform one collection into another, so it’s important to study it thoroughly.

A few problems understanding parallel HTTP processing (W.I.P.)

Then there is this Task.async/await deal. If you’re from a language that has Threads, it’s quite similar: you start several different Threads and await all of them returning before continuing. But a Task in Elixir is not a real thread, it’s a “green thread” or, in Erlang lingo, a very lightweight “process”. Erlang uses processes for everything, and so does Elixir. Under the hood, the “Task” module encapsulates the entire OTP framework for supervisors/workers. But instead of having to deal right now with OTP GenServer I decided to go the simpler route for now, and the “Task” module accomplishes just that.

Then I ended up with a problem. If I just kept going like this, spawning hundreds of async HTTP calls, I quickly ended up with this exception:

17:10:55.882 [error] Task #PID<0.2217.0> started from #PID<0.69.0> terminating
** (HTTPotion.HTTPError) req_timedout
    (httpotion) lib/httpotion.ex:209: HTTPotion.handle_response/1
    (ex_manga_downloadr) lib/ex_manga_downloadr/mangareader/page.ex:6: ExMangaDownloadr.MangaReader.Page.image/1
    (elixir) lib/task/supervised.ex:74: Task.Supervised.do_apply/2
    (elixir) lib/task/supervised.ex:19: Task.Supervised.async/3
    (stdlib) proc_lib.erl:240: :proc_lib.init_p_do_apply/3
Function: #Function<12.106612505/0 in ExMangaDownloadr.Workflow.images_sources/1>
    Args: []

That’s why there is @maximum_fetches 80 at the top of the Workflow module, together with this other odd construction:

def images_sources(pages_list) do
  pages_list
    |> chunk(@maximum_fetches)
    |> Enum.reduce([], fn pages_chunk, acc ->
      result = pages_chunk
        |> Enum.map(&(Task.async(fn -> Page.image(&1) end)))
        |> Enum.map(&(Task.await(&1, @http_timeout)))
        |> Enum.map(fn {:ok, image} -> image end)
      acc ++ result
    end)
end

This gets a huge list (such as all the pages of a very long manga like Naruto), breaks it down into smaller lists of 80 elements and then proceeds to fire up the asynchronous Tasks, reducing the results back to a plain List. The chunk/2 private function just gets the smaller size between the list length and the maximum fetches value.

Sometimes it breaks down if the maximum is larger, sometimes it doesn’t, so my guess is that my code isn’t dealing with network instabilities (with some retry logic) or that the MangaReader site is queueing up above my designated timeout (which I set to 30 seconds). Either way, keeping the maximum value lower than 100 seems to be a good balance without crashing the workflow.

This is one part where I’m not entirely sure what to do to deal with uncertainties like the external website not responding or the network going down for a little while. HTTPotion has support for asynchronous calls, but I don’t know what the difference is between using that and just making synchronous calls within parallel processes with Task, the way I’m doing. And in either case, they are supervised workers, how do I handle the exceptions, how should I implement logic to retry the call once it fails? If anyone has more knowledge about this, a comment below will be really appreciated.

Finally, there is one dirty trick behind why I like to use MangaReader: it’s very friendly to scrapers because on each page of the manga the image is annotated with an “alt” attribute with the format “[manga name] [chapter number] - [page number]”. So I just had to reformat it a bit, adding a pad of zeroes before the chapter and page number so a simple sort of the downloaded files gives me the correct order. MangaFox is not so friendly. This is how to reformat it:

defp normalize_metadata(image_src, image_alt) do
  extension      = String.split(image_src, ".") |> Enum.at(-1)
  list           = String.split(image_alt)      |> Enum.reverse
  title_name     = Enum.slice(list, 4, Enum.count(list) - 1) |> Enum.join(" ")
  chapter_number = Enum.at(list, 3) |> String.rjust(5, ?0)
  page_number    = Enum.at(list, 0) |> String.rjust(5, ?0)
  {image_src, "#{title_name} #{chapter_number} - Page #{page_number}.#{extension}"}
end

Once I have all the images, I spawn another external process using Porcelain to deal with shelling out to run ImageMagick’s Mogrify and Convert tools to resize all the images down to 600x800 pixels (Kindle Voyage resolution) and pack them together into a PDF file. This results in PDF files with 250 pages and around 20Mb in size. Now it is just a matter of copying the files to my Kindle through USB.

The ImageMagick code is quite boring, I just generate the commands in the following format for Mogrify:

"mogrify -resize #{@image_dimensions} #{directory}/*.jpg"

And compile the PDFs with this other command:

"convert #{volume_directory}/*.jpg #{volume_file}"

(By the way, notice the Ruby-like String interpolation we’re used to.)

Technically I could copy the MangaReader module files into a new MangaFox module and repurpose the same Workflow logic once I tweak the parsers to deal with MangaFox page format. But I leave that as an exercise to the reader.

The MangaReader module tests do real calls to their website. I left it that way on purpose because if the test fails it means that they changed the website format and the parser needs tweaking to conform. But after a few years, I never saw them change enough to break my old Ruby parser.

Just as a final exercise I imported the Mock package, to control what some inner pieces of the Workflow implementation return. It’s called Mock but it’s more like stubbing particular functions of a module. I can declare a block where I override the File.rename/1 so it doesn’t actually try to move a file that doesn’t exist in the test environment. This makes the test more brittle because it depends on a particular implementation, which is never good, but when we are dealing with external I/O, this might be the only way to isolate it. This is how the Workflow test was done. Again, if there is a better way I am eager to learn it, please comment down below.

This is how a unit test with Mock looks like, stubbing both the HTTPotion and File modules:

test "workflow tries to download the images" do
  with_mock HTTPotion, [get: fn(_url, _options) -> %HTTPotion.Response{ body: nil, headers: nil, status_code: 200 } end] do
    with_mock File, [write!: fn(_filename, _body) -> nil end] do
      assert Workflow.process_downloads([{"http://src_foo", "filename_foo"}], "/tmp") == [{:ok, "http://src_foo", "/tmp/filename_foo"}]
      assert called HTTPotion.get("http://src_foo", [timeout: 30_000])
      assert called File.write!("/tmp/filename_foo", nil)
    end
  end
end

Conclusion

This has been a very fun experience, albeit very short, and good enough to iron out what I have learned so far. Code like this makes me smile:

[destination_file|_rest] = String.split(file, "/") |> Enum.reverse

The way I can pattern match to extract the head of a list is a different way of thinking. Then there is the other most important way of thinking: everything is a transformation chain, an application is a way to start from some input argument (such as a URL) and go step by step to “transform” it into a collection of PDF files, for example.

Instead of thinking about how to architect classes and objects, we start thinking about what the initial arguments are and what result we want to achieve, and go from there, one small transformation function at a time.

The Workflow module is an example. I actually started writing everything in a single large function in the CLI module. Then I refactored into smaller functions and chained them together to create the Workflow. Finally, I just moved all the functions into the Workflow module and called that from the CLI module.

Because of no global state, and thinking in smaller, isolated functions, both refactoring and test-driven development are much smoother than in OOP languages. This way of thinking is admittedly slow to get a grip on, but then it starts to feel very natural and it quickly steers your way of programming into leaner code.

And the dynamic aspects of both Erlang and Elixir make me feel right at home, just like having an “improved Ruby”.

The code of the downloader is all on Github, please fork it.

I am eager to exercise more. I hope this motivates you to learn Elixir. And if you’re already an advanced programmer in Elixir or Erlang, don’t forget to comment below and even send me a Pull Request to improve this small exercise. I am still a beginner and there is a lot of room to learn more. All contributions are greatly appreciated.