And make sure to subscribe to my new newsletter The M.Akita Chronicles!

–

Let me be blunt: most developers treat background jobs as if they were scripts that run once and that’s it. “Ah, throw it in a Sidekiq and we’re good.” No, we’re not good. Jobs that handle important things — sending emails, publishing content, calling external APIs — need to be treated as first-class citizens in your architecture.

In this post I’ll show how Rails 8 with ActiveJob and SolidQueue changed my perspective on async processing. This is not theory — these are patterns that emerged from a real system running in production every week.

The Problem: Jobs Are Fragile by Nature

Picture the scenario: you have a job that assembles a newsletter, publishes it on a blog through the GitHub API, waits for a podcast to be ready, and then fires off emails to hundreds of subscribers. Any step can fail. The API can time out. The podcast can take longer than expected. The server can restart in the middle of the delivery.

If you treat this as a linear script, you’re going to suffer. The right question is not “how do I make this job run?” — it’s “how do I make this job recover when something goes wrong?”

Pattern 1: retry_on With Specific Exceptions

ActiveJob’s retry_on is absurdly powerful, but most people use it wrong. Look at the classic anti-pattern:

class MeuJob < ApplicationJob
  retry_on StandardError, wait: 5.seconds, attempts: 3
end

This retries any error 3 times and gives up. It’s useless for almost anything in the real world. The pattern that actually works is to create specific exceptions for transient states:

class PodcastNotReady < StandardError; end

class PublishAndSendJob < ApplicationJob
  retry_on PodcastNotReady, wait: 15.minutes, attempts: 16

  def perform(newsletter_id)
    metadata = check_podcast_metadata(newsletter_id)
    raise PodcastNotReady, "Aguardando podcast" unless metadata
    # keep working...
  end
end

See what’s happening? The job waits up to 4 hours (15min × 16) for the podcast to be ready, polling every 15 minutes. This is not an infinite loop — it has a clear limit. And when it hits that limit, you handle the timeout explicitly:

rescue PodcastNotReady => e
  if executions >= 16
    # Publish without podcast and notify the team
    publish_without_podcast(newsletter)
    notify_team("Podcast não ficou pronto a tempo")
  else
    raise # re-raise so SolidQueue schedules the next retry
  end
end

This is graceful degradation. The system doesn’t stop because one component failed — it adapts and keeps going.

Pattern 2: Distributed Locks

When you have multiple paths that can trigger the same job (a safety cron, a manual trigger, an API), you need to make sure only one instance runs at a time.

The concept is simple: a file-based (or database-based) lock that expires automatically:

class DeployLock
  LOCK_DIR = Rails.root.join("tmp/locks")
  DEFAULT_TTL = 30.minutes

  def self.with_lock(name, ttl: DEFAULT_TTL)
    acquire!(name, ttl: ttl)
    yield
  ensure
    release(name)
  end
end

With SQLite, this is trivial — no Redis, no external coordination. The lock has a TTL so it doesn’t hang forever if the process dies. And the ensure guarantees it releases even if the block raises an exception.

Pattern 3: Atomic Email Delivery

This is where most newsletter systems fail spectacularly. The scenario: you’re sending 500 emails and the server restarts on email 247. What happens when the job re-runs?

If you just iterated a list, you’re going to resend the first 247. Your subscribers will love getting the newsletter twice.

The solution is atomic claiming per recipient:

# Before sending, create a record per subscriber
subscribers.each do |sub|
  EmailDelivery.create!(
    newsletter: newsletter,
    subscriber: sub,
    status: "pending"
  )
end

# At send time, do an atomic claim
delivery = EmailDelivery
  .where(status: ["pending", "failed"])
  .lock
  .first

delivery.update!(status: "sending")
send_email(delivery)
delivery.update!(status: "sent")

If the server dies between sending and sent, the record stays as “sending” — and a recovery job (RecoverStaleDeliveriesJob) periodically moves those records to “unknown” after a timeout, so they never get automatically resent. Ambiguous emails are never resent automatically. That’s the kind of detail that separates a hobby system from a production system.

Pattern 4: Orchestrator Jobs

A common mistake is cramming too much logic into a single job. The pattern that works is having orchestrator jobs that delegate to specialized jobs:

PublishAndSendJob (orchestrator)
  ├── Waits for podcast (retry_on PodcastNotReady)
  ├── PublishToBlogJob.publish(newsletter)
  └── SendNewsletterJob.perform_now(newsletter.id)
        └── SendSingleEmailJob (per subscriber)

The orchestrator coordinates the sequence. Each specialized job knows how to do exactly one thing and can be re-executed independently. If email sending fails, you can retrigger just the SendNewsletterJob without republishing the blog.

Pattern 5: Safety Nets With Cron

Don’t trust a single execution chain. SolidQueue’s cron (config/recurring.yml) acts as a safety net:

send_newsletter:
  class: SendNewsletterJob
  schedule: "0 12 * * 1"  # Monday 9am BRT (UTC-3)

If the 7am PublishAndSendJob failed catastrophically, the 9am cron fires the SendNewsletterJob as a fallback. The job checks whether the newsletter has already been sent and no-ops if so. Idempotency is the keyword here — the job can run as many times as you want and the result is always the same.

But watch out for the inverse: a job that reschedules itself infinitely when it finds no work is a ticking time bomb. If the job doesn’t find a ready newsletter, it simply returns. The cron handles trying again the next week.

Pattern 6: Status Notifications

Every long-running job should communicate its progress. A simple concern handles it:

module DiscordStatus
  extend ActiveSupport::Concern

  def notify_start(message)
    DiscordNotifier.send(channel: status_channel, text: "▶️ #{message}")
  end

  def notify_done(message)
    DiscordNotifier.send(channel: status_channel, text: "✅ #{message}")
  end

  def notify_error(message)
    DiscordNotifier.send(channel: status_channel, text: "❌ #{message}")
  end
end

Each job includes the concern and calls notify_start at the beginning, notify_done on success, notify_error on rescue. When something goes wrong at 3am, you wake up and see exactly where it stopped — no digging through logs required.

SolidQueue: The End of Redis Dependency

A note on SolidQueue, which ships as default in Rails 8: using the same SQLite database for jobs and application data drastically simplifies operations. No need for a separate Redis running. No need to worry about Redis restarting and losing jobs that were in memory.

Jobs are persisted in the database. If the server restarts, they’re there waiting. The retry state is preserved. It’s absurdly simpler than the alternative, and for 99% of cases, the performance is more than enough.

Conclusion

Rails 8 with ActiveJob and SolidQueue didn’t invent anything revolutionary. What it did was make it ridiculously easy to implement patterns that used to require heavy infrastructure:

• retry_on with specific exceptions and clear limits
• Distributed locks with automatic TTL
• Atomic claiming for operations that can’t duplicate
• Orchestrator jobs that delegate and recover
• Safety crons as fallback
• Notifications for operational visibility

None of these patterns is complicated on its own. Put them together, though, and that’s what makes the difference between a system that breaks at the first failure and one that holds up in production — and lets you sleep peacefully on Monday.

As a quick recap, the exercise is that I want to scrape MangaReader.net for a bunch of images, organized in pages and chapters, and in the end it should compile organized PDFs so I can load them on a Kindle device.

Web scraping is a simple loop of HTTP GETs over a ton of URLs, scraping the HTML, and fetching more URLs to download.

In many simple languages, people usually solve this naively in 2 ways:

• A simple nested loop. One single thread, sequential fetches. So if you have 5,000 links and each links takes 10 seconds to fetch, it’s basically 10 * 5,000 = 50,000 seconds, which is a stupid long time.
• A simple spawn of processes, fibers, threads or parallel I/O in a reactor, all at once. An attempt to parallelize all fetches at once.

Everybody probably agrees that the first option is stupid. Now, the second one is tricky.

The tricky part is CONTROL.

Anyone in Go would say “oh, this is easy, just put a loop and spawn a bunch of goroutines” or anyone in Node.js would say “oh, this is easy, just put a loop, make the fetch - they all will run asynchronously - and add callbacks, a basic async/await.”

They’re not wrong, but this is still too naive an implementation. It’s trivial to trigger hundreds or thousands of parallel requests. Now, what happens if one fails and you have to retry? What happens if the MangaReader has a throttling system that will either start cutting down connections or timing them out? Or if your internet bandwidth is just not enough, and after a certain amount of requests you start having diminishing returns and time outs?

The message is: it’s damn trivial to spawn parallel stuff. it’s damn complicated to control parallel stuff.

This is why, in my first implementation in Elixir, I introduced a complicated implementation using a combination of a custom GenServer, Elixir’s own Task infrastructure for async/await pattern, and Poolboy to control the rate of the parallelism. This is how you control the bottleneck out to slow resources: using pools and queues (which is why every good database has a connection pool, remember C3P0?)

This is one snippet of my older implementation:

def chapter_page([chapter_link, source]) do
  Task.Supervisor.async(Fetcher.TaskSupervisor, fn ->
    :poolboy.transaction :worker_pool, fn(server) ->
      GenServer.call(server, {:chapter_page, chapter_link, source}, @genserver_call_timeout)
    end, @task_async_timeout
  end)
end

Yes, it’s very ugly, and there are boilerplates for the GenServer, the custom Supervisor to initialize Poolboy and so on. And the higher level workflow code looks like this:

def pages({chapter_list, source}) do
   pages_list = chapter_list
     |> Enum.map(&Worker.chapter_page([&1, source]))
     |> Enum.map(&Task.await(&1, @await_timeout_ms))
     |> Enum.reduce([], fn {:ok, list}, acc -> acc ++ list end)
   {pages_list, source}
end

So, inside the Worker module each public method wraps the GenServer internal calls into a Task async and in the collection iteration we add Task.await to actually wait for all parallel calls to finish, so we can finally reduce the results.

Elixir now comes with this very interesting GenStage infrastructure that promises to replace GenEvent and the use case is when you have a producer-consumer situation with back-pressure. Basically when you have slow endpoints and you would end up having to control bottlenecks.

Then, Flow is an easier high abstraction that you can use almost the same way you would use Enum in your collections, but instead of sequential mapping, it takes care of parallel traversing and control of batches. So the code is very similar but without you having to control the parallelization controls manually.

This is the full commit where I could remove Poolboy, remove my custom GenServer, reimplement the Worker as simple module of functions and then the workflow could get rid off the async/await pattern and use Flow instead:

def pages({chapter_list, source}) do
   pages_list = chapter_list
     |> Flow.from_enumerable(max_demand: @max_demand)
     |> Flow.map(&MangaWrapper.chapter_page([&1, source]))
     |> Flow.partition()
     |> Flow.reduce(fn -> [] end, fn {:ok, list}, acc -> acc ++ list end)
     |> Enum.to_list()
   {pages_list, source}
end

The only boilerplate left is the Flow.from_enumerable() and Flow.partition() wrapping the Flow.map, and that’s it!

Notice I configured @max_demand to be 60. You must tweak it to be larger or smaller depending on your internet connection, you have to experiment it. By default, Flow will trigger 500 processes in parallel, which is way too much for web scraping on a normal home internet connection and you will suffer diminishing returns. That’s what I had to do previously with Poolboy, by initiating a pool of around 60 transactions at most.

Unfortunately not everything is as straight forward as it seems. Running this new version on the test mode I get this result:

58,85s user 13,93s system 37% cpu 3:13,78 total

So a total time of more than 3 minutes, using around 37% of the available CPU.

My immediate previous version using all the shenanigans of Poolboy, Task.Supervisor, GenServer, etc still gives me this:

100,67s user 20,83s system 152% cpu 1:19,92 total

Less than HALF the time, albeit using all my CPU cores. So my custom implementation still uses my resources to the maximum. There is still something in the Flow implementation I didn’t quite get right. I already tried to bump up the max_demand from 60 up to 100 but that didn’t improve anything. Leaving it to the default 500 slows everything down to more than 7 minutes.

All in all, at least it makes the implementation far easier on the eyes (hence, way easier to maintain), but either the Flow implementation has bottlenecks or I am using it wrong at this point. If you know what it is, let me know in the comments section below.

I’ve published an article about ExAdmin and Coherence and another on Deploying Elixir to DigitalOcean this month.

The idea is very simple, it is a homage to Pusher. If you used Pusher before, this is very similar (albeit way less feature complete, of course).

I built an entire solution inspired by Pusher, using the Phoenix framework, deployed to Digital Ocean and you can test it out right now, just sign up at expusherlite.cm42.io.

Once you sign up, you will have a secret token (don’t disclose that, of course) and you will be under an Organization. Then you can go on and create Applications within that Organization. Each Application will have a unique token to identify it.

Now, let’s say you want to create a Rails application with a Chat feature. Any version of Rails, you don’t need 5.0 and you don’t need ActionCable.

First off, let’s configure config/secrets.yml:

development:
  secret_key_base: b9a1...e7aa
  pusher_host: "expusherlite.cm42.io"
  org_id: acme-inc
  app_key: 0221...f193
  secret_token: 4036...f193
...
production:
  secret_key_base: <%= ENV["SECRET_KEY_BASE"] %>
  pusher_host: <%= ENV['PUSHER_LITE_HOST'] %>
  org_id: <%= ENV['PUSHER_LITE_ORGANIZATION'] %>
  app_key: <%= ENV["PUSHER_LITE_APP_KEY"] %>
  secret_token: <%= ENV["PUSHER_LITE_SECRET_TOKEN"] %>

Replace the pusher_host, org_id, app_key, and secret_token for the ones you created before.

Now I want to add a PageController:

require "net/http"
require "uri"
class PageController < ApplicationController
  def index
    uri = URI.parse("http://#{Rails.application.secrets.pusher_host}/api/sessions")
    response = Net::HTTP.post_form(uri, {"token" => Rails.application.secrets.secret_token})
    @guardian_token = JSON.parse(response.body)["jwt"]
    Rails.logger.info @guardian_token
  end
end

(If you’re connecting to my online server, you must use SSL, so change the URL above for “https”)

What this piece does is submit the secret token in the server-side, to my service, to get a JSON Web Token (JWT) back. Now you can pass this JWT to the front-end to enable authentication.

In the front-end we can have this simple app/views/page/index.html.erb:

<h1>Ex Pusher Lite - Rails Integration Example</h1>

<script type="text/javascript" charset="utf-8">
  window.guardian_token = "<%= @guardian_token %>";
  window.org_id = "<%= Rails.application.secrets.org_id %>"
  window.app_key = "<%= Rails.application.secrets.app_key %>";
  window.pusher_host = "<%= Rails.application.secrets.pusher_host %>";
</script>

<div id="chat" class="fixedContainer">
</div>

<input type="text" name="name" id="name" value="" placeholder="Name"/>
<input type="text" name="message" id="message" value="" placeholder="Message"/>
<input type="checkbox" name="channel" id="channel" value="api"/>
<label for="channel">send through API</label>

Super simple, we can tweak the CSS (app/assets/stylesheets/application.css) just to make it look nicer:

...
.fixedContainer {
  height: 250px;
  width: 100%;
  padding:3px;
  border: 1px solid black;
  margin: 5px;
  overflow: auto;
}

body {
  font-family: Helvetica, Arial
}

Finally, we need to load the main Javascript from the ExPusherLite server, so edit the layout at app/views/layouts/application.html.erb and add this line right after the closing </body> tag:

<script src="http://<%= Rails.application.secrets.pusher_host %>/js/pusher.js"></script>

And we can now use this Javascript in the app/assets/javascripts/application.js to hook everything up. This is the relevant bit:

$(document).ready(function() {
  var PusherLite = require("pusher_lite").default;

  var pusher = new PusherLite(window.app_key, {
    host: window.pusher_host,
    jwt: window.guardian_token,
    uniqueUserId: "robot" })
    // ssl: true - if you're connecting to my online server

  var publicChannel = pusher.subscribe("general")

  publicChannel.bind("new_message", function(payload) {
    var chat = $("#chat")
    chat.append("<p><strong>" + payload.name + "</strong> " + payload.message + "</p>");
    chat.scrollTop(chat.prop("scrollHeight"));
  })

  pusher.joinAll();

We can now continue in the same file with the Javascript that binds to the message input field, listening to the “Enter” key press event to send the messages:

  var message_element = $("#message");
  message_element.on('keypress', function(event) {
    if (event.keyCode != 13) { return; }

    var name_element    = $("#name");
    var check_element   = $("#channel");
    var payload = { name: name_element.val(), message: message_element.val() };

    if(!check_element.prop("checked")) {
      sendPusher(payload);
    } else {
      sendAPI(payload)
    }
    message_element.val('');
  });

  window.publicChannel = publicChannel;
})

And this is how we send messages to ExPusherLite, either directly through the full-duplex WebSockets:

function sendPusher(payload) {
  console.log("sending through socket")
  window.publicChannel.trigger('new_message', payload );
}

Or Posting to the available API:

function sendAPI(payload) {
  console.log("sending through API")
  $.ajax({
    type : 'POST',
    crossDomain: true,
    url : makeURL("new_message"),
    headers : { Authorization : 'Bearer ' + window.guardian_token },
    data : payload,
    success : function(response) {
      console.log(response);
      console.log("sent through API successfully");
    },
    error : function(xhr, status, error) {
      console.log(error);
    }
  });
}

function makeURL(event) {
  return "http://" + window.pusher_host + "/api/organizations/" + window.org_id + "/applications/" + window.app_key + "/event/" + event;
}

By the way, you can send messages using the API from the server-side if you want. Specifically from an ActiveJob process so you can keep your Rails web application fast, and you can use the opportunity to store the message in your database, or apply any filters.

And this is it! You now have a Rails application with WebSockets. You can have your lunch and eat it too.

If you want to see this example working, I published a demo app over at Heroku. It’s just a demo, it has no authentication, no cross-scripting sanitization, no nothing.

In summary: this is a Rails app (you could do it in Django, Laravel, ASP.NET MVC, it doesn’t matter) talking through WebSocket + APIs to a Phoenix cluster.

Next Steps

Keep following my blog (or my Twitter at @akitaonrails ) for more posts to come.

I am still considering if I will open the ExPusherLite code as open source, so let me know if you’re interested.

I am also considering if I will keep the current servers online as a cheap service. You can use it for free right now to play with it, but don’t use for production-level apps yet. As I am still heavily coding it, I will keep updating the servers, so there is no SLA. Let me know if you’re interested in such a service that keeps the code open source so you can trust it better.

There are important features still missing, such as proper SSL support, encrypted channels, better Presence tracking APIs and so on, but what’s available right now already covers most use cases for WebSockets.

And better: because this is Phoenix, because this is Elixir, and because this is Erlang, we get distributed PubSub for “free”. As I explained in my deployment post, this is a setup with a server in New York and another in London, just to showcase the distributed nature of Erlang.

It’s been very fun to play with Elixir for the past few days and how fast I was able to put together a full-featured solution like this. There were many puzzles that made me scratch my head, figuring out how to deal with cross origin issues, how to make the nodes find each other through the edeliver deployment, figuring out the missing bits in replacing exrm for distillery (which is a transition still taking place in the community), etc.

Now I am quite comfortable with the basics, from bootstrapping a project all the way to deploying in a cluster scenario. And I hope this service proves useful to more people.

As this is possibly my last post of the year: Happy New Year! And I will see you again in 2017!

For starters, Erlang does not need multiple processes like Ruby (which is limited to one connection per process, or per thread if you’re using a threaded-server like Puma). One single Erlang process will take over the entire machine, if you need to. Internally it will keep one real thread per machine-core. And each thread will have its own Scheduler to manage as many micro-processes as you need. You can read all about it in my post titled “Yocto Services”.

Moreover, Erlang has built-in capabilities to form a cluster, where each Erlang instance acts as a peer-to-peer Node, without the need for a centralized coordinator. You can read all about it in my post about Nodes. The power of Erlang is in how “easy” it is to form reliable distributed systems.

You can fire up many Phoenix instances and from one of the instances, it can broadcast messages to Users subscribed in Channels even if their sockets are connected to different instances. It’s seamless and you don’t need to do anything special in your code. Phoenix, Elixir and Erlang are doing all the heavy lifting for you behind the scenes.

No Heroku for You :-(

Because you want to take advantage of this scalability and high availability feature for distributed systems (in the small example of a real-time chat system) you will need to have more control over your infrastructure. This requirement rules out the majority of Platform as a Service (PaaS) offerings out there, such as Heroku. Heroku’s model revolves around single, volatile processes in isolated containers. Those jailed processes (dynos) are not aware of other processes or the internal networking, so you can’t fire up Dynos and have them form a cluster because they won’t be able to find each other.

If you already know how to configure Linux related stuff: Postgresql, HAproxy, etc, go ahead directly to the Phoenix-specific section.

IaaS (DigitalOcean) to the rescue!

You want long lived processes in network reachable servers (either through private networking, VPN, or plain simple - insecure! - public networks).

In this example I want to walk you through a very simple deployment using DigitalOcean (you can choose any IaaS, such as AWS, Google Cloud, Azure or whatever you feel more comfortable with).

I have created 4 droplets (all using the smallest size of 512Mb of RAM):

• 1 Postgresql database (single point of failure: it’s not the focus of this article to build a highly available, replicated database setup);
• 1 HAProxy server (single point of failure: again, it’s not the focus to create a highly available load balancing scheme);
• 2 Phoenix servers - one in the NYC datacenter and another in the London datacenter, to demonstrate how easy it is for Erlang to form clusters even with geographically separated boxes.

Basic Ubuntu 16.04 configuration

• Goals: configure locale, assure unattended updates are up, upgrade packages, install and configure Elixir and Node.
• You should also do: change SSH to another port and install [fail2ban](https://www.digitalocean.com/community/tutorials/how-to-protect-ssh-with-fail2ban-on-ubuntu-14-04, disallow login through password).

You will want to read my post about configuring Ubuntu 16.04. To summarize, start by configuring proper UTF-8:

sudo locale-gen "en_US.UTF-8"
sudo dpkg-reconfigure locales
sudo update-locale LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8
echo 'LC_ALL=en_US.UTF-8' | sudo tee -a /etc/environment
echo 'LANG=en_US.UTF-8' | sudo tee -a /etc/environment

Make sure you add a proper user into the sudo group and from now on do not use the root user. I will create a user named pusher and I will explain in another post why. You should create a username that suits your application.

adduser pusher
usermod -aG sudo pusher

Now log out and log in again through this user. ssh pusher@server-ip-address. If you’re on a Mac copy the public key of your SSH certificate like this:

ssh-copy-id -i ~/.ssh/id_ed25519.pub pusher@server-ip-address

It creates the .ssh/authorized_keys if it doesn’t exist, sets the correct permission bits and appends your public key. You can do it manually, of course.

DigitalOcean’s droplets start without a swap file and I’d recommend adding one, especially if you want to start with the smaller boxes with less than 1GB of RAM:

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
sudo cp /etc/fstab /etc/fstab.bak
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
sudo sysctl vm.swappiness=10
sudo sysctl vm.vfs_cache_pressure=50
echo 'vm.swappiness=10' | sudo tee -a /etc/sysctl.conf
echo 'vm.vfs_cache_pressure=50' | sudo tee -a /etc/sysctl.conf

Make sure you have unattended upgrades configured. You will want at least to have security updates automatically installed when available.

sudo apt install unattended-upgrades

Now, let’s install Elixir and Node (Phoenix needs Node.js):

wget https://packages.erlang-solutions.com/erlang-solutions_1.0_all.deb && sudo dpkg -i erlang-solutions_1.0_all.deb
curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash -
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install build-essential nodejs esl-erlang elixir erlang-eunit erlang-base-hipe
sudo npm install -g brunch
mix local.hex
mix archive.install https://github.com/phoenixframework/archives/raw/master/phoenix_new.ez # this is optional, install if you want to manually test phoenix in your box

Now you have an Elixir capable machine ready. Create a image snapshot over DigitalOcean, move it to the regions where you want to create your other droplets and use this image to create as many droplets as you need.

For this example, I created a second droplet in the London region, a third droplet for postgresql in the NYC1 region and a fourth droplet in the NYC3 region for HAProxy.

I will refer to their public IP addresses as “nyc-ip-address”, “lon-ip-address”, “pg-ip-address”, and “ha-ip-address”.

Basic PostgreSQL configuration

• Goal: basic configuration of Postgresql to allow the Phoenix servers to connect.
• To do: create a secondary role just to connect to the application database and another superuser role to create the database and migrate the schema. Also lock down the machine and configure SSH tunnels or another secure way, at least private networking, rather than allowing plain 5432 TCP port connections from the public Internet.

Now you can connect to ssh pusher@pg-ip-address and follow this:

sudo apt-get install postgresql postgresql-contrib

You should create a new role with the same name of the user you added above (“pusher” in our example):

$ sudo -u postgres createuser --interactive

Enter name of role to add: pusher
Shall the new role be a superuser? (y/n) y

$ sudo -u postgres createdb pusher

Postgresql expects to find a database with the same name as the role and the role should have the same name as the Linux user. Now you can use psql to set a password for this new role:

$ sudo -u postgres psql
\password pusher

Register a secure password, take note and let’s move on.

Postgresql comes locked down to external connections. One way to connect from the outside is to configure your servers to create an SSH tunnel to the database server and keep external TCP connections through port 5432 disavowed.

But for this example, we will just allow connections from the public Internet to the 5432 TCP port. Warning: this is VERY insecure!

Edit the /etc/postgresql/9.5/main/postgresql.conf and find the listen_addresses configuration line and allow it:

listen_addresses = '*'    # what IP address(es) to listen on;

This should bind the server to the TCP port. Now edit /etc/postgresql/9.5/main/pg_hba.conf and edit the end to look like this:

# IPv4 local connections:
host    all             all             127.0.0.1/32            trust
host    all             all             your-local-machine-ip-address/24        trust
host    all             all             nyc-ip-address/24       trust
host    all             all             lon-ip-address/24       trust
# IPv6 local connections:
host    all             all             ::1/128                 trust

Save the configuration file and restart the server:

sudo service postgresql restart

See what I did there? I only allowed connections coming from the public IPs of the Phoenix servers. This does not make the server secure, just a little bit less vulnerable. If you’re behind a DHCP/NAT based network, just Google for “what’s my IP” to see your public facing IP address - which is probably shared by many other users, remember you’re allowing connections from an insecure IP to your database server! Once you make initial tests, create your new schema, then you should remove that your-local-machine-ip-address/24 line from the configuration.

From your Phoenix application, you can edit your local config/prod.secret.exs file to look like this:

# Configure your database
config :your_app_name, ExPusherLite.Repo,
  adapter: Ecto.Adapters.Postgres,
  username: "pusher",
  password: "your-super-secure-pg-password",
  database: "your-app-database-name",
  hostname: "pg-ip-address",
  pool_size: 20

Replace the information for your server and database and now you can test it out like this:

MIX_ENV=prod iex -S mix phoenix.server

If you see a :econnrefused message from postgrex, then you’re in trouble. Re-check your configuration, restart the server and try again. If everything connects, you can run MIX_ENV=prod mix do ecto.create, ecto.migrate to prepare your database.

Finally, you will want to lock down the rest of your server with UFW, at the very least. UFW should come pre-installed in Ubuntu 16, so you can just do:

ufw allow 5432
ufw allow ssh
ufw enable

That’s it. And again, this does not make your server secure, it just makes it less insecure. There is a huge difference!

And by the way, if you’re a Docker fan:

DO NOT INSTALL A DATABASE INSIDE A DOCKER CONTAINER!

You have been warned!

Basic HAProxy configuration

• Goals: Provide a simple solution to load balance between our 2 Phoenix servers.
• To do: There is something wrong with session checking or something like that as sometimes I have to refresh my browser so I am not sent back to the login form in my application. Phoenix uses cookie-based sessions so I don’t think it is missing sessions.

Now let’s ssh pusher@ha-ip-address. This one is easy, let’s just install HAProxy:

sudo apt-get install haproxy

Edit /etc/haproxy/haproxy.cfg:

...
listen your-app-name
  bind 0.0.0.0:80
  mode http
  stats enable
  stats uri /haproxy?stats
  stats realm Strictly\ Private
  stats auth admin:some-secure-password-for-admin
  option forwardfor
  option http-server-close
  option httpclose
  balance roundrobin
  cookie SERVERID insert indirect nocache
  server us your_us_server_IP:8080 check cookie us1
  server uk your_uk_server_IP:8080 check cookie uk1

You can avoid the stats lines if you have other means of monitoring, otherwise set a secure password for the admin user. One very important part is the option http-server-close as explained in this other blog post, otherwise you may have trouble with Websockets.

For some reason I am having some trouble with my application after I login and it sets the session, sometimes I have to refresh to be sent to the correct page, not sure why yet and I believe it’s something in the HAProxy configuration. If anyone knows what it is, let me know in the comments section below. For now, I am just relying on Sticky sessions (server affinity) by making HAProxy write back a cookie with the server.

Now you can restart the server and enable UFW as well:

sudo service haproxy restart
sudo ufw allow http
sudo ufw allow https
sudo ufw allow ssh
sudo ufw enable

You can easily add SSL support to your application by configuring HAProxy (and not the Phoenix nodes). The DigitalOcean documentation is comprehensive, so just follow it. At the end, my haproxy.cfg looks like this:

global
   log /dev/log    local0
   log /dev/log    local1 notice
   chroot /var/lib/haproxy
   stats socket /run/haproxy/admin.sock mode 660 level admin
   stats timeout 30s
   user haproxy
   group haproxy
   daemon
   maxconn 2048
   tune.ssl.default-dh-param 2048

defaults
   log global
   mode http
   option httplog
   option dontlognull
   option redispatch
   option forwardfor
   option http-server-close
   timeout connect 5000
   timeout client  50000
   timeout server  50000

frontend www-http
   bind your_ha_proxy_IP:80
   reqadd X-Forwarded-Proto:\ http
   default_backend www-backend

frontend www-https
   bind your_ha_proxy_IP:443 ssl crt /etc/haproxy/certs/your_domain.pem
   reqadd X-Forwarded-Proto:\ https
   acl letsencrypt-acl path_beg /.well-known/acme-challenge/
   use_backend letsencrypt-backend if letsencrypt-acl
   default_backend www-backend

backend www-backend
   redirect scheme https if !{ ssl_fc }
   # setting session stickiness
   cookie SERVERID insert indirect nocache
   server us your_us_server_IP:8080 check cookie us1
   server uk your_uk_server_IP:8080 check cookie uk1

backend letsencrypt-backend
   server letsencrypt 127.0.0.1:54321

Finally, I will assume you have a DNS server/service somewhere where you can register the IP of this HAproxy server as an A record so you can access it by a full name such as “your-app-name.mydomain.com”.

Basic Phoenix Configuration

• Goal: configure the Phoenix app to be deployable. Configure the servers to have the necessary configuration files.
• To do: find out a way to cut down the super slow deployment times.

Finally, we have almost everything in place.

I will assume that you have a working Phoenix application already in place, otherwise create one from the many tutorials out there.

I have assembled this information from posts such as this very helpful one from Pivotal about an AWS-based deployment. In summary you must do a number of changes to your configuration.

When you’re developing your application, you will notice that whenever you run it, it delta-compiles what changed. The binary bits are in the _build/dev or _build/test in the form of .beam binaries (similar to what .class are for Java).

Different from Ruby or Python or PHP, you are not deploying source-code to production servers. It’s more akin to Java, where you must have everything compiled into binary bits and packaged into what’s called a release. It’s like a “.war” or “.ear” if you’re from Java.

To create this package people usually use “exrm”, but it’s being replaced by “distillery”, so we will use it.

Then, if you’re from Ruby you’re familiar with Capistrano. Or if you’re from Python, you know Capistrano’s clone, Fabric. Elixir has a similar tool (much simpler at this point), called “edeliver”. It’s your basic SSH automation tool.

You add them to mix.exs just like any other dependency:

...
def application do
  [mod: {ExPusherLite, []},
   applications: [..., :edeliver]]
end

defp deps do
  [...,
   {:edeliver, "~> 1.4.0"},
   {:distillery, "~> 1.0"}]
end
...

From the Pivotal blog post, the important thing to not forget is to edit this part in the config/prod.exs file:

http: [port: 8080],
url: [host: "your-app-name.yourdomain.com", port: 80],
...
config :phoenix, :serve_endpoints, true

You MUST hardcode the default PORT of the Phoenix web server and the allowed domain (remember the domain name you associated to your HAProxy server above? That one). And you MUST uncomment the :serve_endpoints, true line!

For edeliver to work you have to create a .deliver/config file like this:

USING_DISTILLERY=true

# change this to your app name:
APP="your-app-name"

# change this to your own servers IP and add as many as you want
US="nyc-ip-address"
UK="lon-ip-address"

# the user you created in your Ubuntu machines above
USER="pusher"

# which server do you want to build the first release?
BUILD_HOST=$US
BUILD_USER=$USER
BUILD_AT="/tmp/edeliver/$APP/builds"

# list the production servers declared above:
PRODUCTION_HOSTS="$US $UK"
PRODUCTION_USER=$USER
DELIVER_TO="/home/$USER"

# do not change here

LINK_VM_ARGS="/home/$USER/vm.args"

# For *Phoenix* projects, symlink prod.secret.exs to our tmp source
pre_erlang_get_and_update_deps() {
  local _prod_secret_path="/home/$USER/prod.secret.exs"
  if [ "$TARGET_MIX_ENV" = "prod" ]; then
    __sync_remote "
      ln -sfn '$_prod_secret_path' '$BUILD_AT/config/prod.secret.exs'

      cd '$BUILD_AT'

      mkdir -p priv/static

      mix deps.get

      npm install

      brunch build --production

      APP='$APP' MIX_ENV='$TARGET_MIX_ENV' $MIX_CMD phoenix.digest $SILENCE
    "
  fi
}

Remember the information we’ve been gathering since the beginning of this long recipe? These are the options you MUST change to your own. Just follow the comments in the file content above and add it to your git repository. By the way, your project is in a proper GIT repository, RIGHT??

If you like to use passphrase protected SSH private keys, then it’s going to be a huge pain to deploy because for each command, edeliver will issue an SSH command that will keep asking for you passphrase, a dozen times through everything. You’ve been warned! If you still don’t mind that, and you’re in a Mac you will have an extra trouble because the Terminal will not be able to create a prompt for you to input your passphrase. You must create an /usr/local/bin/ssh-askpass script:

#!/bin/bash
# Script: ssh-askpass
# Author: Mark Carver
# Created: 2011-09-14
# Licensed under GPL 3.0

# A ssh-askpass command for Mac OS X
# Based from author: Joseph Mocker, Sun Microsystems
# http://blogs.oracle.com/mock/entry/and_now_chicken_of_the

# To use this script:
#   Install this script running INSTALL as root
#
# If you plan on manually installing this script, please note that you will have
# to set the following variable for SSH to recognize where the script is located:
#   export SSH_ASKPASS="/path/to/ssh-askpass"

TITLE="${SSH_ASKPASS_TITLE:-SSH}";
TEXT="$(whoami)'s password:";
IFS=$(printf "\n");
CODE=("on GetCurrentApp()");
CODE=(${CODE[*]} "tell application \"System Events\" to get short name of first process whose frontmost is true");
CODE=(${CODE[*]} "end GetCurrentApp");
CODE=(${CODE[*]} "tell application GetCurrentApp()");
CODE=(${CODE[*]} "activate");
CODE=(${CODE[*]} "display dialog \"${@:-$TEXT}\" default answer \"\" with title \"${TITLE}\" with icon caution with hidden answer");
CODE=(${CODE[*]} "text returned of result");
CODE=(${CODE[*]} "end tell");
SCRIPT="/usr/bin/osascript"
for LINE in ${CODE[*]}; do
      SCRIPT="${SCRIPT} -e $(printf "%q" "${LINE}")";
done;
eval "${SCRIPT}";

Now do this:

sudo chmod +x /usr/local/bin/ssh-askpass
sudo ln -s /usr/local/bin/ssh-askpass /usr/X11R6/bin/ssh-askpass

Remember, Macs only. And now every time you try to deploy you will receive a number of graphical prompt windows asking for the SSH private key passphrase. It’s freaking annoying! And you must have XQuartz installed, by the way.

Now you must manually create 3 files in all Phoenix servers. Start with the your-app-name/vm.args:

-name us@nyc-ip-address
-setcookie @bCd&fG
-kernel inet_dist_listen_min 9100 inet_dist_listen_max 9155
-config /home/pusher/your-app-name.config
-smp auto

The /home/pusher/your-app-name is the directory where the release will be uncompressed after edeliver deploys the release tarball.

You must create this file in all Phoenix machines, by the way, changing the -name bit for the same name you declared in the .deliver/config file. The -setcookie should be any name, as long as it’s the same in all servers.

See the -config /home/pusher/your-app-name.config? Create that file with the following:

[{kernel,
  [
    {sync_nodes_optional, ['uk@lon-ip-address']},
    {sync_nodes_timeout, 30000}
  ]}
].

This is an Erlang source-code. On the NYC machine you must declare the London name, and vice-versa. If you have several machines, all of them but the one you’re in right now. Get it?

Finally, for the Phoenix app itself, you always have a config/prod.secret.exs that should never be git added to the repository, remember? This is where you put the Postgresql server information and random secret key to sign the session cookies:

use Mix.Config

config :your_app_name, YourAppName.Endpoint,
  secret_key_base: "..."

# Configure your database
config :your_app_name, YourAppName.Repo,
  adapter: Ecto.Adapters.Postgres,
  username: "pusher",
  password: "your-super-secure-pg-password",
  database: "your-app-database-name",
  hostname: "pg-ip-address",
  pool_size: 20

# if you have Guardian, for example:
config :guardian, Guardian,
  secret_key: "..."

How do you create a new random secret key? From your development machine just run: mix phoenix.gen.secret and copy the generated string into the file above.

So now you must have those 3 files in each Phoenix server, in the /home/pusher home folder:

~/your-app-name/vm.args
~/prod.secret.exs
~/your-app-name.config

As per distillery documentation, you have to set an environment variable to let it know about the vm.args file, and this is SUPER important otherwise it will generate a default one that doesn’t set the proper name and cookie, so the nodes will not find each other after booting up.

Using sudo, edit the /etc/environment and add the line:

RELEASE_CONFIG_DIR=/home/pusher/your-app-name
VMARGS_PATH=/home/pusher/your-app-name/vm.args

You have to do it in all the phoenix servers.

In your local development directory you still have to run this:

mix release.init

It will generate a standard rel/config.exs file that you must add to your git repository with the following changes to the bottom:

...
environment :prod do
  plugin Releases.Plugin.LinkConfig
  ...
end

release :your_app_name do
  set version: current_version(:your_app_name)

  set applications: [
    your_app_name: :permanent
  ]
end

The plugin should enable the release to find the local vm.args file in the server (this is super important, otherwise the remote commands such as start, stop, ping, etc will not work and the nodes will not load the correct information to boot up and form a cluster).

Finally, all set, you can issue this command:

$ mix edeliver build release production --skip-mix-clean --verbose

If it finishes without errors you will have a message like the following in the end:

...
==> Release successfully built!
    You can run it in one of the following ways:
      Interactive: _build/prod/rel/your-app-name/bin/your-app-name console
      Foreground: _build/prod/rel/your-app-name/bin/your-app-name foreground
      Daemon: _build/prod/rel/your-app-name/bin/your-app-name start
--> Copying release 0.0.1 to local release store
--> Copying your-app-name.tar.gz to release store

RELEASE BUILD OF YOUR-APP-NAME WAS SUCCESSFUL!

Now, this will take an absurdly long time to deploy. That’s because it will git clone the source code of your app, fetch all Elixir dependencies (every time!), it will have to compile everything, then it will run the super slow npm install (every time!), brunch your assets, create the so-called “release”, tar and gzip it, download it and SCP it to the other machines you configured.

In the .deliver/config file you set a BUILD_HOST option. This is the machine where all this process takes place, so you will want to have at least this machine be beefier than the others. As I am using small 512Mb droplets, the process takes forever.

The last command will download the generated tarball. It has to do it to make sure you have the NIFs compiled in the native environment where they run, because if you use a Mac, binaries from Macs won’t run on Linux.

Now we must upload and decompress this tarball in each server with the following command:

mix edeliver deploy release to production

The slower your network, the longer this will take, as it’s uploading a big tarfile across the public internet, so make sure you’re in a fast connection. And when it finishes, we can restart the servers (if you already had instances running):

mix edeliver stop production
mix edeliver start production

If you do everything right, the edeliver process finishes without any error and it leaves a daemon running in your server, like this:

/home/pusher/your-app-name/erts-8.2/bin/beam -- -root /home/pusher/your-app-name -progname home/pusher/your-app-name/releases/0.0.1/your-app-name.sh -- -home /home/pusher -- -boot /home/pusher/your-app-name/releases/0.0.1/your-app-name -config /home/pusher/your-app-name/running-config/sys.config -boot_var ERTS_LIB_DIR /home/pusher/your-app-name/erts-8.2/../lib -pa /home/pusher/your-app-name/lib/your-app-name-0.0.1/consolidated -name us@nyc-ip-address -setcookie ex-push&r-l!te -kernel inet_dist_listen_min 9100 inet_dist_listen_max 9155 -config /home/pusher/your-app-name.config -mode embedded -user Elixir.IEx.CLI -extra --no-halt +iex -- console

It will still take a long time, but it should be easier. So this is a pro-tip for you, Linux users. Follow this Gist for more details, you must emulate what’s run from the .deliver/config file’s bottom half.

Also notice that I ran the migrations manually, but you can do it using mix edeliver migrate.

Read their documentation for more commands and configurations.

Also, do not forget to enable UFW:

sudo ufw allow ssh
sudo ufw allow 8080
sudo ufw allow 4369
sudo ufw allow proto tcp from any to any port 9100:9155
sudo ufw default allow outgoing
sudo ufw enable

Debugging Production bugs

Right after I deployed, obviously it failed. And the problem is that the /home/pusher/your-app-name/log/erlang.log files (they are automatically rotated so you may find several files ending in a number), you won’t see much.

What I recommend you to do is to change the config/prod.exs file ONLY in your development machine and change the log setting to config :logger, level: :debug, use the same prod.secret.exs you edited in the servers above and run it locally with MIX_ENV=prod iex -S mix phoenix.server.

For example, in development mode I had a code in the controller that was checking the existence of an optional query string parameter like this:

if params["some_parameter"] do
 ...

That was working fine in development but crashing in production, so I had to change it to:

if Map.has_key?(params, "some_parameter") do
 ...

Another thing was that Guardian was working normally in development, but in production I had to declare its application in the mix.exs like this:

def application do
  [mod: {ExPusherLite, []},
   applications: [..., :guardian, :edeliver]]
end

I was getting :econnrefused errors because I forgot to run MIX_ENV=prod mix do ecto.create, ecto.migrate as I instructed above. Once I figured those out, my application was up and running through the http://your-app-name.yourdomain.com, HAProxy was correctly forwarding to the 8080 port in the servers and everything runs fine, including the WebSocket connections.

Conclusion

As I mentioned above, this kind of procedure makes me really miss an easy to deploy solution such as Heroku.

The only problem I am facing right now is that when I log in through Coherence’s sign in page, I am not redirected to the correct URI I am trying ("/admin" in my case), sometimes reloading after sign in works, sometimes it doesn’t. Sometimes I am inside a “/admin” page but when I click one of the links it sends me back to the sign in page even though I am already signed in. I am not sure if it’s a bug in Coherence, ExAdmin, Phoenix itself or an HAProxy misconfiguration. I will update this post if I find out.

Edeliver also takes an obscene amount of time to deploy. Even waiting for sprockets to process in a git push heroku master deploy feels way faster in comparison. And this is for a very bare-bone Phoenix app. Having to fetch everything (because Hex doesn’t keep a local global cache, all dependencies are statically vendored in the project directory). Having to run the super slow npm doesn’t help either.

I still need to research if there are faster options, but for now what I have “works”.

And more importantly, now I have a scalable cluster for real-time bi-directional WebSockets, which is the main reason one might want to use Phoenix in the first place.

If you want to build a “normal” website, keep it simple and do it in Rails, Django, Express or whatever is your web framework of choice. If you want real-time communications the easy way, I might have a better solution. Keep an eye on my blog for news to come soon! ;-)

• MRI Ruby 2.3.1: 57 seconds
• JRuby 9.1.1.0: 45 seconds
• Crystal 0.17.0: 59 seconds
• Elixir: 14 seconds

I also said how it was an unfair comparison as the Elixir version uses a different - and obviously more efficient - algorithm.

It was the “first-version-that-worked” so I decided to go ahead and improve the implementations. In the Ruby/JRuby version I added the thread gem to have a good enough implementation of a Thread pool that works for Ruby and JRuby. I probably should have used Concurrent-Ruby but I was having some trouble making FixedThreadPool work.

Anyway, now all versions will have a constant pool of requests running and we can make a better comparison.

Another thing that may have skewed the results against Crystal is that it seems to have a faulty DNS resolver implementation, so for now I just added the Manga Reader IP address directly to my /etc/hosts file to avoid getaddrinfo: Name or service not known exceptions.

To begin, let’s test the same Elixir implementation, and as expected the result is still the same:

11,49s user 0,82s system 77% cpu 15,827 total

Now, the MRI Ruby with the “batch burst” algorithm was taking 57 seconds and this new implementation using a ThreadPool runs much better:

12,67s user 0,92s system 50% cpu 27,149 total

In Crystal, it’s a bit more complicated as there is no way to implement the equivalent of a “Fiber Pool”. What we have to do is run an infinite loop until the last process signals a loop break. Within the loop we create a maximum number of fibers, wait for each one to signal that it finished through an individual channel and loop again to create a new fiber, and so on. Compared to yesterday’s 59 seconds, this is much better:

5,29s user 0,33s system 26% cpu 21,166 total

The JRuby version is not so fast though. Still better than yesterday’s 45 seconds but now it’s losing even to MRI:

49,24s user 1,41s system 146% cpu 34,602 total

I tried to use the --dev flag for faster start up time and it does improve a bit, getting it closer to MRI:

22,26s user 0,99s system 76% cpu 30,320 total

Not sure if it can be improved more though, any tips are welcome - don’t forget to comment below.

So, Elixir is still at least twice as fast as Crystal at this point. But this also demonstrates how a different algorithm does make a huge difference where it matters. I can probably tweak it a bit more but this should suffice for now.

Changing the Ruby implementation to use ThreadPool

require "thread/pool"

module MangaDownloadr
  class Concurrency
    ...
    def fetch(collection, &block)
      pool    = Thread.pool(@config.download_batch_size)
      mutex   = Mutex.new
      results = []

      collection.each do |item|
        pool.process {
          engine  = @turn_on_engine ? @engine_klass.new(@config.domain) : nil
          reply = block.call(item, engine)&.flatten
          mutex.synchronize do
            results += ( reply || [] )
          end
        }
      end
      pool.shutdown

      results
    end
    ...

The idea is that we will have a fixed number of spawned native threads (in this case, determined by @config.download_batch_size). As one thread finishes it will pop a new link from the collection Array, essentially working as a “queue” to be depleted.

The results are accumulated in the results Array. Because many threads may want to modify it at once we have to synchronize access through a Mutex.

This way we always have a fixed amount of workers performing requests constantly instead of slicing the collection Array and doing bursts as in the previous version.

Changing the Crystal implementation to simulate a Fibers Pool

The Crystal version became a bit more complicated as I didn’t find a pool library to pull in. I found a rough implementation of a pool in this stackoverflow post and I was able to implement an improved version into a new shard so you can take advantage of it in your projects. Check out the source code at akitaonrails/fiberpool. This is how I added it to my project:

dependencies:
  ...
  fiberpool:
    github: akitaonrails/fiberpool

And this is how I used it. Notice that the logic itself is very close to the MRI Ruby version, but using a “Fiber Pool” instead of a Thread Pool.

require "fiberpool"

module CrMangaDownloadr
  struct Concurrency
    def initialize(@config : Config, @turn_on_engine = true); end

    def fetch(collection : Array(A)?, engine_class : E.class, &block : A, E? -> Array(B)?) : Array(B)
      results = [] of B
      if collection
        pool = Fiberpool.new(collection, @config.download_batch_size)
        pool.run do |item|
          engine = if @turn_on_engine
                     engine_class.new(@config.domain, @config.cache_http)
                   end
          if reply = block.call(item, engine)
            results.concat(reply)
          end
        end
      end
      results
    end
  end
end

But again, this is very I/O intensive and both the Ruby and Crystal versions take advantage of the fact that they can do more work while waiting for one HTTP request to finish.

Reproducing the Tests

I implemented a “Test Mode” in all 3 implementations. You can clone from my repositories:

• Crystal version
• Ruby/JRuby version
• Elixir version

And you can run the test mode like this:

# crystal:
time ./cr_manga_downloadr --test

# MRI:
time bin/manga-downloadr --test

# JRuby (you have to edit Gemfile to uncomment the JRuby engine):
time jruby --dev -S bin/manga-downloadr --test

# Elixir:
time ./ex_manga_downloadr --test

This will run only the fetching of chapters, pages and image links, skipping the actual downloading of the images, optimization through mogrify and PDF compilation. Those skipped parts take too long and don’t say anything about the tested languages.

And if you want to test just the CPU intensive parts and avoid all networking interference altogether, you can turn on HTTP Cache mode and run the tests twice so the first run will cache everything first, like this:

# crystal:
time ./cr_manga_downloadr --test --cache

# MRI:
time bin/manga-downloadr --test --cache

# JRuby (you have to edit Gemfile to uncomment the JRuby engine):
time jruby --dev -S bin/manga-downloadr --test --cache

# Elixir:
time CACHE_HTTP=true ./ex_manga_downloadr --test

So, with all requests already cached these are the results:

Elixir:

7,13s user 0,24s system 331% cpu 2,227 total

MRI Ruby:

5.590000   0.180000   5.770000 (  5.678714)
5,87s user 0,21s system 101% cpu 5,996 total

JRuby:

10.580000   0.180000  10.760000 (  3.184472)
14,54s user 0,44s system 262% cpu 5,716 total

Crystal:

1.610000   0.050000   1.660000 (  1.344123)
1,62s user 0,06s system 124% cpu 1,350 total

The Ruby/JRuby/Crystal versions have internal benchmarks to take away startup time (this is why they have 2 lines of times).

So, Elixir is very fast. It takes roughly 2 seconds to parse all the 1,900 HTML files to find the links.

Ruby is the slowest, obviously. It takes almost 6 seconds.

The JRuby version also takes almost 6 seconds but internally it processes in 3 seconds, the rest is startup time and warming up of the JVM underneath.

And Crystal is the fastest, as you would expect because it’s a super optimized binary doing CPU bound operations, clocking in at a bit more than 1 second.

Conclusion

Even though the algorithms are roughly similar, Elixir is still winning by a very large margin in the total process (with the external HTTP requests).

There is more than just interpreter/compiler speeds, there is more than single-thread, multi-thread, fibers infrastructure.

We also have the maturity of the respective standard libraries (including TCP stack, HTTP client libraries, String/Array/Regex operations, etc) and 3rd party libraries (libXML, Nokogiri, etc). So there is a lot that can interfere with the tests. I’d guess that Crystal’s standard library, especially the networking parts, are not battle-tested enough at this point (pre 1.0!).

So, the summary with the new results is this:

• MRI Ruby 2.3.1: 27 seconds
• JRuby 9.1.1.0: 30 seconds
• Crystal 0.17.0: 21 seconds
• Elixir: 15 seconds

Let me know if you have ideas to make them even faster!

In summary, most people are just exposing their contents through simple JSON API endpoints and consuming them from other microservices through simple HTTP GETs. The more microservices they add to the chain, the longer the last endpoint takes to return. There are many techniques to improve this situation, but I want to show just a simple one that can solve many common situations without too much hassle.

First of all, if you’re dealing with caching, never try to expire cache entries. The most important thing to learn about caching is how to generate proper cache keys. Do it right and most problems with caching are gone.

Second, if you’re using HTTP, try to use everything you can from it, instead of reinventing the wheel.

The “TL;DR” version is: make your APIs return proper ETAGs and handle “If-None-Match” headers properly, return the correct 304 status code instead of full blown 200 with full content body in the responses every time. And in the consumer end, cache the ETAG with the corresponding response body and use it from cache when you receive 304s. You will save at least expensive rendering time in the consumed end and slow bandwidth from the consumer end. In the end you should be able to at least be 100% faster, or more, with just a few tweaks.

The Example Applications

In a very very contrived example we could have a Rails API controller like this:

## 1st application
class Api::ProductsController < ApplicationController
  def index
    @products = Product.page (params[:page] || 1)
    render json: [num_pages: @products.num_pages, products: @products]
  end
  ...
end

For the purposes of this contrived post example, we load it up on localhost port 3001. Now, whenever we call “http://localhost:3001/api/products?page=1” the API server dumps something like this in the log:

Started GET "/api/products?page=1" for 127.0.0.1 at 2016-03-23 13:29:34 -0300
Processing by Api::ProductsController#index as */*
  Parameters: {"page"=>"1"}
   (0.3ms)  SELECT COUNT(*) FROM "products"
  Product Load (0.9ms)  SELECT  "products".* FROM "products" LIMIT 100 OFFSET 0
Completed 200 OK in 26ms (Views: 23.0ms | ActiveRecord: 1.2ms)

In summary, it’s taking around 26ms to send back a JSON with the first page of products of this application. Not too bad.

Then we can create another Rails API application that consumes this first one. Something also very contrived and stupid like this:

# 2nd application
class Api::ProductsController < ApplicationController
  def index
    # never, ever, hard code hostnames like this, use dotenv-rails or secrets.yml
    url = "http://localhost:3001/api/products?page=?" % (params[:page] || "1")
    json = Net::HTTP.get_response(URI(url)).body

    response.headers["Content-Type"] = "application/json"
    render plain: json
  end
  ...

We load this other app in localhost port 3000 and when we call “http://localhost:3000/api/products?page=1” the server dumps the following log:

Started GET "/api/products?page=1" for 127.0.0.1 at 2016-03-23 13:31:59 -0300
Processing by Api::ProductsController#index as HTML
  Parameters: {"page"=>"1"}
  Rendered text template (0.0ms)
Completed 200 OK in 51ms (Views: 7.1ms | ActiveRecord: 0.0ms)

Now, this second application is taking twice the time compared to the first one. We can assume that part of those 51ms are the 26ms of the first application.

The more APIs we add on top of each other, the more time the entire flow will take. 26ms for the first, another 25ms for the second, and so on.

There are many things we could do. But I’d argue that we should start simple: by actually using a bit more of the HTTP protocol.

Sending proper ETAGs and handling “If-None-Match”

In a nutshell, we can tag any HTTP response with an ETAG, an identifier for the content of the response. If the ETAG is the same, we can assume the content hasn’t changed. Web browsers receive the ETAGs and send them back if we choose to refresh the content as a “If-None-Match” header. When a web server receives this header it compares against the ETAG of the response and doesn’t send back any content, just a “304 Not Modified” HTTP header, which is much, much lighter and faster to transport back.

An ETAG can be as complicated as an entire SHA256 hexdigest of the entire response content or as simple as just the “updated_at” timestamp if this indicates that the record has changed (in a “show” controller action, for example). It must be a digest that represents the content and it must change if the content changes.

Rails has had support for ETAGs for a long time in the ActionController::ConditionalGet API.

In our contrived example, the 1st application on port 3001 fetches a page of ActiveRecord objects and sends back an array represented in JSON format. If we choose to digest the final content we would have to let ActionView do its job, but it is by far the most expensive operation so we want to avoid it.

One thing that we could do is just check the “updated_at” fields of all the records and see if they changed. If any one of them changed, we would need to re-render everything and send a new ETAG and a new response body. So the code could be like this:

class Api::ProductsController < ApplicationController
  layout false
  def index
    @products = Product.page (params[:page] || 1)

    if stale?(freshness @products)
      render json: [num_pages: @products.num_pages, products: @products]
    end
  end
  ...
  private

  def freshness(collection)
    dates = collection.map(&:updated_at).sort
    etag = dates.map(&:to_i).reduce(&:+)
    {etag: Digest::MD5.hexdigest(etag.to_s), last_modified: dates.last, public: true}
  end

Now, when we try to “curl -I http://localhost:3001/api/products?page=1” we will see the following headers:

HTTP/1.1 200 OK 
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Etag: "ccf372c24cd259d0943fa3dc99830b10"
Last-Modified: Wed, 23 Mar 2016 16:25:53 GMT
Content-Type: application/json; charset=utf-8
Cache-Control: public
X-Request-Id: 601f22bc-72a9-4960-97cb-c30a0b56dbf4
X-Runtime: 0.053529
Server: WEBrick/1.3.1 (Ruby/2.3.0/2015-12-25)
Date: Wed, 23 Mar 2016 17:00:13 GMT
Content-Length: 0
Connection: Keep-Alive

Great! We have an ETAG that uniquely represents this page of products. Now we can go one step further and add the following:

# Gemfile
gem 'dalli'
gem 'rack-cache'

# config/environments/[development|production].rb
...
config.cache_store = :dalli_store
client = Dalli::Client.new
config.action_dispatch.rack_cache = {
  :metastore    => client,
  :entitystore  => client
}
config.static_cache_control = "public, max-age=2592000"
...

This configuration is assuming that we have Memcached installed and running in the same localhost machine (our development environment), but in production you can follow this good documentation from Heroku.

Now, our 1st application has an internal HTTP cache, with the same role as something more advanced such as Varnish in front of it. It will cache all HTTP 200 responses from the application, together with the ETAGs. Whenever a new call comes for the same URI, it will check the cache first, and if the application sends back the same ETAG, it will send the content back from the cache.

So if we call the above “curl” command multiple times, we will see this from the Rails server log:

Started GET "/api/products?page=1" for 127.0.0.1 at 2016-03-23 14:05:16 -0300
Processing by Api::ProductsController#index as */*
  Parameters: {"page"=>"1"}
  Product Load (0.8ms)  SELECT  "products".* FROM "products" LIMIT 100 OFFSET 0
  Couldn't find template for digesting: products/index
   (1.0ms)  SELECT COUNT(*) FROM "products"
Completed 200 OK in 31ms (Views: 16.3ms | ActiveRecord: 1.8ms)
cache: [HEAD /api/products?page=1] miss, store

Notice the last line: it says that it tried to find the returning ETAG in the cache and it “missed”, so it “stored” the new content. Now, if we run the same “curl” command again, we will see this:

Started GET "/api/products?page=1" for 127.0.0.1 at 2016-03-23 14:05:59 -0300
Processing by Api::ProductsController#index as */*
  Parameters: {"page"=>"1"}
  Product Load (0.5ms)  SELECT  "products".* FROM "products" LIMIT 100 OFFSET 0
  Couldn't find template for digesting: products/index
Completed 304 Not Modified in 12ms (ActiveRecord: 0.5ms)
cache: [HEAD /api/products?page=1] stale, valid, store

The simple curl command is not sending the “If-None-Match” header, so it expects to receive the full response body. But because we have Rack Cache it is adding the “If-None-Match” ETAG digest from the cache to the request before hitting the Rails app. The Rails app now compares the received “If-None-Match” digest through the “stale?” methods with the ETAG it just computed and because they match, it just sends an empty body response with the status code of 304. Rack Cache receives the 304 and fetches the cached JSON from Memcached and changes the HTTP response from the 304 to a normal 200 with the full body, which is what Curl can receive.

Because we are skipping the expensive ActionView rendering, the response time went from the previous 26ms to around 12ms: we are now twice as fast!

Consuming APIs with ETAGs

But we can go one step further. If we change nothing about the 2nd application, it will keep receiving just HTTP 200 with full body responses from Rack Cache of the 1st application. Let’s see the code again

# 2nd application
class Api::ProductsController < ApplicationController
  def index
    # never, ever, hard code hostnames like this, use dotenv-rails or secrets.yml
    url = "http://localhost:3001/api/products?page=?" % (params[:page] || "1")
    json = Net::HTTP.get_response(URI(url)).body

    response.headers["Content-Type"] = "application/json"
    render plain: json
  end
  ...

We can do better. How about the following:

# 2nd application - upgrade!
class Api::ProductsController < ApplicationController
  def index
    page = params[:page] || "1"
    url = "http://localhost:3001/api/products?page=?" % page
    # 1 - fetch the ETAG for the URL
    etag = Rails.cache.fetch(url)
    # 2 - fetch from external API or fetch from internal cache
    json = fetch_with_etag(url, etag)

    response.headers["Content-Type"] = "application/json"
    render plain: json
  end
  ...
  private

  def fetch_with_etag(url, etag)
    uri = URI(url)

    req = Net::HTTP::Get.new(uri)
    # 3 - add the important If-None-Match header
    req['If-None-Match'] = etag if etag

    res = Net::HTTP.start(uri.hostname, uri.port) {|http|
      http.request(req)
    }

    etag = res['ETAG']
    etag = etag[1..-2] if etag.present?
    if res.is_a?(Net::HTTPNotModified)
      # 4 - if got a 304, then we already have the content in the internal cache
      Rails.cache.read(etag)
    elsif res.is_a?(Net::HTTPSuccess)
      # 5 - if we got a 200 it's new content to store in internal cache before returning
      Rails.cache.write(url, etag)
      Rails.cache.write(etag, res.body)
      res.body
    else
      "{}"
    end
  end
end

I know, feels overwhelming, but it’s actually quite simple. Let’s go over it step-by-step:

1. First we see if we already have an ETAG associated with the URL we want to fetch (be aware of query parameters!)

2. Now we call the separated “fetch_with_etag” method

3. This is all boilerplate “Net::HTTP” setup, but the important piece is that we add the “If-None-Match” header if we found an ETAG for the URL in the cache.

4. After we make the external call we can have 2 responses. The first being the very very short, body-less, header-only, “304 Not Modified”. In this case, it means that we already have the full content in the internal cache and it is still valid, so we use it.

5. In case we receive the normal HTTP “200” status code, we either didn’t send any ETAG or the one we sent was invalidated and a new ETAG and content body were returned, so we must update them in our internal cache before exiting.

Now, the first time we call “curl http://localhost:3000/api/products?page=1” for the 2nd application endpoint we will see this log:

Started GET "/api/products?page=1" for 127.0.0.1 at 2016-03-23 14:14:05 -0300
Processing by Api::ProductsController#index as */*
  Parameters: {"page"=>"1"}
  Rendered text template (0.0ms)
Completed 200 OK in 62ms (Views: 5.6ms | ActiveRecord: 0.0ms)

Caches are cold, it is taking the same “around 50ms” like we had before, in this case, it’s more like 62ms.

Just to recap, this call to the 2nd application made it call the 1st application API, which shows the following in its log:

Started GET "/api/products?page=?" for 127.0.0.1 at 2016-03-23 14:14:05 -0300
Processing by Api::ProductsController#index as */*
  Parameters: {"page"=>"?"}
/"32b82ebbd99854ea2ca0d49ff4a7c07c
  Product Load (0.9ms)  SELECT  "products".* FROM "products" LIMIT 100 OFFSET 0
  Couldn't find template for digesting: products/index
   (1.2ms)  SELECT COUNT(*) FROM "products"
Completed 200 OK in 37ms (Views: 21.5ms | ActiveRecord: 2.2ms)
cache: [GET /api/products?page=?] miss, store

Cache miss, new content stored!

Now, we call “curl” against the same URL for the 2nd application again and we now see what we wanted in the log:

Started GET "/api/products?page=1" for 127.0.0.1 at 2016-03-23 14:14:10 -0300
Processing by Api::ProductsController#index as */*
  Parameters: {"page"=>"1"}
  Rendered text template (0.0ms)
Completed 200 OK in 24ms (Views: 0.3ms | ActiveRecord: 0.0ms)

Down from 62ms to 24ms!! And in the 1st application log we see:

Started GET "/api/products?page=?" for 127.0.0.1 at 2016-03-23 14:14:10 -0300
Processing by Api::ProductsController#index as */*
  Parameters: {"page"=>"?"}
"ccf372c24cd259d0943fa3dc99830b10", ccf372c24cd259d0943fa3dc99830b10
  Product Load (1.2ms)  SELECT  "products".* FROM "products" LIMIT 100 OFFSET 0
  Couldn't find template for digesting: products/index
Completed 304 Not Modified in 12ms (ActiveRecord: 1.2ms)
cache: [GET /api/products?page=?] stale, valid, store

A cache hit! Content is stale and valid, so return just 304, the 2nd application acknowledges and fetches the still valid content from its own cache and returns to Curl.

Conclusion

If you remove ETAGs from the 1st application, the 2nd one will not break and vice-versa, because it’s optional. If “ETAG” and “If-None-Match” headers are present in received HTTP response, we can use them, otherwise they will work as before.

If the 2nd application is itself another API you should also add ETAGs for it, and so on. In this example we didn’t, just because I wanted to simplify the scenario. But instead of being just a simple one-to-one proxy, it could be one of those “porcelain” APIs that fetch data from several other smaller microservices, compile them down into a single structure and return it. You should create ETAGs that could be the returning ETAGs from all the other microservices digested together in a single ETAG, for example. Because you’re just receiving headers and fetching their content from an internal cache, it’s quite cheap. Something like this pseudo-code:

def index
  url1 = "http://somehost1.foo/some_endpoint/1"
  url2 = "http://somehost1.foo/some_endpoint/2"
  etag1 = etag_from(url1)
  etag2 = etag_from(url2)
  etag = Digest::MD5.hexdigest(etag1 + etag2)
  if stale?(etag: etag, public: true)
    body1 = JSON.parse fetch_from(url1)
    body2 = JSON.parse fetch_from(url2)
    result = do_some_processing(body1, body2)
    render json: result.to_json
  end
end

Another thing: you can add any vanilla HTTP Cache between your microservices, to add authorization, security, or just plain extra caching, it’s just HTTP with proper headers. But the more you exchange “304” between your services, the less processing and the less bandwidth you’re spending. It should be noticeably efficient in most cases. But again, it’s not always cheap or trivial to generate the cache keys/ETAGs to begin with, so this is the point to take more care.

And if you’re creating heavy Javascript apps that also consume those APIs, I “believe” the Ajax calls properly cache HTTP content and send back the correct “If-None-Match” and in case they receive 304s, your application should get the normal “success” triggers. I didn’t test this when I was writing this post but I think this is the case indeed. So you should automatically get better performance in your front-end application for free if you add proper ETAGs in your APIs.

This is particularly useful for APIs that return data that doesn’t change too often. If it changes every second, or every minute, you should not see too much gains. But if it’s something like this example: products lists that only change once every day or every week, or ZIP code lists, or Previous Orders in an e-commerce. Any data that changes infrequently is a good candidate. And the larger the dataset, the larger the benefits you will see (if it’s a megabyte long listing, for example). As usual, this is also no Silver Bullet, but in this case it is not so much work to add ETAGs and there are near-zero side-effects, so why not?

ETAG is just one of many other HTTP features you should be using, CORS is another one (Research Rack Cors).

If you’re from Brazil, you should watch Nando Vieira’s entire course on the broad subject of Rails Caching.

To be honest, I’m not sure how effective this technique can actually be in all kinds of scenarios so I am very interested in hearing your feedback in case you use something like this in your applications.

It’s basically a complete solution on top of vanilla Rails so you can implement WebSocket based applications (the usual real time chats and notifications) with full access to your Rails assets (models, view templates, etc). For small to medium apps, this is a terrific solution that you might want to use instead of having to go to Node.js.

In summary you control Cable Channels that can receive messages sent through a WebSocket client wiring. The new Channel generator takes care of the boilerplate and you just have to fill in the blanks for what kinds of messages you want to send from the client, what you want to broadcast from the server, and to what channels that your clients are subscribed to.

For a more in-depth introduction, DHH himself published a bare-bones Action Cable screencast that you should watch just to get a feeling of what the fuzz is all about. If you watched it already and have experience in programming, you may have spotted the problem I mention in the title, so just jump to “The Problem” section below for a TL;DR.

In the end you will end up with a code base like the one I reproduced in my Github repository up until the tag “end_of_dhh”. You will have a (very) bare-bones single-room real time chat app for you to play with the main components.

Let’s just list the main components here. First, you will have the ActionCable server mounted in the “routes.rb” file:

# config/routes.rb
Rails.application.routes.draw do
  root to: 'rooms#show'

  # Serve websocket cable requests in-process
  mount ActionCable.server => '/cable'
end

This is the main server component, the channel:

# app/channels/room_channel.rb
class RoomChannel < ApplicationCable::Channel
  def subscribed
    stream_from "room_channel"
  end

  def unsubscribed
    # Any cleanup needed when channel is unsubscribed
  end

  def speak(data)
    Message.create! content: data['message']
  end
end

Then you have the boilerplate Javascript:

# app/assets/javascripts/cable.coffee
#= require action_cable
#= require_self
#= require_tree ./channels
#
@App ||= {}
App.cable = ActionCable.createConsumer()

And the main client-side Websocket hooks:

# app/assets/javascripts/channels/room.coffee
App.room = App.cable.subscriptions.create "RoomChannel",
  connected: ->
    # Called when the subscription is ready for use on the server

  disconnected: ->
    # Called when the subscription has been terminated by the server

  received: (data) ->
    $('#messages').append data['message']

  speak: (message) ->
    @perform 'speak', message: message

$(document).on "keypress", "[data-behavior~=room_speaker]", (event) ->
  if event.keyCode is 13
    App.room.speak event.target.value
    event.target.value = ''
    event.preventDefault()

The view template is a bare-bones HTML just to hook a simple form and div to list the messages:

<!-- app/views/rooms/show.html.erb -->
<h1>Chat room</h1>

<div id="messages">
  <%= render @messages %>
</div>

<form>
  <label>Say something:</label><br>
  <input type="text" data-behavior="room_speaker">
</form>

The Problem

In the “RoomChannel”, you have the “speak” method that saves a message to the database. This is already a red flag for a WebSocket action that is supposed to have very short-lived, light processing. Saving to the database is to be considered heavyweight, specially under load. If this is processed inside EventMachine’s reactor loop, it will block the loop and prevent other concurrent processing from taking place until the database releases the lock.

# app/channels/room_channel.rb
class RoomChannel < ApplicationCable::Channel
  ...
  def speak(data)
    Message.create! content: data['message']
  end
end

I would say that anything that goes inside the channel should be asynchronous!

To add insult to injury, this is what you have in the “Message” model itself:

class Message < ApplicationRecord
  after_create_commit { MessageBroadcastJob.perform_later self }
end

A model callback (avoid those as the plague!!) to broadcast the received message to the subscribed Websocket clients as an ActiveJob that looks like this:

class MessageBroadcastJob < ApplicationJob
  queue_as :default

  def perform(message)
    ActionCable.server.broadcast  'room_channel', message: render_message(message)
  end

  private

  def render_message(message)
    ApplicationController.renderer.render(partial: 'messages/message', locals: { message: message })
  end
end

It renders the HTML snippet to send back for the Websocket clients to append to their browser DOMs.

DHH even goes on to say “I’d like to show it because this is how most apps will end up.”

Indeed, the problem is that most people will just follow this pattern and it’s a big trap. So, what’s the solution instead?

The Proper Solution

For just the purposes of a simple screencast, let’s make a quick fix.

First of all, if at all possible you want your channel code to block as little as possible. Waiting for a blocking operation in the database (writing) is definitely not one of them. The Job is underused, it should be called straight from the channel “speak” method, like this:

# app/channels/room_channel.rb
 class RoomChannel < ApplicationCable::Channel
   ...
   def speak(data)
-    Message.create! content: data['message']
+    MessageBroadcastJob.perform_later data['message']
   end
 end

Then, we move the model writing to the Job itself:

# app/jobs/message_broadcast_job.rb
 class MessageBroadcastJob < ApplicationJob
   queue_as :default

-  def perform(message)
-    ActionCable.server.broadcast  'room_channel', message: render_message(message)
+  def perform(data)
+    message = Message.create! content: data
+    ActionCable.server.broadcast 'room_channel', message: render_message(message)
   end
   ...

And finally, we remove that horrible callback from the model and make it bare-bone again:

# app/models/message.rb
class Message < ApplicationRecord
end

This returns quickly, defers processing to a background job and should sustain more concurrency out-of-the-box. The previous, DHH solution, has a built-in bottleneck in the speak method and will choke as soon as the database becomes the bottleneck.

It’s by no means a perfect solution yet, but it’s less terrible for a very quick demo and the code ends up being simpler as well. You can check out this code in my Github repo commit.

I may be wrong in the conclusion that the channel will block or if this is indeed harmful to concurrency. I didn’t measure both solutions, it’s just a gut feeling from older wounds. If you have more insight into the implementation of Action Cable, leave a comment down below.

By the way, be careful before considering migrating your Rails 4.2 app to Rails 5 just yet. Because of the hard coded dependencies on Faye, Eventmachine, Rails 5 right now rules out Unicorn (even Thin seems to be having problems booting up). It also rules out JRuby and MRI on Windows as well because of Eventmachine.

If you want the capabilities of Action Cable without having to migrate, you can use solutions such as “Pusher.com”, or if you want your own in-house solution, follow my evolution on the subject with my mini-Pusher clone written in Elixir.

In Part 2 I will add the proper mechanisms to make a minimal core that is actually useful and deploy it to Heroku. In order to do that I need to implement the following:

• a simple Administration authentication (a hardcoded admin_username and admin_password will do for now)
• an admin restricted "/api/admin/apps" endpoint to manage new Applications. Each Application should have a randomly generated key and secret.
• the existing “/events” endpoint from Part 1 should be moved to "/api/apps/:app_id/events" and have access restricted to the authentication of the key and secret of the Application identified as “app_id”.
• for now, the Event will just broadcast to the appropriate topic. We want to be able to broadcast to everyone from an Application as well to specific topics within the Application scope.

As usual, the code for this section will be tagged as v0.2 in both the client demo and server-side Github repositories.

The App Resource

If this project of ours is to behave like Pusher.com, we need a way to create new “Applications”. Each client connecting to this service will be bound to this Application. Events should be restricted to the Application boundary. This is how we will isolate different clients connecting to the same server. So you can have one core serving several different web applications.

Once a new application is created, the client/consumer web app will have the pair of key and secret tokens that it will use to connect both the server-side triggers as well as the client-side Websocket listeners.

As a disclaimer, at this stage of development I will not implement any sophisticated authentication system such as OAuth2 or JWT. I will save this for posts to follow. For now I will use the Application’s key and secret just as simple username and password in an HTTP Basic Auth. This should be good enough for our purposes for the time being.

So, the very first step is to create such an “Application” resource and we can resort to Phoenix’s built-in JSON scaffold generator:

mix phoenix.gen.json App apps name:string slug:string key:string secret:string active:boolean

Most tutorials will show you the “phoenix.gen.html” generator, which behaves like Rails’s “scaffold”, generating HTML templates for each of the CRUD verbs. This is similar but it skips HTML and assumes this is going to be a JSON CRUD API.

We need to manually update the “web/router.ex” file like this:

# web/router.ex
scope "/api", ExPusherLite do
  pipe_through :api
  post "/apps/:app_slug/events", EventsController, :create
  scope "/admin" do
    resources "/apps", AppController, except: [:new, :edit]
  end
end

The “EventsController” is the one we implemented in Part 1 and that we will overhaul during this Part 2.

The generator gave us this new “AppController”, and similarly to Rails’ routes, the DSL is remarkably similar here. If you’re a Railer I bet you can instantly recognize the routes this DSL is generating.

The generator also created a proper migration for us:

# priv/repo/migrations/20151210131528_create_app.exs
defmodule ExPusherLite.Repo.Migrations.CreateApp do
  use Ecto.Migration

  def change do
    create table(:apps) do
      add :name, :string
      add :slug, :string
      add :key, :string
      add :secret, :string
      add :active, :boolean, default: false

      timestamps
    end
    create index(:apps, [:name], unique: true)
    create index(:apps, [:slug], unique: true)
  end
end

Again, remarkably similar to ActiveRecord’s Migration DSL. Migrations behave as you expect. You must run:

mix ecto.create # if you haven't already
mix ecto.migrate

This App resource will need the ability to create slugs out of the names (which we will use as “app_id”) and also generate random key and secret values. So we must add these dependencies to the “mix.exs” file:

# mix.exs
defp deps do
  [...,
   {:secure_random, "~> 0.2.0"},
   {:slugger, "~> 0.0.1"}]
end

The final App model is quite long, so I will break it down for you:

# web/models/app.ex
defmodule ExPusherLite.App do
  use ExPusherLite.Web, :model
  alias ExPusherLite.Repo

  schema "apps" do
    field :name, :string
    field :slug, :string
    field :key, :string
    field :secret, :string
    field :active, :boolean, default: true

    timestamps
  end

  @required_fields ~w(name)
  @optional_fields ~w()
  ...

This block declares the model Schema. Be careful if you generate a migration and then change its fields settings: you must remember to update the schema in the model. In my first attempt I didn’t include the “slug” field, so I rolled back the database migration (with “mix ecto.rollback”), changed the migration to add the “slug” field and re-ran the “ecto.migrate” task.

I was puzzled with the model not picking up the new field; after some time I remembered that Ecto models don’t attempt to fetch the real database schema and generate accessors dynamically, instead it relies on the explicitly declared schema block as shown above. After I added the new “slug” field in the schema block, then the model would properly use the new field.

# web/models/app.ex
...
def get_by_slug(slug) do
  Repo.get_by!(__MODULE__, slug: slug, active: true)
end
def hashed_secret(model) do
  Base.encode64("#{model.key}:#{model.secret}")
end
...

These are just helper functions to use in the AppController. The odd bit might be “MODULE” but this is just a shortcut for the atom representation of the current module, which is “ExPusherLite.App”. This is how you make a simple query to the model, it resembles Rails’ “App.get_by(slug: slug, active: true)”.

In Elixir convention, Ecto has functions with and without bangs ("get_by!" and “get_by”). If you want to catch an error you use the version without bangs and it will return either a “{:ok, result}” tuple or a “{:error, result}” and you can pattern match them. Or you can use the bang version and it will raise an exception. Depends on what you want to do.

# web/models/app.ex
...
def changeset(model, params \\ :empty) do
  model
  |> cast(params, @required_fields, @optional_fields)
  |> validate_length(:name, min: 5, max: 255)
  |> unique_constraint(:name)
  |> generate_key
  |> generate_secret
  |> slugify
  |> unique_constraint(:slug)
end
...

Second only to the Schema block I mentioned above, this “changeset/2” function is the most important part of a Model.

In Rails you just have the concept of a “Model” which is considered “Fat” because it deals with database operations, business logic and framework hooks all in the same place. In Phoenix you have to deal with at least 3 different concepts:

• You have a Repo, which receives a Changeset and uses it to insert or update the designated rows in the database table. You will see “Repo.get” or “Repo.insert”, not “App.find” or “App.save”.
• Then you have the Changeset which are just Elixir Maps (the ones with the syntax “%{key => value}”). It’s just a Hash, a Dictionary, a collection of key-value pairs. A Repo accepts Maps for its operations.
• Finally, you have the Model which actually give meaning and context to the raw Changesets. And the function “changeset/2” above is the one responsible for receiving a raw Map, piping it through a validation and transformation chain and return a valid Changeset for the Repo to use.

So, in a controller you will usually find code like this:

# web/controllers/app_controller.ex
...
def create(conn, %{"app" => app_params}) do
  changeset = App.changeset(%App{}, app_params)
  
  case Repo.insert(changeset) do
    {:ok, app} ->
      conn
      |> ...
    {:error, changeset} ->
      conn
      |> ...
  end
  ...

This is how you create a new, validated, changeset and then pass it to the Repo, treating the results in a pattern match block. Just compare the above changeset line with the beginning of the “changeset/2” function:

...
def changeset(model, params \\ :empty) do
  model
  |> cast(params, @required_fields, @optional_fields)
  |> validate_length(:name, min: 5, max: 255)
...

It maps the “%App{}” empty record to the “model” argument and the “app_params” that comes from the request (a map of the format “%{name => ‘foo’, active: => ’true’}”) to the argument “params”. Then it pipes the model and params to the “cast/4” function which will copy the values from the params map to the model map/changeset. And it keep passing the resulting changeset to the following functions, such as “validated_length/3” below, and so on. If the chain ends with no exceptions, you end up with a clean, validated changeset that you can just pass to the Repo to blindly insert to the database.

In the above implementation we are chaining filters to generate the key, secret and slug, and this is the implementation as private functions:

# web/models/app.ex
...
  defp generate_key(model) do
    if get_field(model, :key) do
      model
    else
      model
      |> put_change(:key, SecureRandom.uuid)
    end
  end

  defp generate_secret(model) do
    if get_field(model, :secret) do
      model
    else
      model
      |> put_change(:secret, SecureRandom.uuid)
    end
  end

  defp slugify(model) do
    if name = get_change(model, :name) do
      model
      |> put_change(:slug, Slugger.slugify_downcase(name))
    else
      model
    end
  end
end

The logic is set so new key/secret are generated only if the fields are empty and a new slug is generated only if the name has changes. And this is it, I told you the model code would be a bit large. You can see how to use the Slugger and SecureRandom libraries we added in the “mix.exs” before.

I also want to add the equivalent of a Rails seed file to create a test application so it’s easier for newcomers to know what to do. Phoenix has seeds and you can implement it like this:

# priv/repo/seeds.exs

alias ExPusherLite.App
alias ExPusherLite.Repo

# not using the App.changeset should just avoid all validations and generations
Repo.insert! %App{ slug: "test-app", name: "Test App", key: "test-app-fake-key", secret: "test-app-fake-secret", active: true }

Remember how I detailed the role of the “changeset/2” function in creating a clean and validated changeset, which is just a Map? You can skip that function altogether and hand craft your own final Map and pass it to the Repo. The Repo doesn’t care if this is a valid Map or not it will just try to insert it into the database regardless. And in this case the App Model prevents us from hardcoding keys and secrets, so this is how we do it in a seed file.

We can just run it directly like this:

mix run priv/repo/seeds.exs

The AppController just needs 2 changes. The first is to search the App through the slug field instead of the default ‘id’ field. This is simple enough, we just replace all calls to “app = Repo.get!(App, id)” to “app = App.get_by_slug(id)”, which is why we implemented this function in the model above.

The second thing is Authentication.

Adding Authentication

Now that we have an App model that can generate secure random UUIDs for key and secret, I will add a second level of authentication for administrators to be able to create such Apps.

For that I will just hard-code a secret in the config file of the application itself to serve as a development default. Like this:

# config/config.exs
...
config :ex_pusher_lite, :admin_authentication,
  username: "pusher_admin_username",
  password: "pusher_admin_password"
...
import_config "#{Mix.env}.exs"

You must add this block before the “import_config” function. Then you can override those values in the “config/prod.secret.exs” file, for example, like this:

# config/prod.secret.exs
...
config :ex_pusher_lite, :admin_authentication,
  username: "14e86e5fee3335fa88b0",
  password: "2b94ff0f07ce9769567f"

Of course, generate your own pair of secure username and password and replace it in the production environment if you intend to actually use this. For Heroku, we will still have to tweak this further, so keep this in mind.

Just to make the process easier, I also added the following helper function:

# lib/ex_pusher_lite.ex
...
  # Return this applicaton administration Basic HTTP Auth hash
  def admin_secret do
    admin_username = Application.get_env(:ex_pusher_lite, :admin_authentication)[:username]
    admin_password = Application.get_env(:ex_pusher_lite, :admin_authentication)[:password]
    secret = Base.encode64("#{admin_username}:#{admin_password}")
  end
end

This is how you fetch the configuration values. I am generating a simple Base64 encoded string out of the username concatenated with the password with a comma, which is what Basic HTTP Auth requires. I will use this admin hash for the “AppController” and each client must provide the key/secret in its own App instance to be able to trigger the “EventsController”.

For both controllers I will create a single Authentication Plug, like this:

# lib/ex_pusher_lite/authentication.ex
defmodule ExPusherLite.Authentication do
  import Plug.Conn

  alias ExPusherLite.App

  def init(assigns \\ [admin: false]), do: assigns

  def call(conn, assigns) do
    token =
      if assigns[:admin] do
        ExPusherLite.admin_secret
      else
        params = fetch_query_params(conn).params
        params["app_slug"] |> App.get_by_slug |> App.hashed_secret
      end

    "Basic " <> auth_token = hd(get_req_header(conn, "authorization"))
    if Plug.Crypto.secure_compare(auth_token, token) do
      conn
    else
      conn |> send_resp(401, "") |> halt
    end
  end
end 

As I explained in previous articles, a Plug is like a chainable Rails Middleware or even a Rack application. It must have a single “call/2” that receives a Plug.Conn structure and returns it back, allowing to form a chain/pipeline of Plugs.

We check if we want to compare with the Admin token or the App token and then retrieve the Basic HTTP authorization token that’s in the HTTP request connection structure (we retrieve individual header values through the “get_req_header/2” function). Finally we make a secure compare between the tokens.

To enable this plug in the controllers we just add it like this:

# web/controllers/app_controller.ex
defmodule ExPusherLite.AppController do
  use ExPusherLite.Web, :controller

  alias ExPusherLite.App
  plug ExPusherLite.Authentication, [admin: true]
  ...

 defmodule ExPusherLite.EventsController do
   use ExPusherLite.Web, :controller

-  plug :authenticate
+  plug ExPusherLite.Authentication
   ...

In Part 1 we had a simpler “plug :authenticate” in the EventsController. We can remove it and also the “authenticate/2” function. We just refactored it into a better function that also serves administration authentication now, but the idea is the same.

This is it: the basics for API authentication. Again, this is not the best solution as the username/password pair goes in the URL and it’s open to man-in-the-middle attacks. SSL only encrypts the HTTP body but the URL is still open.

For example, if an administrator wants to create a new application, he must do the following:

curl --data "app[name]=foo-app" http://pusher_admin_username:pusher_admin_password@localhost:4000/api/admin/apps

And this would be one example of the resulting JSON representation of the new app:

{"data":{"slug":"foo-app","secret":"8ef69064-0d7e-c9ef-ac14-b6b1db303e7a","name":"foo-app","key":"9400ad21-eed8-117a-bce5-845262e0a09e","id":5,"active":true}}%

With this new key and secret in hand, we can update our client demo to make use of the new app.

Configuring the Client Demo

We must start by adding the proper Application details in the “.env” file:

PUSHER_URL: "localhost:4000"
PUSHER_APP_ID: "foo-app"
PUSHER_KEY: "9400ad21-eed8-117a-bce5-845262e0a09e"
PUSHER_SECRET: "8ef69064-0d7e-c9ef-ac14-b6b1db303e7a"
PUSHER_CHANNEL: "foo-topic"

We must also tweak the “config/secrets.yml” to reflect the new metadata (development, test, and production must follow this):

development:
  secret_key_base: ded7c4a2a298c1b620e462b50c9ca6ccb60130e27968357e76cab73de9858f14556a26df885c8aa5004d0a7ca79c0438e618557275bdb28ba67a0ffb0c268056
  pusher_url: <%= ENV['PUSHER_URL'] %>
  pusher_app_id: <%= ENV['PUSHER_APP_ID'] %>
  pusher_key: <%= ENV['PUSHER_KEY'] %>
  pusher_secret: <%= ENV['PUSHER_SECRET'] %>
  pusher_channel: <%= ENV['PUSHER_CHANNEL'] %>
  ...

And we can create an initializer to make it easier to use this metadata properly:

# config/initializers/pusher_lite.rb
module PusherLite
  def self.uri
    key    = Rails.application.secrets.pusher_key
    secret = Rails.application.secrets.pusher_secret
    app_id = Rails.application.secrets.pusher_app_id
    url    = Rails.application.secrets.pusher_url

    uri = "http://#{key}:#{secret}@#{url}/api/apps/#{app_id}/events"
    URI.parse(uri)
  end
end 

Again, the Rails app will trigger the ExPusherLite server using the Basic HTTP Auth. Do not be fooled into thinking this is “secure”, it just “feels a bit secure through obscurity”. You have been warned, wait for the next articles on this subject. But this is usable in controlled environments.

To finalize the upgrades, we must change the client-side access to the new metadata, first changing the application layout:

<!-- app/views/layouts/application.html.erb -->
...
+  <meta name="pusher_host" content="<%= Rails.application.secrets.pusher_url %>">
-  <meta name="pusher_key" content="<%= Rails.application.secrets.pusher_key %>">
+  <meta name="pusher_app_id" content="<%= Rails.application.secrets.pusher_app_id %>">
   <meta name="pusher_channel" content="<%= Rails.application.secrets.pusher_channel %>">
...

The javascript “index.es6” fetches from this meta headers, so we must change them there:

# app/assets/javascripts/application/pages/home/index.es6
...
     let guardianToken = $("meta[name=guardian-token]").attr("content")
     let csrfToken     = $("meta[name=guardian-csrf]").attr("content")
 
+    let pusherHost    = $("meta[name=pusher_host]").attr("content")
-    let pusherKey     = $("meta[name=pusher_key]").attr("content")
+    let pusherApp     = $("meta[name=pusher_app_id]").attr("content")
     let pusherChannel = $("meta[name=pusher_channel]").attr("content")
 
-    let socket = new Socket("ws://localhost:4000/socket", {
+    let socket = new Socket(`ws://${pusherHost}/socket`, {
       params: { guardian_token: guardianToken, csrf_token: csrfToken }
     })
     socket.connect()
 
     // Now that you are connected, you can join channels with a topic:
-    let channel = socket.channel(pusherChannel, {})
+    let channel = socket.channel(`public:${pusherApp}`, {})
     channel.join()
       .receive("ok", resp => { console.log("Joined successfully", resp) })
       .receive("error", resp => { console.log("Unable to join", resp) })
 
-    channel.on("msg", data => {
+    channel.on(`${pusherChannel}:msg`, data => {
       let new_line = `<p><strong>${data.name}<strong>: ${data.message}</p>`
       $(".message-receiver").append(new_line)
     })
+
+    channel.on("msg", data => {
+      let new_line = `<p><strong>Broadcast to all channels</strong>: ${data.message}</p>`
+      $(".message-receiver").append(new_line)
+    })
   }

One important modification from Part 1 is that the WebSocket host was hardcoded to “localhost” and here we are making it configurable through the meta tags. Right now, for localhost tests, we are using the plain “ws://” protocol but when we deploy to Heroku we will change it to “wss://” for SSL. Same thing for the “PusherLite” initializer. Keep that in mind.

Now it’s subscribing to a different format of topic/channel. In Part 1 it would be something like: “public:test_channel” now we are listening to “public:foo-app”, so the application is the Websocket subscription “topic”.

Then we are changing the socket listener to listen for 2 different events. The first one is in the format “test_channel:msg”. So this is how we must now send messages to a specific “channel” within an “app/topic”.

And last we still listen to the old “msg” event, but this serves as a “broadcast” event for all connected clients subscribed to this particular “foo-app” Application. Now web clients can listen to specific “channels” within the “app” but also receive system wide “broadcast” messages. This is a big improvement and it didn’t require much on the Javascript side.

But what more does it take to make this “channel-only and broadcast” system work? First, we start changing the web form to allow a user to choose between sending a channel-only message or a broadcast, like this:

<!-- app/views/home/index.html.erb -->
...
     <%= f.text_field :name, placeholder: "Name" %>
     <%= f.text_field :message, placeholder: "Message" %>
+    <%= f.check_box :broadcast %>
     <%= f.submit "Send message", class: "pure-button pure-button-primary" %>
   </fieldset>
...

Now the EventsController must accept this new parameter:

# app/controllers/events_controller.rb
...
  def event_params
    params.require(:pusher_event).permit(:name, :message, :broadcast)
  end
end

Finally, the Model must use this new information before posting to the ExPusherLite server:

# app/models/pusher_event.rb
class PusherEvent
  include ActiveModel::Model

  attr_accessor :name, :message, :broadcast
  validates :name, :message, presence: true

  def save
    topic = if broadcast == "1"
              "#general"
            else
              Rails.application.secrets.pusher_channel
            end

    Net::HTTP.post_form(PusherLite.uri, {
      "topic" => topic,
      "event" => "msg",
      "scope" => "public",
      "payload" => {"name" => name, "message" => message}.to_json
    })
  end
end

I am just assuming a hard-coded “#general” string to serve as the broadcast trigger for the server. Now we must make the server accept this new protocol schema, so let’s go back to Elixir.

First we must start with the counterpart for the previous POST trigger, ExPusherLite.EventsController:

# web/controllers/events_controller.ex
 defmodule ExPusherLite.EventsController do
   use ExPusherLite.Web, :controller
 
-  plug :authenticate
+  plug ExPusherLite.Authentication
 
-  def create(conn, params) do
-    topic = params["topic"]
-    event = params["event"]
+  def create(conn, %{"app_slug" => app_slug, "event" => event, "topic" => topic, "scope" => scope} = params) do
     message = (params["payload"] || "{}") |> Poison.decode!
-    ExPusherLite.Endpoint.broadcast! topic, event, message
+    topic_event =
+      if topic == "#general" do
+        event
+      else
+        "#{topic}:#{event}"
+      end
+    ExPusherLite.Endpoint.broadcast! "#{scope}:#{app_slug}", topic_event, message
     json conn, %{}
   end
   ...

The first difference is that I am pattern matching from the arguments directly to the “topic” and “event” variables. This function is also aware of the “#general” string the client can send to indicate an app-wide broadcast. And the new topic is the concatenation of “topic” and “event” to allow for “channel-only” messages.

To connect this all to the WebSocket handler, we must make the following changes:

# web/channels/room_channel.ex
-  def handle_in("msg", payload, socket = %{ topic: "public:" <> _ }) do
-    broadcast socket, "msg", payload
+  def handle_in(topic_event, payload, socket = %{ topic: "public:" <> _ }) do
+    broadcast socket, topic_event, payload
     { :noreply, socket }
   end
 
-  def handle_in("msg", payload, socket) do
+  def handle_in(topic_event, payload, socket) do
     claims = Guardian.Channel.claims(socket)
     if permitted_topic?(claims[:publish], socket.topic) do
-      broadcast socket, "msg", payload
+      broadcast socket, topic_event, payload
       { :noreply, socket }
   ...

Now, the Channel does not pattern match on a specific event, it lets it through without further validation, trusting that the EventsController is doing the right thing. I will come back to this piece for improvements in the future, possibly.

Deploying our first Phoenix app to Heroku!

In this section we will just follow the official documentation, so read it if you want more details.

Let’s get started:

heroku apps:create your-expusherlite --buildpack "https://github.com/HashNuke/heroku-buildpack-elixir.git"
heroku buildpacks:add https://github.com/gjaldon/heroku-buildpack-phoenix-static.git

I am naming the application “your-expusherlite” but you should change it to your own name, of course. And the rest of the configuration data are all examples that you must change for your own needs.

Heroku relies on environment variables. So we start by erasing “config/prod.secret.exs” and change “config/prod.exs” to look like this:

config :ex_pusher_lite, ExPusherLite.Endpoint,
  http: [port: {:system, "PORT"}],
  url: [scheme: "https", host: "your-expusherlite.herokuapp.com", port: 443], force_ssl: [rewrite_on: [:x_forwarded_proto]],
  cache_static_manifest: "priv/static/manifest.json",
  secret_key_base: System.get_env("SECRET_KEY_BASE")

# Configure your database
config :ex_pusher_lite, ExPusherLite.Repo,
  adapter: Ecto.Adapters.Postgres,
  url: System.get_env("DATABASE_URL"),
  pool_size: 20

config :ex_pusher_lite, :admin_authentication,
  username: System.get_env("PUSHER_ADMIN_USERNAME"),
  password: System.get_env("PUSHER_ADMIN_PASSWORD")

# remove this line:
# import_config "prod.secret.exs"

Now we must configure the environment variables “SECRET_KEY_BASE”, “PUSHER_ADMIN_USERNAME” and “PUSHER_ADMIN_PASSWORD”. Use the included “mix phoenix.gen.secret” to generate those.

heroku config:set SECRET_KEY_BASE="`mix phoenix.gen.secret`"
heroku config:set PUSHER_ADMIN_USERNAME="FPO0QUkqbAP6EGjElqBzDQuMs8bhFS3"
heroku config:set PUSHER_ADMIN_PASSWORD="n78DPGmK3DBQy8YAVyshiGqcXjjSXSD"

Then it’s just a matter of waiting for the good old “git push heroku master” to finish compiling everything for the first time. And because this is the first deploy you should not forget to run “heroku run mix ecto.migrate” to create the database table.

Now, if I did everything right, as an Administrator that knows the above hardcoded secrets I should be able to create a new Application like this:

curl --data "app[name]=shiny-new-app" https://FPO0QUkqbAP6EGjElqBzDQuMs8bhFS3:n78DPGmK3DBQy8YAVyshiGqcXjjSXSD@your-expusherlite.herokuapp.com/api/admin/apps

And this is the result I got!

{"data":{"slug":"shiny-new-app","secret":"42560373-0fe1-506e-28ca-35ab5221fb3d","name":"shiny-new-app","key":"958c16e7-ab93-dac0-0fc6-6cb864e26358","id":1,"active":true}}

Great, now that we have a valid Application key and secret we can configure our Rails Client Demo and deploy it to Heroku as well.

Deploying the Rails Client to Heroku

This is a simple Rails application, we can just create the app and deploy right away:

heroku create your-expusherlite-client
heroku config:set PUSHER_URL=your-expusherlite.herokuapp.com
heroku config:set PUSHER_APP_ID=shiny-new-app
heroku config:set PUSHER_KEY=958c16e7-ab93-dac0-0fc6-6cb864e26358
heroku config:set PUSHER_SECRET=42560373-0fe1-506e-28ca-35ab5221fb3d
heroku config:set PUSHER_CHANNEL=shiny-new-topic
git push heroku master

I’m assuming the readers of this post already know how to configure a Rails app properly for Heroku. Just to mention it, I configure this app with the 12 factor and Puma gems and added a proper Procfile. Another very small change was changing the “pusher_lite.rb” initializer to create a URI with “https” because the ExPusherLite we deployed to production requires SSL by default.

There is one more caveat. Being led by experienced web programmers, they made sure that, unlike this bare-bones exercise here, the Phoenix framework itself is secure. One such example is to disallow Websocket connections from different hosts.

Out of the box, the “phoenix.js” Socket will fail connection when we try to connect from the “your-expusherlite-client.herokuapp.com” Rails app host to the Phoenix app in “your-expusherlite.herokuapp.com” with the following error:

WebSocket connection to 'wss://your-expusherlite.herokuapp.com/socket/websocket?guardian_token=N_YCG6hGK7…iOlsicHVibGljOioiXX0._j6s2LiaKde9rBhnTMxDkm0XV5u89pNh1AdLFY6Rlt8&vsn=1.0.0' failed: Error during WebSocket handshake: Unexpected response code: 403

And in the Phoenix log we will see this very helpful message:

[error] Could not check origin for Phoenix.Socket transport.

This happens when you are attempting a socket connection to
a different host than the one configured in your config/
files. For example, in development the host is configured
to "localhost" but you may be trying to access it from
"127.0.0.1". To fix this issue, you may either:

  1. update [url: [host: ...]] to your actual host in the
     config file for your current environment (recommended)

  2. pass the :check_origin option when configuring your
     endpoint or when configuring the transport in your
     UserSocket module, explicitly outlining which origins
     are allowed:

        check_origin: ["https://example.com",
                       "//another.com:888", "//other.com"]

Unless you know what a Cross-Site Web Socket Hijacking is, you will prefer to keep the default settings as they are. In a green-field Phoenix app, the Web part will connect to the Web Socket in the same app and, therefore, in the same host, so this is not an issue.

In this case I am making a separate micro-service to mimic Pusher.com behavior so it should be able to accept Web Socket connections from different hosts.

If you control the applications being created, you will likely prefer to make the “check_origin” setting read from your database for the exact hosts. As a feature for next time I could add a “host” field in the “App” model and use it to validate connections in the transport configuration. For the time being I will just make it accept any hosts:

# web/channels/user_socket.ex
defmodule ExPusherLite.UserSocket do
  use Phoenix.Socket

  ## Channels
  channel "*", ExPusherLite.RoomChannel

  ## Transports
  transport :websocket, Phoenix.Transports.WebSocket, check_origin: false
  ...

And this is it! Now the Rails app should be able to connect and send messages! And you should be able to create any number of new apps and connect all of them to this same service.

Conclusion

Right now, we have a functional, albeit bare-bones, Pusher.com clone that will work for any number of use cases where Pusher.com would be used.

As I warned many times before, the security part is still flaky and needs work. I will still extend on what Daniel began with Guardian to authenticate Web Socket users to private channels as well. And the core should also receive auditing and reporting capabilities (to be able to report usage, number of active connections, throughput of events, keep at least a short history of events so new connections can retrieve the last sent messages, and so on).

But from here it’s a matter of adding features to an already working core. And this is nothing more than Phoenix out-of-the-box without too much added on top of it! It says a lot about the current state of maturity of this very capable framework.

In terms of performance, for this very simple example, I spun up 1 free Heroku dyno for each app.

The Rails app is able to respond to the front-end user interface in around 2ms. And the Sucker Punch job - which does the heavy HTTP POST to ExPusherLite - takes in the order of 30ms or less.

The Phoenix server receives the HTTP POST and performs the broadcast in around less than 6ms. Also quite fast. The times will vary a lot because I believe the free dyno is not only slow but also stays in highly shared metal boxes, getting impact from other neighbor apps running in the same box.

Because we already have an administrative API to create and manage apps (create new ones, delete, update, etc) we can already create a separate application in any other framework to make a dashboard for admins or for a self-serving front-end for developers to register new apps and receive key/secret pair to add in their own applications.

Both the ExPusherLite server and the demo client are deployed in Heroku and you can test the client right now clicking here. The admin keys are different from what I showed in this post, of course, so you won’t be able to create new apps, but you can deploy it yourself to your own environment.

As a Rails developer there are a few things we can’t do in Rails. One of them is to deal with real-time messaging.

Rails 5 will bring Action Cable, and it might be good enough for most cases. It uses Faye, which in turn is based on Eventmachine. You can implement a good enough solution for Websockets using Faye in your Rails 4.2 app right now.

Another option is to avoid the trouble altogether and use a messaging service. One option I always recommend for zero friction is to use Pusher.com.

The Basics

You will want to clone from my example repository, like this:

git clone https://github.com/akitaonrails/pusher_lite_demo
cd pusher_lite_demo
git checkout tags/v0.1 -b v0.1
bundle

This is a very very simple implementation of a real-time, websocket-based, chat using Pusher. The idea goes like this:

We start by having a front-end Form to send messages

<!-- app/views/home/index.html.erb -->
<%= form_for @event, url: events_path, remote: true, html: {class: "pure-form pure-form-stacked"} do |f| %>
  <fieldset>
    <legend>Send your message remotely</legend>
    <%= f.text_field :name, placeholder: "Name" %>
    <%= f.text_field :message, placeholder: "Message" %>
    <%= f.submit "Send message", class: "pure-button pure-button-primary" %>
  </fieldset>
<% end %>

It’s using Rails built-in jQuery support for Ajax posting the form to the “EventsController#create” method:

# app/controllers/events_controller.rb
class EventsController < ApplicationController
  def create
    SendEventsJob.perform_later(event_params)
  end

  def event_params
    params.require(:pusher_event).permit(:name, :message)
  end
end

Just to annotate the process, the “routes.rb” looks like this:

# config/routes.rb
Rails.application.routes.draw do
  resources :events, only: [:create]

  root 'home#index'
end

The HTML layout looks like this:

<!-- app/views/layout/application.html.erb -->
<!DOCTYPE html>
<html>
<head>
  <title>Pusher Lite Demo</title>
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <meta name="pusher_key" content="<%= Rails.application.secrets.pusher_key %>">
  <meta name="pusher_channel" content="<%= Rails.application.secrets.pusher_channel %>">
  <%= stylesheet_link_tag    'application', media: 'all', 'data-turbolinks-track' => true %>
  <%= javascript_include_tag 'application', 'data-turbolinks-track' => true %>
  <script src="https://akitaonrails.github.io//js.pusher.com/3.0/pusher.min.js"></script>
  <%= csrf_meta_tags %>
</head>
<body>

<div class="pure-menu pure-menu-horizontal">
    Pusher Client Demo
...
</div>

<div class="pure-g-r">
  <div class="pure-u-1-3 message-form">
    <%= yield %>
  </div>
  
  <div class="pure-u-1-3 message-receiver">
  </div>
</div>

</body>
</html>

This layout imports the default “application.js” which configures Pusher, establishes the Websocket connection and subscribes to messages on a specific topic with specific events:

// app/assets/javascript/application.js
//= require jquery
//= require jquery_ujs
//= require turbolinks
//= require_tree .

$(document).on("page:change", function(){
  var pusherKey = $("meta[name=pusher_key]").attr("content");
  var pusher = new Pusher(pusherKey, { encrypted: true });

  var pusherChannel = $("meta[name=pusher_channel]").attr("content");
  var channel = pusher.subscribe(pusherChannel);
  
  channel.bind('new_message', function(data) {
    var new_line = "<p><strong>" + data.name + "<strong>: " + data.message + "</p>";
    $(".message-receiver").append(new_line);
  });
});

It gets the configuration metadata from the layout meta tags which grabs the values from “config/secrets.yml”:

development:
  secret_key_base: ded7c4a2a298c1b620e462b50c9ca6ccb60130e27968357e76cab73de9858f14556a26df885c8aa5004d0a7ca79c0438e618557275bdb28ba67a0ffb0c268056
  pusher_url: <%= ENV['PUSHER_URL'] %>
  pusher_key: <%= ENV['PUSHER_KEY'] %>
  pusher_channel: test_chat_channel

test:
  secret_key_base: f51ff494801ff0f9e1711036ef6f2f6f1e13544b02326adc5629c6833ae90f1a476747fae94b792eba8a444305df8e7a5ad53f05ea4234692ac96cc44f372029
  pusher_url: <%= ENV['PUSHER_URL'] %>
  pusher_key: <%= ENV['PUSHER_KEY'] %>
  pusher_channel: test_chat_channel

# Do not keep production secrets in the repository,
# instead read values from the environment.
production:
  secret_key_base: <%= ENV["SECRET_KEY_BASE"] %>
  pusher_url: <%= ENV['PUSHER_URL'] %>
  pusher_key: <%= ENV['PUSHER_KEY'] %>
  pusher_channel: <%= ENV['PUSHER_CHANNEL'] %>