And make sure to subscribe to my new newsletter The M.Akita Chronicles!

–

Unit tests are every developer’s comfort zone. You mock everything, isolate everything, run it in milliseconds, and the green in the terminal gives you a false sense of security. Until the day you deploy and discover that service A writes in a format service B can’t read. Or that the Hacker News RSS switched from Atom to RSS and your parser blew up. Or that Yahoo Finance started returning 429 after the third ticker because the rate limit that worked in tests doesn’t exist in real life.

Unit tests prove the pieces work in isolation. Integration tests prove the system works. And in a project with multiple applications sharing data via the filesystem, the difference between the two is the difference between “the code compiles” and “the newsletter lands in the inbox”.

The Problem: Three Apps, One Filesystem, Zero Guarantees

The project has an unusual architecture: three Rails/Hugo applications in a monorepo, integrated through a shared content/ directory:

akitando-news/
├── marvin-bot/    # Gera conteúdo (Discord bot, AI, scraping)
├── newsletter/    # Monta e envia a newsletter
├── blog/          # Hugo + Hextra → Netlify
├── podcast-tts/   # Servidor TTS (Python/FastAPI)
└── content/       # Markdown + YAML, a cola entre tudo

marvin-bot writes Markdown with YAML frontmatter into content/anime/, content/hacker_news/, etc. The newsletter reads those files to assemble the newsletter. The blog serves the same content as static pages. The podcast reads the assembled newsletter to generate the dialogue script.

The contract between these systems is implicit. There’s no schema. No versioned API. No protobuf. The contract is: “the Markdown file has this frontmatter, with these fields, in this structure”. If marvin-bot changes the format of a field — say, switches image from string to array — the newsletter breaks silently. No unit test in either app detects it.

That’s the fundamental flaw of testing distributed systems with unit tests: you’re testing each side of the contract separately, but never the contract itself.

Layer 1: Unit Tests With Paranoid Isolation

Before talking about integration, I need to explain what the unit tests do in this project — because without that foundation, the integration tests would have nothing to stand on.

The 977 marvin-bot tests run in parallel using every core on the machine. Each test worker has its own directory:

parallelize(workers: :number_of_processors)

parallelize_setup do |worker|
  Rails.application.config.content_dir =
    Rails.root.join("tmp", "test_content_w#{worker}").to_s
end

But worker-level isn’t enough. Inside each worker, multiple tests run sequentially — and if one test leaves a file in content/stories/, the next test might find spurious data. The solution: one unique directory per test:

setup do
  @base_content_dir = Rails.application.config.content_dir
  @test_content_dir = File.join(@base_content_dir, SecureRandom.hex(8))
  Rails.application.config.content_dir = @test_content_dir

  FileUtils.mkdir_p(File.join(@test_content_dir, "stories"))
  FileUtils.mkdir_p(File.join(@test_content_dir, "newsletters"))
  FileUtils.mkdir_p(File.join(@test_content_dir, "images"))
end

teardown do
  FileUtils.rm_rf(@test_content_dir) if @test_content_dir
  Rails.application.config.content_dir = @base_content_dir if @base_content_dir
end

Pay attention to the teardown line that restores @base_content_dir. Without it, the next test would create its directory inside the previous test’s directory — tmp/test_content_w3/abc123/def456/ — and by test 50 you’d have a path 50 levels deep. I learned this burning an entire afternoon debugging why the tests kept getting slower as the suite progressed.

This pattern — worker-level base + per-test subdirectory + restore on teardown — is identical across both apps. And it’s what lets you run 977 tests in 7 seconds with 32 parallel workers.

The golden rule: the test content_dir must NEVER point to the real ../content. If it does, the teardown’s rm_rf wipes out your production prompts. This isn’t hypothetical — it happened during development.

Layer 2: The Integration Environment

Unit tests mock everything: external APIs, AI, Discord, SES. That’s correct — you don’t want to burn $2 in GPT-5.2 tokens every time you run bin/rails test. But it means the tests never exercise the real pipeline.

The project has a third environment beyond development and test: integration. It’s not test. It’s not staging. It’s a specific mode for running the full pipeline with real data, using real APIs, but without sending actual emails:

# config/environments/integration.rb

# Jobs rodam inline (perform_later vira perform_now)
config.active_job.queue_adapter = :inline

# Emails salvos em arquivo ao invés de enviados via SES
config.action_mailer.delivery_method = :file
config.action_mailer.file_settings = {
  location: Rails.root.join("tmp/integration_emails")
}

The queue_adapter = :inline is crucial. In production, perform_later enqueues the job into SolidQueue and a separate worker executes it. In integration, it runs right there, in the same process. That turns the asynchronous production pipeline into a synchronous pipeline you can run, observe, and debug sequentially.

SES is swapped for writing to disk — every email becomes an HTML file in tmp/integration_emails/. After running the pipeline, you open those HTMLs in the browser to visually validate that the email renders correctly.

Layer 3: DevCache — LLMs Aren’t Idempotent

Here’s a problem anyone working with AI in production knows: calling GPT-5.2 with the same prompt twice doesn’t return the same answer. And every call costs money. If you’re iterating on the integration pipeline — running, tweaking a prompt, running again — spending $3 per iteration gets expensive fast.

The solution is DevCache: a file-based cache that only activates in the integration env:

module DevCache
  CACHE_TTL = 1.day
  CACHE_ENVS = %w[integration].freeze

  def self.fetch(namespace, key)
    return yield unless enabled?

    path = cache_path(namespace, key)
    if path.exist? && File.mtime(path) > CACHE_TTL.ago
      return JSON.parse(path.read, symbolize_names: true)
    end

    result = yield
    path.write(JSON.generate(result))
    result
  end
end

In production and in test, DevCache.fetch runs the block directly. In integration, it saves the result as JSON and returns the cached value on the next run. TTL of 1 day — enough to iterate within the same day without recomputing, short enough to not serve stale data.

The generation code uses it like this:

summary = DevCache.fetch("summarize", url_key) do
  AiChat.summarize(article[:content], source_url: url)
end

First run: calls GPT-5.2, pays $0.02, saves to tmp/dev_cache/summarize/{md5}.json. Second run: reads from the cache, zero cost. FORCE=1 busts the cache when you want to force regeneration — as an explicit flag, not a default.

DevCache exists in both apps (marvin-bot and newsletter) with identical implementation. When you run integration:clean, both caches are wiped.

The point: LLM caching in a development environment is a requirement, not an optimization. Without it, iterating on a prompt that needs 8 API calls becomes financially unworkable.

Layer 4: Real Data via Rsync

This is where most projects stop. They have unit tests, maybe a staging environment, and ship it. The problem: staging uses test data. Test data is too clean. It doesn’t have the edge cases real data has:

• Articles with titles in Japanese (anime ranking)
• URLs that redirect 3 times before reaching the destination
• Images that return 200 with Content-Type text/html (CloudFlare error page)
• Previous newsletters with slightly different frontmatter formats
• Stories with score: null because they were created before the field existed

The solution: rsync from production.

# Limpa conteúdo gerado (preserva stories e imagens de produção)
cd marvin-bot && bin/rails integration:clean

# Sincroniza content/ de produção pra local
rsync -avz --delete production:/path/to/content/ ../content/

# Roda o pipeline completo com dados reais
RAILS_ENV=integration bin/rails integration:pipeline

integration:clean is surgical: it removes generated directories (anime/, hacker_news/, newsletters/, podcasts/) but preserves stories/ and images/ — which are user-submitted data coming in via Discord. After the rsync, the local directory is an exact copy of production.

The pipeline then generates every section from that real data. If a parser breaks on a Korean title, you find out here — not in production at 5pm on a Sunday when the 8 jobs run in parallel.

Layer 5: The Full Pipeline

The heart of the integration system is integration:pipeline. It simulates the entire production week in a single run:

RAILS_ENV=integration bin/rails integration:pipeline

The pipeline runs in waves, honoring dependencies:

Wave 1 (paralelo, 8 jobs):
  Book, Holidays, History, Geek History,
  Anime, Hacker News, YouTube, World Events

Wave 2 (depende de World Events):
  Market Recap

Wave 3 (depende de tudo):
  Closing Remarks

Preflight → Newsletter Assembly → Podcast

Each wave runs its jobs in parallel threads, exactly like production does with SolidQueue. Parallel execution isn’t an implementation detail here — it’s a test feature. If two jobs try to write to the same file, or if one job reads a file another is in the middle of writing, the bug shows up here.

threads = to_run.map do |name, klass|
  Thread.new do
    ActiveRecord::Base.connection_pool.with_connection do
      run_single_job(name, klass, target_date, results)
    end
  end
end
threads.each(&:join)

with_connection is necessary because each thread needs its own SQLite connection — without it, connections leak and the pool gets exhausted.

At the end, the pipeline prints a status table and a billing summary:

That billing summary isn’t vanity — it’s cost control. If a refactored prompt triples the output tokens, you see it here before going to production and burning 10x on a Sunday with 200 stories.

Layer 6: Preflight — Automated Structural Validation

After generating every section, the pipeline runs ContentPreflight. It doesn’t validate the content — it validates the structure. Each section gets checked:

• Does the file exist? If content/anime/2026-02-16.md doesn’t exist, something failed.
• Minimum item count? The anime section needs at least 3 items. If it has 1, the generator had a problem.
• Required markers? The Markdown must contain [COMMENTARY] and [AKITA] — without them, newsletter assembly doesn’t know where to cut.
• Specific data? Market recap needs at least 5 ticker rows.

The result is a status per section: pass, degraded, fail, or skip. The pipeline uses this to decide whether the newsletter can be assembled automatically or needs human intervention:

In production, this result goes to Discord as an embed. In the integration pipeline, it’s printed to the terminal. The logic is the same — the notification channel changes.

Layer 7: Cross-App — Newsletter Assembly

Here’s where the monorepo pays dividends. marvin-bot needs to invoke the newsletter to assemble the final newsletter. In production, these are different servers. In the integration pipeline, it’s a system() call:

env_vars = "RAILS_ENV=integration CONTENT_DIR=#{Shellwords.escape(content_dir)}"
cmd = "cd #{newsletter_dir} && #{env_vars} bin/rails integration:generate_all"
system(cmd)

The newsletter reads the files marvin-bot just generated in content/, assembles the newsletter, and saves the result in content/newsletters/. CONTENT_DIR is passed explicitly — both apps point to the same directory.

If the frontmatter format changed and the newsletter can’t parse it, the error shows up here. If the newsletter’s SectionParser has diverged from marvin-bot’s SectionParser (they’re identical copies, but maintained by hand), here’s where you find out.

After the newsletter, the podcast: the pipeline invokes podcast:integration, which generates the dialogue script from the assembled newsletter, synthesizes audio via TTS, and builds the final MP3. It’s the longest pipeline — from Markdown newsletter to Spotify-ready MP3, passing through a two-pass LLM and per-turn voice synthesis.

The Value of Testing With Production Data

Let me give concrete examples of bugs that only showed up with real data:

1. Japanese anime titles broke the slugify. ContentWriter.slugify used Rails’ parameterize, which strips non-ASCII characters. Title: “進撃の巨人 Season 4” → slug: “season-4”. Two anime with the subtitle “Season 4” collided on the filesystem. It never showed up in tests because the fixtures used English titles.

2. The HackerNews parser assumed RSS, but the feed switched to Atom. rss/channel/item became feed/entry. Unit tests mocked the XML. The parser itself was never tested against the real feed. Running integration with the real rsynced feed, the parser returned zero items — caught by preflight as fail.

3. Yahoo Finance returned 429 after the 5th ticker. Rate limiting didn’t exist in tests (HTTP was mocked). In the real integration, the first 5 tickers went through, then everything failed. The fix — sleeping between requests — would never have been implemented without real data exposing the problem.

4. Images with a valid URL but the wrong Content-Type. A CDN returned 200 for image URLs, but with an HTML body (Cloudflare error page). UrlValidator.reachable? returned true. ImageProcessor.download_image downloaded the HTML and tried to resize it. mini_magick crashed with an obscure error. The fix: validate Content-Type on the HEAD response — but this scenario would never surface with mocked URLs.

5. Newsletter assembly failed on stories with score: null. Old stories, created before the score field had a default, had nil in the frontmatter. Assembly tried to do sort_by { |s| s[:frontmatter]["score"] } — nil <=> "high" raises ArgumentError. It only appeared when I rsynced production data that had 3 months of accumulated stories.

None of these bugs would be caught by a unit test. All of them were caught by the integration pipeline with real data.

Practical Tips

1. Separate Cost From Fidelity

The full integration pipeline costs around $0.40 in API calls. Not much, but if you run it 10 times in a day while iterating, that’s $4. DevCache solves this: the first run pays, the following runs are free.

But there are scenarios where you want to pay again — when you change a prompt and need to see the real result. FORCE=1 busts the cache selectively. And SKIP_EXISTING=1 skips sections that have already been generated, saving time when you only want to re-run a specific section.

2. The Integration Environment Is Not Staging

Staging is a server running 24/7 simulating production. The integration environment is a local execution mode you turn on when needed and turn off when you’re done. No dedicated infrastructure required. No separate database required. Same code, same machine, different flags.

The difference is philosophical: staging tests “does the system work on the staging server?”. Integration tests “does the system work with this specific data?”. The first is environment-dependent. The second is data-dependent. And data bugs are more common than environment bugs.

3. Clean Before Rsync, Not After

Order matters: clean the generated content before syncing from production. If you do the opposite — rsync first, clean after — you lose the production data you just downloaded.

integration:clean is surgical by design. It removes anime/, hacker_news/, newsletters/ (generated), but preserves stories/ and images/ (production). That enables an iterative workflow: rsync once, run the pipeline, tweak a prompt, run again — without re-downloading 500MB of content.

4. Preflight Is Your Smoke Test

Don’t trust “ran without exception = correct”. A generation job can run, produce a Markdown with 2 items instead of 10, and return success. Preflight catches that by verifying minimum counts, required markers, and expected structure.

In practice, every LLM bug I’ve found was caught by preflight: truncated responses, wrong format, missing sections. GPT-5.2 is good, but it isn’t deterministic — and when it fails, it fails in creative ways no schema validation would catch.

5. Billing Summary Is Not Optional

If you’re using LLMs in production, every test pipeline needs to report cost. Not for accounting — for engineering. A prompt that works but costs 3x more than the previous one is a performance bug. Input and output tokens per provider, cost per job, total cost — all visible on every run.

6. Monorepo Helps, But Doesn’t Solve

Having the three apps in the same repository helps: a git bisect covers changes on both sides of the contract. A PR that changes the frontmatter format in marvin-bot can also update the parser in newsletter — in the same commit.

But “can” isn’t “guarantees”. The contract between the apps is file-based and implicit. There’s no shared type, no common interface. SectionParser is copied manually between the two apps. If someone edits one and forgets the other, they diverge. The integration test is the only verification that both sides agree.

7. Preview Before Sending

The pipeline doesn’t end at generation. The newsletter has a newsletter:preview that renders the final result as HTML in light and dark mode:

cd newsletter
FILE=../content/newsletters/2026-02-16.md bin/rails newsletter:preview
# → tmp/preview/newsletter_light.html
# → tmp/preview/newsletter_dark.html

Opening those HTMLs in the browser is the visual integration test. The Markdown can be perfect, the frontmatter correct, every field present — but an anime image shows up stretched because the aspect ratio changed. Or Marvin’s commentary block doesn’t have enough contrast in dark mode. Those bugs are visual — no automated test catches them.

8. The CI Pipeline Is Different From the Integration Pipeline

These are different things. CI runs on every commit:

bin/ci  # rubocop + bundler-audit + brakeman + tests (~22s)

Integration runs when you want to validate the pipeline end-to-end — before a deploy, after changing prompts, or after rsyncing fresh data from production. CI is fast and cheap. Integration is slow and costs money. Both are necessary.

The Cost of Not Having Integration

Let me be direct: if this project didn’t have the integration pipeline, I would have shipped at least 3 broken newsletters.

One with the anime section missing because the XML parser assumed RSS and the feed was Atom. One with every YouTube image pointing to 404 because the thumbnail URL pattern changed. And one with the market recap showing last month’s tickers because Yahoo Finance blocked the requests and the job failed silently.

Unit tests would have passed in all those scenarios. CI would have gone green. The deploy would have shipped. And at 7am on a Monday, 300 people would have received a newsletter full of holes.

The integration pipeline is expensive — in setup time, in API cost, in maintenance complexity. But it’s orders of magnitude cheaper than sending broken content to real subscribers. Especially when you only get one shot a week.

Conclusion

The testing hierarchy in this project is:

1. Unit tests (1,330 across both apps, 7 seconds in parallel, $0): prove each piece works in isolation
2. CI pipeline (rubocop + audit + brakeman + tests, ~22 seconds, $0): prove the code is healthy
3. Integration pipeline (real data, real APIs, ~3 minutes, ~$0.40): proves the system works
4. Visual preview (HTML in the browser, manual, $0): proves the final result is presentable

Each layer catches a different type of bug. None of them is enough alone. And the one that surprises most — the one that catches the bugs you swore didn’t exist — is the third: real data, real pipeline, no mocks.

The confidence you have in your system is proportional to how real the data in your tests is. Mock everything and you get 100% coverage and zero confidence. Test with production data and you get the truth — however inconvenient.

I have been working very hard the past few weeks, quite literally coding non-stop, 7 days a week. Remember that “stop complaining you don’t have time, what do you do from midnight to 6 AM?” thing? Yeah, won’t apply to me.

And it’s exactly when you’re sleep deprived and exhausted that you start to make mistakes. Even though your code runs, locally, you will shoot yourself in the foot in one of those 3 AM coding spree sessions.

The test suite was really my sidekick, co-pilot, bringing me back to my senses whenever the night got too dark.

I maintain my own GitLab custom server for all of my company’s client and internal projects. Just a bit of paranoia, to make sure my code is owned by me alone. And GitLab is a fantastic tool. Even more so because it has its own Continuous Integration companion.

Just add your .gitlab-ci.yml file to your project, create a new branch, push it and create a Merge Request and the CI kicks in, with multiple parallel jobs if you want. Even when I was forgetting to run my tests, the CI would not let me fail.

Keeping a CI up and running does require you to stop and maintain your specs though. Sometimes you will wonder, why does a test run on your local machine but keeps failing in the CI server?

That’s when it’s super useful to run the CI docker image locally, to iron out remaining environment-dependent quirks.

And you can do just that by running the GitLab CI Runner itself. It will pick up your project’s .gitlab-ci.yml and run it locally through docker. So whatever problem you’re seeing on the server will quite definitely happen locally as well. With the added benefit that you don’t need to wait in line in case there is a queue of jobs waiting to run (many developers working, not as many machines to chew the CI work immediately). And you won’t be the one adding useless jobs to the same queue.

And, if you’ve followed my recommendations and installed the awesome Arch Linux distro (or my personal favorite derivative, Manjaro Gnome) you can easily install the runner through AUR like this:

pacaur -S gitlab-runner

Otherwise you will have to check your particular distro repos or download the binary from here. For example:

wget https://gitlab-runner-downloads.s3.amazonaws.com/master/binaries/gitlab-runner-linux-amd64
sudo mv gitlab-runner-linux-amd64 /usr/local/bin
sudo chmod +x /usr/local/bin/gitlab-runner

Once you have it installed, let’s say you have a snippet like this in your .gitlab-ci.yml configuration:

backend:
  stage: test
  script:
    - ./bin/cc-test-reporter before-build
    - bundle exec rspec --exclude-pattern "**/features/**/*_spec.rb"
  after_script:
    - ./bin/cc-test-reporter after-build --exit-code $? || true

You can configure many test jobs to run in parallel. I recommend that you separate front-end related JS tests, Rspec/minitest back-end unit tests, Capybara-based feature tests, Brakeman related security checks, for instance. Each will run in parallel and you won’t have to wait for everything to run if you’re only interested in the JS tests, for example. So you have faster feedback and can start fixing immediately.

Locally, from your project root directory, just run:

gitlab-runner exec docker backend

Of course, this is assuming that you already have Docker installed and properly configured. If you don’t, make sure you read Arch Wiki’s excellent entry on Docker but it’s basically installing the Docker package and adding yourself to the docker group.

This is it! Super easy, barely an inconvenience, and runner will do everything your GitLab CI server is doing.

One caveat is that unfortunately, it seems like you don’t have the ability to save cache or artifacts between job runs (gems, npm packages, etc) so every time this local runner runs, it will start from scratch. This makes it less useful to run all the time, but it’s still faster than pushing to the server and waiting in line for your turn in a busy server.

Saved me a couple times already.

But WebKit is known for being very complicated to deal with. So I can only imagine the nightmare to maintain PhantomJS, which is akin to maintaining a full-blown web browser like Chrome or Safari.

So it’s no wonder that when the Chrome team announced the availability of the Chrome Driver, then the maintainer of PhantomJS decided to step down.

If you know the contributors of PhantomJS, say thank you, as it helped us build more solid user features.

That being said, fear not. You can easily replace Poltergeist/PhantomJS with Selenium WebDriver/Chrome Driver in your RSpec/Capybara setup.

My friend Lucas Caton wrote about it in June this year. Follow his blog as well.

If you’re using Linux with the Chromium browser, you don’t need to install anything, as the Chrome Driver comes with Chromium. Otherwise, you need to install the proper packages for your operating system. For example, brew install chromedriver for OS X or pacaur -S chromedriver on Arch if you don’t like to have Chromium around. You may need to tweak your PATH on Ubuntu though.

Rule of thumb: install Chromium.

In my case, this is what had to change:

# Gemfile
- gem "poltergeist"
+ gem "selenium-webdriver"
+ gem "rspec-retry"

Then in the Capybara setup:

Capybara.server = :puma # Until your setup is working
Capybara.server = :puma, { Silent: true } # To clean up your test output

- Capybara.register_driver :poltergeist do |app|
-   options = {
-     timeout: 3.minutes,
-     phantomjs_options: [
-       '--proxy-type=none',
-       '--load-images=no',
-       '--ignore-ssl-errors=yes',
-       '--ssl-protocol=any',
-       '--web-security=false'
-     ]
-   }
-   Capybara::Poltergeist::Driver.new(app, options)
- end
- Capybara.javascript_driver = :poltergeist

+ Capybara.register_driver :chrome do |app|
+   capabilities = Selenium::WebDriver::Remote::Capabilities.chrome(
+     chromeOptions: {
+       args: %w[ no-sandbox headless disable-popup-blocking disable-gpu window-size=1280,1024]
+     }
+   )
+ 
+   Capybara::Selenium::Driver.new(app, browser: :chrome, desired_capabilities: capabilities)
+ end
+ 
+ Capybara::Screenshot.register_driver :chrome do |driver, path|
+   driver.save_screenshot(path)
+ end
+ 
+ Capybara.javascript_driver = :chrome

Capybara.default_max_wait_time = 5 # you may want to increase this timeout if your app is heavy to load

In feature specs, sometimes Rails itself takes a long while to load up, compile assets, etc., and the first feature spec may time out. To avoid a failure in the test run, it’s recommended to add the rspec-retry gem, as I did above, and add the following to your spec/rails_helper.rb:

require 'rspec/retry'

RSpec.configure do |config|
  ...
  # show retry status in spec process
  config.verbose_retry = true
  # Try twice (retry once)
  config.default_retry_count = 2
  # Only retry when Selenium raises Net::ReadTimeout
  config.exceptions_to_retry = [Net::ReadTimeout]
  ...
end

And that should be it. I didn’t have to touch any of my feature specs and they all ran beautifully. So kudos to the respective teams that maintain Capybara, Selenium-WebDriver for supporting this.

If you’re a Node.js developer as well, you probably used something like Casper, which is said to support Chrome Headless as well. But while we’re at it, you should check out Puppeteer as well, from the Google team itself. It is a promise-based library where you can code like this:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://example.com');
  await page.screenshot({path: 'example.png'});

  await browser.close();
})();

So yeah, Chrome Headless seems like a very good option as most users actually use the Chrome browser, so it means we should have more reliable feature specs and also web crawling tools.

For each application you create, it even offers you separate development, staging, and production environments, with separate key/secret tokens.

One problem I stumbled upon these days is that I was quickly zeroing out the free development environment message quota (200,000 messages a day). That’s because all my team was using the same key and also the Continuous Integration server was doing live connections whenever it ran in the same environment. That can quickly consume all the quota you have and block both your development and CI.

It’s actually not a good practice to do live connections to external systems in testing situations. The tests can fail randomly for a number of reasons, so we should always mock those. But mocking a complex system (WebSockets and HTTP) like Pusher is not trivial.

Fortunately, I found this Pusher Fake. It basically implements all the necessary APIs and WebSocket endpoints to mimic Pusher and fool both server and js/client to communicate with it.

The idea is for your Rails app, in development mode, to fork a separate server process for the Pusher client to connect to. This gem is both a simple Pusher-clone server and a series of wrappers to load it up in your setup.

First things first.

In your application, you will have both a Ruby side Pusher client connection setup to push messages to a channel in the Pusher server. And a Javascript, client-side, Pusher instance mainly to subscribe to a channel in a WebSocket connection and receive messages.

First we need to set up the Ruby side. Usually it’s in a config/initializers/pusher.rb configuration like this:

Pusher.app_id = ENV['PUSHER_APP']
Pusher.key    = ENV['PUSHER_KEY']
Pusher.secret = ENV['PUSHER_SECRET']
Pusher.cluster = ENV['PUSHER_CLUSTER']
# never set Pusher.host or Pusher.port

Notice that I am using environment variables to hold the configuration. You should use something like the figaro gem or the dotenv-rails gem. For example:

PUSHER_APP: "xpto"
PUSHER_KEY: "abcd1234"
PUSHER_SECRET: "abcd1234"
PUSHER_CLUSTER: "us2"

At the very least you must have an application ID, a key, a secret, and a cluster name. All of these are provided by Pusher whenever you register a new application there.

Second, we need to set up the Javascript instance. Usually, you have something in the assets/javascripts directory like this:

// .js.erb example
window.pusher = new Pusher(<%= ENV['PUSHER_APP] %>, {
  cluster: <%= ENV['PUSHER_KEY'] %>,
  encrypted: <%= ENV['PUSHER_ENCRYPTED'] %>})

In a Chrome Development Console, you can inspect this instance by typing:

Pusher.instances[0]

This way you can make sure it’s picking up the correct configurations for the connection and also debug problems.

The dependencies are the pusher gem in your Gemfile and the javascript client.

# Gemfile
gem 'pusher'

In the case of the javascript file you can either add it to your project and rely on Webpacker:

yarn add pusher

Then, in your ES6 javascript file, you can do:

const Pusher = require('pusher-js');

Or you can link it directly in your layout:

<script src="https://js.pusher.com/4.2/pusher.min.js"></script>

For more information on the Pusher-js client, read its README file.

Adding Pusher Fake

At this point, you should be able to connect to the real Pusher account and see the real-time magic happening.

You’re also already consuming the free quota you have available in your development environment on Pusher. For most people, this should suffice.

But we want to NOT connect to Pusher over the internet and keep everything local for development and testing. Let’s start by adding the Pusher Fake to our Gemfile:

group :development, :test do
  gem 'pusher-fake'
end

Now, this is where the Pusher Fake setup can get convoluted if you don’t understand what’s happening. As I said before, PusherFake must fork a new process to load a local server that mimics Pusher.

To load it up you must point to the local server. Remember our config/initializers/pusher.rb ? We just need to require a simple file like this:

Pusher.app_id = ENV['PUSHER_APP']
Pusher.key    = ENV['PUSHER_KEY']
Pusher.secret = ENV['PUSHER_SECRET']
Pusher.cluster = ENV['PUSHER_CLUSTER']

if Rails.env.development?
  require "pusher-fake/support/base"
end

This alone presents a LOT of problems if you’re not careful. This require will fork a new process. If you’re using a single-process web server like Webrick or Thin, you should be ok. If you’re using Puma, Unicorn, or Passenger with a maximum of just one process, you should also be good. But if you load a web server that itself forks new processes, you will have trouble.

In practice, I’d rather load the Pusher Fake server separately, in stand-alone mode. Fortunately it provides a command line command to start it up. And it’s good practice to set that up in a Procfile.dev file and use foreman to start everything for you. The Procfile.dev looks like this:

web: bundle exec rails s -p 3000
db: postgres -D /usr/local/var/postgres
redis: redis-server /usr/local/etc/redis.conf
mailcatcher: mailcatcher -f
pusherfake: pusher-fake -i ${PUSHER_APP:-xpto} --socket-host 0.0.0.0 --socket-port ${PUSHER_WS_PORT:-45449} --web-host 0.0.0.0 --web-port ${PUSHER_PORT:-8888} -k ${PUSHER_KEY:-abcd1234} -s ${PUSHER_SECRET:-abcd1234}

As a bonus, look how I configure other services such as PostgreSQL, Redis, etc.

If you didn’t know, you can use ${VARIABLE_NAME:-default_value} to use an environment variable or have a default value in case the variable doesn’t exist. This means that your environment variable configured with Figaro or Dotenv must have the same values.

PUSHER_APP: "xpto"
PUSHER_KEY: "abcd1234"
PUSHER_SECRET: "abcd1234"
PUSHER_CLUSTER: "us2"
PUSHER_HOST: "127.0.0.1"
PUSHER_PORT: "8888"
PUSHER_WS_HOST: "127.0.0.1"
PUSHER_WS_PORT: "45449"

Now your config/initializers/pusher.rb should be something like this:

Pusher.app_id = ENV['PUSHER_APP']
Pusher.key    = ENV['PUSHER_KEY']
Pusher.secret = ENV['PUSHER_SECRET']
Pusher.cluster = ENV['PUSHER_CLUSTER']

if Rails.env.development?
  Pusher.host = ENV['PUSHER_HOST']
  Pusher.port = ENV['PUSHER_PORT']
end

And Pusher-js config somewhere in your app/assets/javascripts/ directory will resemble something like this:

<% if defined?(PusherFake) %>
    <% if Rails.env.test? %>
    var pusher = <%= PusherFake.javascript(cluster: ENV["PUSHER_CLUSTER"]) %>
    <% else %>
    window.pusher = new Pusher(<%= ENV['PUSHER_KEY'] %>, {
      cluster: <%= ENV['PUSHER_CLUSTER'] %>,
      wsHost: <%= ENV['PUSHER_WS_HOST'] %>,
      wsPort: <%= ENV['PUSHER_WS_PORT'] %>,
      encrypted: <%= ENV['PUSHER_ENCRYPTED'] %>})    
    <% end %>
<% else %>
    window.pusher = new Pusher(<%= ENV['PUSHER_KEY'] %>, {
      cluster: <%= ENV['PUSHER_CLUSTER'] %>,
      encrypted: <%= ENV['PUSHER_ENCRYPTED'] %>})
<% end %>

Remember that this is a Javascript+ERB file, so we can fetch the same environment variables for the configuration.

Now, whenever you foreman start -f Procfile.dev -p 3000 it will load the Pusher Fake server with the proper development configurations and both your ruby server-side and javascript client-side should connect to it without any problems.

Also, notice the if Rails.env.test? bit. This is for your RSpec test suite. In the case of the testing environment, we won’t load the fake server manually, instead, we will create something like spec/support/pusher-fake.rb with:

require "pusher-fake/support/rspec"

And that’s it. The PusherFake.javascript will define default WebSocket connection configuration and the require above will both fork the fake server and set Rspec to clean the channels on each test run (through PusherFake::Channel.reset).

This way your testing environment will also avoid connecting to the external, real, Pusher server.

The key to all this is the environment variables. You must make sure that every piece is loading the same configuration, otherwise you will have the fake server binding to a different port than the Pusher-js is connecting to, and you will have errors. Debug with care.

Most importantly: if you did it all correctly, you are now independent of the real Pusher server for development and testing environments, you won’t ever reach any quota limits, and your team and your CI will be able to work uninterruptedly, with a deterministic behavior.

That would be the quick answer if you ask any developer how to share business logic between applications.

Although it makes sense in many situations, it’s not a good answer all the time.

My TL;DR for some situations is that you can organize your model logic as ActiveSupport::Concerns (or plain Ruby Modules if you will) and move them out to a Rubygem that your applications can consume.

Notice that I am only speaking of Models, not Controllers or Views. To share those you would need a full-blown Rails Engine instead. But in many cases I’ve seen, the goal was to just share the business logic between applications while having separate front-end logic.

A small example of this scenario is the open sourced project I’ve been working on in the last few weeks. Central, which is a Pivotal Tracker/Trello alternative - if you’re interested.

A few days ago I started a new project (for internal use only) that would query the same models as Central. I didn’t want to implement HTTP APIs at this point, and the new application would itself have models with relationships to the models in Central (while treating them as read-only).

After a few refactorings, most of Central’s models look like this one:

class Team < ActiveRecord::Base
  include Central::Support::TeamConcern::Associations
  include Central::Support::TeamConcern::Validations
  include Central::Support::TeamConcern::Scopes
  include Central::Support::TeamConcern::DomainValidator
  ...
end

And I have this dependency in the Gemfile:

gem 'central-support', github: 'Codeminer42/cm42-central-support', branch: 'master', require: 'central/support'

Whenever I change the concerns, I do a bundle update central-support in the projects (this is the one caveat to keep in mind to avoid dealing with outdated models).

This was possible because most of those models were mature and stable and I will not be changing them often. I don’t recommend exposing unstable dependencies (as gems or APIs, it doesn’t matter), because this is a recipe for huge headaches of cascading breaking changes due to outdated dependencies that are changing too often.

You should ONLY expose business logic that is reasonably stable (changes only every week or so).

The whole endeavor was to build a certain Rubygems structure, organize the original models into Concerns (which breaks no behavior), make sure specs are still passing, and then move the content (models and specs) over to the new Rubygems and make sure the specs pass there.

That’s how I built a secondary open source dependency for Central, called Central Support. Like many gems, its main file lib/central/support.rb is nothing but a bunch of ‘require’s to load all the dependencies.

So I methodically organized logic as concerns, such as lib/central/support/concerns/team_concern/association.rb, which is just the extraction of the Active Record associations from the ‘Team’ model.

Cut from Central, Paste into Support. When all relevant logic has been moved, I could move the entire Team model spec, mostly without any changes, and make it run. Every time I moved a piece, I bundle updated the gem and ran the main spec suite to make sure nothing broke.

And this is the difficult part: make a sandbox where those concerns could run and be tested.

To begin, I needed to build a minimal Rails app inside the spec folder, at spec/support/rails_app. And there I could put fake models that include the concerns I had just extracted from Central.

There is scarce documentation on how to do that, but I think you can just do rails new and start from there, or copy my rails_app folder for the bare minimum. My case is simpler because this gem is not meant to be general-purpose, so I don’t need to run it against different Rails versions, for example.

This internal test app must have a carefully crafted Gemfile:

...
gem 'central-support', path: File.expand_path("../../../..", __FILE__)

gem 'devise'
gem 'pg'
gem 'friendly_id'
gem 'foreigner'

group :test do
  gem 'test-unit'
  gem 'rspec-rails'
...

You don’t have to add the gems from the main gemspec. But you can remove the development dependencies that you would put in the gemspec and keep them in the test app Gemfile.

Now, from the main Gemfile you can do:

source 'https://rubygems.org'

eval_gemfile File.join(File.dirname(__FILE__), "spec/support/rails_app/Gemfile")

Most tutorials to build a Rubygem will add a line to load dependencies from the gemspec, but here we are replacing it with the test app’s Gemfile. This is the manifest that will be loaded when we run bundle exec rspec, for example.

Speaking of which, this is the spec/rails_helper.rb:

ENV['RAILS_ENV'] ||= 'test'

require 'rails/all'

require 'factory_girl'
require 'factory_girl_rails'
require 'rspec/rails'
require 'shoulda/matchers'

`cd spec/support/rails_app ; bin/rails db:drop db:create db:schema:load RAILS_ENV=test`

require 'support/rails_app/config/environment'

require 'support/database_cleaner'
require 'support/factory_girl'
require 'support/factories'
require 'spec_helper'

To wrap your head around it:

• bundle exec rspec will load the main Gemfile
• the main Gemfile will load from the internal test app’s Gemfile
• that internal test app’s Gemfile will require the gemspec from ../../../.. and the development and test groups of gems (including Rspec, Factory Girl, etc)
• the gemspec will require the runtime dependencies such as “activesupport”, “enumerize”, etc
• finally, the rails_helper.rb listed above will load.

There at line 11, the runner will execute a command to cd into the internal test app’s root folder and run the db:schema:load, therefore you need a db/schema.rb ready to load, as well as config/database.yml.

The spec/spec_helper.rb is more standard, with optional configurations for test coverage, etc.

The models inside the internal test app are the important parts, because they are the means to include the extracted concerns into a runnable format. The ‘spec/support/rails_app/app/models/team.rb’ is such an example:

class Team < ActiveRecord::Base
  include Central::Support::TeamConcern::Associations
  include Central::Support::TeamConcern::Validations
  include Central::Support::TeamConcern::Scopes
  include Central::Support::TeamConcern::DomainValidator
end

And with that, I could move the unmodified specs directly from the main project (Central), such as spec/central/support/team_spec.rb:

require 'rails_helper'

describe Team, type: :model do
  it { is_expected.to validate_presence_of :name }
  it { is_expected.to have_many :enrollments }
  it { is_expected.to have_many :users }
  it { is_expected.to have_many :ownerships }
  it { is_expected.to have_many :projects }
  ...
end

If you go back in the Central project, some commits back, you will find the very same file as spec/models/team_spec.rb. And the main advantage of this approach is exactly being able to move most of the code out of the main project, together with their specs, into a dependency gem, without having to “rewrite” anything.

If I had to rewrite all or a big chunk of the code, it would’ve been a more expensive choice and I would probably have deferred it to another time and focus on more valuable features first.

This approach is not perfect but it was super cheap. I could move all the relevant business logic out of the main project without having to rewrite anything but a few lines of wiring code. The new dependency gem received all the relevant bits and specs, and everything just runs.

So, if you have 2 or more Rails apps that could share the same models, this is how you can start it. Of course, there are always a lot of caveats to keep in mind.

In my case, the Central project is the one that can read-and-write to the database. My internal secondary app is just using the models as read-only. When 2 different apps write to the same database, you may have a number of conflicts to deal with.

This approach is useful if your secondary application is akin to an Administration dashboard, for example. You need to have some of the same associations, scopes, even validations for eventual editing, but it’s limited to a few, controlled users.

This is also useful if you’re doing data analysis, and again you can use the same associations, scopes, to build reports and dashboards. Essentially, if you need read-only access, this is a no-brainer.

In the next article I will explain how I wired a secondary application, using the central-support gem and dealing with 2 different databases at the same time.

When I was testing only with MangaReader.net as a source, everything worked almost flawlessly. But adding MangaFox in Part 3, with its more restrictive rules towards scraper tools like mine (timing out more frequently, not allowing too many connections from the same place, etc), the process just kept crashing and I had to manually restart it (the resuming features I added in Part 4 paid off, but it’s not a reliable tool anymore).

To recap, the Workflow just organizes each step of the process. Its functions are similar to this:

def process_downloads(images_list, directory) do
  images_list
    |> Enum.map(&Worker.page_download_image(&1, directory))
    |> Enum.map(&Task.await(&1, @await_timeout_ms))
  directory
end

It deals with a large list, maps over each element sending it to a Worker function to run, like this:

def page_download_image(image_data, directory) do
  Task.async(fn ->
    :poolboy.transaction :worker_pool, fn(server) ->
      GenServer.call(server, {:page_download_image, image_data, directory}, @genserver_call_timeout)
    end, @task_async_timeout
  end)
end

It returns an asynchronous Task waiting for 2 things: for Poolboy to release a free process to use, and for the Worker/GenServer function to finish running inside that process. As I explained in Part 2 this is so we can limit the maximum number of connections to the external source. If we didn’t have this restriction, sending tens of thousands of asynchronous requests at once, the external source would just fail them all.

The first thing to bear in mind is that a “Task.async/2” links itself to the caller process, so if something goes wrong, the parent process dies as well.

The correct thing to do is to add a Task.Supervisor and make it deal with each Task child. To do that, we can just add the Supervisor in our supervised tree at “pool_management/supervisor.ex”:

defmodule PoolManagement.Supervisor do
  use Supervisor
  ...
  children = [
    supervisor(Task.Supervisor, [[name: Fetcher.TaskSupervisor]]),
    :poolboy.child_spec(:worker_pool, pool_options, [])
  ]
  ...
end

And we can replace the “Task.async/2” calls to “Task.Supervisor.async(Fetcher.TaskSupervisor, …)” like this:

def page_download_image(image_data, directory) do
  Task.Supervisor.async(Fetcher.TaskSupervisor, fn ->
    :poolboy.transaction :worker_pool, fn(server) ->
      GenServer.call(server, {:page_download_image, image_data, directory}, @genserver_call_timeout)
    end, @task_async_timeout
  end)
end

This still creates Tasks that we need to await on, and as before, if the function inside crashes, it still brings down the main process. Now my refactoring found a dead end.

This is the 2nd problem I mentioned in the beginning of the article: a flaw in my design.

Instead of just mapping through each element of a large list, I should have created an Agent based GenServer to keep the list as the state and make the entire Workflow system a new supervised GenServer. If fetching one URL crashed the GenServer, its supervisor would restart it and pick up the next element in the list.

But, as I am in no mood for this refactoring right now (it’s Sunday afternoon) I will concentrate on a quick fix (yes, jerry-rigged patch), just so the function in the async call does not raise exceptions.

OMG! It’s a Try/Catch block!

Turns out that everything I run inside the Poolboy processes are HTTP get requests through HTTPotion. Fortunately I had already refactored every HTTPotion get call into a neat macro:

defmacro fetch(link, do: expression) do
  quote do
    Logger.debug("Fetching from #{unquote(link)}")
    case HTTPotion.get(unquote(link), ExMangaDownloadr.http_headers) do
      %HTTPotion.Response{ body: body, headers: headers, status_code: 200 } ->
        { :ok, body |> ExMangaDownloadr.gunzip(headers) |> unquote(expression) }
      _ ->
        { :err, "not found"}
    end
  end
end

Now I only need to replace 1 line in this macro:

-    case HTTPotion.get(unquote(link), ExMangaDownloadr.http_headers) do
+    case ExMangaDownloadr.retryable_http_get(unquote(link)) do

And define this new retryable logic in the main module:

defmodule ExMangaDownloadr do
  require Logger

  # will retry failed fetches over 50 times, sleeping 1 second between each retry
  @max_retries  50
  @time_to_wait_to_fetch_again 1_000
  ...
  def retryable_http_get(url, 0), do: raise "Failed to fetch from #{url} after #{@max_retries} retries."
  def retryable_http_get(url, retries \\ @max_retries) when retries > 0 do
    try do
      Logger.debug("Fetching from #{url} for the #{@max_retries - retries} time.")
      response = HTTPotion.get(url, ExMangaDownloadr.http_headers)
      case response do
        %HTTPotion.Response{ body: _, headers: _, status_code: status } when status > 499 ->
          raise %HTTPotion.HTTPError{message: "req_timedout"}
        _ ->
          response
      end
    rescue
      error in HTTPotion.HTTPError ->
        case error do
          %HTTPotion.HTTPError{message: message} when message in @http_errors ->
            :timer.sleep(@time_to_wait_to_fetch_again)
            retryable_http_get(url, retries - 1)
          _ -> raise error
        end
    end
  end
  ...
end

I strongly stated that in Elixir we should not use “try/catch” blocks, but there you have it.

This is the consequence of the flaw in my initial Workflow design. If I had coded the Workflow module to be a GenServer, with each list managed by an Agent, each failed HTTPotion call would allow the supervisor to restart it and try again. Without resorting to the ugly “try/catch” code.

Maybe this will force me to write Part 6 as being the code to remove this ugly try/catch later, so consider this a Technical Debt to make everything work now so we can refactor later and pay the debt back.

“HTTPotion.get/2” calls can raise “HTTPotion.HTTPError” exceptions. I am catching those errors for the time being, matching the messages against a list of errors I had already, sleeping for a certain amount of time (just a heuristic to see if the external sources respond better that way) and I recurse to itself through a limited number of “retries”, until it reaches zero, in which case it may even be the case that the internet connection is down or some other severe error that we would not be able to recover soon.

With this code in place, now even fetching from MangaFox, without tweaking down the POOL_SIZE, will run until the end, and this solves my needs for now. If anyone is interested in suggesting a better, GenServer based Workflow design, I would really appreciate a Pull Request.

Cheers.

You can see everything I did since yesterday to clean things up through Github’s awesome compare page

First things first: now the choice to have added a reasonable amount of tests will pay off. In this refactoring I changed function signatures, response formats, moved a fair amount of code around, and without the tests this endeavor would have taken me the entire day or more, rendering the refactor efforts questionable to begin with.

At each step of the refactoring I could run “mix test” and work until I ended up with the green status:

Finished in 13.5 seconds (0.1s on load, 13.4s on tests)
12 tests, 0 failures

The tests take a long time because I made a choice for the MangaReader and Mangafox unit tests to actually go online and fetch from the sites. It takes longer to run the suite but I know that if it breaks and I didn’t touch that code, the source websites changed their formats and I need to change the parser. I could have added fixtures to make the tests run faster, but the point of my parser is for them to be correct.

Macros to the Rescue!

Each source module has 3 sub-modules: ChapterPage, IndexPage and Page. All of them have a main function that resembles this piece of code:

defmodule ExMangaDownloadr.Mangafox.ChapterPage do
  require Logger
  ...
  def pages(chapter_link) do
    Logger.debug("Fetching pages from chapter #{chapter_link}")
    case HTTPotion.get(chapter_link, [headers: ["User-Agent": @user_agent, "Accept-encoding": "gzip"], timeout: 30_000]) do
      %HTTPotion.Response{ body: body, headers: headers, status_code: 200 } ->
        body = ExMangaDownloadr.Mangafox.gunzip(body, headers)
        { :ok, fetch_pages(chapter_link, body) }
      _ ->
        { :err, "not found"}
    end
  end
  ...
end

(Listing 1.1)

It calls “HTTPotion.get/2” sending a bunch of HTTP options and receives a “%HTTPotion.Response” struct that is then decomposed to get the body and headers. It gunzips the body if necessary and goes to parse the HTML itself.

Similar code exists in 6 different modules, with different links and different parser functions. It’s a lot of repetition, but what about making the above code look like the snippet below?

defmodule ExMangaDownloadr.Mangafox.ChapterPage do
  require Logger
  require ExMangaDownloadr

  def pages(chapter_link) do
    ExMangaDownloadr.fetch chapter_link, do: fetch_pages(chapter_link)
  end
  ...
end

(Listing 1.2)

Changed 9 lines to just 1. And by the way, this same line can be written like this:

ExMangaDownloadr.fetch chapter_link do
  fetch_pages(chapter_link)
end

Seems familiar? It’s like every block in the Elixir language, you can write it in the “do/end” block format or the way it really is under the covers: a keyword list with a key named “:do”. And the way this macro is defined is like this:

defmodule ExMangaDownloadr do
  ...
  defmacro fetch(link, do: expression) do
    quote do
      Logger.debug("Fetching from #{unquote(link)}")
      case HTTPotion.get(unquote(link), ExMangaDownloadr.http_headers) do
        %HTTPotion.Response{ body: body, headers: headers, status_code: 200 } ->
          { :ok, body |> ExMangaDownloadr.gunzip(headers) |> unquote(expression) }
        _ ->
          { :err, "not found"}
      end
    end
  end
  ...
end

(Listing 1.3)

There are a lot of details to consider when writing a macro and I recommend reading the documentation on Macros. The code is basically copying the function body from the “ChapterPage.pages/1” (Listing 1.1) and pasting it into the “quote do .. end” block (Listing 1.3).

Inside that code we have “unquote(link)” and “unquote(expression)”. You also must read the documentation on “Quote and Unquote”. It just expands this “external” code inside the macro code to defer execution until the macro quote code is actually executed, instead of running it at that exact moment. I know, tricky to wrap your head around this the first time.

The bottom line is: whatever code is inside the “quote” block will be “inserted” where we called “ExMangaDownloadr.fetch/2” in the “pages/1” function in Listing 1.2, together with the unquoted code you passed as a parameter.

The resulting code will resemble the original code in Listing 1.1.

To make it simpler, if you were in Javascript this would be similar code:

function fetch(url) {
    eval("doSomething('" + url + "')");
}
function pages(page_link) {
    fetch(page_link);
}

“Quote” would be like the string body in an eval and “unquote” just concatenating the value you passed inside the code being eval-ed. This is a crude metaphor as “quote/unquote” is way more powerful and cleaner than ugly “eval” (which you shouldn’t be using, by the way!) But this metaphor should do to make you understand the code above.

Another place I used a macro was to save the images list in a dump file and load it later if the tool crashes for some reason, in order not to have to start over from scratch. The original code was like this:

dump_file = "#{directory}/images_list.dump"
images_list = if File.exists?(dump_file) do
                :erlang.binary_to_term(File.read!(dump_file))
              else
                list = [url, source]
                  |> Workflow.chapters
                  |> Workflow.pages
                  |> Workflow.images_sources
                File.write(dump_file, :erlang.term_to_binary(list))
                list
              end

(Listing 1.4)

And now that you understand macros, you will understand what I did here:

defmodule ExMangaDownloadr do
  ...
  defmacro managed_dump(directory, do: expression) do
    quote do
      dump_file = "#{unquote(directory)}/images_list.dump"
      images_list = if File.exists?(dump_file) do
          :erlang.binary_to_term(File.read!(dump_file))
        else
          list = unquote(expression)
          File.write(dump_file, :erlang.term_to_binary(list))
          list
        end
    end
  end
  ...
end

defmodule ExMangaDownloadr.CLI do
  alias ExMangaDownloadr.Workflow
  require ExMangaDownloadr
  ...
  defp process(manga_name, directory, {_url, _source} = manga_site) do
    File.mkdir_p!(directory)

    images_list = 
      ExMangaDownloadr.managed_dump directory do
        manga_site
          |> Workflow.chapters
          |> Workflow.pages
          |> Workflow.images_sources 
      end
    ...
  end
  ...
end

And there you have it! And now you see how “do .. end” blocks are implemented. It just passes the expression as the value in the keyword list of the macro definition. Let’s define a dumb macro:

defmodule Foo
  defmacro foo(do: expression) do
     quote do
       unquote(expression)
     end
  end
end

And now the following calls are all equivalent:

require Foo
Foo.foo do
  IO.puts(1)
end
Foo.foo do: IO.puts(1)
Foo.foo(do: IO.puts(1))
Foo.foo([do: IO.puts(1)])
Foo.foo([{:do, IO.puts(1)}])

This is macros combined with Keyword Lists which I explained in previous articles and it’s simply a List with tuples where each tuple has an atom key and a value.

More Macros

Another opportunity to refactor was the “mangareader.ex” and “mangafox.ex” modules that were just used in the unit tests “mangareader_test.ex” and “mangafox_test.ex”. This is the old “mangareader.ex” code:

defmodule ExMangaDownloadr.MangaReader do
  defmacro __using__(_opts) do
    quote do
      alias ExMangaDownloadr.MangaReader.IndexPage
      alias ExMangaDownloadr.MangaReader.ChapterPage
      alias ExMangaDownloadr.MangaReader.Page
    end
  end
end 

And this is how it was used in “mangareader_test.ex”:

defmodule ExMangaDownloadr.MangaReaderTest do
  use ExUnit.Case
  use ExMangaDownloadr.MangaReader
  ...

It was just a shortcut to alias the modules in order to use them directly inside the tests. I just moved the entire module as a macro in “ex_manga_downloadr.ex” module:

  ...
  def mangareader do
    quote do
      alias ExMangaDownloadr.MangaReader.IndexPage
      alias ExMangaDownloadr.MangaReader.ChapterPage
      alias ExMangaDownloadr.MangaReader.Page
    end
  end

  def mangafox do
    ...
  end

  defmacro __using__(which) when is_atom(which) do
    apply(__MODULE__, which, [])
  end
end

And now I can use it like this in the test file:

defmodule ExMangaDownloadr.MangaReaderTest do
  use ExUnit.Case
  use ExMangaDownloadr, :mangareader
  ...

The special “using” macro is called when I “use” a module, and I can even pass arguments to it. The implementation then uses “apply/3” to dynamically call the correct macro. This is exactly how Phoenix imports the proper behaviors for Models, Views, Controllers, Router, for example:

defmodule Pxblog.PageController do
  use Pxblog.Web, :controller
  ...

The macros are open in a Phoenix file and available in the “web/web.ex” module, so I just copied the same behavior. And now I have 2 fewer files to worry about.

Tiny refactorings

In the previous code I used the “String.to_atom/1” to convert the string of the module name to an atom, to be later used in “apply/3” calls:

defp manga_source(source, module) do
  case source do
    "mangareader" -> String.to_atom("Elixir.ExMangaDownloadr.MangaReader.#{module}")
    "mangafox"    -> String.to_atom("Elixir.ExMangaDownloadr.Mangafox.#{module}")
  end
end

I changed it to this:

    "mangareader" -> :"Elixir.ExMangaDownloadr.MangaReader.#{module}"
    "mangafox"    -> :"Elixir.ExMangaDownloadr.Mangafox.#{module}"

It’s just a shortcut to do the same thing.

In the parser I was also not using Floki correctly. So take a look at this piece of old code:

defp fetch_manga_title(html) do
  Floki.find(html, "#mangaproperties h1")
  |> Enum.map(fn {"h1", [], [title]} -> title end)
  |> Enum.at(0)
end
defp fetch_chapters(html) do
  Floki.find(html, "#listing a")
  |> Enum.map fn {"a", [{"href", url}], _} -> url end
end

Now using the better helper functions that Floki provides:

defp fetch_manga_title(html) do
  html
  |> Floki.find("#mangaproperties h1")
  |> Floki.text
end
defp fetch_chapters(html) do
  html
  |> Floki.find("#listing a")
  |> Floki.attribute("href")
end

This was a case of not reading the documentation as I should have. Much cleaner!

I did other bits of cleanup but I think this should cover the major changes. And finally, I bumped up the version to “1.0.0” as well!

   def project do
     [app: :ex_manga_downloadr,
-     version: "0.0.1",
+     version: "1.0.0",
      elixir: "~> 1.1",

And speaking of versions, I’m using Elixir 1.1 but pay attention as Elixir 1.2 is just around the corner and it brings some niceties. For example, that macro that aliased a few modules could be written this way now:

def mangareader do
  quote do
    alias ExMangaDownloadr.MangaReader.{IndexPage, ChapterPage, Page}
  end
end

And this is just 1 feature among many other syntax improvements and support for the newest Erlang R18. Keep an eye on both!

I recently came across an interesting post by my colleague Pat Kua where he talks about how some patterns he’s noticed in code can be linked to Conway’s Law, which suggests that the structure of systems designed in organisations will mirror the communication structure of that organisation. (AkitaOnRails: also read The Mythical Man-Month, where Conway’s topic is also explored, and understand Cross Functional Teams to understand one of the solutions.)

I recently read a book called The Five Dysfunctions of Teams which describe some behaviours in teams which aren’t working in an effective way.

Playing the devil’s advocate I became intrigued as to if there is some sort of link between these dysfunctions and whether they manifest themselves in our code as anti patterns.

The five dysfunctions are:

1. Absence of Trust – team members are unwilling to be vulnerable within the group
2. Fear of Conflict – team cannot engage in unfiltered and passionate debate of ideas
3. Lack of Commitment – team members rarely have buy in or commit to decisions
4. Avoidance of Accountability – team members don’t call their peers on actions/behaviours which hurt the team
5. Inattention to Results – team members put their individual needs before those of the team

Absence of Trust

I think having null checks all over the code is the most obvious indicator that people don’t trust the code that they are working with.

If the person writing the code had faith in their colleagues who had written the code they now need to interact with then I think it would be more likely that they would trust the code to do the right thing and they wouldn’t feel driven to such a defensive approach.

Fear of Conflict

Fear of conflict in a team seems to manifest itself most obviously in code when we have a lot of duplication happening (the “copy & paste” syndrome) - there are several reasons why duplication can happen but I think one of them is when people aren’t engaging in discussions when they disagree with something that a colleague has written and therefore end up writing their own version of something that’s already been done.

This probably manifests itself even more obviously when you end up with multiple different frameworks all in the same code base and all doing the same thing just because people don’t want to engage in a conversation to choose which one the team is going to use.

Lack of Commitment

This is one which seems to overlap a lot with the previous two although perhaps one specific way that this would manifest itself in the code might be if we see sloppy mistakes or lack of care being shown with the code (the “sloppy job” syndrome) - an example of this could be changing the name of a class but then not ensuring that all the places where the old name was used in variables have been changed accordingly.

This leaves the code in a half baked state which becomes quite difficult for other people to work with and they have to do some clean up work before being able to effectively make changes to the code.

Avoidance of Accountability

The coding anti pattern that stands out for me here is when we allow people to write code without tests and then check those into source control.

From my experience so far this never seems to work out well and I think it shows a lack of respect for the rest of the team since we don’t have an easy way of verifying whether this code actually works and other people can’t make use of it elsewhere in the code base with any degree of confidence.

Inattention to Results

Team members putting their individual needs before the team manifests itself in code when we end up with code that has been written in such a way that only the person who wrote it is really able to understand it.

I think this manifests itself in “clever code” which is fine in your own projects but in a team context is very detrimental as you become a bit of a bottleneck when people want to make changes in this area of the code and can’t do it because they can’t understand what’s going on.

Something else that falls under this dysfunctions is when there is a convention for how to do certain things in the code but we decide to go off and do it our own way. Now granted sometimes it’s fine to do this if you’re working the code towards a better state and the rest of the team are aware you’re trying to work towards this goal but otherwise it’s not a very effective approach.

In Summary

I found it quite intriguing that in my mind at least some of the problems we see in code do seem to have some correlation to the problems that we see in teams.

One thing I remember from reading Gerald Weinberg’s The Secrets of Consulting is his claim that no matter what the problem is it’s always a people problem - if indeed this is true then in theory problems that we see in code should be indicative of a people problem which I think probably to an extent is true.

I think certainly not all problems in code are linked to the dysfunctions of teams - certainly some anti patterns creep into our code due to a lack of experience of team members of how to do things better but then again maybe that’s indicative of the team not having senior members working closely enough with their colleagues!

Maybe we can therefore work out how we can identify ways to improve our team by starting with a look at the code.

Rant

by AkitaOnRails: Honestly, I’m thoroughly convinced that the problems that show up in code are just symptoms of structural problems in teams and organizations.

It starts with a lack of respect. When the team watches the boss (don’t call a hierarchical boss a “leader”; they’re almost never real leaders) pull rank to get what he wants out of other teams, the developers themselves turn it into an arm-wrestling match: “I did my part, and if the other guy complains I’ll have my boss talk to his and that’s that.” In my experience, almost every problem in an inefficient, dysfunctional team traces back to the manager.

Managers who run on “command-and-control” are exactly the types who should be run out of an organization. They’re the ones who don’t trust the team, who yell, who lean on the power of the title, who insist on being the communication bottleneck and demand that everything pass through them. They have no real knowledge, and it shows the moment they have to actually argue a point.

Then there’s the manager who never gives honest, daily feedback, good or bad, and instead saves it all up to throw in your face months later. To me, that’s the loudest display of cowardice from someone who is supposed to be a “leader.”

There’s the manager who loves to “micro-manage” whenever it suits him but never spells out what he expects. His feedback tracks his mood that day instead of the work you delivered, which keeps things nice and comfortable on his end. And there’s the manager who, to look a little less bad, spends his time trying to make the other teams look worse. These are the nitpickers, the ones who dig up trivial stuff (what time you showed up, what you wore, some hallway chat) as an excuse to badmouth people, instead of looking at real results like profit and cost.

If you’re genuinely worried about why your teams aren’t performing the way they should, look at the management layer. Especially the ones who’ve spent many years in the same organization, hooked on it, who already know every “workaround” in the place. They’re dangerous: they smile at everyone, they seem confident, they seem efficient, and most of their reports praise them (under coercion, obviously).

The team reflects the system and the structure dropped on top of it. Strip away their autonomy, treat them like children, leave them confused and afraid, and this is exactly what you get: sloppy work, poor quality, defects all the time, high cost.

Some people call that an exaggeration, especially a distant, barely present upper management (anything short of every day isn’t presence) when it comes to what happens on the shop floor. Then they’re shocked to find their employees have gone stagnant, incompetent, and obsolete, and grown distant and indifferent to the future of the organization. At that point the only thing they care about is their own job.

Meanwhile those “managers” stay comfortable. When things go well (by luck, and only luck), they take the credit. When things go wrong, the blame lands on the team members he was going to get rid of anyway, or on the other teams, or on the whole organization that didn’t back him up, or on historical decisions that can’t be undone now. The blame is never his.

Remember this: years of tenure can’t buy immunity. Years of tenure should be irrelevant when an organization wants to be efficient. And don’t be afraid to let a manager like that go out of some fear that “only he knows how things work.” Things won’t fall apart. Trust the teams, give the autonomy back, they know what to do.

So yes: the vast majority of a team’s dysfunctions are a direct result of the organization’s dysfunctions. There’s no point applying localized techniques, preaching pair programming, test-driven development, refactoring and the like, if the organization stays the same. Want to change? Change everything. Or don’t even bother.

There’s a mess I’ve heard about with quite a few projects recently. It works out like this:

• They want to use an agile process, and pick Scrum
• They adopt the Scrum practices, and maybe even the principles
• After a while progress is slow because the code base is a mess

What’s happened is that they haven’t paid enough attention to the internal quality of their software. If you make that mistake you’ll soon find your productivity dragged down because it’s much harder to add new features than you’d like. You’ve taken on a crippling Technical Debt and your scrum has gone weak at the knees. (And if you’ve been in a real scrum, you’ll know that’s a Bad Thing.)

I’ve mentioned Scrum because when we see this problem, Scrum seems to be particularly common as the nominative process the team is following. For many people, this situation is exacerbated by Scrum because Scrum is process that’s centered on project management techniques and deliberately omits any technical practices, in contrast to (for example) Extreme Programming.

In defense of Scrum, it’s important to point out that just because it doesn’t include technical activities within its scope should not lead anyone to conclude that it doesn’t think they are important. Whenever I’ve listened to prominent Scrummers they’ve always emphasized that you must have good technical practices to succeed with a Scrum project. They don’t mandate what those technical practices should be, but you do need them. After all projects get into trouble for poor internal quality all the time, the fact that a lot crop up under Scrum’s flag may be more due to the fact that Scrum is so popular at the moment then anything particular to Scrum. Popularity and Semantic Diffusion tend to go together.

So what to do about it?

The scrum community needs to redouble its efforts to ensure that people understand the importance of strong technical practices. Certainly any kind of project review should include examining what kinds of technical practices are present. If you’re involved or connected to such a project, make a fuss if the technical side is being neglected.

If you’re looking to introduce scrum, make sure you pay good attention to technical practices. We tend to apply many of those from Extreme Programming and they fit just fine. XPers often joke, with some justification, that Scrum is just XP without the technical practices that make it work. Sniping aside, the XP practices make a good starting point - and are certainly going to be much better than doing nothing at all.

I always like to point out that it isn’t methodologies that succeed or fail, it’s teams that succeed or fail. Taking on a process can help a team raise its game, but in the end it’s the team that matters and carries the responsibility to do what works for them. I’m sure that the many Flaccid Scrum projects being run will harm Scrum’s reputation, and probably the broader agile reputation as well. But since I see Semantic Diffusion as an inevitability I’m not unduly alarmed. Teams that fail will probably fail whatever methodology they mis-apply, teams that succeed will build their practices on good ideas and the scrum community’s role is to spread these good ideas around widely.

Many people are looking to Lean as the Next Big Agile Thing. But the more popular lean becomes the more it will run into the same kind of issues as Scrum is facing now. That doesn’t make Lean (or Scrum) worthless, it just reminds us Individuals and Interactions are more valuable than Processes and Tools.

Akita’s Notes

In practice, what I’ve noticed is that the planning and management parts meet the least resistance: User Stories, Release Planning, Small Releases, Project Velocity, Iterations, Iteration Planning, Stand Up Meetings. The reason is simple. When a company decides to adopt Scrum, the decision came from some level above; without that, it never gets off the ground. And managers and bosses tend to understand and swallow these concepts well enough, especially if they’ve already been through traditional project management methodologies without success.

And this is where most people go wrong: if the team isn’t yet mature enough to be self-managing, that is, to produce quality code on its own, it’s up to upper management to lead in that direction. That’s hard because many managers aren’t technical, which leaves them blind to the problem of missing good technical practices.

The mess Martin Fowler refers to lives in design, coding, and testing. The ones who struggle most with these practices are developers who came from the old ways of building software, the “suicidal cowboy” style, or who have little experience and little study of software development in general.

Most developers don’t understand Simplicity, the famous “do the simplest thing that works,” or YAGNI (You Ain’t Gonna Need It). In the younger ones, the “cowboy” spirit is hard to tame: they always want to build things in more complex ways than necessary. This leads precisely to the problem of adding things too early, complexity for the pleasure of complexity. The team needs to police itself against this. Fortunately Spike Solutions help: the team stops for a moment, understands the problem, and studies solution alternatives.

Another mix-up I’ve seen a lot is confusing Refactoring with rewriting. A rewrite in itself isn’t a problem, but it becomes one when applied with the notion that “just because it’s new, it’s better.” First and foremost, Refactoring is a set of techniques aimed at rejuvenating code, making it more manageable, without modifying its behavior.

This leads to the Achilles’ heel of most amateur developers: the aversion to testing. “I’m too good to make mistakes, I don’t need tests.” That’s the developer who will effectively lead his own work to certain failure. Testing first is an effective way to refine design, and it also leads to simplicity, doing only what is truly needed.

Without good tests, effective refactoring is impossible: how do you guarantee that a change didn’t alter the code’s behavior? The corollary is that these developers also don’t practice Refactoring, which leads to the mass of messy code. The irony is that even rewrites, where it was assumed the new version would be better, end up becoming “legacy” very quickly.

Worse yet, when a bug is found, tests to prevent a regression to the same bug are rarely created. And everyone must have seen bugs that were supposedly already fixed coming back shortly after. Another problem is that many teams never settled on a good definition of “story done” and therefore also struggle to maintain acceptance tests for those stories. One of the reasons: instead of writing User Stories (“as X, I want Y because of Z,” defining what to implement, for whom, and what value it brings), they write tasks (“do X”), skipping the for-whom and the why.

Still in the “cowboy” spirit, inexperienced programmers don’t understand the concept of collective code ownership. What happens is that each developer tries to limit himself to the code he thinks he’s responsible for and doesn’t worry about the whole. It should work the other way around: every developer must feel responsible for all the code. This also explains one more reason for the importance of unit tests: without them it’s impossible to help with code you didn’t write, and impossible to know if your own code doesn’t break something someone else made. In summary: this leads to an immeasurable accumulation of “Technical Debt” that only becomes visible when it’s already too late, turning your “new system” prematurely into an unmanageable “legacy.”

In that same spirit, “cowboy” developers don’t integrate frequently as they should. It’s not uncommon to see developers spend the whole week with new code only on their own machine and, at the end of the week, commit to the repository and consider it done. No tests created, no full test suite run. One more developer deliberately creating Technical Debt.

To make matters worse, it’s not hard to find teams that don’t even understand the value of a repository: for them, it’s enough to change things directly on the production server or edit files right there. That’s the prehistory of development. A code versioning repository is mandatory, and treating it as a sanctuary is the greatest responsibility of a team. That means all code in the repository must always be code with no deliberately introduced problems, from not writing tests, not integrating frequently, not refactoring where needed. Bugs will always exist and must be fixed, but deliberate errors make the developer irresponsible and a problem for the entire team.

Finally, it’s common to see developers who think they’re very clever optimizing very early in the project, based on nothing but guesswork, without measurements. By the way, most programmers I’ve met have as much aversion to measurement as they do to testing. And optimizing without being based on metrics is the purest waste of time. You can make a certain piece of code 100 times faster, but if in overall time that doesn’t represent even a 0.5% gain, it was a waste of time. Again, inexperience.

Still regarding “suicidal cowboys,” they’re normally attached to their limited tools, the ones they’re already used to. They’re too lazy to learn new things, whether from inability or lack of will, and they always try to take the apparently easiest route, like using automatic code generators, which generally produce code that’s hard to maintain and outside the most modern standards accepted as good practices.

In the end, the result is the same: inexperienced developers who think they know everything, or long-tenured developers addicted to anti-practices and too stubborn to learn the good ones. Building an efficient and truly agile development team is very hard. And the reality is that simply not everyone is cut out for the job. An agile developer is someone proactive, self-taught, sociable, and communicative. And all of that matters more than their supposed technical competence.

PS: some people might take this personally. But in reality, this is more common than you’d imagine; I’ve seen it at many clients and will keep seeing it. To be fair, I myself have done many projects (more than I’d like) that immediately became “legacy,” code poorly tested and hard to maintain. Complaining is easy; doing something to improve is the hard part. And yesterday’s cowboy developer has everything it takes to become a good programmer tomorrow; they just need to want it.