And make sure you subscribe to my new newsletter The M.Akita Chronicles!

–

274 commits. 8 days. 4 applications. 1,323 tests. A complete system in production — with real subscribers receiving a newsletter every Monday at 7am, an AI-generated podcast on Spotify, a static blog on Netlify, and a Discord bot that’s my operational interface.

I did this with a partner: Claude Code, an AI agent running in the terminal. And the most important lesson I took away is not that “AI programs by itself” — AI programming by itself is a recipe for disaster. What works is something that already has a name and decades of history: Extreme Programming. Except now the pair in pair programming isn’t human.

I need all of you to go through an experience: a 1-WEEK IMMERSION completely DETACHED from the code, using AI as pair programming, going from zero to production on any small project. Trust me, until you do it at least once, you won’t get it. But after that, everything changes.

The “One-Shot Prompt” Myth

There’s a growing narrative that “vibe coding” means writing a detailed spec, handing it to the AI agent, and it delivers the system ready. Like those “I built a SaaS in 10 minutes with Cursor” videos. It looks magical until you look at what was actually delivered: a prototype with no tests, no security, no error handling, no deploy, nothing that makes a system survive contact with real users.

The reality of my 274 commits tells a different story:

Only 37% of the commits are features. The rest is the work that makes real software: fixing bugs, hardening against failures, protecting against attacks, deploying, testing, documenting. Anyone showing only the features part is selling an illusion.

More than that: many of the features emerged because iteration revealed they were needed. ContentPreflight wasn’t in the original plan — it was born when the second newsletter went out with an empty section because a job failed silently. RecoverStaleDeliveriesJob was born when I realized a crash in the middle of sending left emails in eternal limbo. StealthBrowser was born when Morningstar blocked HTTParty. No spec in the world predicts this. The correct system emerges from iteration, not from specification.

Pair Programming With an AI Agent

I don’t use Claude Code as a code generator. I use it as a pair programming partner.

The difference is fundamental. A code generator takes a spec and returns output. A pair programming partner talks. I interrupt when it takes the wrong path. I give it context it doesn’t have. I question decisions. I ask for alternatives. Sometimes I say “stop, this is getting too complicated” and it simplifies.

Some real examples from the 274 commits:

Interruption for over-engineering. When I asked for the email delivery system, the first proposal had a state machine with 8 states, separate retry queues, and dead letter handling. I interrupted: “Simplify. Four states: pending, sending, sent, unknown. When in doubt, don’t resend.” The result was SendSingleEmailJob with the ses_accepted variable as the pivot — 40 lines that solve the whole problem.

Context the agent doesn’t have. When we were on Yahoo Finance, Claude tried to use HTTParty. I explained: “Yahoo does TLS fingerprinting, it’ll block any HTTP client that isn’t a real browser.” Result: headless Chromium with crumb authentication. That’s in no tutorial — it’s the kind of knowledge that comes from trying and failing.

Joint architecture decision. The podcast pipeline could have been a Python monolith. I suggested: “Rails orchestrates, Python serves TTS over HTTP, content integrates by filesystem.” The result — QwenTtsClient in Ruby calling podcast-tts in Python — keeps each language doing what it does best.

Iterative prompt refinement. The first newsletters came out generic. The Akita and Marvin comments sounded the same. I tuned prompts with it: “Akita never says ‘maybe’. Marvin uses ‘well…’ and ‘anyway’. The LLM will soften opinions if you’re not explicit.” Each tweak was a prompt commit, tested with real data.

The pattern is: I bring the what and the why. The agent brings the how — and the how is often better than I’d do alone. But how without what produces correct code that solves the wrong problem. And what without how produces specs that never leave paper.

Extreme Programming Isn’t a 2000s Fad

Everything we did has a name in XP. Not because I planned it — because these are the practices that naturally emerge when you program with fast feedback:

Pair Programming

Already explained. The pair is the agent. The dynamic is the same: one drives, the other navigates. I navigate more than drive — I define direction, question decisions, correct course. The agent drives more than navigates — writes code, runs tests, proposes solutions. When it inverts (me dictating exact code, it just typing), the result always gets worse.

Small Releases

Out of 274 commits, every single one passes CI. No exceptions. There’s no “broken commit that’ll be fixed in the next one”. Every commit on master is production-ready.

CI runs on every commit: RuboCop (style) → bundler-audit (gem vulnerabilities) → Brakeman (static security) → full tests. It’s ~22 seconds. Sounds like nothing, but multiplied by 274 commits that’s 100+ minutes of automated validation I didn’t have to do manually.

The distribution of commits by day shows the rhythm:

The day-2 “explosion” isn’t chaos — it’s small releases stacked up. Each one adds a feature, runs CI, is ready to deploy. It isn’t a 2-week waterfall followed by a big-bang merge. It’s real continuous integration.

Test-Driven Development

1,323 tests. 970 in marvin-bot (6 seconds in parallel), 353 in newsletter (1 second in parallel). Ratio:

More lines of test than of code. In both applications. And this isn’t vanity — it’s the infrastructure that enables speed.

Coverage from SimpleCov:

It’s not 100% — and never will be. The 17-18% uncovered is mostly integration with external APIs (Discord, SES, OpenRouter) that are mocked in tests. Real coverage on business logic is above 95%.

The question that matters: are these tests useful? The empirical proof: over 274 commits, I saw CI fail and catch a real bug more than 50 times. Without those tests, every one of those bugs would have gone to production. The agent proposes a change, the test catches the regression, the change is fixed before committing. Without tests, the agent hands you plausible code that silently breaks something that worked yesterday.

TDD with an AI agent is multiplicative. The agent is good at generating tests — it knows the patterns, knows the common edge cases, knows the Minitest structure. And the tests it generates become the safety net for the changes it makes later. It’s a virtuous cycle: tests enable speed, speed generates more tests.

Continuous Refactoring

27 explicit refactoring commits. Some examples:

• Extract StealthBrowser wrapper — The headless browser code with stealth patches was duplicated in 4 clients. Extracted into a service, the 4 clients were left with 5 lines each.
• DRY section rendering: collapse 5 bubble methods + 7 Hugo shortcodes — 5 bubble rendering methods in MarkdownToHtml that were copy-paste with different colors, collapsed into 1 parameterized method. 7 Hugo shortcodes delegating to 1 partial.
• Replace market recap LLM generation with Yahoo Finance real data — The LLM invented market numbers. Replaced with real data from Yahoo Finance + LLM commentary. Better result, lower cost.
• Replace DuckDuckGo with Brave Search API — DuckDuckGo doesn’t have a stable API. Brave Search has an official API with 2,000 queries/month free. Clean swap.
• Rework ArticleFetcher: browser-first with anti-detection — Two full rewrites. The first tried HTTP-first with fallback to browser. The second inverted: browser-first because most reader links are SPAs. Each rewrite kept tests passing.

Refactoring is where the pair programming partner shines. I say “this code is duplicated, extract a concern” and the agent does in 2 minutes what would take me 20. But I decide what to extract and how the interface should look. The agent alone doesn’t refactor — it piles new code on top of existing code.

Continuous Integration

The CI already described above: rubocop + bundler-audit + brakeman + tests. Runs on every commit, in both apps. Brakeman deserves a mention:

Zero security warnings in both. The 2 ignored in newsletter are documented false positives (SQL interpolation with config paths, not user input).

Brakeman caught real problems during development: SQL injection in a search query, path traversal in the images controller, open redirect in unsubscribe. Each one fixed before going to production, in the same commit where CI flagged it.

The Anatomy of a Feature: From Request to Deploy

Let me take a concrete example to illustrate how the flow works in practice. List-Unsubscribe (commit 5950ccc).

Context: confirmation emails weren’t reaching Gmail. Everything green — DKIM, SPF, DMARC, SES accepting. But Gmail was silently dropping them into spam.

Investigation (me + agent): we researched. Since February 2024, Gmail requires List-Unsubscribe + List-Unsubscribe-Post on every bulk sender email. Without them, it penalizes the entire domain — including transactional emails.

Implementation (agent executes, I validate): header in SesMailer, RFC 8058 endpoint in UnsubscriptionsController (skip CSRF for email clients, returns 200 instead of redirect), tests covering both flows.

CI: rubocop OK, brakeman flagged the skip_forgery_protection as a warning (correct — we validated it’s conditional). Tests pass.

Deploy: kamal deploy. In 2 minutes, the fix is in production. Domain reputation takes 1-2 days to recover. The next day, confirmation emails start arriving again.

Total time: ~25 minutes from “why is Gmail rejecting this?” to production deploy. Without the agent, it would be hours of RFC reading, manual implementation, and hoping I didn’t forget anything.

What AI Does Well (And What It Does Badly)

After 274 commits with an AI agent, I have a pretty clear sense of the strengths and weaknesses:

Does well:

• Boilerplate and scaffolding: jobs, services, tests, migrations — the agent produces at typing speed
• Tests: surprisingly good at identifying edge cases and generating comprehensive tests
• Mechanical refactoring: renaming, extracting methods, moving code between files — fast and precise
• Contextual research: “how does RFC 8058 work?” produces an actionable answer in seconds
• Consistency: follows patterns established in the project (concerns, naming conventions, test structure) without forgetting

Does badly:

• Architecture decisions: tends to over-engineer. Needs constant human braking
• Specific domain knowledge: doesn’t know that Yahoo does TLS fingerprinting or that Qwen’s VoiceDesign produces a European accent
• Opinions: softens everything. Personality prompts become generic mush without explicit instructions
• Proactive security: implements what you ask for, but rarely suggests protections you didn’t ask for (SSRF, rate limiting, encryption at rest)
• Prioritization: executes anything with equal enthusiasm. Never says “this is a waste of time” or “do X before Y”

AI doesn’t program by itself, and it isn’t useless. It’s a programmer multiplier, not a substitute. A mediocre programmer with AI produces mediocre code faster. An experienced programmer with AI produces good code at a speed that was previously only possible with a team.

The Numbers That Matter

Let’s look at the real size of what was built in 8 days:

Code

Total functional code: ~16,000 lines. Total with tests: ~36,000 lines.

Components

• 43 services in marvin-bot (scrapers, AI, parsers, validators, uploaders)
• 38 jobs in marvin-bot (content generation, daily digest, podcast, infrastructure)
• 12 services in newsletter (email delivery, publishing, assembly)
• 11 jobs in newsletter (sending, recovery, backup, blog publishing)
• 23 AI prompt templates (personality, generation, podcast, tools)
• 17 Hugo shortcodes (section rendering on the blog)
• 3 RubyLLM tools (image generation, web search, web fetch)

Pipeline

• 10 content sections generated per week (anime, hacker news, youtube, market, world events, history x3, holidays, closing remarks)
• 10 daily digest jobs posting to Discord at 8am
• 1 podcast of ~30 minutes generated per week (script + TTS + assembly + S3)
• 1 newsletter assembled, published to the blog, and sent by email every Monday

Is This 200 User Stories?

I looked at the 274 commits and tried to map them to user stories in the agile sense. Not every commit is a story — many are fixes inside the same feature, prompt tweaks, or CI cleanup. But the conservative count of distinct features, each one with implicit acceptance criteria (works, has tests, passes CI, deploys):

Estimate: ~180-200 user stories.

That includes:

• Each newsletter section (10 distinct generators with scraping, AI, validation, fallback)
• The email pipeline (delivery tracking, atomic claiming, recovery, List-Unsubscribe)
• The podcast pipeline (LLM script, dual-model TTS, assembly, S3, RSS)
• The Discord bot (15+ commands, embeds, context, multi-turn /ask, daily digest)
• The blog (Hugo + Hextra, 17 shortcodes, RSS podcast, SEO, GA)
• The subscription page (confirmation, themes, LGPD, terms of service)
• Infrastructure (Kamal deploy, Cloudflare, SES config, RunPod, backups)
• 21 security commits (SSRF, CSP, rate limiting, encryption, path traversal)

In traditional Scrum with a team of 2-3 devs, a well-done story (with code review, QA, deploy) takes on average 1-2 days. 200 stories × 1.5 days / 2 devs = ~150 days of work, or ~30 weeks in 2-week sprints (counting planning, review, retro, and the inevitable overhead of people coordinating with each other).

Being generous — senior team, well-defined stories, no impediments — maybe 10-15 weeks.

We did it in 8 days. One person and one AI agent.

But — and this is crucial — I’m not just any dev and this isn’t just any project. I have 30 years of software experience. I know which decisions matter and which don’t. I know when to stop engineering and ship. I know when the agent is wrong before it finishes generating the code. And the project is solo-developer, without external code review, without separate QA, without coordination between teams.

The agent’s real multiplier isn’t “10x more code”. It’s eliminating dead time. Without an agent, 70% of my time would be typing boilerplate, reading API docs, writing mechanical tests, debugging typos. With an agent, that time goes to zero and I focus 100% on architecture, domain, and quality decisions.

The CLAUDE.md: The Spec That Evolves

One thing that works exceptionally well with AI agents is having a living document that describes the project. The project’s CLAUDE.md has 702 lines and covers:

• Architecture overview
• Full tech stack
• All environment variables
• Content directory structure
• Services, jobs, and models of each app
• 12 “common hurdles” with documented solutions
• 14 project design patterns
• Full weekly pipeline with schedules
• Post-implementation checklist

This document wasn’t written all at once. It evolved with the project. Every time we found a hurdle (Yahoo Finance needs Chromium, HN RSS changes format, LLM softens opinions), we documented it in CLAUDE.md. In the next session, the agent already knows.

It’s the equivalent of a team’s onboarding doc — except the “new team member” is the agent, and it reads the whole doc in 2 seconds before every interaction. Investing in documentation pays back exponentially with AI agents because they actually read it.

The Lessons

1. Vibe coding without discipline is a throwaway prototype. 274 commits with CI, tests, security scanning, and small releases is the opposite of vibe coding. It’s software engineering with a copilot that types fast.

2. TDD is more important with AI, not less. The agent modifies code with confidence because it has 1,323 tests as a safety net. Without tests, every change is a gamble.

3. The human decides the what. The agent decides the how. Flip it and the result drops dramatically.

4. Continuous refactoring is mandatory. The agent piles code. If you don’t prune regularly — extract concerns, DRY duplications, simplify interfaces — the codebase becomes a monolith of 500 lines per file.

5. Documentation is an investment with immediate return. Every hour documenting in CLAUDE.md saves hours of lost context in future sessions.

6. Small releases are the key. Every commit is production-ready. If something goes wrong, revert one commit. If the feature was a bad idea, it never depended on another. This only works with automated CI.

7. Security isn’t a phase — it’s a habit. 21 security commits spread across 8 days, not concentrated in a “security sprint” at the end. Brakeman runs on every commit. Vulnerabilities are fixed the moment they appear.

8. The agent never says “no”. That’s a bug, not a feature. If you ask for something over-engineered, it implements with enthusiasm. If you ask for something insecure, it implements without complaining. You are the brake. You are the code review. You are the adult in the room.

The Counter-Example: FrankMD

Everything I described above may sound theoretical. “Of course TDD and continuous refactoring are important” — everyone agrees in theory. So let me prove it empirically with a counter-example from the same developer, with the same AI agent, two weeks earlier.

Before the M.Akita Chronicles, I built FrankMD — a Markdown editor in Rails, Obsidian/VSCode style. 212 commits in 19 days. Same setup: me + Claude Code as a pair programming partner.

The difference: on FrankMD, I did what any developer does when “it’s just a little personal project”. I focused on features. Tests? Later. Refactoring? When necessary. CI? Runs when I remember.

The app_controller.js Spiral

FrankMD is a SPA-like with Stimulus. In the beginning, all the JavaScript lived in a single file: app_controller.js. Here’s what happened to it across commits:

The pattern is clear: build-build-build-STOP-refactor. The file grew 10x in 54 commits, hit almost 5,000 lines, and then I needed 6 rounds of refactoring to bring it back to acceptable. Each stop was a big commit (hundreds of lines moved, dozens of files created), not a surgical tweak.

The CSS followed the same pattern: 1,053 lines in a single application.css, until the refactoring commit that exploded it into 10 theme files + 9 component files.

Tests: Before vs. After

On FrankMD, the 2nd commit was “Add comprehensive test suite” — system tests (browser integration). Sounds responsible, but those tests covered the happy path and nothing else. The JavaScript unit tests only came in commit #52, after app_controller.js had already hit 5,000 lines — because I needed tests to refactor safely. Retroactive tests.

Ruby coverage reflects this: it had to be pushed from 77% to 89% in an explicit “Add tests for untested success paths” commit. That wasn’t TDD — it was filling holes afterward.

The Numbers Side by Side

The number that matters: commits per day. The M.Akita Chronicles had 3x more throughput — with TDD, CI, and continuous refactoring — because it never had to stop. No “stop everything and refactor 5,000 lines into 10 files” commit. No “I need to add retroactive tests so I can touch this code without fear” moment.

The Concrete Lesson

FrankMD’s 6 big refactors weren’t “continuous refactoring”. They were emergency surgery on accumulated technical debt. Each one took hours — not the 2 minutes of the “extract a concern” described above. And each one introduced risk: when you move 1,500 lines from one file to 10 new files, the chance of breaking something is proportional to the size of the change.

With TDD and continuous refactoring, the M.Akita Chronicles never accumulated that debt. The 27 refactors are small commits — “Extract StealthBrowser”, “DRY bubble methods” — that take minutes and are protected by the 1,323 tests that already existed when the refactoring was done.

Same developer. Same AI agent. Two different approaches. Result: the disciplined project delivered more, in less time, with less pain.

Conclusion

This project isn’t an AI showcase. It’s a showcase of software engineering accelerated by AI. And the strongest proof is that the same developer with the same agent produced radically different results depending on the process.

On FrankMD, without discipline, the result was predictable: a 5,000-line file, 6 emergency surgeries, retroactive tests, and 11 commits per day. On the M.Akita Chronicles, with XP, the result was 34 commits per day, zero forced stops, 1,323 organic tests, and a production system in 8 days. The variable wasn’t the AI — it was the process.

274 commits in 8 days don’t happen because the agent is magical. They happen because Extreme Programming — pair programming, TDD, small releases, continuous refactoring, continuous integration — works as well with an AI pair as it does with a human pair. Maybe better in some ways: the agent never gets tired, never argues from ego, never says “but we’ve always done it this way”, and reads 700 lines of documentation before every interaction without complaining.

The future of development isn’t “AI replacing programmers”. It’s programmers who know how to use AI as a tool — with the same discipline they’ve always used with any other tool — producing in a week what entire teams take quarters to deliver.

Not because the AI is brilliant. Because the process is brilliant. XP was right from the start. It was just missing a pair that never gets tired and types at 1,000 words per minute.

Remember:

“AI is your mirror, it reveals faster who you are. If you’re incompetent, it’ll produce bad things faster. If you’re competent, it’ll produce good things faster.” - by Akita

And make sure to subscribe to my new newsletter The M.Akita Chronicles!

–

Unit tests are every developer’s comfort zone. You mock everything, isolate everything, run it in milliseconds, and the green in the terminal gives you a false sense of security. Until the day you deploy and discover that service A writes in a format service B can’t read. Or that the Hacker News RSS switched from Atom to RSS and your parser blew up. Or that Yahoo Finance started returning 429 after the third ticker because the rate limit that worked in tests doesn’t exist in real life.

Unit tests prove the pieces work in isolation. Integration tests prove the system works. And in a project with multiple applications sharing data via the filesystem, the difference between the two is the difference between “the code compiles” and “the newsletter lands in the inbox”.

The Problem: Three Apps, One Filesystem, Zero Guarantees

The project has an unusual architecture: three Rails/Hugo applications in a monorepo, integrated through a shared content/ directory:

akitando-news/
├── marvin-bot/    # Gera conteúdo (Discord bot, AI, scraping)
├── newsletter/    # Monta e envia a newsletter
├── blog/          # Hugo + Hextra → Netlify
├── podcast-tts/   # Servidor TTS (Python/FastAPI)
└── content/       # Markdown + YAML, a cola entre tudo

marvin-bot writes Markdown with YAML frontmatter into content/anime/, content/hacker_news/, etc. The newsletter reads those files to assemble the newsletter. The blog serves the same content as static pages. The podcast reads the assembled newsletter to generate the dialogue script.

The contract between these systems is implicit. There’s no schema. No versioned API. No protobuf. The contract is: “the Markdown file has this frontmatter, with these fields, in this structure”. If marvin-bot changes the format of a field — say, switches image from string to array — the newsletter breaks silently. No unit test in either app detects it.

That’s the fundamental flaw of testing distributed systems with unit tests: you’re testing each side of the contract separately, but never the contract itself.

Layer 1: Unit Tests With Paranoid Isolation

Before talking about integration, I need to explain what the unit tests do in this project — because without that foundation, the integration tests would have nothing to stand on.

The 977 marvin-bot tests run in parallel using every core on the machine. Each test worker has its own directory:

parallelize(workers: :number_of_processors)

parallelize_setup do |worker|
  Rails.application.config.content_dir =
    Rails.root.join("tmp", "test_content_w#{worker}").to_s
end

But worker-level isn’t enough. Inside each worker, multiple tests run sequentially — and if one test leaves a file in content/stories/, the next test might find spurious data. The solution: one unique directory per test:

setup do
  @base_content_dir = Rails.application.config.content_dir
  @test_content_dir = File.join(@base_content_dir, SecureRandom.hex(8))
  Rails.application.config.content_dir = @test_content_dir

  FileUtils.mkdir_p(File.join(@test_content_dir, "stories"))
  FileUtils.mkdir_p(File.join(@test_content_dir, "newsletters"))
  FileUtils.mkdir_p(File.join(@test_content_dir, "images"))
end

teardown do
  FileUtils.rm_rf(@test_content_dir) if @test_content_dir
  Rails.application.config.content_dir = @base_content_dir if @base_content_dir
end

Pay attention to the teardown line that restores @base_content_dir. Without it, the next test would create its directory inside the previous test’s directory — tmp/test_content_w3/abc123/def456/ — and by test 50 you’d have a path 50 levels deep. I learned this burning an entire afternoon debugging why the tests kept getting slower as the suite progressed.

This pattern — worker-level base + per-test subdirectory + restore on teardown — is identical across both apps. And it’s what lets you run 977 tests in 7 seconds with 32 parallel workers.

The golden rule: the test content_dir must NEVER point to the real ../content. If it does, the teardown’s rm_rf wipes out your production prompts. This isn’t hypothetical — it happened during development.

Layer 2: The Integration Environment

Unit tests mock everything: external APIs, AI, Discord, SES. That’s correct — you don’t want to burn $2 in GPT-5.2 tokens every time you run bin/rails test. But it means the tests never exercise the real pipeline.

The project has a third environment beyond development and test: integration. It’s not test. It’s not staging. It’s a specific mode for running the full pipeline with real data, using real APIs, but without sending actual emails:

# config/environments/integration.rb

# Jobs rodam inline (perform_later vira perform_now)
config.active_job.queue_adapter = :inline

# Emails salvos em arquivo ao invés de enviados via SES
config.action_mailer.delivery_method = :file
config.action_mailer.file_settings = {
  location: Rails.root.join("tmp/integration_emails")
}

The queue_adapter = :inline is crucial. In production, perform_later enqueues the job into SolidQueue and a separate worker executes it. In integration, it runs right there, in the same process. That turns the asynchronous production pipeline into a synchronous pipeline you can run, observe, and debug sequentially.

SES is swapped for writing to disk — every email becomes an HTML file in tmp/integration_emails/. After running the pipeline, you open those HTMLs in the browser to visually validate that the email renders correctly.

Layer 3: DevCache — LLMs Aren’t Idempotent

Here’s a problem anyone working with AI in production knows: calling GPT-5.2 with the same prompt twice doesn’t return the same answer. And every call costs money. If you’re iterating on the integration pipeline — running, tweaking a prompt, running again — spending $3 per iteration gets expensive fast.

The solution is DevCache: a file-based cache that only activates in the integration env:

module DevCache
  CACHE_TTL = 1.day
  CACHE_ENVS = %w[integration].freeze

  def self.fetch(namespace, key)
    return yield unless enabled?

    path = cache_path(namespace, key)
    if path.exist? && File.mtime(path) > CACHE_TTL.ago
      return JSON.parse(path.read, symbolize_names: true)
    end

    result = yield
    path.write(JSON.generate(result))
    result
  end
end

In production and in test, DevCache.fetch runs the block directly. In integration, it saves the result as JSON and returns the cached value on the next run. TTL of 1 day — enough to iterate within the same day without recomputing, short enough to not serve stale data.

The generation code uses it like this:

summary = DevCache.fetch("summarize", url_key) do
  AiChat.summarize(article[:content], source_url: url)
end

First run: calls GPT-5.2, pays $0.02, saves to tmp/dev_cache/summarize/{md5}.json. Second run: reads from the cache, zero cost. FORCE=1 busts the cache when you want to force regeneration — as an explicit flag, not a default.

DevCache exists in both apps (marvin-bot and newsletter) with identical implementation. When you run integration:clean, both caches are wiped.

The point: LLM caching in a development environment is a requirement, not an optimization. Without it, iterating on a prompt that needs 8 API calls becomes financially unworkable.

Layer 4: Real Data via Rsync

This is where most projects stop. They have unit tests, maybe a staging environment, and ship it. The problem: staging uses test data. Test data is too clean. It doesn’t have the edge cases real data has:

• Articles with titles in Japanese (anime ranking)
• URLs that redirect 3 times before reaching the destination
• Images that return 200 with Content-Type text/html (CloudFlare error page)
• Previous newsletters with slightly different frontmatter formats
• Stories with score: null because they were created before the field existed

The solution: rsync from production.

# Limpa conteúdo gerado (preserva stories e imagens de produção)
cd marvin-bot && bin/rails integration:clean

# Sincroniza content/ de produção pra local
rsync -avz --delete production:/path/to/content/ ../content/

# Roda o pipeline completo com dados reais
RAILS_ENV=integration bin/rails integration:pipeline

integration:clean is surgical: it removes generated directories (anime/, hacker_news/, newsletters/, podcasts/) but preserves stories/ and images/ — which are user-submitted data coming in via Discord. After the rsync, the local directory is an exact copy of production.

The pipeline then generates every section from that real data. If a parser breaks on a Korean title, you find out here — not in production at 5pm on a Sunday when the 8 jobs run in parallel.

Layer 5: The Full Pipeline

The heart of the integration system is integration:pipeline. It simulates the entire production week in a single run:

RAILS_ENV=integration bin/rails integration:pipeline

The pipeline runs in waves, honoring dependencies:

Wave 1 (paralelo, 8 jobs):
  Book, Holidays, History, Geek History,
  Anime, Hacker News, YouTube, World Events

Wave 2 (depende de World Events):
  Market Recap

Wave 3 (depende de tudo):
  Closing Remarks

Preflight → Newsletter Assembly → Podcast

Each wave runs its jobs in parallel threads, exactly like production does with SolidQueue. Parallel execution isn’t an implementation detail here — it’s a test feature. If two jobs try to write to the same file, or if one job reads a file another is in the middle of writing, the bug shows up here.

threads = to_run.map do |name, klass|
  Thread.new do
    ActiveRecord::Base.connection_pool.with_connection do
      run_single_job(name, klass, target_date, results)
    end
  end
end
threads.each(&:join)

with_connection is necessary because each thread needs its own SQLite connection — without it, connections leak and the pool gets exhausted.

At the end, the pipeline prints a status table and a billing summary:

That billing summary isn’t vanity — it’s cost control. If a refactored prompt triples the output tokens, you see it here before going to production and burning 10x on a Sunday with 200 stories.

Layer 6: Preflight — Automated Structural Validation

After generating every section, the pipeline runs ContentPreflight. It doesn’t validate the content — it validates the structure. Each section gets checked:

• Does the file exist? If content/anime/2026-02-16.md doesn’t exist, something failed.
• Minimum item count? The anime section needs at least 3 items. If it has 1, the generator had a problem.
• Required markers? The Markdown must contain [COMMENTARY] and [AKITA] — without them, newsletter assembly doesn’t know where to cut.
• Specific data? Market recap needs at least 5 ticker rows.

The result is a status per section: pass, degraded, fail, or skip. The pipeline uses this to decide whether the newsletter can be assembled automatically or needs human intervention:

In production, this result goes to Discord as an embed. In the integration pipeline, it’s printed to the terminal. The logic is the same — the notification channel changes.

Layer 7: Cross-App — Newsletter Assembly

Here’s where the monorepo pays dividends. marvin-bot needs to invoke the newsletter to assemble the final newsletter. In production, these are different servers. In the integration pipeline, it’s a system() call:

env_vars = "RAILS_ENV=integration CONTENT_DIR=#{Shellwords.escape(content_dir)}"
cmd = "cd #{newsletter_dir} && #{env_vars} bin/rails integration:generate_all"
system(cmd)

The newsletter reads the files marvin-bot just generated in content/, assembles the newsletter, and saves the result in content/newsletters/. CONTENT_DIR is passed explicitly — both apps point to the same directory.

If the frontmatter format changed and the newsletter can’t parse it, the error shows up here. If the newsletter’s SectionParser has diverged from marvin-bot’s SectionParser (they’re identical copies, but maintained by hand), here’s where you find out.

After the newsletter, the podcast: the pipeline invokes podcast:integration, which generates the dialogue script from the assembled newsletter, synthesizes audio via TTS, and builds the final MP3. It’s the longest pipeline — from Markdown newsletter to Spotify-ready MP3, passing through a two-pass LLM and per-turn voice synthesis.

The Value of Testing With Production Data

Let me give concrete examples of bugs that only showed up with real data:

1. Japanese anime titles broke the slugify. ContentWriter.slugify used Rails’ parameterize, which strips non-ASCII characters. Title: “進撃の巨人 Season 4” → slug: “season-4”. Two anime with the subtitle “Season 4” collided on the filesystem. It never showed up in tests because the fixtures used English titles.

2. The HackerNews parser assumed RSS, but the feed switched to Atom. rss/channel/item became feed/entry. Unit tests mocked the XML. The parser itself was never tested against the real feed. Running integration with the real rsynced feed, the parser returned zero items — caught by preflight as fail.

3. Yahoo Finance returned 429 after the 5th ticker. Rate limiting didn’t exist in tests (HTTP was mocked). In the real integration, the first 5 tickers went through, then everything failed. The fix — sleeping between requests — would never have been implemented without real data exposing the problem.

4. Images with a valid URL but the wrong Content-Type. A CDN returned 200 for image URLs, but with an HTML body (Cloudflare error page). UrlValidator.reachable? returned true. ImageProcessor.download_image downloaded the HTML and tried to resize it. mini_magick crashed with an obscure error. The fix: validate Content-Type on the HEAD response — but this scenario would never surface with mocked URLs.

5. Newsletter assembly failed on stories with score: null. Old stories, created before the score field had a default, had nil in the frontmatter. Assembly tried to do sort_by { |s| s[:frontmatter]["score"] } — nil <=> "high" raises ArgumentError. It only appeared when I rsynced production data that had 3 months of accumulated stories.

None of these bugs would be caught by a unit test. All of them were caught by the integration pipeline with real data.

Practical Tips

1. Separate Cost From Fidelity

The full integration pipeline costs around $0.40 in API calls. Not much, but if you run it 10 times in a day while iterating, that’s $4. DevCache solves this: the first run pays, the following runs are free.

But there are scenarios where you want to pay again — when you change a prompt and need to see the real result. FORCE=1 busts the cache selectively. And SKIP_EXISTING=1 skips sections that have already been generated, saving time when you only want to re-run a specific section.

2. The Integration Environment Is Not Staging

Staging is a server running 24/7 simulating production. The integration environment is a local execution mode you turn on when needed and turn off when you’re done. No dedicated infrastructure required. No separate database required. Same code, same machine, different flags.

The difference is philosophical: staging tests “does the system work on the staging server?”. Integration tests “does the system work with this specific data?”. The first is environment-dependent. The second is data-dependent. And data bugs are more common than environment bugs.

3. Clean Before Rsync, Not After

Order matters: clean the generated content before syncing from production. If you do the opposite — rsync first, clean after — you lose the production data you just downloaded.

integration:clean is surgical by design. It removes anime/, hacker_news/, newsletters/ (generated), but preserves stories/ and images/ (production). That enables an iterative workflow: rsync once, run the pipeline, tweak a prompt, run again — without re-downloading 500MB of content.

4. Preflight Is Your Smoke Test

Don’t trust “ran without exception = correct”. A generation job can run, produce a Markdown with 2 items instead of 10, and return success. Preflight catches that by verifying minimum counts, required markers, and expected structure.

In practice, every LLM bug I’ve found was caught by preflight: truncated responses, wrong format, missing sections. GPT-5.2 is good, but it isn’t deterministic — and when it fails, it fails in creative ways no schema validation would catch.

5. Billing Summary Is Not Optional

If you’re using LLMs in production, every test pipeline needs to report cost. Not for accounting — for engineering. A prompt that works but costs 3x more than the previous one is a performance bug. Input and output tokens per provider, cost per job, total cost — all visible on every run.

6. Monorepo Helps, But Doesn’t Solve

Having the three apps in the same repository helps: a git bisect covers changes on both sides of the contract. A PR that changes the frontmatter format in marvin-bot can also update the parser in newsletter — in the same commit.

But “can” isn’t “guarantees”. The contract between the apps is file-based and implicit. There’s no shared type, no common interface. SectionParser is copied manually between the two apps. If someone edits one and forgets the other, they diverge. The integration test is the only verification that both sides agree.

7. Preview Before Sending

The pipeline doesn’t end at generation. The newsletter has a newsletter:preview that renders the final result as HTML in light and dark mode:

cd newsletter
FILE=../content/newsletters/2026-02-16.md bin/rails newsletter:preview
# → tmp/preview/newsletter_light.html
# → tmp/preview/newsletter_dark.html

Opening those HTMLs in the browser is the visual integration test. The Markdown can be perfect, the frontmatter correct, every field present — but an anime image shows up stretched because the aspect ratio changed. Or Marvin’s commentary block doesn’t have enough contrast in dark mode. Those bugs are visual — no automated test catches them.

8. The CI Pipeline Is Different From the Integration Pipeline

These are different things. CI runs on every commit:

bin/ci  # rubocop + bundler-audit + brakeman + tests (~22s)

Integration runs when you want to validate the pipeline end-to-end — before a deploy, after changing prompts, or after rsyncing fresh data from production. CI is fast and cheap. Integration is slow and costs money. Both are necessary.

The Cost of Not Having Integration

Let me be direct: if this project didn’t have the integration pipeline, I would have shipped at least 3 broken newsletters.

One with the anime section missing because the XML parser assumed RSS and the feed was Atom. One with every YouTube image pointing to 404 because the thumbnail URL pattern changed. And one with the market recap showing last month’s tickers because Yahoo Finance blocked the requests and the job failed silently.

Unit tests would have passed in all those scenarios. CI would have gone green. The deploy would have shipped. And at 7am on a Monday, 300 people would have received a newsletter full of holes.

The integration pipeline is expensive — in setup time, in API cost, in maintenance complexity. But it’s orders of magnitude cheaper than sending broken content to real subscribers. Especially when you only get one shot a week.

Conclusion

The testing hierarchy in this project is:

1. Unit tests (1,330 across both apps, 7 seconds in parallel, $0): prove each piece works in isolation
2. CI pipeline (rubocop + audit + brakeman + tests, ~22 seconds, $0): prove the code is healthy
3. Integration pipeline (real data, real APIs, ~3 minutes, ~$0.40): proves the system works
4. Visual preview (HTML in the browser, manual, $0): proves the final result is presentable

Each layer catches a different type of bug. None of them is enough alone. And the one that surprises most — the one that catches the bugs you swore didn’t exist — is the third: real data, real pipeline, no mocks.

The confidence you have in your system is proportional to how real the data in your tests is. Mock everything and you get 100% coverage and zero confidence. Test with production data and you get the truth — however inconvenient.

And make sure to subscribe to my new newsletter The M.Akita Chronicles!

–

If I told you that you can run a full Rails application in production — with database, job queue, cache — on a single $12/month VPS without installing PostgreSQL, Redis, or any other external service, would you believe me?

Well, you can. That’s Rails 8 with SQLite and Kamal.

SQLite on Rails 8: now it’s for real

Rails 8 brought SQLite as a real production option, not as a development toy. And when I say “real”, I mean rails new generates everything ready to go: main database in SQLite, SolidQueue (jobs) in SQLite, SolidCache (cache) in SQLite, SolidCable (WebSocket) in SQLite.

Four services that used to require PostgreSQL + Redis + Memcached are now .sqlite3 files on disk.

WAL mode (Write-Ahead Logging) enables concurrent reads while a write is happening. For applications with tens of thousands of requests per day — which is the vast majority of Rails applications in the world — SQLite is more than enough. And read latency is measured in microseconds, not milliseconds. It’s local disk access, not a network round-trip.

# database.yml — production with SQLite
production:
  primary:
    <<: *default
    database: storage/production.sqlite3
  queue:
    <<: *default
    database: storage/production_queue.sqlite3
    migrations_paths: db/queue_migrate
  cache:
    <<: *default
    database: storage/production_cache.sqlite3
    migrations_paths: db/cache_migrate

Three files. No TCP connection. No pooling. No “is PostgreSQL accepting connections?”. Backup is a VACUUM INTO and that’s it — a single SQL command that produces an atomic copy. Restore? Paste the file back. Try doing that with a 2GB PostgreSQL dump.

Kamal: Docker Deploy Without Kubernetes

Kamal is the Rails 8 deployer. Think of it as “SSH + Docker, but smart”. No Kubernetes. No Helm charts. No 300-line YAML describing an Ingress Controller.

The entire configuration fits in one file:

# config/deploy.yml
service: minha-app
image: meuregistry/minha-app

servers:
  web:
    hosts:
      - 107.170.70.49
    options:
      network: minha-network
    volumes:
      - minha-app-storage:/rails/storage
      - minha-app-content:/rails/content

proxy:
  ssl: true
  host: app.meudominio.com

builder:
  arch: amd64

env:
  clear:
    RAILS_LOG_LEVEL: info
  secret:
    - RAILS_MASTER_KEY
    - DATABASE_URL

What Kamal does when you run kamal deploy:

1. Builds the Docker image locally (or in CI)
2. Pushes to the registry
3. SSHs into the server
4. Pulls the new image
5. Runs migrations
6. Restarts the container with zero downtime via kamal-proxy

The kamal-proxy is the secret: it acts as a reverse proxy in front of your containers. When the deploy happens, it brings up the new container, waits for it to be healthy, redirects traffic, and then tears down the old one. Your users never notice a thing.

Docker Volumes: The Secret of SQLite in Production

The critical point of SQLite in Docker is: the data can’t die with the container. Docker volumes solve this:

volumes:
  - minha-app-storage:/rails/storage

The storage/ directory contains all the SQLite databases. With the Docker volume, they persist across deploys. The container is ephemeral; the data is permanent.

But there’s a detail that trips up a lot of people: if you run two different services that share data through files (not database), you need a shared volume:

# Service A writes content
volumes:
  - content-compartilhado:/rails/content

# Service B reads content
volumes:
  - content-compartilhado:/rails/content

The same Docker volume mounted on two containers. Simple, it works, and you don’t need NFS, S3, or anything fancy. It’s a directory on the server’s disk.

Multiple Services on the Same VPS

A rarely discussed advantage of Kamal: you can run multiple applications on the same server. The kamal-proxy routes by hostname:

app.meudominio.com    → container-app
bot.meudominio.com    → container-bot

Each service has its own deploy.yml, its own Docker image, its own deploy cycle. But they all run on the same VPS, sharing the same Docker network for internal communication.

SSL is automatic via Let’s Encrypt — kamal-proxy handles that. Each hostname gets its certificate without you configuring anything.

Hooks: Deploy Automation

Kamal supports hooks at various points of the deploy cycle. The most useful is pre-deploy, which runs before the new container replaces the old one:

# .kamal/hooks/pre-deploy
#!/bin/sh
# Run migrations before the container goes live
ssh root@$KAMAL_HOSTS \
  "docker exec minha-app-web bin/rails db:migrate"

Other useful hooks:

• post-deploy — notify the team, clear cache
• pre-connect — check server health
• docker-setup — install dependencies on the host

The Dockerfile Rails Generates

Rails 8 generates an optimized production Dockerfile. Some highlights:

# Multi-stage build: large build stage, minimal runtime stage
FROM ruby:3.3-slim AS base

# Install only runtime dependencies
RUN apt-get install -y libsqlite3-0

FROM base AS build
# Here install everything to compile native gems
RUN apt-get install -y build-essential libsqlite3-dev

# Copy and install gems
COPY Gemfile* ./
RUN bundle install --without development test

# Final stage: only what's needed to run
FROM base
COPY --from=build /usr/local/bundle /usr/local/bundle
COPY . .
RUN bundle exec bootsnap precompile --gemfile app/ lib/

Multi-stage build means the final image has no compilers, no development headers, nothing unnecessary. The production image stays lean.

Secrets: No .env in Production

Kamal has a secrets system that reads from a local file (.kamal/secrets) and injects them as environment variables into the container. That file never goes to git:

# .kamal/secrets (gitignored)
RAILS_MASTER_KEY=abc123...
AWS_ACCESS_KEY_ID=AKIA...

In deploy.yml, you reference:

env:
  secret:
    - RAILS_MASTER_KEY
    - AWS_ACCESS_KEY_ID

Kamal reads from the local file and configures it on the server. Simple, secure, no HashiCorp Vault or AWS Secrets Manager (unless you want — Kamal supports adapters).

SQLite Backup: Absurdly Simple (If You Do It Right)

Want a backup? One SQL command:

VACUUM INTO '/tmp/backup/newsletter.sqlite3';

VACUUM INTO is atomic — it produces a consistent, defragmented copy of the database, even with writes happening. It runs while the application is serving requests normally. And the resulting copy is a complete, functional SQLite database — you can open it, query it, restore it.

Warning: copying the .sqlite3 file directly (cp, tar, rsync) from a database in use can corrupt the backup. If a write is in progress at the moment of the copy, you end up with a half-written file. SQLite has WAL (Write-Ahead Log) and journaling that protect the active database — but the raw copy doesn’t inherit that protection.

In practice, I automated this with a Rails job that runs every hour:

class BackupDatabaseJob < ApplicationJob
  def perform
    backup_dir = File.join(Rails.configuration.content_dir, "backups")
    FileUtils.mkdir_p(backup_dir)
    dest = File.join(backup_dir, "newsletter.sqlite3")
    ActiveRecord::Base.connection.execute("VACUUM INTO '#{dest}'")
  end
end

The backup goes to the shared content directory — the same one that’s already synced with rsync to my local machine. No extra backup agent, no external service, no volume snapshot. One SQL, one file, one rsync.

Compare that with pg_dump of a production PostgreSQL database with dozens of tables and constraints. The SQLite backup is one command that produces a working file. Restore? Copy it back and restart.

When NOT to Use SQLite

I’ll be honest: SQLite isn’t for everything.

• Multiple web servers writing to the same database? No. SQLite is single-writer. If you need horizontal scaling with multiple nodes, go PostgreSQL.
• Massive datasets (hundreds of GB)? PostgreSQL has better query planning and parallelism.
• Replication and high availability? PostgreSQL with streaming replication.

But the honest question is: how many applications really need that? The vast majority run on a single server and will never need more. And for those, SQLite + Kamal is the most productive combination out there today.

The Real Cost

Let’s do the math:

And the cognitive cost? Instead of debugging “why did Redis lose my jobs?”, “why is PostgreSQL rejecting connections?”, “why did the cert expire?”, you focus on what matters: the product.

Conclusion

SQLite + Kamal is not a downgrade. The truth is that most applications never needed the complexity we thought they needed. Rails 8 embraced that and delivered a deploy experience that is, without exaggeration, the simplest that ever existed in the Ruby ecosystem.

One VPS. One command. Zero external services. And an application that runs as fast as any enterprise setup with 15 services in docker-compose.

And don’t forget to subscribe to my new newsletter The M.Akita Chronicles!

–

Every Rails project starts the same way: you build an admin panel. Active Admin, Administrate, or some homegrown CRUD. Tables, forms, buttons. And every week, you end up using 5% of the screens you built.

Now picture your admin panel as a conversation. You paste a link; the system processes it. You type /week and see the summary. /approve and the newsletter is scheduled. /ask akita how does the podcast pipeline work? and an AI explains the architecture of your own system to you.

That’s Discord as an operational interface. It’s not a hack — it’s the best CRUD interface there is for a team of one.

Why Discord Instead of a Web Admin?

Let’s be honest about what an admin panel demands:

1. Authentication: login, session, CSRF, password reset
2. Authorization: roles, permissions, who can do what
3. UI: tables, pagination, forms, client-side validation
4. Deployment: another route, another controller, another set of tests
5. Mobile: responsive? App? PWA?

And what Discord already gives you for free:

1. Auth: Discord handles it. You just check the author_id
2. Authorization: a list of admin IDs and you’re done
3. UI: embeds with colors, fields, thumbnails, reactions as status
4. Deployment: zero. Uses what’s already deployed
5. Mobile: native iOS and Android app, push notifications

The trade-off? It doesn’t fit everything. A complex CRUD with 20-field forms? Better off with a web admin. But for command/response operations — which are 90% of what you do day to day — Discord is unbeatable.

discordrb: A Gem That Works

The discordrb gem (shardlab fork, version 3.7.2) is the most mature one for Ruby. It’s not perfect — it has its quirks. But it genuinely works in production.

The minimal setup:

bot = Discordrb::Bot.new(
  token: token,
  intents: Discordrb::INTENTS[:server_messages] |
           Discordrb::INTENTS[:direct_messages] |
           (1 << 15)  # message_content intent
)

bot.message do |event|
  # Every message flows through here
end

bot.run

That (1 << 15) is the Message Content Intent. Without it, your bot receives events but event.content comes in empty. You need to enable it in the Discord Developer Portal > Bot > Privileged Gateway Intents. That’s the first trap that catches everyone.

I won’t walk through how to set up a Discord bot, there are plenty of tutorials for that already, easy to find online. I’ll focus only on my use case as an example.

Pattern 1: Parser + Dispatcher

Mixing command parsing with business logic is a recipe for spaghetti. Split it into three layers:

Discord Event → Parser → Dispatcher → Job/Service

The runner (the thing that talks to the Discord API) is deliberately dumb:

bot.message do |event|
  next if event.author.bot_account?

  parsed = SlashCommandParser.parse(event.content, attachments: urls)

  case parsed
  when SlashCommandParser::Command
    CommandDispatcher.dispatch(parsed, channel_id: channel_id)
    event.message.react("\u{2705}")  # checkmark
  when SlashCommandParser::Submission
    ProcessMessageJob.perform_later(url: parsed.url, tags: parsed.tags)
    event.message.react("\u{1F916}")  # robot
  end
rescue StandardError => e
  event.message.react("\u{274C}")  # red X
end

Three instant visual results: checkmark (command accepted), robot (URL being processed), red X (error). The user knows what happened without waiting for any text response.

The parser returns typed objects, not strings. The dispatcher runs a giant case but each handler is an isolated method. Each handler calls a job or service. No business logic in the runner.

Evolution: Discord Application Commands

The initial version parsed raw text — the user typed /week and the parser interpreted it. It works, but it has limits: no autocomplete, no argument validation, no inline documentation.

Discord has a native Application Commands system — slash commands registered with the API that show up with autocomplete, description, and typed parameters. When the user types /, Discord displays every available command with its description.

bot.register_application_command(:week, "List this week's stories", server_id: guild_id)
bot.register_application_command(:approve, "Approve newsletter for sending", server_id: guild_id)
bot.register_application_command(:ask, "Ask AI a question", server_id: guild_id) do |cmd|
  cmd.string("question", "Your question", required: true)
  cmd.string("personality", "AI personality", choices: { "Akita" => "akita", "Marvin" => "marvin" })
end

The handler uses bot.application_command instead of parsing text:

bot.application_command(:week) do |event|
  CommandDispatcher.dispatch_week(channel_id: event.channel.id)
  event.respond(content: "Listing stories...")
end

Both systems coexist: Application Commands for those who prefer autocomplete, raw text for those who prefer typing fast. The dispatcher is the same — the only difference is how the command arrives.

Pattern 2: Reactions as Status

The most underrated feature of Discord for bots is reactions. They’re instant visual feedback that doesn’t pollute the channel with messages:

• Checkmark = command received and queued
• Robot = async processing started
• Red X = error (details in the embed that follows)

The user pastes a link. Sees the robot appear. 30 seconds later, a rich embed shows up with title, summary, image and AI commentary. If something fails, a red embed appears with the error message.

That beats any loading spinner in a web UI.

Pattern 3: Rich Embeds as Dashboard

Discord embeds are surprisingly powerful:

embed = {
  title: "Story Preview",
  url: source_url,
  color: 0xFF4500,           # orange-red for high priority
  description: summary,
  thumbnail: { url: image },  # article image
  fields: [
    { name: "M.Akita diz", value: akita_comment, inline: false },
    { name: "M.Arvin diz", value: marvin_comment, inline: false },
    { name: "File", value: "`stories/2026-02-16-titulo.md`", inline: false }
  ],
  footer: { text: "fire high | tech_news | verified" }
}

Discordrb::API::Channel.create_message(token, channel_id, "", false, [embed])

Colors signal priority (red = high, amber = medium, green = low). The footer shows score, section and fact-check status all on a single line. You glance at it and know the state of things.

For reports, multiple embeds side by side (Discord accepts up to 10 per message) work as a mini-dashboard. The /week command lists every story with inline actions — /load @a3f2b1 to see details, /delete @a3f2b1 to remove it. The hash is stable (based on the file path), so it doesn’t shift when you delete another story.

Pattern 4: Context Per Channel

When the user types /load #3, the #3 refers to the most recent listing. But what if two channels are being used at the same time?

class ChannelContext < ApplicationRecord
  EXPIRY_DURATION = 2.hours

  def self.for_channel(channel_id)
    find_or_create_by!(channel_id: channel_id.to_s)
  end

  def set_week_stories!(hash)
    update!(week_stories_json: hash.to_json, expires_at: EXPIRY_DURATION.from_now)
  end

  def resolve_story_reference(ref)
    if ref.match?(/\A@[a-f0-9]+\z/i)
      # Hash reference (stable across re-listings)
      hash = ref.delete("@").downcase
      week_stories.values.find { |path| md5_prefix(path) == hash }
    elsif ref.match?(/\A#?\d+\z/)
      week_stories[ref.delete("#")]
    end
  end
end

Each channel has its own context with a 2-hour expiration. /week stores the number→file mapping. Then /load #3 queries that context. If it expired, the bot says “run /week first”.

And the most useful detail: hash-based references (@a3f2b1) instead of numbers. When you delete story #2, #3 becomes #2 — but the hash @a3f2b1 still points to the same file. That avoids the classic “I deleted the wrong one because the list shifted”.

Pattern 5: Simple Admin Check

No complex RBAC. A list of IDs:

ADMIN_COMMANDS = %i[delete update_score approve newsletter publish_blog].freeze

def self.dispatch(command, channel_id:, author_id:)
  if ADMIN_COMMANDS.include?(command.type) && !admin?(author_id)
    send_error(channel_id, "Permission denied. Admin privileges required.")
    return
  end
  # ...
end

def self.admin?(author_id)
  admin_ids = ENV["DISCORD_ADMIN_USER_IDS"].to_s.split(",")
  return true if admin_ids.empty?  # dev: no IDs = everything allowed
  admin_ids.include?(author_id)
end

In dev, without DISCORD_ADMIN_USER_IDS set, everything is allowed. In production, it’s a list of Discord user IDs. No database, no roles table, no middleware. Ten lines handle authorization.

Pattern 6: Async Operations with Feedback

Slow operations (generating an image, processing an article, calling an AI) go to background jobs. Visual feedback comes in three beats:

def self.handle_ask(personality, question, attachments, channel_id)
  # 1. Immediate feedback: "I'm thinking"
  thinking_id = DiscordNotifier.send_message(channel_id, content: "Thinking...")

  # 2. Job does the heavy lifting in the background
  AskJob.perform_later(
    question,
    channel_id: channel_id,
    thinking_message_id: thinking_id
  )
end

# In AskJob:
def perform(question, channel_id:, thinking_message_id: nil)
  result = AiChat.ask_multimodal(question, tools: [GenerateImageTool])

  # 3. Remove "Thinking..." and show result
  DiscordNotifier.delete_message(channel_id, thinking_message_id)
  DiscordNotifier.send_message(channel_id, content: "**Marvin:** #{result[:text]}")
end

The “Thinking…” shows up instantly, the job runs in the background, and when it finishes the temporary message disappears and the real result appears. If the job fails, “Thinking…” disappears and an error embed shows up.

Without this, the user sends a question and stares at the screen not knowing if the bot heard them. Terrible UX.

Pattern 7: RubyLLM + Tool Calling on Discord

The most powerful part: giving the LLM tools. /ask has three tools available: GenerateImageTool (generates images via Gemini), WebSearchTool (searches the web via Brave Search), and WebFetchTool (fetches and summarizes web pages). The LLM decides on its own which one to use based on the question. When the user asks “generate an image of a cat programming”, the LLM picks the right tool:

class GenerateImageTool < RubyLLM::Tool
  description "Generate an image using Gemini. Use when the user asks to create or draw."
  param :prompt, desc: "Image description"
  param :aspect_ratio, desc: "1:1 or 16:9", required: false

  def execute(prompt:, aspect_ratio: "1:1")
    image_data = ImageGenerator.generate(prompt: prompt)
    return "Failed to generate image" unless image_data

    path = save_temp_file(image_data)
    Thread.current[:generated_images] << path

    "Image generated successfully. Tell the user it's ready."
  end
end

The Thread.current[:generated_images] is the trick to pass data back from the tool to the job. The LLM calls the tool during chat.ask, the tool saves the file and registers the path. After chat.ask returns, the job grabs the paths and sends them to Discord as file uploads.

Why thread-local and not a direct return? Because the LLM can call multiple tools within a single response, and the tool’s return value goes back to the LLM (not to your code) — the LLM decides what to say to the user. Thread-local is the side channel for your code to know what happened.

Pattern 8: Production Status Notifications

Every content generation job notifies a dedicated channel:

module DiscordStatus
  def notify_start(label)
    DiscordNotifier.send_to_status_channel("Starting #{label}...")
  end

  def notify_done(label)
    DiscordNotifier.send_to_status_channel("#{label} complete.")
  end

  def notify_error(label, error)
    DiscordNotifier.send_to_status_channel("#{label} FAILED: #{error}")
  end
end

The status channel is separate from the operations channel. When the 8 content generation jobs run on Sunday at 5pm, the status channel turns into a real-time log:

Starting Anime Ranking...
Starting Hacker News...
Starting YouTube Digest...
Anime Ranking complete.
YouTube Digest complete.
Hacker News complete.
Starting Market Recap...  (wave 2, after world events)
...

In the morning, if something went wrong, you open Discord on your phone and you know exactly what — no SSH into the server, no grep in logs.

And when the bot restarts after a deploy:

bot.ready do |_event|
  status_channel = ENV["DISCORD_STATUS_CHANNEL_ID"]
  DiscordNotifier.send_message(status_channel, content: "Bot reconnected and ready.")
end

Simple, but you know the deploy worked without checking anything.

Pattern 9: Cross-App Communication

The system has two Rails apps that need to talk to each other. The newsletter needs to notify Discord when it finishes assembling an edition. But the newsletter has no Discord connection — marvin-bot is the one that does.

Solution: HTTP relay.

# In the newsletter — send it to marvin-bot
class DiscordRelayClient
  def self.send_embed(title:, description:, color:)
    post("/api/discord_notify",
      type: "embed",
      payload: { title: title, description: description, color: color }
    )
  end
end

# In marvin-bot — receive it and forward to Discord
class Api::DiscordNotifyController < ApplicationController
  def create
    case params[:type]
    when "embed"
      DiscordNotifier.send_embed(status_channel, params[:payload])
    when "message"
      DiscordNotifier.send_message(status_channel, content: params[:payload][:content])
    end
  end
end

The newsletter speaks HTTP. marvin-bot translates it to Discord. A shared ADMIN_TOKEN authenticates. Every method rescues errors and returns nil — if Discord is down, the newsletter doesn’t lock up. Fire and forget.

But there’s a second, more interesting cross-app flow: the podcast trigger. When the newsletter is approved, it needs to tell marvin-bot to generate the podcast:

# In the newsletter — trigger podcast generation on marvin-bot
class MarvinBotClient
  def self.trigger_podcast(date:)
    post("/api/trigger/podcast", date: date.to_s)
  end
end

That kicks off an entire pipeline on marvin-bot: two-pass LLM script → per-turn TTS on the GPU → assembly with loudnorm → S3 upload. And the newsletter’s PublishAndSendJob waits (via retry_on PodcastNotReady) until the podcast is ready. Two apps, two backend languages (Ruby and Python), connected by fire-and-forget HTTP with result polling. It’s not elegant — but it works every single week with no intervention.

Pattern 10: Integrated Cost Tracking

Every AI call is tracked with tokens and estimated cost:

def self.create_chat(caller:, model:, provider:)
  chat = RubyLLM.chat(model: model, provider: provider)
  chat.on_end_message do |message|
    ApiUsage.create!(
      caller_name: caller,
      input_tokens: message.input_tokens,
      output_tokens: message.output_tokens,
      cost_usd: estimate_cost(model, message)
    )
  end
  chat
end

And on Discord: /cost shows spending by month, year, and by provider. No separate dashboard, no Grafana, nothing. One command and the information that matters.

Pattern 11: Never Fail Silently

This is the pattern I should have implemented from the start, but I only learned it the hard way — in production.

The project has ~10 daily jobs that fetch data from external sources (Hacker News, GitHub Trending, Morningstar, etc.) and post embeds to Discord. Each client follows a “defensive” pattern:

# The pattern that LOOKS right:
class DailyDevClient
  def self.fetch_popular(limit: 10)
    response = HTTParty.get(API_URL, headers: { "Authorization" => "Bearer #{token}" })
    return [] unless response.success?
    parse(response.body)
  rescue StandardError => e
    Rails.logger.warn("DailyDevClient failed: #{e.message}")
    []
  end
end

# And in the job:
class DailyDailyDevJob < ApplicationJob
  def perform
    posts = DailyDevClient.fetch_popular(limit: 10)
    return if posts.empty?  # ← THE PROBLEM
    # ... posts embed to Discord
  end
end

This code never crashes. Never raises an exception. Never fills the log with errors. It’s the most “robust” code possible.

It’s also completely useless for diagnosis. When I deployed and ran the daily jobs manually, only one of ten worked. But there was no error in the logs. None. return [] unless response.success? swallowed the HTTP 401 without logging. return if posts.empty? discarded the result without telling anyone. All green, all silent, zero data.

The fix has two parts:

Part 1: Clients log the HTTP code:

unless response.success?
  Rails.logger.warn("DailyDevClient: API returned HTTP #{response.code}")
  return []
end

It sounds obvious, but it’s easy to forget when you write return [] as a defensive reflex. response.code is the information that distinguishes “expired token” (401) from “rate limited” (429) from “API down” (503). Without it, everything looks the same: empty array.

Part 2: Jobs warn on Discord when they have no data:

module DailyDiscordEmbed
  def report_empty(title)
    Rails.logger.warn("#{self.class.name}: #{title} returned no results")
    channel_id = Rails.application.config.discord.channel_ids.first
    return unless channel_id.present?

    embed = {
      title: "⚠️ #{title}",
      description: "#{self.class.name} returned no results. Check logs.",
      color: 0xFFA500,  # orange = warning, not red = error
      timestamp: Time.current.iso8601
    }
    DiscordNotifier.send_embed(channel_id, embed)
  rescue StandardError => e
    Rails.logger.error("#{self.class.name} empty report failed: #{e.message}")
  end
end

Now when a daily job brings back no data:

def perform
  posts = DailyDevClient.fetch_popular(limit: 10)
  if posts.empty?
    report_empty("daily.dev Popular")
    return
  end
  # ...
end

An orange embed appears on Discord. In the morning, I open the channel and see: “⚠️ daily.dev Popular returned no results”. I check the logs and find: “DailyDevClient: API returned HTTP 401”. Expired token. Fixed in 2 minutes.

Without this pattern, I would have discovered it weeks later — when someone asked “why are there never any daily.dev posts?”.

The lesson: rescue => [] is not error handling. It’s error hiding. Every failure point needs two signals: a log with the technical detail (HTTP code, exception message) and a visual notification in the channel where you’re already looking (Discord). If the bot IS your operational interface, bot failures need to show up ON the interface.

Pattern 12: Conversations With Memory (Multi-Turn /ask)

The first version of /ask created a brand new RubyLLM::Chat on every invocation. It worked, but it was like talking to someone with amnesia:

/ask what is Ruby?
→ Ruby is a programming language created by Matz in 1995...

/ask compare it with Python
→ Compare what with Python? Could you be more specific?

ruby_llm already keeps history internally — every chat.ask() appends to the message list and sends the full context to the model. What was missing was persisting the Chat object between invocations.

The solution is an in-memory store with three safeguards:

class AskSessionStore
  MAX_SESSIONS = 100       # total across all users
  SESSION_TTL = 2.hours    # inactivity timeout
  MAX_MESSAGES = 50        # ~25 exchanges, then auto-reset

  Session = Struct.new(:chat, :last_used, keyword_init: true)

  def get_or_create(author_id, personality, system_prompt:, tools:)
    synchronize do
      cleanup_expired
      key = [author_id, personality]

      if (session = @sessions[key])
        session.last_used = Time.current
        if session.chat.messages.length > MAX_MESSAGES
          @sessions.delete(key)
          return create_session(key, system_prompt: system_prompt, tools: tools)
        end
        session.chat
      else
        evict_oldest if @sessions.size >= MAX_SESSIONS
        create_session(key, system_prompt: system_prompt, tools: tools)
      end
    end
  end
end

The key is (author_id, personality) — each user has separate sessions for Marvin and for Akita. Volatile on purpose: restart the server, lose the history. No database, no serialization, no complexity. It’s working memory, not persistence.

/ask clear resets the session. Without it, there’s no way to “start over” when the conversation gets tangled.

Now:

/ask what is Ruby?
→ Ruby is a programming language created by Matz in 1995...

/ask compare it with Python
→ Ruby and Python have similar philosophies — both prioritize programmer
  happiness — but Ruby embraces blocks and metaprogramming while Python
  prefers explicitness...

The LLM knows “compare” refers to Ruby because the session history includes the previous question. That’s the difference between a chatbot and an assistant.

Pattern 13: Registering Application Commands (Native Autocomplete)

The Application Commands section above showed the simplified concept. In practice, registering 18 commands with subcommands, typed parameters and choices demands a dedicated structure. Here’s how the project handles it.

The Problem With Raw Text

When the bot only parses text, the user has to memorize the syntax:

/update score #1 #2 #3 high       ← works
/update score #1 #2 #3 alta       ← fails silently
/update scoree #1 high             ← ignored

No autocomplete, no parameter validation, no inline documentation. The user types / and has no idea what’s available.

The DiscordCommands Module

All command definitions live in a separate module with two public methods: register! (registers the commands with the Discord API) and setup_handlers! (wires the handlers to the bot):

module DiscordCommands
  SCORE_CHOICES  = { "High" => "high", "Medium" => "medium", "Low" => "low" }.freeze
  SECTION_CHOICES = {
    "Tech" => "tech", "Global" => "global", "Financial" => "financial",
    "Q&A" => "qa", "Entertainment" => "entertainment", "Open Source" => "opensource",
    "Product" => "product", "Book" => "book"
  }.freeze
  PERSONALITY_CHOICES = { "Marvin (default)" => "marvin", "Akita" => "akita" }.freeze

  def self.register!(bot)
    server_id = Rails.application.config.discord.server_id
    sid = server_id.present? ? server_id.to_i : nil

    register_week(bot, sid)
    register_add(bot, sid)
    register_update(bot, sid)
    register_ask(bot, sid)
    # ... 14 more commands
  end

  def self.setup_handlers!(bot)
    setup_week_handlers(bot)
    setup_add_handler(bot)
    setup_update_handlers(bot)
    setup_ask_handlers(bot)
    # ... 14 more handlers
  end
end

Guild vs Global

The server_id (or sid) controls the registration scope. With server_id, the command shows up instantly on the specified server. Without it, the command is global — available on every server, but can take up to an hour to propagate.

For a private bot doing internal operations, always go guild-scoped. The DISCORD_SERVER_ID env var is set at deploy time:

sid = server_id.present? ? server_id.to_i : nil
bot.register_application_command(:week, "Story list and weekly report", server_id: sid)

If sid is nil, discordrb registers globally. In dev it’s convenient; in production, guild-scoped is faster and lets you have different commands per server.

Anatomy of a Command

The simplest method — no parameters:

def self.register_count(bot, sid)
  bot.register_application_command(:count, "Show story count for this week", server_id: sid)
end

With typed parameters:

def self.register_add(bot, sid)
  bot.register_application_command(:add, "Submit a URL for the newsletter", server_id: sid) do |cmd|
    cmd.string(:url, "The URL to submit", required: true)
    cmd.string(:tags, "Tags (e.g. #tech #high)")
    cmd.string(:comment, "Optional comment about this link")
  end
end

Discord renders this as an inline form: required url field, optional tags and comment. The user sees each parameter’s description as they fill it in.

Subcommands: Grouping Actions

Complex commands use subcommands. /update on its own does nothing — the user has to pick what to update:

def self.register_update(bot, sid)
  bot.register_application_command(:update, "Update story properties", server_id: sid) do |cmd|
    cmd.subcommand(:score, "Update story priority score") do |sub|
      sub.string(:stories, "Story numbers (e.g. #1 #2 #3 or 1,2,3)", required: true)
      sub.string(:level, "Priority level", required: true, choices: SCORE_CHOICES)
    end
    cmd.subcommand(:tag, "Update story section tag") do |sub|
      sub.string(:stories, "Story numbers (e.g. #1 #2 #3 or 1,2,3)", required: true)
      sub.string(:section, "Section tag", required: true, choices: SECTION_CHOICES)
    end
    cmd.subcommand(:image, "Update story image") do |sub|
      sub.string(:stories, "Story numbers (e.g. #1 #2 #3 or 1,2,3)", required: true)
      sub.attachment(:file, "Image file (omit for AI generation)")
    end
    # ... more subcommands: title, description, comment, source
  end
end

When the user types /update, Discord shows a list of subcommands: score, tag, title, description, comment, image, source. Pick score and the stories field (free text) and level field (dropdown with High/Medium/Low) appear. Syntax errors become impossible.

Choices: Native Dropdowns

The SCORE_CHOICES hash becomes a dropdown on Discord. The key is the visible label (“High”), the value is what arrives in the handler (“high”):

SCORE_CHOICES = { "High" => "high", "Medium" => "medium", "Low" => "low" }.freeze

# In the registration:
sub.string(:level, "Priority level", required: true, choices: SCORE_CHOICES)

The user doesn’t have to remember whether it’s “high”, “HIGH”, “alta”, or “3”. Pick it from the dropdown and done. That eliminates an entire category of input errors.

Attachments: File Uploads

Discord supports attachment-type parameters — the user can drag a file straight into the command:

cmd.subcommand(:image, "Update story image") do |sub|
  sub.string(:stories, "Story numbers (e.g. #1 #2 #3 or 1,2,3)", required: true)
  sub.attachment(:file, "Image file (omit for AI generation)")
end

In the handler, the resolved attachments come through as Discord CDN URLs:

def self.resolve_attachments(event)
  return [] unless event.resolved&.attachments&.any?
  event.resolved.attachments.values.map(&:url)
end

Lifecycle: Where to Register, Where to Wire Handlers

This is the point that confuses people most. The handlers have to be registered before bot.run. The registration with the Discord API can happen inside bot.ready (which fires after the WebSocket connection):

class MarvinBotRunner
  def self.run!
    bot = Discordrb::Bot.new(token: token, intents: ...)

    # 1. Handlers BEFORE bot.run (otherwise the bot doesn't know how to respond)
    DiscordCommands.setup_handlers!(bot)

    bot.ready do |_event|
      # 2. API registration AFTER connection (idempotent, safe on reconnect)
      DiscordCommands.register!(bot)
    end

    bot.run
  end
end

Why inside bot.ready? Because the bot can reconnect (Discord closes WebSockets periodically). register! is idempotent — registering the same command twice doesn’t duplicate it, it just updates. Putting it in ready makes sure the commands exist even after a reconnect.

Bridge: Application Command → CommandDispatcher

The most elegant part of the architecture: the Application Command handler has no business logic. It translates Discord’s parameters into the format the CommandDispatcher already expects:

def self.dispatch_interaction(event, type, match: [nil], attachments: [])
  event.respond(content: "\u{2705}", ephemeral: true)  # checkmark visible only to the author
  command = SlashCommandParser::Command.new(
    type: type,
    args: { match: match, attachments: attachments }
  )
  CommandDispatcher.dispatch(command,
    channel_id: event.channel.id.to_s,
    message_id: nil,
    author_id: event.user.id.to_s
  )
rescue StandardError => e
  Rails.logger.error("Discord interaction error (#{type}): #{e.message}")
end

Each subcommand handler is a one-liner that assembles the args and calls dispatch_interaction:

bot.application_command(:update).subcommand(:score) do |event|
  stories = event.options["stories"]
  level = event.options["level"]
  dispatch_interaction(event, :update_score, match: [nil, stories, nil, level])
end

CommandDispatcher doesn’t know if the command came from raw text or from an Application Command. Same logic, same authorization, same jobs. No duplicated code.

The Result

The project registers 18 Application Commands with combinations of subcommands, typed parameters, dropdowns and file uploads. The user types / and sees everything available with descriptions. Pick a command and Discord guides you field by field. Syntax errors are impossible — Discord validates before sending.

And both systems coexist: people who prefer autocomplete use Application Commands, people who prefer to type fast use raw text. CommandDispatcher receives both the same way. Adding a new command means adding a register_*, a setup_*_handler, and a dispatcher method — without touching the runner.

The discordrb Pitfalls

It’s not all roses. Some real pain points:

Gateway disconnects: The WebSocket to Discord drops every once in a while. discordrb reconnects automatically, but your bot.ready and bot.message handlers need to be resilient to reconnects.

Message Content Intent: Since 2022, bots in more than 75 servers need approval to read message content. For private bots (which is the case for internal operations), that’s a non-issue. But if you plan to distribute the bot, get ready for paperwork.

API Rate Limits: Discord rate-limits per route. Sending 10 embeds at once to the same channel? You’ll hit the limit. discordrb handles rate limits internally, but calls that are too fast cause visible delays.

Embed Limits: Title max 256 chars. Description 4096. Field value 1024. Exceeded them? Silent error. Always truncate before sending.

File Uploads: 25MB limit on the free plan, 50MB with Nitro. For AI-generated images that’s never an issue, but if you’re shipping PDFs or audio, be careful.

When NOT to Use Discord as Admin

• Complex forms: editing 15 fields on a record? Web screen
• Bulk operations: importing a CSV with 1000 records? Script
• Data visualization: charts, analytical dashboards? Grafana/Metabase
• Multi-step approval: workflows with 5 approval stages? You need persisted state

But for day-to-day operations — curating content, approving newsletters, monitoring jobs, checking costs, talking to AI — Discord is the most productive interface I’ve ever used. And the best part: the subscribers will never know their weekly newsletter was curated by a guy typing commands into Discord on his phone while drinking coffee.

Conclusion

Using Discord as an admin panel isn’t laziness — it’s pragmatism. The best interface is the one you already use all day long. Auth, push notifications, mobile, rich formatting, file sharing — all free. You only build the business logic.

The ruby_llm gem with tool calling takes the bot from “command executor” to “assistant that thinks”. Add session memory and it thinks and remembers. When the LLM can generate images, search the web, keep context across questions, and answer questions about the architecture of its own system — that’s when it stops being CRUD and becomes a co-pilot.

And the Parser → Dispatcher → Job pattern keeps everything testable. The parser knows nothing about Discord. The dispatcher knows nothing about AI. The jobs know nothing about the interface. Swap any one of those layers and the others don’t even notice.

In the end, the best admin UI is the one that doesn’t need a separate deploy. And the worst bug is the one that fails in silence — if the bot is your operational interface, the bot’s failures need to show up on the interface, not hidden away in a log nobody reads.

And make sure to subscribe to my new newsletter The M.Akita Chronicles!

–

Let me be blunt: most developers treat background jobs as if they were scripts that run once and that’s it. “Ah, throw it in a Sidekiq and we’re good.” No, we’re not good. Jobs that handle important things — sending emails, publishing content, calling external APIs — need to be treated as first-class citizens in your architecture.

In this post I’ll show how Rails 8 with ActiveJob and SolidQueue changed my perspective on async processing. This is not theory — these are patterns that emerged from a real system running in production every week.

The Problem: Jobs Are Fragile by Nature

Picture the scenario: you have a job that assembles a newsletter, publishes it on a blog through the GitHub API, waits for a podcast to be ready, and then fires off emails to hundreds of subscribers. Any step can fail. The API can time out. The podcast can take longer than expected. The server can restart in the middle of the delivery.

If you treat this as a linear script, you’re going to suffer. The right question is not “how do I make this job run?” — it’s “how do I make this job recover when something goes wrong?”

Pattern 1: retry_on With Specific Exceptions

ActiveJob’s retry_on is absurdly powerful, but most people use it wrong. Look at the classic anti-pattern:

class MeuJob < ApplicationJob
  retry_on StandardError, wait: 5.seconds, attempts: 3
end

This retries any error 3 times and gives up. It’s useless for almost anything in the real world. The pattern that actually works is to create specific exceptions for transient states:

class PodcastNotReady < StandardError; end

class PublishAndSendJob < ApplicationJob
  retry_on PodcastNotReady, wait: 15.minutes, attempts: 16

  def perform(newsletter_id)
    metadata = check_podcast_metadata(newsletter_id)
    raise PodcastNotReady, "Aguardando podcast" unless metadata
    # keep working...
  end
end

See what’s happening? The job waits up to 4 hours (15min × 16) for the podcast to be ready, polling every 15 minutes. This is not an infinite loop — it has a clear limit. And when it hits that limit, you handle the timeout explicitly:

rescue PodcastNotReady => e
  if executions >= 16
    # Publish without podcast and notify the team
    publish_without_podcast(newsletter)
    notify_team("Podcast não ficou pronto a tempo")
  else
    raise # re-raise so SolidQueue schedules the next retry
  end
end

This is graceful degradation. The system doesn’t stop because one component failed — it adapts and keeps going.

Pattern 2: Distributed Locks

When you have multiple paths that can trigger the same job (a safety cron, a manual trigger, an API), you need to make sure only one instance runs at a time.

The concept is simple: a file-based (or database-based) lock that expires automatically:

class DeployLock
  LOCK_DIR = Rails.root.join("tmp/locks")
  DEFAULT_TTL = 30.minutes

  def self.with_lock(name, ttl: DEFAULT_TTL)
    acquire!(name, ttl: ttl)
    yield
  ensure
    release(name)
  end
end

With SQLite, this is trivial — no Redis, no external coordination. The lock has a TTL so it doesn’t hang forever if the process dies. And the ensure guarantees it releases even if the block raises an exception.

Pattern 3: Atomic Email Delivery

This is where most newsletter systems fail spectacularly. The scenario: you’re sending 500 emails and the server restarts on email 247. What happens when the job re-runs?

If you just iterated a list, you’re going to resend the first 247. Your subscribers will love getting the newsletter twice.

The solution is atomic claiming per recipient:

# Before sending, create a record per subscriber
subscribers.each do |sub|
  EmailDelivery.create!(
    newsletter: newsletter,
    subscriber: sub,
    status: "pending"
  )
end

# At send time, do an atomic claim
delivery = EmailDelivery
  .where(status: ["pending", "failed"])
  .lock
  .first

delivery.update!(status: "sending")
send_email(delivery)
delivery.update!(status: "sent")

If the server dies between sending and sent, the record stays as “sending” — and a recovery job (RecoverStaleDeliveriesJob) periodically moves those records to “unknown” after a timeout, so they never get automatically resent. Ambiguous emails are never resent automatically. That’s the kind of detail that separates a hobby system from a production system.

Pattern 4: Orchestrator Jobs

A common mistake is cramming too much logic into a single job. The pattern that works is having orchestrator jobs that delegate to specialized jobs:

PublishAndSendJob (orchestrator)
  ├── Waits for podcast (retry_on PodcastNotReady)
  ├── PublishToBlogJob.publish(newsletter)
  └── SendNewsletterJob.perform_now(newsletter.id)
        └── SendSingleEmailJob (per subscriber)

The orchestrator coordinates the sequence. Each specialized job knows how to do exactly one thing and can be re-executed independently. If email sending fails, you can retrigger just the SendNewsletterJob without republishing the blog.

Pattern 5: Safety Nets With Cron

Don’t trust a single execution chain. SolidQueue’s cron (config/recurring.yml) acts as a safety net:

send_newsletter:
  class: SendNewsletterJob
  schedule: "0 12 * * 1"  # Monday 9am BRT (UTC-3)

If the 7am PublishAndSendJob failed catastrophically, the 9am cron fires the SendNewsletterJob as a fallback. The job checks whether the newsletter has already been sent and no-ops if so. Idempotency is the keyword here — the job can run as many times as you want and the result is always the same.

But watch out for the inverse: a job that reschedules itself infinitely when it finds no work is a ticking time bomb. If the job doesn’t find a ready newsletter, it simply returns. The cron handles trying again the next week.

Pattern 6: Status Notifications

Every long-running job should communicate its progress. A simple concern handles it:

module DiscordStatus
  extend ActiveSupport::Concern

  def notify_start(message)
    DiscordNotifier.send(channel: status_channel, text: "▶️ #{message}")
  end

  def notify_done(message)
    DiscordNotifier.send(channel: status_channel, text: "✅ #{message}")
  end

  def notify_error(message)
    DiscordNotifier.send(channel: status_channel, text: "❌ #{message}")
  end
end

Each job includes the concern and calls notify_start at the beginning, notify_done on success, notify_error on rescue. When something goes wrong at 3am, you wake up and see exactly where it stopped — no digging through logs required.

SolidQueue: The End of Redis Dependency

A note on SolidQueue, which ships as default in Rails 8: using the same SQLite database for jobs and application data drastically simplifies operations. No need for a separate Redis running. No need to worry about Redis restarting and losing jobs that were in memory.

Jobs are persisted in the database. If the server restarts, they’re there waiting. The retry state is preserved. It’s absurdly simpler than the alternative, and for 99% of cases, the performance is more than enough.

Conclusion

Rails 8 with ActiveJob and SolidQueue didn’t invent anything revolutionary. What it did was make it ridiculously easy to implement patterns that used to require heavy infrastructure:

• retry_on with specific exceptions and clear limits
• Distributed locks with automatic TTL
• Atomic claiming for operations that can’t duplicate
• Orchestrator jobs that delegate and recover
• Safety crons as fallback
• Notifications for operational visibility

None of these patterns is complicated on its own. Put them together, though, and that’s what makes the difference between a system that breaks at the first failure and one that holds up in production — and lets you sleep peacefully on Monday.

And be sure to subscribe to my new newsletter The M.Akita Chronicles!

–

There’s an obsession in the market that frontend = React. Or Vue. Or Svelte. Or the framework of the week. What if I told you it’s possible to have a modern, responsive frontend with dark mode, email-compatible, multi-platform — without a single line of JavaScript framework?

I’m not talking about static sites from the 2000s. I’m talking about Tailwind CSS v4 on Rails, Hugo with the Hextra theme for the blog, and email templates that work from Gmail to Outlook 2016. All in the same project, with one developer.

Tailwind CSS v4: The End of Handcrafted CSS

I have a confession: CSS has always been my kryptonite. Not because it’s hard — but because it’s unpredictable. You change a margin-top and break the layout of a page you never touched. Cascade, specificity, inheritance — it’s a minefield.

Tailwind solves this in the most pragmatic way possible: utility classes right in the HTML.

<div class="flex items-center gap-4 p-6 bg-white dark:bg-gray-800
            rounded-xl shadow-sm border border-gray-200 dark:border-gray-700">
  <img class="w-12 h-12 rounded-full" src="/images/avatar.jpg">
  <div>
    <h3 class="font-semibold text-gray-900 dark:text-gray-100">Título</h3>
    <p class="text-sm text-gray-500 dark:text-gray-400">Descrição</p>
  </div>
</div>

“But this is just inline style with extra steps!” — No, it isn’t. It’s a design system with constraints. You don’t pick “margin: 17px” — you pick m-4 (16px) or m-5 (20px). That constraint forces visual consistency without needing a design system documented across 40 pages.

And Tailwind v4, which came to Rails via tailwindcss-rails, has a fundamental change: CSS-first configuration. No more tailwind.config.js. Everything is CSS:

@import "tailwindcss";

@theme {
  --color-brand: #8b5cf6;
  --font-sans: "Inter", system-ui, sans-serif;
}

One less config file. One less build step. Simpler.

Dark Mode: Respect Your User

Something that deeply annoys me is a site that ignores the operating system’s theme preference. The user set dark mode on the OS — respect that.

With Tailwind, dark mode is trivial:

<body class="bg-white dark:bg-gray-900 text-gray-900 dark:text-gray-100">

The dark: prefix applies the style when the operating system is in dark mode (via prefers-color-scheme: dark). No JavaScript needed. No toggle needed. No cookie needed. It works.

But “it works” is the bare minimum. The real challenge is maintaining consistency across dozens of components. Every color you use needs a dark: variant. Every background, every text, every border.

The trick: create conventions and follow them. Main background? bg-white dark:bg-gray-900. Primary text? text-gray-900 dark:text-gray-100. Secondary text? text-gray-500 dark:text-gray-400. Borders? border-gray-200 dark:border-gray-700. With 4-5 pairs defined, 90% of components are covered.

Hugo + Hextra: Blog Without a Database

The project’s blog runs Hugo with the Hextra theme. Hugo is a static site generator written in Go — absurdly fast. A full rebuild in <500ms even with dozens of pages.

What makes Hextra special is that it’s a documentation theme adapted for blogs. Navigation sidebar, integrated search, table of contents — everything ready. And customizable via partial layouts:

layouts/
├── _partials/
│   ├── custom/
│   │   └── head-end.html          # Injects analytics, fonts, custom CSS
│   └── section-box.html           # Shared partial (DRY, see below)
├── blog/
│   └── home.html                  # Custom homepage
├── newsletters/
│   └── single.html                # Individual newsletter template
├── podcast-transcripts/
│   └── list.html                  # Episode listing
└── shortcodes/                    # 17 shortcodes — components without JS
    ├── podcast-turn.html          # Speech bubble in the transcript
    ├── akita-comment.html         # Akita's comment
    ├── marvin-comment.html        # Marvin's comment
    ├── history-timeline.html      # Historical events timeline
    ├── market-ticker.html         # Financial tickers table
    ├── market-map.html            # FinViz S&P 500 map
    ├── subscribe-cta.html         # Subscription call-to-action
    ├── score.html                 # Relevance badge (high/medium/low)
    ├── holidays.html              # Holidays and commemorative dates
    ├── ref-stories.html           # Reference links
    ├── book-downloads.html        # Book download links
    ├── hacker-news.html           # ┐
    ├── anime-ranking.html         # │ Section boxes — delegate to
    ├── youtube-digest.html        # │ section-box.html (see below)
    ├── world-events.html          # │
    ├── market-news.html           # │
    └── qa.html                    # ┘

Each content type has its own template. Newsletters have a navigation sidebar and visual sections. Podcast transcripts have an embedded player and dialogue formatting. The same Markdown content is rendered differently depending on the type in the frontmatter.

Shortcodes: Components Without JavaScript

Hugo shortcodes are the equivalent of React components, but without a build step:

<!-- layouts/shortcodes/podcast-turn.html -->
{{ $speaker := .Get "speaker" }}
{{ $isAkita := eq $speaker "akita" }}

<div class="podcast-turn" style="display: flex; {{ if $isAkita }}flex-direction: row{{ else }}flex-direction: row-reverse{{ end }};">
  <img src="/images/{{ $speaker }}.jpg"
       style="width: 40px; height: 40px; border-radius: 50%;">
  <div class="bubble {{ if $isAkita }}akita-bubble{{ else }}marvin-bubble{{ end }}">
    {{ .Inner | markdownify }}
  </div>
</div>

In Markdown, the author writes:

{{< podcast-turn speaker="akita" >}}
Então, vamos falar sobre o que aconteceu essa semana no mundo tech...
{{< /podcast-turn >}}

And Hugo renders a styled chat bubble, with avatar, correct alignment, and processed markdown. Zero JavaScript. Zero framework. HTML + CSS + template logic.

DRY With a Shared Partial

With 17 shortcodes, duplication becomes a problem fast. Seven of them — the “section boxes” that wrap newsletter sections (Hacker News, Anime, YouTube, etc.) — have the same visual structure: icon, title, colored border, Markdown content. Only the color and label change.

The solution is a shared partial that takes parameters:

<!-- layouts/partials/section-box.html -->
<div class="section-box {{ .class }}">
  <div class="section-box-header">
    <span>{{ .icon }}</span> {{ .label }}
  </div>
  <div class="section-box-content">{{ .content | markdownify }}</div>
</div>

And each shortcode becomes a one-liner:

<!-- layouts/shortcodes/hacker-news.html -->
{{ partial "section-box" (dict "class" "hacker-news" "icon" "&#128274;" "label" "The Hacker News" "content" .Inner) }}

It’s the Hugo equivalent of component inheritance: a polymorphic partial parameterized by dict. The CSS in custom.css defines colors per class (.hacker-news, .anime-ranking, etc.) with light/dark variants. Result: 7 shortcodes that used to have duplicated HTML are now 7 lines + 1 partial + CSS.

But there’s a crucial catch: shortcodes are exclusive to Hugo. The same content that goes to the blog also goes out by email. And email doesn’t understand shortcodes.

The Challenge: Same Content, Two Renders

Here’s the central frontend problem in this kind of project: the same Markdown file needs to work in two completely different contexts.

On the blog (Hugo):

• Shortcodes work (\{\{< subscribe-cta >\}\})
• CSS with classes (Tailwind, Hextra)
• Dark mode via prefers-color-scheme
• Responsive images
• Flexible layout

In email (Rails → Amazon SES):

• Shortcodes are dead text — they have to be removed
• CSS must be inline (many email clients ignore <style>)
• Dark mode? Forget it for 80% of clients
• Images with absolute URLs and fixed size
• Layout with <table> (yes, in 2026)

The solution: two independent rendering pipelines. The Markdown content is the source of truth. Hugo transforms it for the web. Rails transforms it for email. Each one with its own rules.

On the Rails side, the MarkdownToHtml service needs to:

1. Process standard Markdown
2. Convert \{\{< shortcode >\}\} into equivalent HTML or strip them — via a SHORTCODE_PATTERNS hash that maps each shortcode to a regex + inline template with CSS
3. Apply the theme (light/dark based on the subscriber’s preference)
4. Inject inline CSS for email client compatibility
5. Rewrite image URLs to absolute

It’s more work than it sounds. Each shortcode needs an email-safe equivalent. Comment avatars need absolute URLs. Sections with colored backgrounds need <td style="background-color:..."> instead of CSS classes.

Email HTML: The Worst Frontend in the World

If you think cross-browser compatibility is hard, try cross-email-client compatibility. Outlook 2016 renders HTML using the Microsoft Word engine. Yes, Word. That’s not a joke.

The survival rules:

1. Layout with <table>. display: flex? grid? They don’t exist in Outlook.
2. Inline CSS on every element. <style> tags are ignored by several clients.
3. No margin on images. Use padding on the parent <td>.
4. Web fonts don’t work. Use a font stack: font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif;
5. Themes are a subscriber preference, not automatic detection — many email clients don’t reliably support prefers-color-scheme.

The project’s email template is an ERB that takes theme variables:

<td style="padding: 8px 28px; color: <%= @theme[:text] %>;
           background: <%= @theme[:card_bg] %>;">
  <%= raw @content %>
</td>

The theme defines all the colors:

THEMES = {
  light: {
    text: "#1a1a1a",
    card_bg: "#ffffff",
    link_color: "#7c3aed",
    # ...
  },
  dark: {
    text: "#e6edf3",
    card_bg: "#161b22",
    link_color: "#a78bfa",
    # ...
  }
}

Each subscriber picks their theme at signup time. The email is rendered with the correct colors for that person. No media queries, no automatic detection — because email clients don’t support it reliably. It’s simpler to give the user the choice.

Color Palette: HSL and the Science Behind the Choices

“But how did you pick these colors?” — Not by eyeballing it. There’s a science behind it, and it’s called HSL (Hue, Saturation, Lightness).

Most developers think about color as hex (#e8eef8) or RGB (rgb(232, 238, 248)). Those are machine formats — useful for rendering, useless for designing. When you look at #3b50a2 and #c93030 side by side, there’s no way to intuitively know if they’ll work together. They’re just numbers.

HSL solves this. Instead of mixing light channels (Red, Green, Blue), you describe color in human terms:

• Hue: the position on the color wheel, from 0° to 360°. Red = 0°, green = 120°, blue = 240°.
• Saturation: how “vivid” the color is. 0% = gray, 100% = pure color.
• Lightness: how light or dark. 0% = black, 100% = white.

With HSL, you control each dimension independently. And that’s where building the palette comes in.

Step 1: Distribute Hues Around the Color Wheel

The newsletter has 8 section types, each with its own color. For all of them to “belong” visually to the same project, the hues are distributed evenly around the 360° wheel:

Eight hues spaced around the wheel. That’s not random — it’s the same logic as a polychromatic palette in color theory. When hues are equidistant, the human eye perceives harmony even with completely different colors. It’s the same principle artists have been using with a 12-color wheel since the 18th century.

Step 2: Lock Saturation and Lightness Per Layer

Here’s the trick that makes it all work together. Each visual “layer” (background, text, border) has fixed saturation and lightness — only the hue changes:

Light theme:

• Backgrounds: ~92-95% lightness, moderate saturation. With lightness that high, even different saturations result in pastel colors with similar visual weight — no section “screams” louder than another.
• Text: ~15-20% lightness. Dark enough to guarantee >7:1 contrast ratio against the light background (above the WCAG AAA minimum of 7:1).
• Borders: ~45-55% lightness. The chromatic accent — visible but not dominant.

Dark theme:

• Backgrounds: ~10-12% lightness. Uniform depth — every section has the same “darkness”.
• Text: ~80-85% lightness. Light enough for >6:1 contrast ratio on the dark background.
• Borders: same lightness range as in the light theme — they work as accents in both contexts.

In practice, this means generating the dark variant of any color is mechanical: keep the hue, invert the lightness. A background that was 94% becomes 11%. Text that was 18% becomes 83%. That’s not design — it’s arithmetic.

Step 3: Check Contrast (WCAG)

Pretty colors nobody can read are worthless. The WCAG standard (Web Content Accessibility Guidelines) defines minimum ratios:

• AA: 4.5:1 for normal text, 3:1 for large text
• AAA: 7:1 for normal text, 4.5:1 for large text

The light backgrounds at ~95% lightness against text at ~20% lightness give a ratio >7:1 — WCAG AAA. The dark backgrounds (~11%) against light text (~82%) give >6:1 — comfortably WCAG AA, near AAA.

This isn’t altruism. Low-contrast email is harder to read, generates less engagement, and in extreme cases is flagged by email clients’ accessibility filters.

The Result

When you open the newsletter and see the Hacker News section (purple), followed by YouTube (red), followed by Anime (blue) — they look like “the same project” despite being completely different colors. That’s HSL at work: different hues, same saturation, same lightness. The eye perceives coherence without knowing how to explain why.

And the best part: adding a new section is trivial. Pick a hue that isn’t taken on the wheel, apply the same saturation/lightness rules per layer, and the new section automatically “belongs” to the existing design.

# The comment in the code sums up the rules:
# - Hues evenly spaced: blue 210, green 140, sepia 35, amber 45, royal 225, purple 270, red 0, teal 180
# - Light backgrounds: ~95% lightness, ~25% saturation
# - Light text: ~20% lightness (>7:1 contrast ratio on light bg)
# - Dark backgrounds: ~10-12% lightness
# - Dark text: ~80-85% lightness (>6:1 contrast ratio on dark bg)
# - Borders: ~45-55% lightness

Six rules. Eight hues. Two themes. 100% deterministic. Zero subjective “I think this blue looks pretty” decisions — it’s a design system derived from geometric properties of the HSL color space.

Well-Formed HTML: The Spam Filter Nobody Tells You About

Everything I described above — tables, inline CSS, font stacks — solves the rendering problem. But there’s a second reason to follow these rules that most developers ignore: email providers analyze the quality of your HTML to decide if it’s spam.

Gmail, Outlook, Yahoo — all of them run your email’s HTML through heuristics before showing it to the user. Malformed HTML is one of the classic spam signals. It makes sense: spammers don’t test templates. They spit out broken HTML generated by cheap mass-mailing tools. If your email looks like spammer output, it gets treated as spam.

The signals that trip filters:

1. Unclosed or improperly nested tags. A <div> that opens and never closes, or a </td> before the </tr>, is an immediate flag. Providers expect HTML that parses without errors — not HTML “that the browser fixes up”.
2. Unbalanced image-to-text ratio. An email that’s basically a giant image with little text? Classic spam. Providers want to see real text — not text inside images. The rule of thumb is to keep at least 60% text, 40% images.
3. Missing alt attribute on images. Beyond accessibility, an empty or missing alt is a sign of carelessness — and carelessness correlates with spam. Every <img> needs a descriptive alt.
4. CSS that hides content. display: none, visibility: hidden, font-size: 0, text the same color as the background — those are classic keyword stuffing techniques. Even if you use them legitimately for responsive email, the filter doesn’t know the difference.
5. Excessively heavy HTML. An email with 500KB of HTML raises suspicion. Keep it light — the actual content rarely needs more than 100KB of HTML.
6. Links with text that doesn’t match the href. <a href="https://site-real.com">https://site-falso.com</a> is textbook phishing. Even subtle discrepancies (domain in the text different from the domain in the link) are penalized.

The point that ties it all together: the same practices that make email render correctly in Outlook are the ones that keep your emails out of spam. Tables with clean structure? Well-formed HTML. Inline CSS on every element instead of hacks with <style>? No hidden CSS. Images with explicit dimensions and alt text? Legitimate content.

It’s no coincidence. Email providers calibrated their filters by watching what spammers do — and spammers do exactly the opposite of well-built HTML email. When you follow best practices for compatibility, for free you’re also signaling legitimacy to spam filters.

The practical lesson: treat your email’s HTML as if it were an XML document. Validate, test, and make sure every tag opens and closes in the right order. Use tools like Email on Acid or Litmus to test in multiple clients — they also report spam scoring issues. And resist the temptation to copy ready-made templates from the internet without validating them — many are a disaster of malformed HTML that slips past Gmail but gives Outlook a heart attack and gets Yahoo’s spam filter to flag you as suspicious.

RSS Feeds: The Forgotten Frontend

A detail web developers ignore: RSS is a frontend. Podcasts on Spotify and Apple Podcasts are consumed via RSS. The feed needs:

• Valid XML (podcast parsers are strict)
• Complete metadata (title, description, duration, file size)
• Image URLs that actually work (Spotify validates them)
• Cover art with minimum dimensions (1400×1400 for Spotify)
• Unique GUIDs per episode

Hugo generates RSS automatically, but the default template doesn’t have the fields podcast platforms require. A custom template with iTunes tags is necessary:

<itunes:duration>{{ .Params.duration_seconds }}</itunes:duration>
<itunes:image href="{{ .Params.cover_image | absURL }}"/>
<enclosure url="{{ .Params.audio_url }}"
           length="{{ .Params.file_size }}"
           type="audio/mpeg"/>

The Philosophy: Content First, Framework Never

The pattern that emerges from this whole setup is: content is sovereign. Markdown with YAML frontmatter is the universal format. Each frontend (blog, email, RSS, podcast player) consumes the same content and renders it its own way.

There’s no “React component that fetches data from an API”. There’s a Markdown file read by whoever needs it. This is radically simple — and radically powerful. Swap the blog engine from Hugo to something else? Change the templates, the content stays the same. Swap the email provider from SES to Sendgrid? Change the mailer, the templates stay the same.

It’s the Unix philosophy applied to frontend: each tool does one thing well, connected by a universal format (Markdown + YAML).

Conclusion

You don’t need React to have a modern frontend. Tailwind v4 gives you a design system with constraints, native dark mode, and zero JavaScript. Hugo gives you an absurdly fast blog with flexible templates. ERB with inline CSS gives you emails that work in any client.

The real investment is understanding the constraints of each platform and building rendering pipelines for each one. The payoff comes from that, not from learning the trendy framework that’ll die in 18 months.

At the end of the day, the best frontend abstraction is none at all. HTML, CSS, and Markdown. Simple, debuggable, and going to work 10 years from now.

And make sure to subscribe to my new newsletter The M.Akita Chronicles!

I’ll talk about the podcast today; remember that the pilot episode is already live (transcripts and RSS to subscribe) and on the channel on Spotify.

–

Every AI tutorial starts with “install the model, run inference, done!”. None of them tell you what happens when you need to run this in production with real uptime, controlled cost, and consistent quality.

I’ll tell you how it was to put a TTS (Text-to-Speech) model into production on a GPU in the cloud. Because if you’re thinking about serving AI models, you need to know what you’re getting yourself into.

The Scenario

The goal: generate a ~30-minute podcast from text, with two distinct voices, broadcast quality, ready for Spotify. Every week, automatically, with no human intervention.

The chosen model: Qwen3-TTS, a text-to-speech model that supports voice cloning and fine-tuning. It runs on GPU — needs at least 6GB of VRAM for inference.

The platform: RunPod, which offers serverless GPUs with billing by the second of usage.

Sounds simple, right? Well, hold on.

Problem 1: Cold Start — The Silent Killer

Serverless GPU works like this: when there’s no request, the worker shuts down. When a request comes in, it boots up, loads the model into VRAM, and processes.

The time to “load the model into VRAM” is the cold start. For a ~4GB model, we’re talking 30 to 90 seconds. Your HTTP request sits there hanging, waiting. If your timeout is 30 seconds, you don’t even get to see the model work — you just see a timeout.

The naive solution: “bump the timeout!”. The real solution: health check with polling.

@app.get("/health")
async def health():
    if not model_loaded:
        return JSONResponse(
            status_code=503,
            content={"status": "loading", "detail": "Model loading..."}
        )
    return {"status": "ready"}

The client (in this case, a Rails job) polls /health before sending the actual request:

def wait_for_server(max_wait: 180)
  deadline = Time.current + max_wait
  loop do
    response = http_get("/health")
    return if response.code == 200
    raise "Server não iniciou a tempo" if Time.current > deadline
    sleep 5
  end
end

Sounds primitive? It is. But it works better than any sophisticated “warm pool” solution that costs 10x more.

Problem 2: Where Do the Model Weights Live?

A 4GB model needs to be available when the worker starts. Three options:

1. Bake into Docker: 4GB+ image. Slow build. Slow push. Slow pull.
2. Download from HuggingFace on boot: Network-dependent. Can fail. Adds minutes to the cold start.
3. Network Volume: Persistent disk mounted on the worker.

Option 3 is the correct one. RunPod offers Network Volumes that persist between runs. The model sits there, always ready. The cold start is just the time to load from disk to VRAM, no download.

But there’s a catch: the volume may or may not be mounted, depending on how the worker is configured. So the setup code needs to be resilient:

NETWORK_VOLUME = "/runpod-volume"
LOCAL_FALLBACK = "/workspace/models"

def resolve_model_path():
    if os.path.isdir(NETWORK_VOLUME):
        base = os.path.join(NETWORK_VOLUME, "huggingface")
        os.environ["HF_HOME"] = base
        return os.path.join(base, "hub", "models--Qwen--Qwen3-TTS")
    # Fallback: local download
    return LOCAL_FALLBACK

Auto-detection. If the volume exists, use it. If not, it works the same way (slower on the first boot). Never assume your infrastructure is perfect.

Problem 3: Two Voice Modes, Two Sets of Parameters

Here’s something no documentation will tell you: TTS models with voice cloning and fine-tuning need completely different sampling parameters depending on the mode.

With a fine-tuned voice (trained specifically on the model), you want conservative parameters:

temperature: 0.5
top_p: 0.8
repetition_penalty: 1.1

With voice cloning (base voice + reference audio), the same parameters produce white noise. Literally. White noise. Because the clone model needs more “creativity” to reconstruct the vocal characteristics:

temperature: 0.9
top_p: 0.95
repetition_penalty: 1.05

I discovered this after hours generating unusable audio. The model’s documentation doesn’t mention it. The academic papers don’t mention it. It’s the kind of knowledge that only comes from experimentation in production.

Problem 4: The Reference Audio Matters More Than the Model

Voice cloning uses an audio clip as a reference to replicate the voice. Sounds simple: record 10 seconds, feed it to the model, done.

Except if the reference audio starts with speech immediately — without even 200 milliseconds of silence at the beginning — the model learns that “the audio starts with no onset”. And every subsequent generation will clip the first syllable.

This is not a model bug. It’s the model doing exactly what you asked for: “generate audio that starts just like this reference”. If the reference starts clipped, the generation starts clipped.

The fix is not post-processing (the audio has already been generated without the syllable). It’s fixing the reference: ensuring at least 300ms of silence before the speech.

For those coming from the software world, this is counterintuitive. We’re used to treating input data as immutable and fixing things on the output. With generative models, input quality is everything.

Problem 5: Timeout and Generation Limits

A TTS model generates audio token by token. If you don’t cap it, it can generate indefinitely — and on RunPod, you pay by the second of GPU. A 30-word turn of dialogue shouldn’t generate 3 minutes of audio, but without a limit, it can.

MAX_NEW_TOKENS = 720  # ~60 seconds of audio at 12Hz
GENERATION_TIMEOUT = 180  # 3 absolute minutes

with ThreadPoolExecutor(max_workers=1) as executor:
    future = executor.submit(model.generate, **params)
    result = future.result(timeout=GENERATION_TIMEOUT)

Double protection: token limit on generation and absolute timeout on the process. If either one trips, the turn fails and can be retried with adjusted parameters.

Problem 6: Post-Processing Is Half the Work

The model generates raw WAV. To get to podcast-quality audio, the path is long:

1. Loudness normalization: Spotify requires -14 LUFS. Without normalization, every turn has a different volume.
2. Re-encoding: The model’s WAVs can have subtly different formats (sample rate, bit depth). Concatenating with -c copy causes silent segments. You need to re-encode everything to the same format.
3. Conversion to MP3: CBR 192kbps, 44.1kHz, with ID3 metadata.

And loudness normalization needs to be two-pass: first measure, then apply. Single-pass is a guess — and guessing in production generates audio with inconsistent volume across episodes.

# Pass 1: analysis
ffmpeg -i input.wav -af loudnorm=I=-14:print_format=json -f null -

# Pass 2: applying the measured values
ffmpeg -i input.wav -af loudnorm=I=-14:measured_I=-18.5:measured_TP=-2.1:... output.wav

All of this runs on the server, after TTS generation. And it needs ffmpeg installed in the container — one more dependency to manage.

Problem 7: The Script Matters More Than the Voice Model

After solving GPU, cold start, voice cloning, sampling, normalization — the podcast comes out. And it sounds… robotic. Not because of the TTS. The TTS is excellent. It sounds robotic because the text is robotic.

This is the problem nobody mentions in TTS pipelines: audio quality starts in the script, not in the voice model. A perfect TTS reading a dry, formal text produces dry, formal audio. Garbage in, garbage out — but at a level you wouldn’t expect.

Two Passes: Draft and Refine

The solution is to generate the script in two LLM passes:

Pass 1 (Draft): Turns the newsletter into dialogue. Here the LLM gets the full content and creates the structure — who says what, in what order, with what transitions. Without changing the content, only the form!

Pass 2 (Refine): Takes the draft and makes it natural. Adds hesitations, reactions, interruptions, self-corrections. Turns “read script” into “real conversation”.

Why not do it all in a single pass? Because these are cognitively different tasks. Structuring content and adding naturalness at the same time results in a text that does neither properly. Two specialized passes consistently produce a better result.

Personality Needs Brutally Specific Instructions

“Make Akita sound confident” doesn’t work. The LLM will produce generic confidence — motivational phrases, positive tone. That’s not it. Akita is confident because he’s assertive and impatient. The difference is in the prompt details:

## Akita Delivery Style

- **Declarative, not tentative**: "Isso é óbvio" not "Eu acho que talvez isso seja..."
- **Dismissive of nonsense**: "Isso é bobagem", "Não tem nenhum mistério aqui"
- **Owns his track record**: "Eu avisei", "Quem assistiu meu canal já sabe"
- **Never hedges**: No "talvez", "quem sabe", "pode ser que"
- **Impatient with the obvious**: "Vocês já sabem como funciona"

And for Marvin (the robot co-host):

- **Filler words** (Marvin only): "bom...", "olha...", "enfim", "pois é"
- **Self-corrections** (Marvin only): "quer dizer...", "ou melhor..."
- Akita doesn't second-guess himself — if he corrects, it's
  "Não, peraí" and he restates with even MORE conviction

Without this level of specificity, both characters sound the same. The LLM needs concrete examples of how each one talks, not abstract personality descriptions.

Normalization for Speech: What Looks Right in Text Sounds Wrong in Audio

Acronyms. The invisible enemy of TTS. “EUA” in text is perfectly readable. Spoken by TTS, it comes out “éu-á” — incomprehensible. The prompt needs explicit normalization rules:

## Spoken Text Normalization (MANDATORY)

- `EUA` → `Estados Unidos`
- `UE` → `União Europeia`
- `IA` → `inteligência artificial`
- `LLM` → `modelo de linguagem`
- `API` → `interface de programação`

This needs to be in the prompt for both passes (draft and refine), because the LLM tends to reintroduce acronyms it removed in the previous pass.

The LLM Will Rewrite Your Opinions (If You Let It)

This was the most subtle lesson. The newsletter has opinion commentary from Akita and Marvin in each section. When the LLM turns it into dialogue, it substitutes the opinions with more “balanced” and “nuanced” versions. Akita, who said “isso é bobagem”, becomes Akita saying “isso tem seus méritos, mas…”. Pessimistic Marvin becomes “mildly skeptical” Marvin.

The LLM does this because it was trained to be helpful and balanced. Strong opinions seem “unhelpful” to the model, so it softens them. The original instruction — “use the newsletter comments as seeds for the conversation” — was too vague. It gave the model freedom to reinterpret.

The fix is to be explicit and inflexible:

**PRESERVE THE ORIGINAL OPINIONS**: The newsletter contains Akita and
Marvin commentary on each topic. These opinions are the author's real
voice — you MUST preserve their substance, stance, and conclusions.
Make the delivery conversational, but do NOT change what they actually
think. If Akita's comment says "isso é bobagem", the podcast version
must convey the same dismissal. Rephrase for spoken delivery, but
never substitute a different opinion.

It’s the difference between “adapt it for conversation” (LLM understands: “rewrite however you want”) and “change the HOW, never the WHAT” (LLM understands: “keep the opinion, swap the packaging”).

And this applies to any prompt engineering with authored content: the LLM will homogenize your voice if you’re not explicit about preserving it. And in a podcast where the hosts’ personality IS the product, losing the voice means losing everything.

The Real Cost of AI in Production

Let’s get practical about costs on RunPod:

The cost per episode is low. The cost of figuring out how to make it work is high. It took dozens of hours of experimentation, incomprehensible CUDA logs, failed generations, and unusable-quality audio before landing on a stable pipeline.

And the cognitive cost: keeping two ecosystems (Python for ML, Ruby for the application) in sync, with HTTP interfaces between them, and debugging that crosses language boundaries.

Anyway, for those who thought “Why not Elevenlabs?”, well, I got similar quality at a fraction of the cost, and I learned a lot in the process (and so did you, reading this).

Lessons

1. Cold start is a requirement, not a bug. Design your system expecting that the model will take time to load.

2. Network volumes are mandatory for large models. Don’t bake weights into Docker, don’t download on boot.

3. Sampling parameters are not universal. Each operation mode of the same model may need completely different configuration.

4. Input quality dominates output quality. Both for the TTS reference audio and for the script. If the reference clips the first syllable, every generation clips. If the script is generic, the audio is generic.

5. Timeouts at every layer. Token limit, process timeout, HTTP timeout. If you don’t cap it, cost scales and generation hangs.

6. Post-processing is half the pipeline. The model generates raw material. Normalization, encoding, metadata — all of that is real work.

7. Separate the ML server from the application. Python for inference, Ruby for orchestration. Trying to do everything in one language is a recipe for frustration.

8. Prompts need two passes for complex content. Structuring and naturalizing are different tasks. Forcing both into one pass produces mediocre results in both.

9. The LLM will homogenize your voice. Strong opinions turn into “nuanced takes” if you’re not explicit about preserving them. Instruct the model to change the packaging, never the content.

Conclusion

AI in production is not a 10-line HuggingFace tutorial. It’s infrastructure, it’s experimentation, it’s debugging problems that don’t exist in traditional software. But when it works — when your system generates an entire podcast automatically every week, with broadcast quality, no human intervention — the feeling is that it was worth every hour of headache.

The trick is not to underestimate the complexity. And to have a robust system (like the Rails jobs I mentioned in the first post) orchestrating it all — because AI without reliable orchestration is just an expensive script that hangs.

And don’t forget to subscribe to my new newsletter The M.Akita Chronicles!

–

Ten years ago, fetching data from a website was trivial. Four lines of Ruby: Net::HTTP.get, Nokogiri parse, CSS selector, done. The whole world was open to any script with a decent User-Agent.

Today? Half the sites I need to read to put together my newsletter return “Are you a robot?” for any conventional HTTP request. The other half renders the content entirely in JavaScript — there’s no HTML at all in the server response. And the most paranoid ones combine both.

My goal was to create a daily routine so that M.arvin would deliver the top news to me in the morning on my Discord. That way I wouldn’t have to go hunting for the news of the day. Sounded simple… at first.

This post is about how I went from “one HTTP call” to “packing an entire Chromium into Docker” — and why this is the reality of any project that needs to read the web programmatically in 2026.

The Golden Age: Simple HTTP

Let’s start with what still works. Some services offer structured data with zero resistance:

# RSS feeds: the most resilient format on the web
uri = URI("https://news.google.com/rss/search?q=when:24h+allinurl:reuters.com/world")
xml = Net::HTTP.get(uri)
doc = REXML::Document.new(xml)
doc.get_elements("//item").each do |item|
  title = item.get_text("title").to_s
  url   = item.get_text("link").to_s
end

# Public APIs: clean JSON, documented, stable
response = HTTParty.get("https://api.jikan.moe/v4/top/anime?filter=airing")
animes = response.parsed_response["data"]

# GitHub trending: server-rendered HTML with no protection
html = HTTParty.get("https://github.com/trending").body
doc = Nokogiri::HTML(html)
repos = doc.css("article.Box-row")

Hacker News, GitHub Trending, anime APIs, RSS feeds, Project Gutenberg — they all work with a GET and a parser. No JavaScript, no implicit authentication, no fingerprinting. Just like the web was supposed to be.

The problem is that these are the exception. Sites with financial data, exclusive news, or premium content — exactly the data that matters most — are increasingly armored.

The First Obstacle: SPAs and JavaScript Rendering

When the server returns something like:

<html>
<body>
  <div id="root"></div>
  <script src="/app-3fa8c2.js"></script>
</body>
</html>

Your Net::HTTP.get returns exactly that: an empty div and a link to a 2 MB bundle. The real content — articles, prices, lists — is assembled in the browser by React, Vue, Angular, or the framework of the week.

For a script, this page is useless. To extract the content, you need to execute the JavaScript. That means running a real browser.

In the Ruby world, the answer is the ferrum gem — a CDP (Chrome DevTools Protocol) client that controls Chromium programmatically:

browser = Ferrum::Browser.new(headless: "new", timeout: 30)
browser.go_to("https://www.morningstar.com/markets")
browser.network.wait_for_idle  # wait for JS to finish loading
html = browser.body  # now the DOM is populated
browser.quit