The problem is that gathering data from social networks is grunt work. Open Instagram, YouTube, X, look at numbers, compare to competitors, check the events calendar, monitor sponsors, read comments, calculate engagement, decide how much to charge for a sponsored post. All manual, all repetitive, all eating time that should go to creating content.

I did what any programmer boyfriend would do: I built a full data mining system, with automated collection, LLM-driven analysis, and a Discord chatbot where she talks to the data in plain language.

This article isn’t about the code itself (the project won’t be open source). It’s about the build process, the decisions that only show up after you start, the technical tricks that saved the project, and why a wishlist document from your user is worth more than any functional spec.

Start with the Wish, Not the Architecture

The first thing I did was sit down with her and ask her to talk freely about what she wanted. No technical jargon, no form, no user stories. Her own words. I recorded it in an IDEA.md file that became the north star for the whole project.

What came out of it:

“My biggest difficulty today is figuring out what kind of content to make that can really take off and why the ones that worked actually worked, what made them get the result they got. I have to read the comments on the videos to get a general sense of why.”

“How much I should/could charge for a sponsored post, based on engagement research, competition, sponsors, events.”

“How to make a sponsored post that doesn’t look like a sponsored post — paid content that brings engagement without looking like just a sales pitch.”

She didn’t ask for a dashboard. She didn’t ask for charts. She didn’t ask for PDF reports. She asked for answers to concrete day-to-day problems. That completely changed how I approached the architecture.

If I had started from a technical spec, I’d have built an analytics platform with pretty charts and CSV exports. The system she actually needed was something more like an assistant that knows the data and answers questions in Portuguese.

The IDEA.md also listed initial competitors and sponsors. Brands she could work with (themed restaurants, figure shops, Crunchyroll, Piticas, Fanlab). Reference profiles on Instagram. All of it became seed data. The document wasn’t a spec — it was a conversation that turned into organic requirements.

58 Commits in 3 Days

The project was built in 3 days. Not 3 weeks, not 3 months. 58 commits, 3 distinct phases:

Day 1 — The Data Engine. 12 commits. Rails 8 scaffold, models, collectors for YouTube/Instagram/X, LLM integration. Discovery pipeline to find new profiles automatically. Per-post performance scoring. Sentiment analysis of comments via Claude. By the end of the day, the system was collecting data from all 3 platforms and classifying every post as viral, above average, average, below or flop.

Day 2 — The Brain and the Voice. 35 commits. The most intense day. Two big subsystems showed up: the “Oracle” (tracking events, conventions, game/movie/anime releases, news) and the Discord chatbot with tool calling via RubyLLM. Also production hardening, Docker, deploy guide. And a big pivot: I swapped the weekly email report for 5 daily Discord digests.

Day 3 — Entertainment and Resilience. 11 commits. Steam games (Store API + SteamSpy), AniList for tracking seasonal anime, growth analytics, image generation via Gemini, an auto-healing system for broken URLs.

The final numbers:

These numbers aren’t to my credit. Claude wrote a good chunk of the code. But the direction, the architectural decisions, and especially the validation of every step were human. Claude doesn’t know what a Brazilian cosplay influencer needs. Neither did I — but she told me.

The Architecture That Emerged

The final system is a headless Rails 8. Zero web interface. No views, no real controllers (just /up for the health check). All functionality is delivered via background jobs, rake tasks, and the Discord chatbot.

The stack:

• Rails 8.1 with Solid Queue (jobs) and Solid Cache
• SQLite3 in production (WAL mode, bind mount between containers)
• RubyLLM for integration with Claude Sonnet via OpenRouter and Grok
• Ferrum for scraping with headless Chrome
• Discordrb for the chatbot
• Docker Compose with 4 services

The docker-compose.yml ended up lean:

x-app: &app
  build: .
  env_file: .env
  restart: unless-stopped
  volumes:
    - ./data/storage:/rails/storage

services:
  app:
    <<: *app
    ports:
      - "127.0.0.1:3000:80"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:80/up"]
    deploy:
      resources:
        limits:
          memory: 512M

  jobs:
    <<: *app
    command: bundle exec rake solid_queue:start
    depends_on:
      app:
        condition: service_healthy
    deploy:
      resources:
        limits:
          memory: 1G

  discord:
    <<: *app
    command: bundle exec rake discord:start
    depends_on:
      app:
        condition: service_healthy

  chrome:
    image: chromedp/headless-shell:stable
    deploy:
      resources:
        limits:
          memory: 1G

4 containers: app (Puma, basically just the health check), jobs (Solid Queue running 25+ scheduled jobs), discord (the bot), and chrome (headless browser for scraping). All state lives in SQLite via a bind mount on the host. No Redis, no Postgres, no extra infrastructure.

Idempotency Above All

If I had to pick one concept to summarize this project, it would be idempotency. Every job can run twice in a row without creating duplicates, without corrupting data, without side effects.

The base pattern:

module Collection
  class BaseCollectorJob < ApplicationJob
    SNAPSHOT_DEDUP_WINDOW = 1.hour

    retry_on StandardError, wait: :polynomially_longer, attempts: 3

    def perform(profile_id: nil)
      scope = profile_id ? SocialProfile.where(id: profile_id) : profiles_to_collect
      scope.find_each do |profile|
        log = find_or_create_log(profile)
        begin
          items = collect_for_profile(profile)
          profile.touch(:last_collected_at)
          log.update!(status: :completed, items_collected: items || 0)
        rescue => e
          log.update!(status: :failed, error_message: "#{e.class}: #{e.message}")
          raise if should_raise?(e)
        end
      end
    end

    def upsert_post(profile, attrs)
      post = profile.social_posts.find_or_initialize_by(
        platform_post_id: attrs[:platform_post_id]
      )
      post.assign_attributes(attrs.except(:platform_post_id))
      post.save!
      post
    end

    def record_snapshot(profile, metrics)
      return if profile.profile_snapshots
        .where("captured_at > ?", SNAPSHOT_DEDUP_WINDOW.ago).exists?
      profile.profile_snapshots.create!(captured_at: Time.current, **metrics)
    end
  end
end

Two protection mechanisms: find_or_initialize_by(platform_post_id) for posts (upsert, not insert), and SNAPSHOT_DEDUP_WINDOW for snapshots (skip if we already collected one in the last hour). If the job crashes mid-run and Solid Queue requeues it, nothing duplicates.

Scraping errors (BlockedError, RateLimitError) get swallowed silently — retrying doesn’t help, the rate limit needs to cool off, and getting blocked is expected. Real errors (database, network, bugs) bubble up to the retry with polynomial backoff.

This decision to “swallow certain errors” feels wrong until you run the system for a week and see that Instagram blocks one in every 5 collection runs. If every block triggered 3 retries with exponential backoff, the system would be perpetually behind.

nil vs Zero

This is one of those conceptual bugs you only discover with real data. When Instagram doesn’t return the share count for a post (because the API simply doesn’t expose that field), should the value be nil or 0?

The difference matters. nil means “we don’t have this data”. Zero means “we have the data and it’s zero”. If you treat nil as zero, the LLM analysis concludes that nobody shares posts on Instagram — which is false. The API just doesn’t expose that metric.

I built a reusable prompt snippet just for this:

When a field is null: don’t say “0 likes” — say “data not available for this platform”. When comparing platforms, warn that certain metrics aren’t comparable. When it’s zero: report it normally — the data is real and confirmed by the API.

That snippet gets included in every prompt that handles numeric data. Without it, the LLM happily mixes “not collected” with “actually zero” and draws wrong conclusions.

The Headless Chrome Trick: Host Header Bypass

The project uses chromedp/headless-shell in a separate container for scraping. Works perfectly. Until you try to connect Ferrum (the Ruby Chrome automation gem) to it over Docker networking.

The problem: chromedp/headless-shell uses a socat proxy on port 9222 that rejects any request whose Host header isn’t localhost or an IP. When Ferrum tries to connect to http://chrome:9222, the Host header goes out as chrome:9222, and socat refuses.

The fix was to discover the WebSocket URL manually, forging the header:

def discover_ws_url(chrome_url)
  uri = URI.parse("#{chrome_url}/json/version")
  req = Net::HTTP::Get.new(uri)
  req["Host"] = "localhost"  # bypass

  response = Net::HTTP.start(uri.hostname, uri.port) { |http| http.request(req) }
  data = JSON.parse(response.body)

  ws_url = data["webSocketDebuggerUrl"]
  return nil unless ws_url

  # WS URL points to 127.0.0.1 inside the container — swap it for the reachable hostname
  remote_uri = URI.parse(chrome_url)
  ws_url.sub(%r{://[^/]+}, "://#{remote_uri.host}:#{remote_uri.port}")
end

I do the GET to /json/version with Host: localhost to slip past socat. I get the WebSocket URL back (which points at 127.0.0.1 — useless from outside the container). I rewrite the hostname to chrome:9222 (the service name in Docker Compose). I hand the ws_url straight to Ferrum, which opens the WebSocket without going through socat’s HTTP layer.

This kind of thing doesn’t come to mind upfront. You discover it after 1 hour of connection refused and reading the source code of chromedp/headless-shell.

Tool Calling: The Bot That Queries the Database on Its Own

The coolest part of the project is the Discord chatbot. She types a question in Portuguese (“how were my posts this week?”) and the bot calls the right tools, queries the database, and answers with real data.

The secret is RubyLLM’s tool calling. Every tool is a Ruby class with description and params:

class PostPerformanceTool < RubyLLM::Tool
  description "Returns post performance stats: baseline, " \
              "breakdown by content type, best times, best hashtags."

  param :username, type: :string, desc: "Profile username (e.g. milaoliveira.png)"
  param :analysis, type: :string,
    desc: "Type: baseline, content_types, timing, hashtags",
    required: false

  def execute(username:, analysis: "baseline")
    profile = SocialProfile.find_by(username: username.to_s.strip)
    return "Profile '#{username}' not found." unless profile

    case analysis.to_s.strip.downcase
    when "baseline"
      baseline = PostPerformance.compute_baseline(profile)
      {
        username: profile.username,
        post_count: baseline[:post_count],
        mean_views: baseline[:mean_views].round(0),
        stddev_views: baseline[:stddev_views].round(0),
        mean_engagement: baseline[:mean_engagement].round(2)
      }
    when "timing"
      PostPerformance.timing_breakdown(profile).first(10).map do |entry|
        { day: days[entry[:day]], hour: "%02d:00" % entry[:hour],
          avg_percentile: entry[:avg_percentile].round(1) }
      end
    # ...
    end
  end
end

The LLM gets the list of 40+ tools with their descriptions, decides which ones to call with which parameters, gets the results back, and formulates the answer. The Discord bot shows real-time status while it works: “Querying profile metrics… Checking growth… Analyzing post performance… (3 queries)”.

The trick was making each tool return structured data (Hashes and Arrays), not formatted text. The LLM is much better at formatting raw data into a conversational reply than at parsing semi-structured text.

Another detail that only showed up in real use: clamping parameters. When the LLM asks for limit: 999 on a parameter that maxes out at 50, instead of returning an error, I do [[val.to_i, 1].max, 50].min. The LLM gets parameters wrong more than you’d think. Every error generates a correction round-trip that costs tokens and time.

Composable Prompts: YAML > Hardcoded Strings

Every prompt in the system lives in YAML under config/prompts/. Each one has a system (fixed instructions), a template (with data interpolation), and optionally includes (reusable snippets):

module Llm
  class PromptBuilder
    PROMPTS_DIR = Rails.root.join("config", "prompts")

    class << self
      def build(name, data = {})
        prompt = load_prompt(name)
        base = load_prompt(:_base_context)

        includes = resolve_includes(prompt["includes"])
        system_parts = [base["system"], includes, prompt["system"]]
          .compact.reject(&:blank?)
        system_message = system_parts.join("\n\n")
        user_message = interpolate(prompt["template"], data)

        { system: system_message, user: user_message }
      end
    end
  end
end

The _base_context.yml carries info every prompt needs (who the influencer is, niches, audience, general guidelines). The snippets in config/prompts/snippets/ solve recurring problems:

• null_vs_zero.yml — the nil/zero distinction explained above
• never_invent.yml — “NEVER invent data, only analyze what’s actually present”
• json_only.yml — “respond ONLY in valid JSON”

When a prompt needs any combination of these, you just list them in includes. No duplication, no risk of conflicting instructions across prompts.

This decision came out of a real bug: two different prompts gave contradictory instructions on how to handle null values. One said “use 0”, the other said “use null”. The LLM obeyed one or the other depending on the day. Centralizing in snippets killed off an entire class of inconsistency.

Discovery Pipeline: Autonomous Profile Mining

The initial system started from a manual list of competitors and sponsors. But the influencer lives in a dynamic ecosystem — new creators show up every week, brands appear and disappear. The discovery pipeline automates that.

It runs every Friday at 2 a.m. It starts by mining data we already have in the database: who’s being @mentioned in posts, who comments often and with high engagement, which profiles show up in bio links, who uses brand hashtags (#publi, #parceria), which Linktree links point to other creators. All pure SQL, no API calls, no rate-limit risk.

The candidates that come out of that get validated against the platforms’ APIs (does the Instagram exist? Is the YouTube channel real?), then enriched with cross-platform connections (if we validate an Instagram, we look for the YouTube and X of the same creator).

A batch LLM call evaluates all the candidates, classifies each one as competitor, potential sponsor or irrelevant, and assigns relevance and niche-fit scores. The top 3 become actually tracked profiles — the system starts collecting data from them automatically.

After that, it groups everything into Creator entities. The influencer is depth 0. Direct competitors, depth 1. Mentions of competitors, depth 2. Maximum 3 degrees of separation, to avoid infinite cascade.

From Weekly Report to Daily Digests: Listening to the User

The original plan was a weekly email report with 9 sections. But midway through I decided it would be more dynamic to not even leave Discord and have that information always available.

I pivoted to 5 daily digests on Discord:

• Monday — Performance Recap (followers, growth, top posts)
• Tuesday — Competitor Radar (snapshot, news, strategies)
• Wednesday — Content Playbook (schedule, trends, hashtags)
• Thursday — Opportunities and Pricing (brands, pricing, packages)
• Friday — Next Week (events, priority actions)

Every digest has numbered items with feedback buttons. She marks what she found useful and what she didn’t. That feedback feeds future analyses — the system learns which kinds of insight she values.

This change, which wasn’t in any plan, is probably what improved her experience the most. Every morning at 9 a.m. she opens Discord and gets a fresh, digestible, actionable summary. The weekly email got demoted to backup, commented out in recurring.yml.

The Oracle: Context the Algorithm Doesn’t See

Social media data without external context is kind of useless. “This post got 3x more views than average” is a fact. “This post got 3x more views because it dropped on the day the new Sonic trailer came out” is an insight.

The Oracle is the subsystem that provides that context.

The most critical part is convention tracking: CCXP, Anime Friends, BGS, and dozens of other geek calendar events. Dates, prices, venues, ticket links, the event’s Instagram/X accounts. It monitors daily for changes (date moved, cancellation, guest announcement). The influencer plans her entire calendar around conventions. If CCXP changes its date, she needs to know that day, not the following week.

It also tracks movie/series releases via TMDB, games via IGDB (with Twitch OAuth), anime via AniList — all filtered for geek content, with a 90-day forward window. Franchise anniversaries (using TMDB, IGDB, AniList, Wikimedia) and Brazilian holidays with floating-date calculation. Yes, there’s an Easter calculator in the code based on the computus algorithm. And news via RSS and X/Twitter scraping every 6 hours, classified by relevance to the niche.

The event tracker scrapes convention sites with headless Chrome and uses the LLM at temperature 0.1 to extract structured metadata (date, price, venue). It has a regex fallback for Brazilian-format dates (“15 de março de 2026”). Deduplication by fuzzy name matching + date proximity — because “CCXP 2026”, “CCXP São Paulo 2026” and “CCXP SP” are the same event.

SQLite in Production: Yes, It Works

SQLite3 in production. With WAL mode, a single writer at a time is enough because the system has a predictable write pattern (sequential jobs, never concurrent in the same second). The data lives in a Docker bind mount (./data/storage:/rails/storage) and backup is cp data/storage/*.db backups/.

Rails 8 treats SQLite as a first-class citizen. Solid Queue and Solid Cache run on SQLite without issues. The overhead of a Postgres for a system that serves one person doesn’t justify itself.

25+ Scheduled Jobs

The config/recurring.yml has 25+ jobs with staggered schedules so the machine doesn’t get overloaded:

production:
  youtube_collection:
    class: Collection::YoutubeCollectorJob
    schedule: every day at 3am

  instagram_collection:
    class: Collection::InstagramCollectorJob
    schedule: every day at 4am

  x_collection:
    class: Collection::XCollectorJob
    schedule: every 2 days at 5am

  oracle_events:
    class: Oracle::EventTrackerJob
    schedule: every Monday at 2am

  convention_monitor:
    class: Oracle::ConventionMonitorJob
    schedule: every day at 5am

  discovery_pipeline:
    class: Discovery::OrchestratorJob
    schedule: every Friday at 2am

  comment_sentiment:
    class: Analysis::CommentSentimentJob
    schedule: every Sunday at 5am

  weekly_analysis:
    class: Analysis::WeeklyAnalysisJob
    schedule: every Sunday at 6am

Collection runs in the early morning (3am-5am). The Oracle intelligence kicks off Monday at 2am. Discovery on Friday at 2am. Heavy analysis (sentiment, strategy) on Sunday when there’s no digest to deliver. Digests Monday through Friday at 9am, each day on a different theme. Maintenance (cleaning up finished jobs, URL health checks, aggregating old data) fills the gaps.

The pattern is deliberate: no heavy job competes with the morning digests. If the weekly analysis runs late, Monday’s digest still goes out because it uses Saturday’s collection data, not Sunday’s analysis.

What You Can’t Predict Without Running It

Some things I only found out after the system was in production.

Instagram blocks scraping at random. It doesn’t matter how stealth your browser is. The initial setup was scraping via Ferrum only. Right off the bat I added Apify as the primary source (paid API, more reliable) and the scraping became fallback.

LLMs hallucinate timestamps. I asked the bot to create a reminder for “tomorrow at 3pm” and it calculated a date in 2024. The fix was to inject current_datetime into every prompt that touches time and instruct it explicitly: “calculate based on current_datetime, convert to ISO 8601 with UTC-3”.

Discord has a 2000-character per-message limit, which seems obvious until your bot sends a 4000-char analysis and Discord just truncates it. I built an automatic split that breaks at the last \n before the limit.

Tool calls fail on parameters more often than I expected. The LLM asks for limit: 100 when the max is 50, or sends the username without the @ when it should have. Silent clamping ([[val.to_i, 1].max, 50].min) and input normalization (.strip, .delete("@")) on every tool killed off an entire category of errors that were burning tokens in retry loops.

Conventions change dates more than I imagined. I added an announcement system that tracks every change (date, venue, price, cancellation) with a timestamp. The event tracker runs every Monday, but the convention monitor runs DAILY, because the last thing the influencer wants is to find out a convention moved its date after she already booked the hotel.

The Bot as an Interface

The smartest decision in the project was not building a web interface. Discord is already where the influencer spends her day. The bot lives there, available any time. She types “what should I post this week?” and gets a contextualized answer with data from her own posts, from competitors, from the events calendar, from hashtag trends, and from the season’s anime/games.

The chatbot combines several tools into one answer. If she asks about content strategy, the LLM calls: content digest + events calendar + hashtag trends + seasonal anime + Steam games. Five database queries, all stitched into a conversational reply in Portuguese.

There’s even a “deep thinking” mode that activates when it detects words like “research”, “analyze”, “investigate”. In that mode, the bot uses ALL relevant tools, runs multiple queries, cross-references data across sources.

Conclusion

The whole project shipped in 3 days, from zero to production, collecting real data, sending out digests every morning. No Jira, no sprint planning, no 6-month discovery phase. An IDEA.md with the user’s wishes in her own words, iterative development commit by commit, and continuous validation with the person who’s going to actually use it.

If I had started from the architecture, I’d have spent weeks designing a dashboard she would never open. Starting from the wish, the system ended up being a Discord chatbot she uses every day. Async jobs that aren’t idempotent are time bombs — the rate limit is going to blow, the scraping is going to fail, the container is going to restart, and if the job can’t survive that without duplicating data, the system doesn’t work.

In the end, the measure of success for the project isn’t lines of code, isn’t the number of passing tests, isn’t elegant architecture. It’s the influencer opening Discord on a Monday morning and having what she needs to plan the week.

So I did what any programmer would do: sat down, wrote a spec document, and with the help of Claude Code (Claude Opus 4.6) I built FrankMega, a simplified, self-hosted Mega clone, in Rails 8, in a single day of work. 21 commits, 3 hours of active development, 210 tests, zero external dependencies beyond SQLite.

And before the smart guy shows up in the comments with “ah, but Vibe Coding is just prompt engineering, anyone can do it” – hold on. This post exists exactly to show that it isn’t.

From Idea to Code: the IDEA.md

Every project starts with a document. Mine was docs/IDEA.md – 56 lines describing what I wanted:

• Single file upload, with a shareable link
• Download counter (default 5, configurable)
• Automatic expiration within 24 hours
• Full authentication: password + 2FA + passkeys
• No public registration page – invites only
• Admin creates invites, manages users, bans abusers
• Aggressive rate limiting, automatic IP banning
• Deploy via Docker + Cloudflare Tunnel on my home server
• Stack: Rails 8, SQLite, Tailwind CSS, Hotwire/Stimulus

I deliberately added at the end: “Suggest important features that may be relevant for a service like this.” Because I know a spec document never covers everything. And that’s where the interesting part begins.

The Big Bang: Commit #1

The first commit (e277226) at 12:17 delivered 207 files, 6,901 lines of code. In a single commit. Yes, I know it’s controversial, but when you’re building from scratch with AI, the first commit is necessarily huge.

What came in that initial commit:

• Full authentication with has_secure_password (bcrypt), TOTP via rotp, passkeys via webauthn
• Invitation system with unique codes and expiration
• Full admin panel (Users, Invitations, Files, MIME Types)
• Upload with Active Storage + drag-and-drop via Stimulus
• Download links with 24-byte hashes (SecureRandom.urlsafe_base64(24))
• Rate limiting via Rack::Attack on every public endpoint
• Automatic IP banning with a Ban model and async job
• Dark/light theme with Tailwind CSS inspired by Mega.nz
• 73 tests (Minitest + FactoryBot)
• Docker configuration with Puma + Thruster
• Solid Queue/Cache/Cable – zero Redis

Got it? All of that in commit #1. The IDEA.md became functional code in a single shot. But here’s the point: the first commit was not production-ready. Not even close.

The Reality: 20 Iteration Commits

After the Big Bang came 20 commits over 3 hours. And this is where the truth about software development lives, the one no “Vibe Coding” tutorial will tell you.

Phase 1: “Works on my machine” (12:40 - 12:46)

Four commits in 6 minutes. You know what happened? I went to build the Docker image and it broke. Missing libssl-dev. Fixed it. Then the CSP (Content Security Policy) from secure_headers clashed with the inline scripts from Rails’ importmap. Fixed it. Changed the port from 3000 to 3100 to avoid colliding with other services on my server.

Look at commit 389ebe8:

# CSP is now handled by the Rails built-in, not by secure_headers
# Because secure_headers doesn't support nonces for importmap inline scripts
config.csp = SecureHeaders::OPT_OUT

This kind of gem incompatibility you only discover in the moment. No LLM will warn you about this up front because it’s a specific combination of versions.

Phase 2: The Security Commit – 22 Issues Fixed (13:10)

This is the most important commit of the entire project: 4a854a6. 35 files, 612 lines, 22 security issues fixed. I’ll go into detail because this is where toy code separates from production code.

What the initial commit got wrong:

CRITICAL – Encryption keys with hardcoded fallbacks:

# BEFORE (dangerous):
config.active_record.encryption.primary_key = ENV.fetch("KEY", "test-primary-key")

# AFTER (fail-fast):
if Rails.env.production?
  config.active_record.encryption.primary_key = ENV.fetch("ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY")
end

If the ENV var didn’t exist in production, the app would happily run with the hardcoded key. Anyone reading the source code could decrypt every user’s OTP secrets. Now in production, if the variable is missing, the app crashes on boot. That’s the correct behavior.

CRITICAL – Replay Attack on OTP:

# BEFORE:
def verify_otp(code)
  totp = ROTP::TOTP.new(otp_secret)
  totp.verify(code, drift_behind: 30, drift_ahead: 30).present?
end

# AFTER:
def verify_otp(code)
  totp = ROTP::TOTP.new(otp_secret)
  timestamp = totp.verify(code, drift_behind: 30, drift_ahead: 30)
  return false unless timestamp

  # Prevent replay: reject if this code's timestamp was already used
  if last_otp_at.present? && Time.at(timestamp) <= last_otp_at
    return false
  end

  update_column(:last_otp_at, Time.at(timestamp))
  true
end

Every TOTP tutorial online shows verify(code).present? and stops there. The problem: a TOTP code is valid for 30 seconds. If I intercept your code (shoulder surfing, camera, clipboard), I can use it multiple times in that window. last_otp_at guarantees real one-time use.

HIGH – Race Condition on the Download Counter:

# BEFORE (race condition):
def increment_download!
  increment!(:download_count)
end

# AFTER (atomic):
def increment_download!
  self.class.where(id: id)
      .where("download_count < max_downloads")
      .where("expires_at > ?", Time.current)
      .update_all("download_count = download_count + 1") == 1
end

increment! does: (1) read the value, (2) add 1 in Ruby, (3) write it back. Two simultaneous requests can both read 4, both write 5, and the user gets two downloads when they should only have had one. The atomic UPDATE ... WHERE in SQL guarantees only one request wins. And the == 1 returns success/failure in a single operation, with no lock.

This is the classic TOCTOU bug (Time of Check, Time of Use). You’ll never see it in local development with one browser. It only shows up in production with concurrent requests.

HIGH – Open Redirect after login:

# AFTER:
def safe_redirect_url?(url)
  uri = URI.parse(url)
  uri.host.nil? || uri.host == request.host
rescue URI::InvalidURIError
  false
end

Without this, an attacker could send a link like frankmega.com/session?return_to=https://evil.com and after login the user would be redirected to the malicious site.

MEDIUM – Predictable CSP nonce:

# BEFORE:
config.content_security_policy_nonce_generator = ->(request) { request.session.id.to_s }

# AFTER:
config.content_security_policy_nonce_generator = ->(_request) { SecureRandom.base64(16) }

Session ID doesn’t change between requests. If the nonce is predictable, an attacker can inject scripts with the correct nonce. It has to be random per request.

And more: Cloudflare IPv6 added to trusted proxies, minimum 12 characters for passwords, server-side MIME detection via Marcel, file size verified from the tempfile (not from the client header that can be forged), protection against deleting/banning/demoting the last admin.

The test count jumped from 73 to 109 with this single commit.

How Rails Makes Security Easier

Rails makes implementing security MUCH easier than doing it from scratch. Look at the comparison:

Rails 8 in particular brought rate_limit built into the controller:

class SessionsController < ApplicationController
  rate_limit to: 10, within: 3.minutes, only: :create,
    with: -> { redirect_to new_session_path, alert: t("flash.sessions.create.rate_limit") }
end

And authentication? Rails 8 now ships with a built-in authentication generator. You no longer need Devise for basic stuff. FrankMega’s app/controllers/concerns/authentication.rb is based on that and extends it with 2FA, passkeys and invitation-only registration.

File Sharing Is More Than Downloads

One thing lots of people don’t realize: a file sharing service has a huge attack surface. It goes way beyond “save file, generate link”. Look at everything you need to consider:

Filename Sanitization

def sanitize_filename(name, content_type = nil)
  sanitized = File.basename(name.to_s)                          # Strip path traversal
  sanitized = sanitized.encode("UTF-8", invalid: :replace,      # Handle invalid UTF-8
                                undef: :replace, replace: "")
  sanitized = sanitized.gsub(/[\x00-\x1f\x7f\/\\:*?"<>|]/, "") # Control chars + unsafe chars
  sanitized = sanitized.sub(/\A\.+/, "")                        # Leading dots (hidden files)
  sanitized = sanitized.gsub(/\s+/, " ").strip                  # Collapse whitespace

  # Windows reserved device names
  base_without_ext = sanitized.sub(/\.[^.]*\z/, "")
  if base_without_ext.match?(/\A(CON|PRN|AUX|NUL|COM[1-9]|LPT[1-9])\z/i)
    sanitized = "_#{sanitized}"
  end

  sanitized = strip_extension_junk(sanitized, content_type)
  sanitized = truncate_filename(sanitized, 255)
  sanitized.presence || "unnamed_file"
end

Every line exists for a reason:

• File.basename blocks path traversal (../../etc/passwd becomes passwd)
• Stripping control chars blocks null byte injection and terminal escape injection
• Windows reserved names (CON, NUL, AUX, LPT1-9) cause problems if storage is accessed from Windows
• Truncating to 255 bytes (not characters!) while preserving the extension – a 4-byte emoji counts as 4 toward the ext4/NTFS limit
• strip_extension_junk handles names that come from URLs with glued-on parameters: photo.jpg_1280x720+quality=80 becomes photo.jpg

And this is server-side. There’s also client-side validation in the Stimulus controller to reject before upload:

isInvalidFilename(name) {
  if (new Blob([name]).size > 255) return true
  if (/[\x00-\x1f\x7f/:*?"<>|\\]/.test(name)) return true
  return false
}

Server-Side MIME Type Detection

@shared_file.content_type = Marcel::MimeType.for(uploaded.tempfile, name: uploaded.original_filename)
@shared_file.file_size = uploaded.tempfile.size

Don’t trust the Content-Type the browser sends. Marcel inspects the file’s magic bytes to determine the real type. And the file size comes from the tempfile on disk, not from the Content-Length header which can be forged.

Quotas and Disk Usage

Each user has a disk quota (5 GB default, admin can customize):

def can_upload?(file_size)
  grace = Rails.application.config.x.security.disk_quota_grace_bytes
  (storage_used + file_size) <= (disk_quota + grace)
end

Without a quota, a single malicious user fills up the server disk. With the 10% grace buffer, an upload that slightly exceeds the limit is accepted so the user isn’t frustrated in the edge case.

Automatic Cleanup

class CleanupExpiredFilesJob < ApplicationJob
  queue_as :default

  def perform
    SharedFile.inactive.find_each do |shared_file|
      shared_file.file.purge if shared_file.file.attached?
      shared_file.destroy
    end
  end
end

Runs every 15 minutes via Solid Queue. If nobody downloaded in 24 hours, gone. If it hit the download limit, gone. Without this, the server disk fills up and the service dies.

Automatic IP Banning

class InvalidHashAccessJob < ApplicationJob
  queue_as :default

  def perform(ip_address)
    security = Rails.application.config.x.security
    return unless security.enable_banning

    cache_key = "invalid_hash:#{ip_address}"
    count = Rails.cache.increment(cache_key, 1, expires_in: 1.hour) || 1

    if count >= security.max_invalid_hash_attempts
      Ban.ban!(ip_address, reason: "Repeated invalid download hash access",
               duration: security.ban_duration)
      Rails.cache.delete(cache_key)
    end
  end
end

Tried 3 invalid hashes? Banned for 1 hour. That stops hash enumeration. With 24 bytes of entropy (192 bits, 2^192 possibilities) brute force is computationally infeasible, but the extra protection costs nothing.

Layered Rate Limiting

Rack::Attack operates at the middleware layer (before Rails processes the request):

# Login: 5 attempts per minute per IP
Rack::Attack.throttle("logins/ip", limit: 5, period: 1.minute) do |req|
  req.ip if req.path == "/session" && req.post?
end

# Login: 5 attempts per minute per email
Rack::Attack.throttle("logins/email", limit: 5, period: 1.minute) do |req|
  if req.path == "/session" && req.post?
    req.params.dig("email_address")&.to_s&.downcase&.strip
  end
end

# Downloads: 60 views and 30 downloads per minute
Rack::Attack.throttle("downloads_get/ip", limit: 60, period: 1.minute) do |req|
  req.ip if req.path.start_with?("/d/") && req.get?
end

# General: 300 requests per 5 minutes (except assets)
Rack::Attack.throttle("requests/ip", limit: 300, period: 5.minutes) do |req|
  req.ip unless req.path.start_with?("/assets")
end

Seven different throttles for specific scenarios. Plus Rails 8’s rate_limit in the controllers. Banned IPs never even hit Rails – they’re rejected at the middleware blocklist with status 403.

And the limits are configurable: 1x in production, 10x in development so you don’t lock yourself out during testing. Everything centralized in config/initializers/security.rb.

The Download Saga: 5 Commits in 21 Minutes

This part is fun because it shows real debugging. Between 14:44 and 15:05 I made 5 commits trying to get downloads working with Turbo Drive:

1. cc3c23d – Added data-turbo=false on the download button. Didn’t work.
2. 5224ed2 – Moved data-turbo=false to the <form>. Didn’t work.
3. 98b0d4f – Dropped redirect_to rails_blob_path and used send_file directly. Partially worked.
4. 1fdf5f9 – Switched from POST to GET link. Worked, but… bots can burn downloads with GET.
5. 4c74b27 – Went back to POST. Added URL artifact cleanup in filenames.

The root cause: Turbo Drive intercepts navigation aggressively, and downloading a file via redirect_to (which generates a chain of 302 redirects from Active Storage) confuses Turbo. The final fix: send_file straight from disk with disposition: "attachment" via POST with data: { turbo: false } on the form.

This kind of problem an LLM won’t solve on its own. You need to test in a real browser, see what happens, try, fail, adjust. It’s pure iterative development.

The Gap Between the Idea and the Final Product

Look at what the IDEA.md didn’t mention and that emerged during development:

1. Per-user disk quotas – without these one user fills the server disk
2. Full I18n (EN + PT-BR) – 66 files, ~250 strings extracted
3. Terms of Service with mandatory acceptance
4. User self-deletion – privacy requirement
5. Blocked downloads for banned users – their links return 410 Gone
6. Client-side upload validation – checks size, quota and filename before upload
7. Inline file previews – images, video and audio on the download page without consuming downloads
8. Styled error pages – instead of plain text “Not Found”, branded pages with i18n

The IDEA.md said “use activeadmin or administrate”. In practice I built a custom admin panel with Tailwind because administrate had compatibility issues with Rails 8.1. The IDEA.md mentioned “progress bar if possible”. In practice I used drag-and-drop with a file preview via Stimulus, which is better than a progress bar.

Nobody can predict every feature on day 1. You discover requirements as you build. And each new feature brings edge cases that need handling.

The Numbers

At the end of the project:

• 21 commits in a ~3-hour session
• 210 tests (Minitest + FactoryBot), 513 assertions, zero failures
• ~3,100 lines of application code (models, controllers, views, JS, CSS, configs)
• ~1,965 lines of tests (including dedicated security tests)
• 9 models, 20 controllers, 36 views, 9 Stimulus controllers
• 24 distinct security measures across 7 layers
• CI: SimpleCov, RuboCop (zero offenses), Brakeman (zero warnings), bundler-audit (zero vulnerabilities)
• Zero external dependencies beyond SQLite. No Redis, no PostgreSQL, no external queue service

The Invitation-Only Server

FrankMega has no public signup. The flow is:

1. On first run, with no users, the setup screen appears to create the admin
2. The /setup route only exists while User.count.zero? – afterward it vanishes entirely:

constraints(->(request) { User.count.zero? }) do
  get "setup", to: "setup#new"
  post "setup", to: "setup#create"
end

3. The admin creates invites in the admin panel (with expiration)
4. Each invite generates a unique 16-byte code
5. Each code can only be used once (with_lock + transaction to prevent a race condition)
6. After signup, the invite is marked as “used”

This is perfect for a personal/family service. Full control over who gets in.

Deploy: Docker + Cloudflare Tunnel

To wrap up, the deploy tutorial in case you want to run this on your own Docker home server. It’s not complicated but there are details that matter.

The Dockerfile

Multi-stage build in 3 stages:

# Stage 1: Base (runtime)
FROM ruby:3.4.8-slim AS base
RUN apt-get install -y curl libjemalloc2 libvips sqlite3
# jemalloc for better memory management

# Stage 2: Build (throwaway)
FROM base AS build
RUN apt-get install -y build-essential git libssl-dev
# Bundle install + asset precompilation with dummy env vars

# Stage 3: Final
# Copies gems and app, runs as non-root user (rails:rails, UID 1000)
CMD ["./bin/thrust", "./bin/rails", "server"]

bin/thrust is Thruster (from Basecamp) – an HTTP proxy in front of Puma that handles gzip/brotli compression, asset caching, X-Sendfile acceleration. It listens on port 80 and proxies to Puma on 3000.

The entrypoint runs db:prepare and db:seed automatically at boot:

#!/bin/bash -e
if [ "${@: -2:1}" == "./bin/rails" ] && [ "${@: -1:1}" == "server" ]; then
  ./bin/rails db:prepare
  ./bin/rails db:seed
fi
exec "${@}"

docker-compose.yml

Two services:

services:
  web:
    image: akitaonrails/frankmega:latest
    ports:
      - "3100:80"
    volumes:
      - /home/seuuser/frankmega/uploads:/rails/storage/uploads
      - /home/seuuser/frankmega/db:/rails/storage
    environment:
      SECRET_KEY_BASE: ${SECRET_KEY_BASE}
      HOST: ${HOST}
      # ... around 15 more variables
    restart: unless-stopped

  tunnel:
    image: cloudflare/cloudflared:latest
    command: tunnel run
    environment:
      - TUNNEL_TOKEN=${TUNNEL_TOKEN}
    depends_on:
      - web
    restart: unless-stopped

The cloudflare/cloudflared sidecar creates an outbound tunnel to Cloudflare’s edge. No need to open ports on your home server’s firewall. Cloudflare handles SSL termination, DDoS protection, and routes traffic to the container.

Step by Step

1. Create the Tunnel on Cloudflare:

• Cloudflare Zero Trust Dashboard > Networks > Tunnels > Create
• Type: Cloudflared
• Copy the TUNNEL_TOKEN
• Configure the hostname: frankmega.yourdomain.com pointing to http://web:80

2. Generate the secrets:

# SECRET_KEY_BASE
openssl rand -hex 64

# Encryption keys (3 separate values)
openssl rand -hex 32  # PRIMARY_KEY
openssl rand -hex 32  # DETERMINISTIC_KEY
openssl rand -hex 32  # KEY_DERIVATION_SALT

3. Configure .env:

SECRET_KEY_BASE=<generated above>
RAILS_MASTER_KEY=<from config/master.key>
HOST=frankmega.seudominio.com
WEBAUTHN_ORIGIN=https://frankmega.seudominio.com
WEBAUTHN_RP_ID=frankmega.seudominio.com
ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY=<generated>
ACTIVE_RECORD_ENCRYPTION_DETERMINISTIC_KEY=<generated>
ACTIVE_RECORD_ENCRYPTION_KEY_DERIVATION_SALT=<generated>
TUNNEL_TOKEN=<from Cloudflare>
FORCE_SSL=true
APP_LOCALE=pt-BR

4. Bring it up:

docker compose pull
docker compose up -d

5. Visit https://frankmega.yourdomain.com, create the admin account, start using it.

Important Caveats

• FORCE_SSL=false to test locally at http://localhost:3100 without Cloudflare in front, otherwise you hit a redirect loop
• The WebAuthn origin has to match exactly, including https://. If you get it wrong, passkeys silently stop working
• Cloudflare IPs are hardcoded in the initializer (because the cloudflare-rails gem isn’t compatible with Rails 8.1). If Cloudflare changes the ranges, you need to rebuild the image
• Encryption keys are permanent. If you swap them after users configure 2FA, the OTP secrets become unreadable and users lose access
• Cloudflare free has a 100 MB upload limit. On the free plan, files above that don’t make it through the tunnel. The app allows up to 1 GB but Cloudflare is the bottleneck
• The container runs as UID 1000. If you use bind mounts instead of named volumes, the directory has to be owned by UID 1000
• There are 4 separate SQLite databases in production (app, cache, queue, cable), all in the db_data volume. Backup means copying the entire volume

The Real Point About “Vibe Coding”

Claude Code is absurdly productive. I wouldn’t have done all of this in 3 hours without it. Remember when you’d bang your head against the wall for a week just to get login working? Yeah.

The I18n commit alone (66 files, ~250 strings extracted for two languages) would take a full day by hand.

But here’s the point nobody wants to hear: the LLM did not make the security decisions. I asked it to run an audit and it found the problems, but I had to know how to ask. And I had to know which questions to ask. “Review security” is a vague instruction. “Check if there’s a race condition on the download counter increment” is an instruction only someone who knows what TOCTOU is can give.

Those 56 lines of IDEA.md carry decades of web development experience. I knew to ask for rate limiting because I’ve seen services taken down by bots. I knew to ask for IP banning because I’ve dealt with abuse. I knew to ask for an atomic download counter because I understand concurrency.

If a beginner wrote the same IDEA.md, they probably wouldn’t have half of those concerns. And the LLM wouldn’t spontaneously suggest them. The result would be a service that’s functional, pretty, and completely insecure.

The experienced programmer’s value in Vibe Coding isn’t writing code. It’s knowing what to ask for and reviewing what was generated. Experience is the filter between “works” and “works in production”.

Those 210 tests and 24 security measures didn’t come from prompt engineering. They came from knowing what to test and why.

The code is open source under AGPL-3.0: github.com/akitaonrails/frank_mega. Deploy it on your server, poke around the code, learn from the commits. And if you find more security flaws, let me know.

Remember:

AI is the mirror of your own competence.

And make sure to subscribe to my new newsletter The M.Akita Chronicles!

–

If I told you that you can run a full Rails application in production — with database, job queue, cache — on a single $12/month VPS without installing PostgreSQL, Redis, or any other external service, would you believe me?

Well, you can. That’s Rails 8 with SQLite and Kamal.

SQLite on Rails 8: now it’s for real

Rails 8 brought SQLite as a real production option, not as a development toy. And when I say “real”, I mean rails new generates everything ready to go: main database in SQLite, SolidQueue (jobs) in SQLite, SolidCache (cache) in SQLite, SolidCable (WebSocket) in SQLite.

Four services that used to require PostgreSQL + Redis + Memcached are now .sqlite3 files on disk.

WAL mode (Write-Ahead Logging) enables concurrent reads while a write is happening. For applications with tens of thousands of requests per day — which is the vast majority of Rails applications in the world — SQLite is more than enough. And read latency is measured in microseconds, not milliseconds. It’s local disk access, not a network round-trip.

# database.yml — production with SQLite
production:
  primary:
    <<: *default
    database: storage/production.sqlite3
  queue:
    <<: *default
    database: storage/production_queue.sqlite3
    migrations_paths: db/queue_migrate
  cache:
    <<: *default
    database: storage/production_cache.sqlite3
    migrations_paths: db/cache_migrate

Three files. No TCP connection. No pooling. No “is PostgreSQL accepting connections?”. Backup is a VACUUM INTO and that’s it — a single SQL command that produces an atomic copy. Restore? Paste the file back. Try doing that with a 2GB PostgreSQL dump.

Kamal: Docker Deploy Without Kubernetes

Kamal is the Rails 8 deployer. Think of it as “SSH + Docker, but smart”. No Kubernetes. No Helm charts. No 300-line YAML describing an Ingress Controller.

The entire configuration fits in one file:

# config/deploy.yml
service: minha-app
image: meuregistry/minha-app

servers:
  web:
    hosts:
      - 107.170.70.49
    options:
      network: minha-network
    volumes:
      - minha-app-storage:/rails/storage
      - minha-app-content:/rails/content

proxy:
  ssl: true
  host: app.meudominio.com

builder:
  arch: amd64

env:
  clear:
    RAILS_LOG_LEVEL: info
  secret:
    - RAILS_MASTER_KEY
    - DATABASE_URL

What Kamal does when you run kamal deploy:

1. Builds the Docker image locally (or in CI)
2. Pushes to the registry
3. SSHs into the server
4. Pulls the new image
5. Runs migrations
6. Restarts the container with zero downtime via kamal-proxy

The kamal-proxy is the secret: it acts as a reverse proxy in front of your containers. When the deploy happens, it brings up the new container, waits for it to be healthy, redirects traffic, and then tears down the old one. Your users never notice a thing.

Docker Volumes: The Secret of SQLite in Production

The critical point of SQLite in Docker is: the data can’t die with the container. Docker volumes solve this:

volumes:
  - minha-app-storage:/rails/storage

The storage/ directory contains all the SQLite databases. With the Docker volume, they persist across deploys. The container is ephemeral; the data is permanent.

But there’s a detail that trips up a lot of people: if you run two different services that share data through files (not database), you need a shared volume:

# Service A writes content
volumes:
  - content-compartilhado:/rails/content

# Service B reads content
volumes:
  - content-compartilhado:/rails/content

The same Docker volume mounted on two containers. Simple, it works, and you don’t need NFS, S3, or anything fancy. It’s a directory on the server’s disk.

Multiple Services on the Same VPS

A rarely discussed advantage of Kamal: you can run multiple applications on the same server. The kamal-proxy routes by hostname:

app.meudominio.com    → container-app
bot.meudominio.com    → container-bot

Each service has its own deploy.yml, its own Docker image, its own deploy cycle. But they all run on the same VPS, sharing the same Docker network for internal communication.

SSL is automatic via Let’s Encrypt — kamal-proxy handles that. Each hostname gets its certificate without you configuring anything.

Hooks: Deploy Automation

Kamal supports hooks at various points of the deploy cycle. The most useful is pre-deploy, which runs before the new container replaces the old one:

# .kamal/hooks/pre-deploy
#!/bin/sh
# Run migrations before the container goes live
ssh root@$KAMAL_HOSTS \
  "docker exec minha-app-web bin/rails db:migrate"

Other useful hooks:

• post-deploy — notify the team, clear cache
• pre-connect — check server health
• docker-setup — install dependencies on the host

The Dockerfile Rails Generates

Rails 8 generates an optimized production Dockerfile. Some highlights:

# Multi-stage build: large build stage, minimal runtime stage
FROM ruby:3.3-slim AS base

# Install only runtime dependencies
RUN apt-get install -y libsqlite3-0

FROM base AS build
# Here install everything to compile native gems
RUN apt-get install -y build-essential libsqlite3-dev

# Copy and install gems
COPY Gemfile* ./
RUN bundle install --without development test

# Final stage: only what's needed to run
FROM base
COPY --from=build /usr/local/bundle /usr/local/bundle
COPY . .
RUN bundle exec bootsnap precompile --gemfile app/ lib/

Multi-stage build means the final image has no compilers, no development headers, nothing unnecessary. The production image stays lean.

Secrets: No .env in Production

Kamal has a secrets system that reads from a local file (.kamal/secrets) and injects them as environment variables into the container. That file never goes to git:

# .kamal/secrets (gitignored)
RAILS_MASTER_KEY=abc123...
AWS_ACCESS_KEY_ID=AKIA...

In deploy.yml, you reference:

env:
  secret:
    - RAILS_MASTER_KEY
    - AWS_ACCESS_KEY_ID

Kamal reads from the local file and configures it on the server. Simple, secure, no HashiCorp Vault or AWS Secrets Manager (unless you want — Kamal supports adapters).

SQLite Backup: Absurdly Simple (If You Do It Right)

Want a backup? One SQL command:

VACUUM INTO '/tmp/backup/newsletter.sqlite3';

VACUUM INTO is atomic — it produces a consistent, defragmented copy of the database, even with writes happening. It runs while the application is serving requests normally. And the resulting copy is a complete, functional SQLite database — you can open it, query it, restore it.

Warning: copying the .sqlite3 file directly (cp, tar, rsync) from a database in use can corrupt the backup. If a write is in progress at the moment of the copy, you end up with a half-written file. SQLite has WAL (Write-Ahead Log) and journaling that protect the active database — but the raw copy doesn’t inherit that protection.

In practice, I automated this with a Rails job that runs every hour:

class BackupDatabaseJob < ApplicationJob
  def perform
    backup_dir = File.join(Rails.configuration.content_dir, "backups")
    FileUtils.mkdir_p(backup_dir)
    dest = File.join(backup_dir, "newsletter.sqlite3")
    ActiveRecord::Base.connection.execute("VACUUM INTO '#{dest}'")
  end
end

The backup goes to the shared content directory — the same one that’s already synced with rsync to my local machine. No extra backup agent, no external service, no volume snapshot. One SQL, one file, one rsync.

Compare that with pg_dump of a production PostgreSQL database with dozens of tables and constraints. The SQLite backup is one command that produces a working file. Restore? Copy it back and restart.

When NOT to Use SQLite

I’ll be honest: SQLite isn’t for everything.

• Multiple web servers writing to the same database? No. SQLite is single-writer. If you need horizontal scaling with multiple nodes, go PostgreSQL.
• Massive datasets (hundreds of GB)? PostgreSQL has better query planning and parallelism.
• Replication and high availability? PostgreSQL with streaming replication.

But the honest question is: how many applications really need that? The vast majority run on a single server and will never need more. And for those, SQLite + Kamal is the most productive combination out there today.

The Real Cost

Let’s do the math:

And the cognitive cost? Instead of debugging “why did Redis lose my jobs?”, “why is PostgreSQL rejecting connections?”, “why did the cert expire?”, you focus on what matters: the product.

Conclusion

SQLite + Kamal is not a downgrade. The truth is that most applications never needed the complexity we thought they needed. Rails 8 embraced that and delivered a deploy experience that is, without exaggeration, the simplest that ever existed in the Ruby ecosystem.

One VPS. One command. Zero external services. And an application that runs as fast as any enterprise setup with 15 services in docker-compose.

And make sure to subscribe to my new newsletter The M.Akita Chronicles!

–

Let me be blunt: most developers treat background jobs as if they were scripts that run once and that’s it. “Ah, throw it in a Sidekiq and we’re good.” No, we’re not good. Jobs that handle important things — sending emails, publishing content, calling external APIs — need to be treated as first-class citizens in your architecture.

In this post I’ll show how Rails 8 with ActiveJob and SolidQueue changed my perspective on async processing. This is not theory — these are patterns that emerged from a real system running in production every week.

The Problem: Jobs Are Fragile by Nature

Picture the scenario: you have a job that assembles a newsletter, publishes it on a blog through the GitHub API, waits for a podcast to be ready, and then fires off emails to hundreds of subscribers. Any step can fail. The API can time out. The podcast can take longer than expected. The server can restart in the middle of the delivery.

If you treat this as a linear script, you’re going to suffer. The right question is not “how do I make this job run?” — it’s “how do I make this job recover when something goes wrong?”

Pattern 1: retry_on With Specific Exceptions

ActiveJob’s retry_on is absurdly powerful, but most people use it wrong. Look at the classic anti-pattern:

class MeuJob < ApplicationJob
  retry_on StandardError, wait: 5.seconds, attempts: 3
end

This retries any error 3 times and gives up. It’s useless for almost anything in the real world. The pattern that actually works is to create specific exceptions for transient states:

class PodcastNotReady < StandardError; end

class PublishAndSendJob < ApplicationJob
  retry_on PodcastNotReady, wait: 15.minutes, attempts: 16

  def perform(newsletter_id)
    metadata = check_podcast_metadata(newsletter_id)
    raise PodcastNotReady, "Aguardando podcast" unless metadata
    # keep working...
  end
end

See what’s happening? The job waits up to 4 hours (15min × 16) for the podcast to be ready, polling every 15 minutes. This is not an infinite loop — it has a clear limit. And when it hits that limit, you handle the timeout explicitly:

rescue PodcastNotReady => e
  if executions >= 16
    # Publish without podcast and notify the team
    publish_without_podcast(newsletter)
    notify_team("Podcast não ficou pronto a tempo")
  else
    raise # re-raise so SolidQueue schedules the next retry
  end
end

This is graceful degradation. The system doesn’t stop because one component failed — it adapts and keeps going.

Pattern 2: Distributed Locks

When you have multiple paths that can trigger the same job (a safety cron, a manual trigger, an API), you need to make sure only one instance runs at a time.

The concept is simple: a file-based (or database-based) lock that expires automatically:

class DeployLock
  LOCK_DIR = Rails.root.join("tmp/locks")
  DEFAULT_TTL = 30.minutes

  def self.with_lock(name, ttl: DEFAULT_TTL)
    acquire!(name, ttl: ttl)
    yield
  ensure
    release(name)
  end
end

With SQLite, this is trivial — no Redis, no external coordination. The lock has a TTL so it doesn’t hang forever if the process dies. And the ensure guarantees it releases even if the block raises an exception.

Pattern 3: Atomic Email Delivery

This is where most newsletter systems fail spectacularly. The scenario: you’re sending 500 emails and the server restarts on email 247. What happens when the job re-runs?

If you just iterated a list, you’re going to resend the first 247. Your subscribers will love getting the newsletter twice.

The solution is atomic claiming per recipient:

# Before sending, create a record per subscriber
subscribers.each do |sub|
  EmailDelivery.create!(
    newsletter: newsletter,
    subscriber: sub,
    status: "pending"
  )
end

# At send time, do an atomic claim
delivery = EmailDelivery
  .where(status: ["pending", "failed"])
  .lock
  .first

delivery.update!(status: "sending")
send_email(delivery)
delivery.update!(status: "sent")

If the server dies between sending and sent, the record stays as “sending” — and a recovery job (RecoverStaleDeliveriesJob) periodically moves those records to “unknown” after a timeout, so they never get automatically resent. Ambiguous emails are never resent automatically. That’s the kind of detail that separates a hobby system from a production system.

Pattern 4: Orchestrator Jobs

A common mistake is cramming too much logic into a single job. The pattern that works is having orchestrator jobs that delegate to specialized jobs:

PublishAndSendJob (orchestrator)
  ├── Waits for podcast (retry_on PodcastNotReady)
  ├── PublishToBlogJob.publish(newsletter)
  └── SendNewsletterJob.perform_now(newsletter.id)
        └── SendSingleEmailJob (per subscriber)

The orchestrator coordinates the sequence. Each specialized job knows how to do exactly one thing and can be re-executed independently. If email sending fails, you can retrigger just the SendNewsletterJob without republishing the blog.

Pattern 5: Safety Nets With Cron

Don’t trust a single execution chain. SolidQueue’s cron (config/recurring.yml) acts as a safety net:

send_newsletter:
  class: SendNewsletterJob
  schedule: "0 12 * * 1"  # Monday 9am BRT (UTC-3)

If the 7am PublishAndSendJob failed catastrophically, the 9am cron fires the SendNewsletterJob as a fallback. The job checks whether the newsletter has already been sent and no-ops if so. Idempotency is the keyword here — the job can run as many times as you want and the result is always the same.

But watch out for the inverse: a job that reschedules itself infinitely when it finds no work is a ticking time bomb. If the job doesn’t find a ready newsletter, it simply returns. The cron handles trying again the next week.

Pattern 6: Status Notifications

Every long-running job should communicate its progress. A simple concern handles it:

module DiscordStatus
  extend ActiveSupport::Concern

  def notify_start(message)
    DiscordNotifier.send(channel: status_channel, text: "▶️ #{message}")
  end

  def notify_done(message)
    DiscordNotifier.send(channel: status_channel, text: "✅ #{message}")
  end

  def notify_error(message)
    DiscordNotifier.send(channel: status_channel, text: "❌ #{message}")
  end
end

Each job includes the concern and calls notify_start at the beginning, notify_done on success, notify_error on rescue. When something goes wrong at 3am, you wake up and see exactly where it stopped — no digging through logs required.

SolidQueue: The End of Redis Dependency

A note on SolidQueue, which ships as default in Rails 8: using the same SQLite database for jobs and application data drastically simplifies operations. No need for a separate Redis running. No need to worry about Redis restarting and losing jobs that were in memory.

Jobs are persisted in the database. If the server restarts, they’re there waiting. The retry state is preserved. It’s absurdly simpler than the alternative, and for 99% of cases, the performance is more than enough.

Conclusion

Rails 8 with ActiveJob and SolidQueue didn’t invent anything revolutionary. What it did was make it ridiculously easy to implement patterns that used to require heavy infrastructure:

• retry_on with specific exceptions and clear limits
• Distributed locks with automatic TTL
• Atomic claiming for operations that can’t duplicate
• Orchestrator jobs that delegate and recover
• Safety crons as fallback
• Notifications for operational visibility

None of these patterns is complicated on its own. Put them together, though, and that’s what makes the difference between a system that breaks at the first failure and one that holds up in production — and lets you sleep peacefully on Monday.

TL;DR: Google is actually doing a pretty good job being a counterbalancer so AWS doesn’t slack off. If you already know everything about AWS, I’d encourage you to test out Google Cloud. Possibly because of muscle memory, I would still be more comfortable with AWS, but now that I forced myself to suffer the learning process, I am pretty confident with Google Cloud and Kubernetes for most of my scenarios.

Full disclaimer that I am not an expert, so take what I say with a grain of salt. This is one of those subjects that I am super eager to talk about but I’m also very reluctant on the proper choice of words so you don’t get the wrong idea about the proposed solutions.

The goal of this exercise is mainly for me to store some snippets and thoughts for future reference. So keep in mind that this is also not a step-by-step tutorial. My first intention was to go that way, but then I realized that it would be almost like writing an entire book, so not this time.

To succeed with something like Google Cloud and Kubernetes you must be battle tested in infrastructure. If you never installed server-grade Linux boxes from scratch, if you never did server optimizations, if you’re not comfortable with bare server-side components, do not attempt a real production deployment. Your safest bet is still something like Heroku.

You have to be that kinda person that likes to tinker (as you’ve probably read me doing in previous blog posts).

I don’t know everything, but I know enough. So I only had to figure out which of the pieces would closely fit my needs. You must outline your needs before attempting to write your first YAML file. Planning is crucial.

First things first, this is what I wanted/needed:

• Scalable web application tier, where I could do both rolling updates (for zero downtime updates) and automatic and manual horizontal scaling of the servers.
• Mountable persistent storage with automatic snapshots/backups.
• Managed robust database (Postgresql) with automatic backups and easy replication to read-only instances.
• Managed solution to store secrets (such as Heroku’s ENV support). Never store production configuration in the source code.
• Docker images support without me having to build custom infrastructure to deploy.
• Static external IP addresses for integrations that required a fixed IP.
• SSL termination so I could connect to CloudFlare (CDN is mandatory, but not enough, in 2018 we need some level of DDoS protection).
• Enough security by default, so everything is - in theory - lockdown unless I decide to open it.
• High-availability in different data center regions and zones.

It’s easy to deploy a simple demo web application. But I didn’t want a demo, I wanted a production-grade solution for the long-run. Improvements to my implementation are super welcome, so feel free to comment down below.

Some of the problems for newcomers:

• The documentation is very extensive, and you will find almost everything - if you know what you are looking for. Also bear in mind that Azure and AWS also implement Kubernetes with some differences, so some documentation doesn’t apply to Google Cloud and vice-versa.
• There are many features in alpha, beta, and stable stages. The documentation kinda keeps up well, but most tutorials that are just a couple months old may not work as intended anymore (this one included - I am assuming Kubernetes 1.8.4-gke).
• There is a whole set of words that apply to concepts you already know but are called differently. Getting used to the vocabulary may get in the way at first.
• It feels like you’re playing with Lego. Lots of pieces that you can mix and match. It’s easy to mess up. This means that you can build a configuration tailored to your needs. But if you just copy and paste from tutorials you will get stuck.
• You can do almost everything through YAML files and the command line, but it’s not trivial to reuse the configuration (for production and staging environments, for example). There are 3rd party tools that deal with parameterizable and reusable YAML bits, but I’d do it all by hand first. Never, ever, try automated templates in infrastructure without knowing exactly what they are doing.
• You have 2 fat command line tools: gcloud and kubectl, and the confusing part is that they name some things differently even though they are the same “things”. At least, kubectl is close to docker, if you’re familiar with that.

Once again, this is NOT a step-by-step tutorial. I will annotate a few steps but not everything.

Scalable Web-Tier (the Web App itself)

The very first thing you must have is a fully 12-factor compliant web app.

Be it Ruby on Rails, Django, Laravel, Node.js or whatever. It must be a fully shared-nothing app, that does not depend on writing anything to the local filesystem. One where you can easily shut down and start up instances independently. No old-style session in local memory or in local files (I prefer to avoid session affinity). No uploads to the local filesystem (if you must, you will have to mount an external persistent storage), always prefer to send binary streams to managed storage services.

You must have a proper pipeline that outputs cache-busting through fingerprinting assets (and like it or not, Rails still has the best out-of-the-box solution in its Asset Pipeline). You don’t want to worry about manually busting caches in CDNs.

Instrument your app, add New Relic RPM, add Rollbar.

Again, this is 2018, you don’t want to deploy naive code with SQL (or any other input) injection, no unchecked eval around your code, no room for CSRF or XSS, etc. Go ahead, buy the license for Brakeman Pro and add it to your CI pipeline. I can wait …

As this is not a tutorial, I will assume you’re more than able to sign up to Google Cloud and find your way to set up a project, configure your region and zone.

It took me a while to wrap my head around the initial structure in Google Cloud:

• You start with a Project, which is the umbrella for everything your app needs.
• Then you create “clusters”. You can have a production or staging cluster, for example. Or a web cluster and a separated services cluster for non-web stuff, and so on.
• A cluster has a “cluster-master”, which is the controller of everything else (the gcloud and kubectl commands talk to its APIs).
• A cluster has many “node instances”, the proper “machines” (or, more accurately, VM instances).
• Each cluster also has at least one “node pool” (the “default-pool”), which is a set of node instances with the same configuration, the same “machine-type”.
• Finally, each node instance runs one or more “pods” which are lightweight containers like LXC. This is where your application actually is.

This is an example of creating a cluster:

gcloud container clusters create my-web-production \
--enable-cloud-logging \
--enable-cloud-monitoring \
--machine-type n1-standard-4 \
--enable-autoupgrade \
--enable-autoscaling --max-nodes=5 --min-nodes=2 \
--num-nodes 2

As I mentioned, it also creates a default-pool with a machine-type of n1-standard-4. Choose what combination of CPU/RAM you will need for your particular app beforehand. The type I chose has 4 vCPUs and 15GB of RAM.

By default, it starts with 3 nodes, so I chose 2 at first but auto-scalable to 5 (you can update it later if you need to, but make sure you have room for initial growth). And you can keep adding extra node-pools for differently sized node instances, let’s say, for Sidekiq workers to do heavy-duty background processing. Then you should create a separated Node Pool with a different machine-type for its set of node instances, for example:

gcloud container node-pools create large-pool \
--cluster=my-web-production \
--node-labels=pool=large \
--machine-type=n1-highcpu-8 \
--num-nodes 1

This other pool controls 1 node of type n1-highcpu-8 which has 8 vCPUs with 7.2 GB of RAM. More CPUs, less memory. You have a category of highmem which is less CPUs with a whole lot more memory. Again, know what you want beforehand.

The important bit here is the --node-labels this is how I will map the deployment to choose between Node Pools (in this case, between the default-pool and the large-pool).

Once you create a cluster, you must issue the following command to fetch its credentials:

gcloud container clusters get-credentials my-web-production

This sets the kubectl command as well. If you have more than one cluster (let’s say, one my-web-production and my-web-staging), you must be very careful to always get-credentials for the correct cluster first, otherwise, you may end up running a staging deployment on the production cluster.

Because this is confusing, I modified my ZSH PROMPT to always show which cluster I am dealing with. I adapted from zsh-kubectl-prompt:

As you will end up having multiple clusters in a big app, I highly recommend you add this PROMPT to your shell.

Now, how do you deploy your application to the pods within those fancy node instances?

You must have a Dockerfile in your application project repository to generate a Docker image. This is one example for a Ruby on Rails application:

FROM ruby:2.4.3
ENV RAILS_ENV production
ENV SECRET_KEY_BASE xpto
RUN curl -sL https://deb.nodesource.com/setup_8.x | bash -
RUN apt-get update && apt-get install -y nodejs postgresql-client cron htop vim
ADD Gemfile* /app/
WORKDIR /app
RUN gem update bundler --pre
RUN bundle install --without development test
RUN npm install
ADD . /app
RUN cp config/database.yml.prod.example config/database.yml && cp config/application.yml.example config/application.yml
RUN RAILS_GROUPS=assets bundle exec rake assets:precompile

From the Google Cloud Web Console, you will find a “Container Registry”, which is a Private Docker Registry.

You must add the remote URL to your local config like this:

git remote add gcloud https://source.developers.google.com/p/my-project/r/my-app

Now you can git push gcloud master. I recommend you also add triggers to tag your images. I add 2 triggers: one to tag it with latest and another to tag it with a random version number. You will need those later.

Once you add the registry repository as a remote on your git configuration (git remote add) and push to it, it should start building your Docker image with the proper tags you configured with the triggers.

Make sure your Ruby on Rails application doesn’t have anything in the initializers that require a connection to the database, as it’s not available. This is something you might get stuck with when your Docker build fails because of the assets:precompile task loaded an initializer that accidentally calls a Model - and that triggers ActiveRecord::Base to try to connect.

Also, make sure the Ruby version in the Dockerfile matches the one in Gemfile, otherwise it will also fail.

Notice the weird config/application.yml above? This is from figaro. I also recommend you using something to make it easy to configure ENV variables in your system. I don’t like Rails secrets, and it’s not exactly friendly to most deployment systems after Heroku made ENV vars ubiquitous. Stick to ENV vars. Kubernetes will also thank you for that.

Now, you can override any ENV variable from the Kubernetes Deployment YAML file. Now it’s a good time to show an example of that. You can name it deploy/web.yml or whatever suits your fancy and - of course - check it into your source code repository.

kind: Deployment
apiVersion: apps/v1beta1
metadata:
  name: web
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  minReadySeconds: 10
  replicas: 2
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - image: gcr.io/my-project/my-app:latest
          name: my-app
          imagePullPolicy: Always
          ports:
          - containerPort: 4001
          command: ["passenger", "start", "-p", "4001", "-e", "production",
          "--max-pool-size", "2", "--min-instances", "2", "--no-friendly-error-pages"
          "--max-request-queue-time", "10", "--max-request-time", "10",
          "--pool-idle-time", "0", "--memory-limit", "300"]
          env:
            - name: "RAILS_LOG_TO_STDOUT"
              value: "true"
            - name: "RAILS_ENV"
              value: "production"
            # ... obviously reduced the many ENV vars for brevity
            - name: "REDIS_URL"
              valueFrom:
                secretKeyRef:
                  name: my-env
                  key: REDIS_URL
            - name: "SMTP_USERNAME"
              valueFrom:
                secretKeyRef:
                  name: my-env
                  key: SMTP_USERNAME
            - name: "SMTP_PASSWORD"
              valueFrom:
                secretKeyRef:
                  name: my-env
                  key: SMTP_PASSWORD
            # ... this part below is mandatory for Cloud SQL
            - name: DB_HOST
              value: 127.0.0.1
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: cloudsql-db-credentials
                  key: password
            - name: DB_USER
              valueFrom:
                secretKeyRef:
                  name: cloudsql-db-credentials
                  key: username

        - image: gcr.io/cloudsql-docker/gce-proxy:latest
          name: cloudsql-proxy
          command: ["/cloud_sql_proxy", "--dir=/cloudsql",
                    "-instances=my-project:us-west1:my-db=tcp:5432",
                    "-credential_file=/secrets/cloudsql/credentials.json"]
          volumeMounts:
            - name: cloudsql-instance-credentials
              mountPath: /secrets/cloudsql
              readOnly: true
            - name: ssl-certs
              mountPath: /etc/ssl/certs
            - name: cloudsql
              mountPath: /cloudsql
      volumes:
        - name: cloudsql-instance-credentials
          secret:
            secretName: cloudsql-instance-credentials
        - name: ssl-certs
          hostPath:
            path: /etc/ssl/certs
        - name: cloudsql
          emptyDir:

There is a lot going on here. So let me break it down a bit:

• The kind, and apiVersion is important, you have to keep an eye on the documentation if those change. This is what is called a Deployment. There used to be a Replication Controller (you will find those in old tutorials), but it’s no longer in use. The recommendation is to use a ReplicaSet.
• Name things correctly, here you have metadata:name with web. Also pay close attention to spec:template:metadata:labels where I am labeling every pod having a label of app: web, you will need this to be able to select those pods later in the Service section down below.
• Then I have spec:strategy where we configure the Rolling Update, so if you have 10 pods, it will terminate one, boot up the new one and keep doing that, without never bringing everything down at once.
• spec:replicas declares how many Pods I want at once. You will have to manually calculate the machine-type of the node-pool then divide how many total CPUs/RAM you have by how much you need for each application instance.
• Remember the Docker image we generated above with the ’latest’ tag? You refer to it in spec:template:spec:containers:image
• I am using Passenger with production configuration (check out Phusion’s documentation, do not just copy this).
• In the spec:template:spec:containers:env section I can override the ENV vars with the real production secrets. And you will notice that I can hard-code values or use this strange contraption:

- name: "SMTP_USERNAME"
  valueFrom:
    secretKeyRef:
      name: my-env
      key: SMTP_USERNAME

Now, it’s referencing a “Secret” storage that I named “my-env”. And this is how you create your own:

kubectl create secret generic my-env \
--from-literal=REDIS_URL=redis://foo.com:18821 \
--from-literal=SMTP_USERNAME=foobar

Read the documentation as you can load text files instead of declaring everything from the command line.

As I said before, I’d rather use a managed service for a database. You can definitely load your own Docker image, but I really don’t recommend it. Same goes for other database-like services such as Redis, Mongo. If you’re from AWS, Google Cloud SQL is like RDS.

After you create your PostgreSQL instance you can’t access it directly from the web application. In the end, you have a boilerplate for a second Docker image, a “CloudSQL Proxy”.

For that to work you must first create a Service Account:

gcloud sql users create proxyuser host --instance=my-db --password=abcd1234

After you create the PostgreSQL instance it will prompt you to download a JSON credential, so be careful and save it somewhere safe. I don’t have to say that you must generate a strong secure password as well. Then you must create extra secrets:

kubectl create secret generic cloudsql-instance-credentials \
--from-file=credentials.json=/home/myself/downloads/my-db-12345.json

kubectl create secret generic cloudsql-db-credentials \
--from-literal=username=proxyuser --from-literal=password=abcd1234

These are referenced in this part of the Deployment:

- image: gcr.io/cloudsql-docker/gce-proxy:latest
  name: cloudsql-proxy
  command: ["/cloud_sql_proxy", "--dir=/cloudsql",
            "-instances=my-project:us-west1:my-db=tcp:5432",
            "-credential_file=/secrets/cloudsql/credentials.json"]
  volumeMounts:
    - name: cloudsql-instance-credentials
      mountPath: /secrets/cloudsql
      readOnly: true
    - name: ssl-certs
      mountPath: /etc/ssl/certs
    - name: cloudsql
      mountPath: /cloudsql
volumes:
- name: cloudsql-instance-credentials
  secret:
    secretName: cloudsql-instance-credentials
- name: ssl-certs
  hostPath:
    path: /etc/ssl/certs
- name: cloudsql
  emptyDir:

See that you must add the database name (“my-db” in this example) in the -instance clause in the command.

And by the way, the gce-proxy:latest refers to version 1.09 at the time when this post was published. But there already was a 1.11 version. That one gave me headaches, dropping connections and adding a super long timeout. So I went back to the 1.09 (latest) and everything worked as expected. So be aware! Not everything that is brand new is good. In infrastructure, you want to stick to stable.

You may also want the option to load a separated CloudSQL instance instead of having it in each pod, so the pods could connect to just one proxy. You may want to read this thread on the subject.

It seems that nothing is exposed to anything unless you say so. So we need to expose those pods through what’s called a Node Port Service. Let’s create a deploy/web-svc.yaml file as well:

apiVersion: v1
kind: Service
metadata:
  name: web-svc
spec:
  sessionAffinity: None
  ports:
  - port: 80
    targetPort: 4001
    protocol: TCP
  type: NodePort
  selector:
    app: web

This is why I highlighted the importance of the spec:template:metadata:labels, so we can use it here in the spec:selector to select the proper pods.

We can now deploy these 2 like this:

kubectl create -f deploy/web.yml
kubectl create -f deploy/web-svc.yml

And you can see the pods being created with kubectl get pods --watch.

The Load Balancer

Many tutorials will expose those pods directly through a different Service, called Load Balancer. I am not so sure how well this behaves under pressure and if it has SSL termination, etc. So I decided to go full blown with an Ingress Load Balancer using the NGINX Controller, instead.

First of all, I decided to create a separated node-pool for it, for example, like this:

gcloud container node-pools create web-load-balancer \
--cluster=my-web-production \
--node-labels=role=load-balancer \
--machine-type=g1-small \
--num-nodes 1 \
--max-nodes 3 --min-nodes=1 \
--enable-autoscaling 

As when we created the example large-pool here you must take care of adding --node-labels to make the controller be installed here instead of the default-pool. You will need to know the node instance name, we can do it like this:

$ gcloud compute instances list
NAME                                             ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP      STATUS
gke-my-web-production-default-pool-123-123       us-west1-a  n1-standard-4               10.128.0.1   123.123.123.12   RUNNING
gke-my-web-production-large-pool-123-123         us-west1-a  n1-highcpu-8                10.128.0.2   50.50.50.50      RUNNING
gke-my-web-production-web-load-balancer-123-123  us-west1-a  g1-small                    10.128.0.3   70.70.70.70      RUNNING

Let’s save it like this for now:

export LB_INSTANCE_NAME=gke-my-web-production-web-load-balancer-123-123

You can manually reserve an external IP and give it a name like this:

gcloud compute addresses create ip-web-production \
        --ip-version=IPV4 \
        --global

For the sake of the example, let’s say that it generated a reserved IP “111.111.111.111”. Then let’s fetch it and save it for now like this:

export LB_ADDRESS_IP=$(gcloud compute addresses list | grep "ip-web-production" | awk '{print $3}')

Finally, let’s hook this address to the load balancer node instance:

export LB_INSTANCE_NAT=$(gcloud compute instances describe $LB_INSTANCE_NAME | grep -A3 networkInterfaces: | tail -n1 | awk -F': ' '{print $2}')
gcloud compute instances delete-access-config $LB_INSTANCE_NAME \
    --access-config-name "$LB_INSTANCE_NAT"
gcloud compute instances add-access-config $LB_INSTANCE_NAME \
    --access-config-name "$LB_INSTANCE_NAT" --address $LB_ADDRESS_IP

Once we do this, we can add the rest of the Ingress Deployment configuration. This will be kinda long but it’s mostly boilerplate. Let’s start by defining another web application that we will call default-http-backend that will be used to respond to HTTP requests in case our web pods are not available for some reason. Let’s call it deploy/default-web.yml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: default-http-backend
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        # Any image is permissable as long as:
        # 1. It serves a 404 page at /
        # 2. It serves 200 on a /healthz endpoint
        image: gcr.io/google_containers/defaultbackend:1.0
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi

No need to change anything here, and by now you may be familiar with the Deployment template. Again, you now know that you need to expose it through a NodePort, so let’s add a deploy/default-web-svc.yml:

kind: Service
apiVersion: v1
metadata:
  name: default-http-backend
spec:
  selector:
    app: default-http-backend
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
  type: NodePort

Again, no need to change anything. The next 3 files are the important parts. First, we will create an NGINX Load Balancer, let’s call it deploy/nginx.yml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller
spec:
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: nginx-ingress-lb
    spec:
      # hostNetwork makes it possible to use ipv6 and to preserve the source IP correctly regardless of docker configuration
      # however, it is not a hard dependency of the nginx-ingress-controller itself and it may cause issues if port 10254 already is taken on the host
      # that said, since hostPort is broken on CNI (https://github.com/kubernetes/kubernetes/issues/31307) we have to use hostNetwork where CNI is used
      hostNetwork: true
      terminationGracePeriodSeconds: 60
      nodeSelector:
        role: load-balancer
      containers:
        - args:
            - /nginx-ingress-controller
            - "--default-backend-service=$(POD_NAMESPACE)/default-http-backend"
            - "--default-ssl-certificate=$(POD_NAMESPACE)/cloudflare-secret"
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          image: "gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.5"
          imagePullPolicy: Always
          livenessProbe:
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            timeoutSeconds: 5
          name: nginx-ingress-controller
          ports:
            - containerPort: 80
              name: http
              protocol: TCP
            - containerPort: 443
              name: https
              protocol: TCP
          volumeMounts:
            - mountPath: /etc/nginx-ssl/dhparam
              name: tls-dhparam-vol
      volumes:
        - name: tls-dhparam-vol
          secret:
            secretName: tls-dhparam

Notice the nodeSelector to make the node label we added when we created the new node-pool.

You may want to tinker with the labels, the number of replicas if you need to. But here you will notice that it mounts a volume that I named as tls-dhparam-vol. This is a Diffie Hellman Ephemeral Parameters. This is how we generate it:

sudo openssl dhparam -out ~/documents/dhparam.pem 2048

kubectl create secret generic tls-dhparam --from-file=/home/myself/documents/dhparam.pem

kubectl create secret generic tls-dhparam --from-file=/home/myself/documents/dhparam.pem

Also, notice that I am using version “0.9.0-beta_5” for the controller image. It works well, no problems so far. But keep an eye on release notes for newer versions as well and do your own testing.

Again, let’s expose this Ingress controller through the Load Balancer Service. Let’s call it deploy/nginx-svc.yml:

apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress
spec:
  type: LoadBalancer
  loadBalancerIP: 111.111.111.111
  ports:
  - name: http
    port: 80
    targetPort: http
  - name: https
    port: 443
    targetPort: https
  selector:
    k8s-app: nginx-ingress-lb

Remember the static external IP we have reserved above and saved in the LB_INGRESS_IP ENV var? This is the one we must put in the spec:loadBalancerIP section. This is also the IP that you will add as an “A record” in your DNS service (let’s say, mapping your “www.my-app.com.br” on CloudFlare).

Finally, we can create the Ingress configuration itself, let’s create a deploy/ingress.yml like this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.org/ssl-services: "web-svc"
    kubernetes.io/ingress.global-static-ip-name: ip-web-production
    ingress.kubernetes.io/ssl-redirect: "true"
    ingress.kubernetes.io/rewrite-target: /
spec:
  tls:
    - hosts:
      - www.my-app.com.br
      secretName: cloudflare-secret
  rules:
    - host: www.my-app.com.br
      http:
        paths:
        - path: /
          backend:
            serviceName: web-svc
            servicePort: 80

Be careful with the annotations above that hooks everything up. It binds the NodePort service we created for the web pods with the nginx ingress controller and adds SSL termination through that spec:tls:secretName. How do you create that? First, you must purchase an SSL certificate - again, using CloudFlare as the example.

When you finish buying, the provider should give you the secret files to download (keep them safe! a public dropbox folder is not safe!). Then you have to add it to the infrastructure like this:

kubectl create secret tls cloudflare-secret \
--key ~/downloads/private.pem \
--cert ~/downloads/fullchain.pem

Now that we edited a whole bunch of files, we can deploy the entire load balancer stack:

kubectl create -f deploy/default-web.yml
kubectl create -f deploy/default-web-svc.yml
kubectl create -f deploy/nginx.yml
kubectl create -f deploy/nginx-svc.yml
kubectl create -f deploy/ingress.yml

This NGINX Ingress configuration is based off of Zihao Zhang’s blog post. There are also examples in the kubernetes incubator repository. You may want to check it out as well.

If you did everything right so far, https://www.my-app-com.br should load your web application. You may want to check for Time to First-Byte (TTFB). You can do it going through CloudFlare like this:

curl -vso /dev/null -w "Connect: %{time_connect} \n TTFB: %{time_starttransfer} \n Total time: %{time_total} \n" https://www.my-app.com.br

Or, if you’re having slow TTFB you can bypass CloudFlare doing this:

curl --resolve www.my-app.com.br:443:111.111.111.111 https://www.my-app.com.br -svo /dev/null -k -w "Connect: %{time_connect} \n TTFB: %{time_starttransfer} \n Total time: %{time_total} \n"

TTFB should be in the neighborhood of 1 second or less. Anything far and above could mean a problem in your application. You must check your node instance machine types, the number of workers loaded per pod, the CloudSQL proxy version, the NGINX controller version and so on. This is a trial and error procedure as far as I know. Sign up to services such as Loader or even Web Page Test for insight.

Rolling Updates

Now, that everything is up and running, how do we accomplish the Rolling Update I mentioned in the beginning? First you git push to the Container Registry repository and wait for the Docker image to build.

Remember that I said to let a trigger tag the image with a random version number? Let’s use it (you can see it from the Build History list in the Container Registry, from the Google Cloud console):

kubectl set image deployment web my-app=gcr.io/my-project/my-app:1238471234g123f534f543541gf5 --record

You must use the same name and image that is declared in the deploy/web.yml from above. This will start rolling out the update by adding a new pod, then terminating one pod and so on and so forth until all of them are updated, without downtime for your users.

Rolling updates must be carried out carefully. For example, if your new deployment requires a database migration, then you must add a maintenance window (meaning: do it when there is little to no traffic, such as in the middle of the night). So you can run the migrate command like this:

kubectl get pods # to get a pod name

kubectl exec -it my-web-12324-121312 /app/bin/rails db:migrate

# you can also bash to a pod like this, but remember that this is an ephemeral container, so files you edit and write there disappear on the next restart:

kubectl exec -it my-web-12324-121312 bash

To redeploy everything without resorting to rolling update you must do this:

kubectl delete -f deploy/web.yml && kubectl apply -f deploy/web.yml

You will find a more thorough explanation in Ta-Ching’s blog post.

Bonus: Auto Snapshots

One item I had in my “I wanted/needed” list, in the beginning, is the ability to have persistent mountable storage with automatic backups/snapshots. Google Cloud provides half of that for the time being. You can create persistent disks to mount in your pods but it doesn’t have a feature to automatically back it up. At least it does have APIs to manually snapshot it.

For this example, let’s create a new SSD disk and format it first:

gcloud compute disks create --size 500GB my-data --type pd-ssd

gcloud compute instances list

The last command is so we can copy the name of a node instance. Let’s say it’s gke-my-web-app-default-pool-123-123. We will attach the my-data disk to it:

gcloud compute instances attach-disk gke-my-web-app-default-pool-123-123 --disk my-data --device-name my-data

gcloud compute ssh gke-my-web-app-default-pool-123-123

The last command ssh’s in the instance. We can list the attached disks with sudo lsblk and you will see the 500GB disk, probably, as /dev/sdb, but make sure that’s correct because we will format it!

sudo mkfs.ext4 -m 0 -F -E lazy_itable_init=0,lazy_journal_init=0,discard /dev/sdb

Now we can exit from the SSH session and detach the disk:

gcloud compute instances detach-disk gke-my-web-app-default-pool-123-123 --disk my-data

You can mount this disk in your pods by adding the following to your deployment yaml:

spec:
  containers:
    - image: ...
      name: my-app
      volumeMounts:
        - name: my-data
          mountPath: /data
          # readOnly: true
   # ...
   volumes:
     - name: my-data
       gcePersistentDisk:
         pdName: my-data
         fsType: ext4

Now, let’s create a CronJob deployment file as deploy/auto-snapshot.yml:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: auto-snapshot
spec:
  schedule: "0 4 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
          - name: auto-snapshot
            image: grugnog/google-cloud-auto-snapshot
            command: ["/opt/entrypoint.sh"]
            env:
            - name: "GOOGLE_CLOUD_PROJECT"
              value: "my-project"
            - name: "GOOGLE_APPLICATION_CREDENTIALS"
              value: "/credential/credential.json"
            volumeMounts:
              - mountPath: /credential
                name: editor-credential
          volumes:
            - name: editor-credential
              secret:
                secretName: editor-credential

As we already did before, you will need to create another Service Account with editor permissions in the “IAM & admin” section of the Google Cloud console, then download the JSON credential, and finally upload it like this:

kubectl create secret generic editor-credential \
--from-file=credential.json=/home/myself/download/my-project-1212121.json

Also notice that, as a normal cron job, there is a schedule parameter that you might want to change. In the example, “0 4 * * *” means that it will run the snapshot every day at 4 AM.

Check out the original repository of this solution for more details.

And this should be it for now!

As I said, in the beginning, this is not a complete procedure, just highlights of some of the important parts. If you’re new to Kubernetes you just read about Deployment, Service, Ingress, but you have ReplicaSet, DaemonSet, and much more to play with.

I think this is also already too long to add a multi-region High Availability setup explanation, so let’s leave it at that.

Any corrections or suggestions are more than welcome as I am still in the learning process, and there are a ton of things that I still don’t know myself.

But WebKit is known for being very complicated to deal with. So I can only imagine the nightmare to maintain PhantomJS, which is akin to maintaining a full-blown web browser like Chrome or Safari.

So it’s no wonder that when the Chrome team announced the availability of the Chrome Driver, then the maintainer of PhantomJS decided to step down.

If you know the contributors of PhantomJS, say thank you, as it helped us build more solid user features.

That being said, fear not. You can easily replace Poltergeist/PhantomJS with Selenium WebDriver/Chrome Driver in your RSpec/Capybara setup.

My friend Lucas Caton wrote about it in June this year. Follow his blog as well.

If you’re using Linux with the Chromium browser, you don’t need to install anything, as the Chrome Driver comes with Chromium. Otherwise, you need to install the proper packages for your operating system. For example, brew install chromedriver for OS X or pacaur -S chromedriver on Arch if you don’t like to have Chromium around. You may need to tweak your PATH on Ubuntu though.

Rule of thumb: install Chromium.

In my case, this is what had to change:

# Gemfile
- gem "poltergeist"
+ gem "selenium-webdriver"
+ gem "rspec-retry"

Then in the Capybara setup:

Capybara.server = :puma # Until your setup is working
Capybara.server = :puma, { Silent: true } # To clean up your test output

- Capybara.register_driver :poltergeist do |app|
-   options = {
-     timeout: 3.minutes,
-     phantomjs_options: [
-       '--proxy-type=none',
-       '--load-images=no',
-       '--ignore-ssl-errors=yes',
-       '--ssl-protocol=any',
-       '--web-security=false'
-     ]
-   }
-   Capybara::Poltergeist::Driver.new(app, options)
- end
- Capybara.javascript_driver = :poltergeist

+ Capybara.register_driver :chrome do |app|
+   capabilities = Selenium::WebDriver::Remote::Capabilities.chrome(
+     chromeOptions: {
+       args: %w[ no-sandbox headless disable-popup-blocking disable-gpu window-size=1280,1024]
+     }
+   )
+ 
+   Capybara::Selenium::Driver.new(app, browser: :chrome, desired_capabilities: capabilities)
+ end
+ 
+ Capybara::Screenshot.register_driver :chrome do |driver, path|
+   driver.save_screenshot(path)
+ end
+ 
+ Capybara.javascript_driver = :chrome

Capybara.default_max_wait_time = 5 # you may want to increase this timeout if your app is heavy to load

In feature specs, sometimes Rails itself takes a long while to load up, compile assets, etc., and the first feature spec may time out. To avoid a failure in the test run, it’s recommended to add the rspec-retry gem, as I did above, and add the following to your spec/rails_helper.rb:

require 'rspec/retry'

RSpec.configure do |config|
  ...
  # show retry status in spec process
  config.verbose_retry = true
  # Try twice (retry once)
  config.default_retry_count = 2
  # Only retry when Selenium raises Net::ReadTimeout
  config.exceptions_to_retry = [Net::ReadTimeout]
  ...
end

And that should be it. I didn’t have to touch any of my feature specs and they all ran beautifully. So kudos to the respective teams that maintain Capybara, Selenium-WebDriver for supporting this.

If you’re a Node.js developer as well, you probably used something like Casper, which is said to support Chrome Headless as well. But while we’re at it, you should check out Puppeteer as well, from the Google team itself. It is a promise-based library where you can code like this:

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://example.com');
  await page.screenshot({path: 'example.png'});

  await browser.close();
})();

So yeah, Chrome Headless seems like a very good option as most users actually use the Chrome browser, so it means we should have more reliable feature specs and also web crawling tools.

For each application you create, it even offers you separate development, staging, and production environments, with separate key/secret tokens.

One problem I stumbled upon these days is that I was quickly zeroing out the free development environment message quota (200,000 messages a day). That’s because all my team was using the same key and also the Continuous Integration server was doing live connections whenever it ran in the same environment. That can quickly consume all the quota you have and block both your development and CI.

It’s actually not a good practice to do live connections to external systems in testing situations. The tests can fail randomly for a number of reasons, so we should always mock those. But mocking a complex system (WebSockets and HTTP) like Pusher is not trivial.

Fortunately, I found this Pusher Fake. It basically implements all the necessary APIs and WebSocket endpoints to mimic Pusher and fool both server and js/client to communicate with it.

The idea is for your Rails app, in development mode, to fork a separate server process for the Pusher client to connect to. This gem is both a simple Pusher-clone server and a series of wrappers to load it up in your setup.

First things first.

In your application, you will have both a Ruby side Pusher client connection setup to push messages to a channel in the Pusher server. And a Javascript, client-side, Pusher instance mainly to subscribe to a channel in a WebSocket connection and receive messages.

First we need to set up the Ruby side. Usually it’s in a config/initializers/pusher.rb configuration like this:

Pusher.app_id = ENV['PUSHER_APP']
Pusher.key    = ENV['PUSHER_KEY']
Pusher.secret = ENV['PUSHER_SECRET']
Pusher.cluster = ENV['PUSHER_CLUSTER']
# never set Pusher.host or Pusher.port