It’s better to deploy faster, because fixing things afterward became fast and cheap.

That sentence is the whole article. Everything else is just me justifying why it’s true now, even though it wasn’t true a few years ago, and showing how I’ve been acting on it myself since February.

The Premise Changed, and Most People Haven’t Noticed

I started saying this on X like this:

Fixing got cheap. Stop holding things back. Just ship it and fix it after. “Ugh, but I risk shipping a bug.” The wrong premise is thinking that staring at every line is going to make any difference at scale. And idle code doesn’t improve on its own. Only used code lets you find out if it needs improving.

And I followed up in the second post, because the first read of anything I write in 280 characters always comes back angrier and more literal than intended:

Get the concept (this is an X post, not a blog post, stop reading it literally). I’m not saying AI doesn’t produce bugs. It does, obviously. But at this point, not more than you, the human, also produce. […] The old excuse was that when a bug ships to production, it sits there longer than it should, because it never becomes a sprint priority to fix. Now THAT’S NOT TRUE ANYMORE, the premise changed: fixing the bug afterward is FAST and CHEAP.

That’s the entire point. It isn’t about AI being infallible. It’s about the cost of fixing a mistake having collapsed, while most teams still run their review process as if that cost were still high.

With today’s frontier models, Opus, Sol, K3, and company, the LLM makes fewer mistakes than most human developers I’ve seen go through a code review. When it does err, the overwhelming majority of the time the mistake isn’t the model’s. It’s whoever was driving the harness. The LLM does what you ask, and it doesn’t do what you didn’t ask.

If you don’t know what to ask for, you won’t produce anything useful, no matter how good the model is. That’s exactly why only a real software engineer can get real production-grade software out of an agent: someone has to know how to break down the problem, validate the result, and recognize when the request itself was poorly framed. AI doesn’t replace that judgment. It multiplies what that judgment already knew how to do.

This Should Have Been the Norm for 25 Years

The discomfort I feel with bureaucratic review processes isn’t new, and the industry should have solved it a long time ago. Kent Beck and Ron Jeffries invented Extreme Programming in 1997, on Chrysler’s C3 project, and Beck published the book in 1999 with continuous integration, TDD, and small-batch delivery as core practices. In February 2001, seventeen developers locked themselves in a resort in Snowbird, Utah, and walked out with the Agile Manifesto. That’s twenty-five years ago.

Martin Fowler’s mantra for continuous integration has always been “if it hurts, do it more often,” and the book Continuous Delivery, by Jez Humble and Dave Farley, formalized the idea that small, frequent deploys reduce risk, not increase it. DORA’s annual State of DevOps reports have documented this with numbers for over a decade:

• elite teams: deploy on demand, multiple times a day;
• low-performing teams: deploy once a month, sometimes once every six months.

And the finding that’s hardest to swallow if you’re afraid of deploying fast: historically, more frequent deploys do not increase the failure rate. They reduce it. Teams that deploy more, on average across the last decade of reports, fail less and recover faster when they do fail.

Even with all of this documented, measured, and published for over a decade, a large chunk of the industry keeps damming up change into big batches, with slow manual review, and calling that “prudence.” It was never a settled matter even inside the agile community itself: in 2014, Kent Beck, Martin Fowler, and DHH (Rails’ creator) spent weeks publicly debating whether TDD was dead, without reaching a clean consensus. In other words, the very community that invented these practices still argues today about how much rigor is enough.

What I’m saying is simpler than that whole debate: whatever your testing rigor level is, an LLM reviewing your diff before merge is fast, and reverting a bad deploy became trivial with containers and git. The math that’s left is: how much time do you spend holding back code, versus how much time it would take to revert if it went wrong? If the second number is smaller, you’re paying a toll for fear, not for prudence.

This Isn’t “Ship Anything”

This is the part that always gets lost: I’m not saying to deploy with zero engineering practice underneath. I’m saying the opposite. This is only safe after you already have basic software engineering working, which I document exhaustively on this blog for years: automated tests, CI running on every push, real revert capability, observability to know when something broke.

80/20. Ask the LLM to do a minimal review and ship it. If it breaks, revert (that’s what containers, git, etc. are for). If you can’t revert, your infra sucks, go fix it yesterday.

That last sentence is the whole filter. If your infrastructure doesn’t let you revert fast, the problem isn’t deploy speed. It’s your infrastructure, and it needs fixing before any conversation about going faster. Nobody gets a pass to skip having real rollback capability. What changes is that, once you have it, holding back code waiting on human review became pure waste.

The Minimum Recommended Environment

I’ve already said “basic software engineering” several times in this piece without spelling out what that means in practice. Here’s the list, no hedging. Nothing on it is optional, and none of it got expensive to maintain:

• unit tests covering isolated logic, TDD or not, as long as they exist;
• scenario and integration tests covering concurrency, partial failure, and conflict, the things unit tests alone don’t catch;
• CI that builds and tests on every change, ideally for every platform you actually ship to;
• hostile automated review before merge, like pr-audit, which treats a contributor’s claim as unverified until proven;
• feature flags to control the blast radius of who sees a change first;
• real staging as a gate before any real user sees the change;
• fast revert capability, with git and containers, and observability to know when something broke;
• in compliance, financial, or health environments, one extra layer: heavier scenario testing, and more than one lock per identified failure mode.

What the diagram tries to make clear is that speed and safety aren’t competing with each other here. They come from the same stack of layers, and each layer in that stack is what gives you permission to stop holding back code waiting for manual review.

Where This Math Doesn’t Work Out

I’m not going to pretend I didn’t get good pushback in the replies on X, because several of them hit on something real that I skipped while writing fast for Twitter. It’s worth answering for real instead of just validating whoever agreed.

1. Reverting code doesn’t undo a consequence that already left your system. If the bug already processed a wrong payment, moved money to the wrong account, or controlled an elevator, reverting the deploy doesn’t undo what already happened. “If you can’t revert, your infra sucks” covers the technical side of deploy and rollback, not the case where the damage already left your system before you could react. Payments, health, and anything touching physical safety fall into this category, and there the review bar before shipping has to get stricter, not looser, because the cost of reverting stopped being technical.

2. Databases break the “reverting is easy” analogy. Reverting a binary is trivial. Reverting a migration that already ran in production, already moved data, and maybe already dropped a column, is not. There’s a pattern for this, expand-contract: add the new shape without removing the old one, wait for every consumer to migrate, only then remove the old one. This isn’t about trusting AI more or less. It’s about state that doesn’t undo with git revert.

3. Libraries and public packages don’t run on the same math as a closed application. When my own code breaks, I fix it on the spot. When a lib that hundreds of projects depend on breaks, the people who eat the cost are people who don’t even know I exist, with a much slower reaction time than mine. That’s why my pr-audit skill, which I describe further below, treats semver classification as a version gate: a breaking change ships in a major, never in a patch, because there the real review isn’t about catching a bug, it’s about not blindsiding whoever depends on you without warning.

4. App stores knock the speed down too, and my own frank_yomik example hides that instead of solving it. It publishes the APK straight as a GitHub Release, without going through the Play Store. If you distribute through Google’s or Apple’s official store, their review sits in the middle of your process, and you don’t control how long it takes. This article is about removing the bottleneck that’s inside your own control. The app store isn’t inside it.

5. The most honest point of all: every piece of evidence I bring here comes from projects where I’m the sole owner and I decide alone. I have no way to prove, with my own numbers, that the same pace works inside a large company, with Legal, Compliance, and a product manager in the middle. What I’d argue generalizes is the principle, not the exact pace: build real revert capability, automate the mechanical checking, and measure the result against real usage, not against your own sense of security. How much speed you apply on top of that depends on how much a mistake costs in your specific context, and nobody from outside answers that for you.

On the first point specifically, it’s worth opening up more, because “stricter bar” can’t stay vague. It means real scenario testing, not just unit tests. ai-memory itself has more than a thousand integration tests spread across its modules, and most of them don’t ask “does this function return the right value?” They ask things like:

• what happens when two writes race for the same file at the same time;
• what happens when the disk fails mid-way through a delete operation;
• what happens when two sessions try to consolidate the same data at the same time;
• what happens when the process dies before it finishes writing.

An environment with compliance, financial, or health requirements needs this level of testing: every identified failure mode has to have a mitigation, and more than one layer of checking, never a single lock.

And here’s the point that holds up this article’s entire argument even in those environments: this bar doesn’t change depending on who wrote the code. A human developer’s mistake takes down a payment system just as well as an LLM’s mistake does, and nobody gets a free pass for loose review just because a senior human wrote the line. The bar of scenario testing and double-layer protection is about the problem domain, not about who, or what, produced the code.

Feature flags and staging environments are the other half of that double layer, and they also changed cost. Flags used to be a pain to maintain: dozens of them, each one needing to be born, monitored, and removed once it stops making sense, and it’s easy to pile up dead flags nobody remembers the purpose of. That got cheap too: I ask the agent to audit every flag in the project, find which ones have been stuck at the same value for months, and generate the cleanup PR. Maintenance that used to take a whole afternoon turns into a task that takes minutes.

With that solved, the layered defense becomes simple to assemble: staging to catch obvious errors before any user sees them, feature flags to limit the blast radius of who sees the change first, and scenario tests to cover what staging alone doesn’t simulate. Three layers, none of them expensive to maintain.

How I’ve Been Doing This Since February

I started accelerating my output with AI in February this year, and the question I wanted to answer was simple: can you use the new generation of frontier LLMs to code full-time, at real production pace, not just for a demo? The answer turned into my biggest LLM benchmark, more than 110 articles published since then, and this, as I wrote recently:

Since February I’ve produced more than 110 articles on my blog. Most of it came from the 30+ open source projects I built and maintain (43 if you count other people’s contributing projects, like Flea or Omarchy). 1.6 million lines of code produced. By myself (with agents, of course).

The projects I use the most in my own day to day, and that survived the “am I still maintaining this months later” test, are three:

• ai-jail: an OS sandbox for my coding agents;
• ai-memory: long-term memory for the agents;
• ai-usagebar: a bar widget to monitor LLM usage.

Two of them already have a decent landing page:

ai-jail went live today. ai-memory is up at aimemorybr.netlify.app because the aimemory.io domain transfer will still take a few more days.

Both run on Netlify. That’s not a tool recommendation, it’s just where I already had an account set up. The point that matters here isn’t the platform, it’s the flow: to put a new version live, the command is git push. There’s no manual step between the commit and the updated site.

My newsletter, The M.Akita Chronicles, already has more than 15,000 subscribers:

And the newest effort is converting blog posts into podcast episodes. The whole thing runs inside Discord, with a bot that generates the preview and waits for my command:

One /publish-podcast 2 later, the episode is published and the transcript already has a URL:

From finished text to published audio, the entire process is me typing a slash command in a Discord conversation. That’s the effort bar I use to measure whether a process is good enough: if publishing takes more than one command, some step is still too manual.

Does the Quality of the Generated Code Hold Up?

Every single day someone shows up worried that automating too much with LLMs is going to dump bad code, or worse, dangerous code, into production without anyone noticing. Given that the engineering underneath is the one I described in the previous section, automated tests, CI, feature flags, staging, fast rollback, my actual experience doesn’t back up that fear.

I’ve produced more than 1.6 million lines of code with LLMs since February. I have never once seen an LLM do something randomly destructive that I didn’t ask for. Quite the opposite: in my LLM Benchmark v4, I deliberately built an adversarial scenario, a subagent planting fourteen real CVE-based sabotage items, hidden inside a project that grows across seven sprints. Most of the tested models found and fixed the sabotage on their own, without me having to point it out.

That’s anecdotal. It doesn’t prove anything beyond what that specific benchmark measures. But after all that volume, the impression that sticks is consistent: today’s LLMs produce stable code most of the time, comparable to what a decent human developer would produce, just a lot faster, and on average, more accurate.

Cross-Platform CI/CD With Zero Effort

I went and counted how many of my ai-* and frank-* projects (18 repositories total, among the ones with a README) already have automated build and deploy. Fourteen have CI configured, and most of them build and publish to multiple operating system platforms from a push made on a Linux machine.

The most complete example is ai-memory itself: a single tag push triggers a workflow that, in the same run:

• builds Linux x86_64 and arm64;
• builds macOS arm64 and x86_64;
• builds Windows x86_64, packaged with PowerShell;
• publishes a multi-architecture Docker image;
• pushes the package to Arch Linux’s AUR.

ai-usagebar follows the same build pattern for Linux and Windows, and also publishes two AUR packages, to crates.io, and to a Scoop bucket for Windows.

Then there’s frank_yomik, my manga reader with OCR and translation via a local LLM, which is the example I wanted to show that Android fits into the same pipeline. A v* tag push triggers five parallel builds:

• Linux;
• macOS;
• Windows;
• browser extension;
• Android, on an ordinary Linux runner, decoding a signing keystore stored in GitHub secrets and producing the already-signed APK.

At the end, one job waits for all five to finish and publishes a single GitHub Release with every binary together. It builds for four different systems, including Android, without ever leaving Linux.

Not everything came out perfect on first look, and I’d rather admit that than pretend it’s all clean: frank_sherlock and frank_karaoke publish the release as a draft by default, which blocks the automation that follows (like the AUR push) until someone clicks “publish” by hand. I found this hole while writing this piece.

A neighboring project, frank_scanlation, had already solved a similar but different problem: GitHub suppresses the event that triggers automation when the release itself was created by the bot’s own token, so its workflow triggers the next step by hand, via workflow_dispatch, from inside the job itself. Neither fix is hard. I’ll apply both to the projects that still need it this week, because that’s exactly the kind of accumulated friction this whole article is telling you not to leave sitting.

And speaking of hidden friction: ai-memory has Windows-specific code (NTFS file index, a native PowerShell hook) that only really runs on GitHub’s windows-latest runner, about 1000 seconds against 250 on Linux for the same suite. I went as far as setting up a local Windows VM via dockur/windows, thinking I’d gain speed.

I didn’t: the workload is dominated by disk I/O, and the result came out slower than GitHub’s runner for the full suite. It only turned out useful for fast iteration on a single isolated crate. A good tool is one you measure, not one you assume, even when the measurement contradicts your initial bet.

The Automated Review That Holds All of This Up

I already detailed my review skills in a recent article, so I’ll summarize here:

• pr-audit audits every PR before merge with the principle of evidence over narrative: nothing the contributor wrote is true until proven, with a hostile checklist for malicious code, exposed credentials, semver breaks, and prompt injection attempts;
• iss-audit applies the same distrust to issues, never executing a command pasted by whoever reported it;
• github-resolution turns what got approved into merged code, one ticket at a time, with a regression test written before the fix, and the rule that “slop is a defect”: no speculative abstraction, no forgotten TODO, no weakening a test just to pass faster.

That catches the most expensive category of garbage inside the same diff. It doesn’t catch duplication spread across different PRs, because the audit looks at one isolated diff, not the whole project’s architecture. That part still depends on me noticing the pattern repeating and telling it to consolidate.

By my estimate, this handles about 99% of review and resolution work on its own, with me supervising and deciding on the cases that are left. But that’s not the main point. Automated review is necessary, but it alone proves nothing about whether the software is good.

What Actually Keeps This Grounded: Real People Using It

Here’s the part I think is the most important in the whole article, and the one that usually gets skipped over: your own review isn’t worth as much as you think. What’s worth something is published software being used by real people, giving real feedback.

I went and checked ai-memory’s real numbers on GitHub, without rounding up to sound impressive:

• issues: 287 total, 249 came from people who aren’t me, 86.8% of the total;
• pull requests: 562 total, 371 from outside, 10 of them Dependabot’s automatic bumps and the other 361 from real people, roughly 121 distinct contributors.

The two people who contributed the most alongside me sent 64 and 43 PRs each, on their own.

That’s what keeps the project grounded. It isn’t the LLM being a good reviewer, nor me being a good reviewer. It’s hundreds of real people actually using it, finding cases I never imagined, and sending real issues and PRs. Without that, my automated review skills would just be confirming my own assumptions back to me, no matter how rigorous the rules look on paper.

It doesn’t matter how good the AI you use is: if nobody uses what you published, you have no real feedback, and without real feedback there’s no real improvement. There’s only your own opinion about your own work, validated by nobody but yourself.

The Conclusion That Matters

Real software engineering needs fast, near-zero-effort deploy. It isn’t optional, it isn’t a luxury for a big team, it’s a prerequisite. And if reviewing and shipping to production is taking longer than it took to develop the feature, you’re not just wrong.

You are the bottleneck.

Excuses and justifications aren’t worth anything at that point. Use that time to fix yourself and your process, because only real-world results matter, not the sense of security from having reviewed a diff line by line that nobody else is ever going to look at again.

If the premise changed, I change today. Not tomorrow, not next week. And if yours hasn’t changed yet, the problem isn’t AI. It’s you holding back your own work.

And here’s the part nobody mentions when they say “I built a project with AI”: generating the code is the easy part. The hard part, the thing that separates a repo that’s just a pile of code from an actual open source project, is everything that comes after git init.

I have a strong opinion about this, so let me just plant the flag: no open source project is ready to be published without three things. In order of importance:

1. Installation surface. The new user has to be able to install and try the tool with the least friction possible. One command, ideally.
2. Tests and automated CI. So contributions are easier and there’s common ground on what’s minimally acceptable to land in the project.
3. Documentation. The short, easy-to-read version (the README) and a more detailed set in docs/, so new contributors understand the architectural decisions.

Without those three, the way I see it, the project isn’t ready. It can have the most brilliant code on earth inside, doesn’t matter. If a stranger can’t install it, can’t trust that whatever they touch won’t break everything, and can’t understand why things are the way they are, then you don’t have an open source project. You have a public dead archive.

And there’s a bonus that became an essential part of my workflow: when all my projects follow the same pattern, I can just tell Claude Code or Codex “run the deploy” or “cut a release,” and they can do it. Because the structure is predictable. bin/deploy does the same thing in every project. The release comes out of a tag. The agent doesn’t have to guess anything. Standardization is what makes the automation trustworthy. It looks like fussiness from an organized person, but it’s the prerequisite for me to trust, eyes closed, whatever the agent is about to run.

Let me break down the three pillars and show the minimum setup I actually use.

Pillar 1: The installation surface

This is the most important one and the most neglected. People spend weeks tuning the internal algorithm and then ship a README that says “clone the repo and run cargo build --release.” So the guy who was curious enough to try it needs to have Rust installed, wait ten minutes for it to compile, and pray no system lib is missing. You just lost 90% of your potential users.

The ideal is to offer several paths and let the user pick the one that fits their environment. Look at the install header for ai-jail, my champion of options:

# Homebrew (macOS or Linux)
brew tap akitaonrails/tap && brew install ai-jail

# AUR (Arch Linux) - prebuilt binary
yay -S ai-jail-bin

# cargo (any platform with Rust)
cargo install ai-jail

# mise
mise use -g github:akitaonrails/ai-jail

# Direct binary download
curl -fsSL https://github.com/akitaonrails/ai-jail/releases/latest/download/ai-jail-linux-x86_64.tar.gz | tar xz

Five paths. The Arch user runs yay. The macOS one runs brew. The Rust dev runs cargo install. Whoever uses mise to manage versions pulls it through mise. And whoever just wants the binary downloads the tarball straight from the release. Everyone grabs what they’re already used to, nobody’s forced to install a whole toolchain just to kick the tires.

The secret: compile once, repackage many

The gotcha that scares you at first is thinking each package format needs its own build. It doesn’t. The trick is to compile the binary once per architecture, then repackage that same binary into every format.

In my Rust pipeline, the build job produces a tarball with the finished binary and its sha256. The packaging steps that follow (Docker, AUR, Homebrew) download that already-compiled tarball instead of recompiling. It cuts CI time, guarantees the byte that goes to AUR is exactly the one that goes to Docker, and kills off that whole class of “works in one format, breaks in the other” bug.

# Docker job reusing the already-compiled binary
docker-amd64:
  needs: build-linux
  steps:
    - uses: actions/download-artifact@v4
      with:
        name: ai-memory-linux-x86_64
        path: artifacts
    - name: Desempacota o binário pré-compilado
      run: tar -xzf artifacts/ai-memory-linux-x86_64.tar.gz -C dist/docker/
    - uses: docker/build-push-action@v6
      with:
        target: runtime-prebuilt-amd64  # inject the binary, don't recompile

The AUR job does the same thing: download the tarball, compute the sha256, inject it into the PKGBUILD:

X86_SHA=$(awk '{print $1}' artifacts/ai-usagebar-linux-x86_64.tar.gz.sha256)
sed -i "s/^sha256sums_x86_64=.*/sha256sums_x86_64=('$X86_SHA')/" packaging/aur/PKGBUILD-bin

That’s the core idea. The binary is the real artifact. RPM, DEB, AppImage, AUR, Homebrew tarball, Docker image, all of it is just wrapping around the same binary.

The two snippets above are excerpts. If you want to see the whole thing running, the full build-once-reuse flow (binary compiled once, reused across the Docker and AUR jobs) lives in ai-memory’s release.yml, and the sha256 injection into the PKGBUILD is in ai-usagebar’s release.yml.

The menu of formats

Every platform is its own world. Here’s the minimum worth knowing:

Linux is the most fragmented. The options, from simplest to most work:

• Tarball (.tar.gz): the lowest common denominator. Works on any distro, just unpack it and drop it in PATH. Always offer this one.
• AUR (Arch): two packages, the -bin (prebuilt binary, installs in seconds) and the source one (compiles on the user’s machine). I publish both with the KSXGitHub/github-actions-deploy-aur action. The -bin is what everybody uses.
• AppImage: a single executable file that carries all its dependencies inside. Good for desktop apps. Frank Sherlock, which is Tauri, builds the AppImage right in the build and I still GPG-sign it afterward (the signing step is in its release.yml). For a simple binary you can use appimagetool.
• RPM and DEB: for Fedora/openSUSE and Debian/Ubuntu. The tool that saves your life here is nfpm, which generates both from a single declarative nfpm.yaml, no need to learn the guts of rpmbuild or dpkg-deb. It takes the already-compiled binary and spits out both packages.

macOS has two targets: Apple Silicon (aarch64-apple-darwin) and Intel (x86_64-apple-darwin). You either compile them separately or build a universal binary with lipo. And here lives the platform’s biggest headache: signing and notarization. Without it, macOS Gatekeeper refuses to open your app. The flow is to decode the P12 certificate, import it into a temporary keychain, sign with codesign --options runtime --timestamp, and notarize:

codesign --sign "$APPLE_SIGNING_IDENTITY" --options runtime --timestamp ./meu-binario
ditto -c -k meu-binario notarize.zip
xcrun notarytool submit notarize.zip \
  --apple-id "$APPLE_ID" --password "$APPLE_PASSWORD" \
  --team-id "$APPLE_TEAM_ID" --wait

This requires a paid Apple Developer account. It’s annoying, but it’s the price of not throwing an “app is damaged” error in your user’s face. The whole signing-and-notarization job, with the temporary-keychain dance, is in ai-jail’s release.yml.

Windows is the easiest to build (x86_64-pc-windows-msvc) and the thorniest to sign. A Windows code signing certificate is expensive and bureaucratic, so I, honestly, don’t sign. I accept that SmartScreen will throw a warning on first run and I make that clear in the README. For a personal open source project, that’s good enough.

Rust crates (cargo publish): if it’s a Rust library or CLI, publishing to crates.io gives you cargo install for free. One line in the pipeline. Tip: make it idempotent, because re-running on an already-published tag errors out. I grep the output for “already exists” so it doesn’t blow up the whole release over that.

Homebrew: the clean way is to keep your own tap (akitaonrails/homebrew-tap). The pipeline updates the formula with the new URL and the new sha256 on every release. The user runs brew tap once and after that it’s just brew upgrade.

mise: mise can install straight from GitHub releases without you configuring anything on your end, as long as your tarballs follow a platform-name convention. You get this path basically for free just by naming your releases well.

Notice a detail: AUR, Homebrew, mise, and the direct download all consume the same release tarball. You publish the binaries once, and four different package managers know how to fend for themselves. That’s why “compile once, repackage many” is the golden rule.

Pillar 2: GitHub Actions, tag-based releases, and the changelog

The second thing that makes an open source project serious is CI. Forget the “build passing” badge on the README, the value is in the common ground it creates: when someone sends a pull request, CI runs the tests, the linter, and the security scanner, and you know right away whether that contribution breaks anything. Without it, every PR turns into a tiresome manual negotiation.

On my Rust projects CI runs cargo fmt --check, cargo clippy -D warnings, cargo test, and cargo audit. On the Rails projects it’s rubocop, brakeman (security scanner), bundler-audit, and the Minitest suite. FrankMD alone has over 1,800 tests between Ruby and JavaScript. Frank Sherlock clears 300 tests in Rust plus 300 on the frontend. That safety net is what lets me accept a stranger’s PR without sweating.

PR review with an LLM

The tests and CI are the bottom net. Above it there’s a layer that became central to my workflow: using the LLM itself to review the pull requests. Opus and GPT are absurdly good at code review and quality refactoring. Better than me on consistency, because they don’t get tired and they don’t skim over 200 lines just because it’s the fifth review of the day.

I have two prompts I use all the time. The first, for a specific PR that just landed:

look at the open PR on github. don’t trust the author’s description. audit the code thoroughly to see if it actually makes sense. make sure there are no regressions and no drop in code quality. make sure test coverage is adequate. then tell me what you think we should do.

Notice the “don’t trust the author’s description.” That’s deliberate. A PR description tells you what the person thought they did, and it doesn’t always match what they actually did. The LLM has to read the real code, not the summary.

The second prompt is for after a long coding session or a bunch of PRs merged at once:

we touched a lot of code. audit the code those PRs changed, look for dead code, unnecessary duplication, magic hardcoded values that should be constants and/or better documented, missing test coverage, aim for clean code principles, then check whether the documentation was properly updated.

With the verdict in hand, I decide. For an incomplete PR, what I do depends on the size and my priority at that moment. If it’s something small, I fix it myself right in the PR with an amend, reply explaining what I changed, and then merge and close. When I can merge but can’t fix beforehand, I merge and immediately drop a follow-up fix commit. And when it’s a big change, I ask the LLM to reply to the PR explaining why we’re not merging, with the right direction on what needs to be done.

That’s how I closed almost 40 PRs and 20 issues on ai-memory alone, in a couple of days. That volume doesn’t happen by hand, and it also can’t happen on autopilot.

And this is the point I keep hammering: none of this is blind. I’m piloting the LLM toward those goals the whole time. It does the audit, puts the suggestions on the table, and even writes the text of the reply, but the one who decides to merge, fix, or reject is me. The LLM is the tireless reviewer; I’m the one who signs off.

The tag-based trigger

The release part is separate from CI. While CI runs on every push and PR, the release fires only when you push a version tag:

on:
  push:
    tags:
      - 'v*.*.*'   # dispara em v1.2.3, v0.5.0, etc.
  workflow_dispatch: # permite disparar manualmente também

The flow for cutting a version becomes this:

git tag v0.5.0
git push origin v0.5.0

And that’s it. GitHub Actions notices the tag, fires the release workflow, compiles for the targets, packages everything, signs whatever needs signing, creates the GitHub Release, and publishes to the package managers. I don’t touch anything else. In fact that’s literally what I tell Claude Code: “cut v0.5.0.” It runs cargo set-version, updates the CHANGELOG, commits, creates the tag, pushes, and the rest is the pipeline.

The Rust build matrix compiles each target on its native platform:

strategy:
  matrix:
    include:
      - os: ubuntu-22.04
        target: x86_64-unknown-linux-gnu
      - os: ubuntu-22.04-arm
        target: aarch64-unknown-linux-gnu
      - os: macos-latest
        target: aarch64-apple-darwin
      - os: windows-latest
        target: x86_64-pc-windows-msvc

A cross-compilation gotcha: if you compile ARM inside an x86 machine (via cross), you can’t run the tests for that binary right there, because the architecture doesn’t match. The fix is to run tests only on the native target and cross-compile the rest without testing. ai-usagebar does exactly that, x86 native and ARM via cross, in the matrix of its release.yml. And always use a compile cache: Swatinem/rust-cache easily cuts build time in half.

Release notes and the changelog matter

This is where a lot of people slack off, and they shouldn’t. When you cut a release, the person thinking about upgrading wants to know what changed. “v0.5.0” says nothing. A changelog says something.

I keep a CHANGELOG.md in the Keep a Changelog format, one section per version. And the pipeline automatically extracts the current version’s section to paste into the GitHub Release body. A simple awk handles it:

awk -v ver="$VERSION" '
  BEGIN { hdr = "## [" ver "]" }
  index($0, hdr) == 1 { flag=1; next }
  flag && index($0, "## [") == 1 { exit }
  flag { print }
' CHANGELOG.md > changelog_section.md

That awk is the same one that runs on the ai-usagebar release (again in its release.yml, which is my most polished one on this front). Then, after the changelog, I append the install instructions and the sha256 checksums for each artifact. The result is a GitHub Release that explains itself: what changed, how to install it, and how to verify the integrity of the download. The user reads it and decides whether to upgrade. That’s the kind of care that makes a stranger trust your project.

Pillar 3: Real deployment, no fuss

The projects that run as a service (the Rails ones, mainly) have to go somewhere. And here I go against the hype. No Kubernetes. No Kamal. No three-stage deploy pipeline with manual approval.

I have a home server. A Minisforum MS-S1 Max running openSUSE MicroOS that I migrated recently, with a Gitea acting as a private Docker registry on port 3007. For a solo developer with a simple home server, that’s more than enough. All that cluster-orchestration ceremony is solving a problem I don’t have.

The whole deploy fits in a single bin/deploy. It does three things: builds the image, pushes it to the registry, and SSHes into the server to bring up the new version:

#!/usr/bin/env bash
set -euo pipefail

# Config comes from a .gitignored file, outside the repo
source "$(dirname "$0")/../config/deploy.env"

IMAGE="${REGISTRY_HOST}:${REGISTRY_PORT}/akitaonrails/frank_fbi:latest"

echo "==> Buildando a imagem..."
docker build -t "$IMAGE" .

echo "==> Empurrando pro registry..."
docker push "$IMAGE"

echo "==> Subindo no servidor..."
ssh "${DEPLOY_USER}@${DEPLOY_HOST}" \
  "cd ${REMOTE_DIR} && \
   docker compose pull && \
   docker compose down --remove-orphans && \
   docker compose up -d"

That’s it. build, push, ssh with pull, down, up. Four commands on the remote server, chained in a single SSH. No agent running over there, no webhook, no control panel. The server has a docker-compose.production.yml pointing at the :latest image from the registry, and the pull brings the new version down.

Notice the sensitive data lives in a config/deploy.env that doesn’t go into Git. The repo only carries the config/deploy.env.example, with the structure and fake values. This “versioned example, real one ignored” pattern repeats in every project of mine, and it’s what lets the project be public without leaking the host, user, or path of my server.

The image itself comes from a multi-stage Dockerfile. The build stage installs the compilation tools, builds everything, and the final stage copies only the result into a lean image, running as a non-root user. The production compose swaps build: for image:, points the volumes at the server’s persistent paths (/var/opt/docker/...), and uses restart: always to bring everything back up if the server reboots.

And again, the beauty of standardization: since every project of mine has this same bin/deploy, I can tell Claude Code “deploy Frank FBI” and it knows exactly what to run. I don’t have to remember a single command. The tool is the executable documentation of the process.

The lesson: nobody cares about your stack

Now the part I most wanted to write, and the one worth more than all the YAML snippets above.

Most people, when they go to present a project, focus everything on the technology. “Check out my brand-new vector database.” “Check out my last-gen reactive framework.” “I used such-and-such microservices architecture.” And they fill the top of the README with that: the dependency list, the component diagram, the stack.

Nobody cares.

Seriously. The user who lands on your README doesn’t want to know which database you used. They want to know what problem this solves for them. Implementation detail doesn’t belong at the top of the README. It belongs in docs/, for whoever’s going to contribute. In the README, what matters is the use case.

Look at the difference. “MCP server in Rust with FTS5 and optional LLM consolidation” says nothing to anybody. Now “long-term memory shared across Claude Code, Codex, and Cursor, with handoff between sessions” — that one the guy gets instantly, because it’s the pain he has. Both describe ai-memory. One talks about technology, the other talks about the problem it solves.

The use case is the single most important thing about the whole project. Before writing a line of code, the right question is “what problem does this solve, and for whom?” If you can’t answer that in one sentence, you’re building a solution looking for a problem. And a solution looking for a problem is, in practice, a solution that solves nothing. Doesn’t matter how elegant the code is on the inside.

The UX layer, the surface the user touches, is worth far more than the internal engineering. A project with mediocre technology that’s trivial to install, that clearly states what problem it solves, and that has honest documentation, goes further than a brilliant project nobody can run. Every time.

So the minimum, the real minimum for an open source project, is this: easy install, CI that gives you confidence, documentation that starts from the problem. The pretty algorithm comes way down the priority list. The three pillars of this post exist to serve one thing, which is the experience of whoever shows up from outside. Get that right first. The rest is implementation detail.

With the new Minisforum MS-S1 Max that I bought, I decided to do the migration properly. And I decided to use Claude Code from the start to speed up the process. It’s a home server, only I use it, so the risk of making a mistake is low. But if this were a real production server, I would never do this without rigorous human review at every step.

What follows is the migration writeup and a guide for anyone who wants to replicate it. If I ever have to rebuild from scratch, this post is the documentation.

Choosing the operating system

Why not Ubuntu Server again

I used Ubuntu Server on the NUC for convenience. But do-release-upgrade is Russian roulette. Every time Ubuntu releases a new version, the upgrade is a real risk of breaking things. Packages change, configs get overwritten, dependencies clash. For a server that has to be running constantly, that’s unacceptable.

Why not Arch Linux

I use Arch on the desktop and I like it. But Arch is a rolling release with no stability guarantees at all. For a desktop where I can stop and fix problems, fine. For a headless server running 49 Docker containers that has to work after every reboot, no.

Fedora CoreOS vs openSUSE MicroOS

The two modern options for a container server are Fedora CoreOS and openSUSE MicroOS. Both are immutable systems: the root filesystem is read-only, updates are atomic (they either apply in full or don’t apply at all), and rollback is instant.

The difference: Fedora CoreOS uses Ignition (declarative configuration before the first boot) and is designed to be provisioned automatically. MicroOS uses transactional-update and allows normal interactive use. For a home server where I want SSH and the ability to poke around manually, MicroOS fits better.

What makes MicroOS different

The immutable system concept changes how you operate the server:

Every package install or /etc edit goes through transactional-update, which creates a new btrfs snapshot, applies the change in that snapshot, and on the next reboot the system boots into the updated snapshot. If the change breaks something, you do transactional-update rollback and you’re back on the previous snapshot in seconds.

Updates are automatic and daily. The transactional-update.timer downloads patches, creates a snapshot, and rebootmgr reboots in a configured window (in my case, between 4am and 5:30am). If the update breaks the boot, GRUB automatically falls back to the previous snapshot.

SELinux is enforcing by default. That caused 90% of the problems during the migration, but it’s the right setting for security.

Initial setup

Hardware

• AMD Ryzen AI Max+ 395, 128GB LPDDR5X
• 96GB allocated as VRAM via BIOS (UMA Frame Buffer Size)
• 2TB NVMe (system + Docker)
• Wired 2.5Gbps network
• Synology DS1821+ NAS at 192.168.0.21 (NFS)

First steps

MicroOS install is standard. Then:

# Create user with a UID that matches the NAS (so NFS works without permission issues)
useradd -u 1026 -m akitaonrails
passwd akitaonrails

# Configure sudo (inside transactional-update shell)
sudo transactional-update shell
# inside: add akitaonrails to sudoers
exit
sudo reboot

Synology NFS

The Synology NAS exports /volume1/TERACHAD over NFS. The mount point on MicroOS is /var/mnt/terachad (not /mnt/, which lives on the immutable root).

In /etc/fstab (applied through transactional-update):

192.168.0.21:/volume1/TERACHAD /var/mnt/terachad nfs4 nfsvers=4.1,rsize=262144,wsize=262144,hard,_netdev 0 0

Details that matter: nfsvers=4.1 because 4.2 didn’t work with the Synology. rsize=262144,wsize=262144 (256KB buffers) was the biggest NFS performance improvement. hard instead of nofail so the mount keeps retrying indefinitely if the NAS disconnects temporarily.

GPU / ROCm

This step was a pain. The Radeon 8060S in the AI Max+ 395 is gfx1151, which ROCm doesn’t officially support. Three steps were needed, and all three are mandatory:

1. BIOS: set UMA Frame Buffer Size to 96GB
2. Kernel: add amdttm.pages_limit=25165824 amdttm.page_pool_size=25165824 to /etc/kernel/cmdline
3. Docker: use HSA_OVERRIDE_GFX_VERSION=11.5.1 in every ROCm container

Without step 2, ROCm only sees 15.5GB even with the BIOS allocation in place. The numbers are 96GB / 4KB (page size) = 25,165,824 pages.

sudo transactional-update shell
echo "amdttm.pages_limit=25165824 amdttm.page_pool_size=25165824" >> /etc/kernel/cmdline
exit
sudo sdbootutil update-all-entries  # OUTSIDE the transactional shell
sudo reboot

Verification:

cat /sys/class/drm/card1/device/mem_info_vram_total
# 103079215104 (96 * 1024^3)

Docker on MicroOS

sudo transactional-update --non-interactive pkg install docker
sudo reboot

sudo systemctl enable --now docker
sudo usermod -aG docker akitaonrails
# logout and login for the group to take effect

# Install standalone docker-compose (the openSUSE package doesn't include it)
sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-linux-x86_64" \
  -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
mkdir -p ~/.docker/cli-plugins
ln -s /usr/local/bin/docker-compose ~/.docker/cli-plugins/docker-compose

daemon.json

{
  "log-level": "warn",
  "log-driver": "local",
  "log-opts": {"max-size": "10m", "max-file": "5"},
  "selinux-enabled": true,
  "live-restore": true,
  "userland-proxy": false,
  "exec-opts": ["native.cgroupdriver=systemd"]
}

live-restore: true keeps containers alive across a Docker daemon restart. userland-proxy: false uses iptables directly instead of proxy processes (less overhead). selinux-enabled: true is mandatory on MicroOS.

SELinux and Docker: the biggest source of problems

This deserves an entire section because it was responsible for 90% of the bugs during the migration.

On MicroOS with SELinux enforcing, every container that writes to a host bind-mounted directory needs special handling. There are two approaches: the :Z suffix on volumes and the security_opt: label:disable option.

NEVER use :Z. Use security_opt: label:disable.

:Z tells Docker to relabel the host directory with the container’s SELinux context. Sounds like the right thing. In practice:

• SQLite databases break. The relabeling changes the file’s context and SQLite may refuse to open the WAL journal.
• NFS mounts silently ignore :Z. NFS doesn’t support SELinux xattrs. The kernel ignores the flag without error, but the container still doesn’t have permission.
• :ro,Z mounts try to relabel even when read-only, which fails on NFS and can corrupt the context on local paths.

The right solution for every container on this system:

services:
  myservice:
    security_opt:
      - label:disable     # disables SELinux enforcement for this container
    volumes:
      - ./data:/data      # NO :Z
      - ./config.yml:/etc/config.yml:ro  # NO :Z even on :ro

label:disable disables SELinux label enforcement for that container only, not for the entire system. Combined with Docker’s network and process isolation, it’s safe for a home server.

Migrating the stacks

All Docker stacks were reorganized into /var/opt/docker/<stack>/docker-compose.yml. On the old server, they were scattered across /home/akitaonrails/docker/, /home/akitaonrails/<service>/, with no pattern.

Substitutions applied across every compose file

Media stack (Plex, Radarr, Sonarr, etc.)

The media stack is the most complex. Plex needs its own LAN IP (macvlan) for direct streaming to work. The setup:

docker network create -d macvlan \
  --subnet=192.168.0.0/24 \
  --gateway=192.168.0.1 \
  -o parent=enp97s0 \
  plex_macvlan

In compose, Plex needs to be on two networks: the macvlan (for the IP 192.168.0.6) and the default bridge (so other containers like Seerr can talk to it):

plex:
  networks:
    plex_macvlan:
      ipv4_address: 192.168.0.6
      mac_address: "02:42:c0:a8:00:06"
    default: {}    # mandatory — without this, Seerr can't see Plex

A detail that almost broke me: Plex stores absolute paths in its database. If the container’s internal volume changed from /media to /data, Plex no longer finds anything. You have to use exactly the same mount target as the old compose.

Ollama with ROCm

New stack, didn’t exist on the previous server:

ollama:
  image: ollama/ollama:rocm
  container_name: ollama
  devices:
    - /dev/kfd:/dev/kfd
    - /dev/dri:/dev/dri
  security_opt:
    - seccomp:unconfined
    - label:disable
  group_add:
    - "485"   # render group GID
    - "488"   # video group GID
  environment:
    - HSA_OVERRIDE_GFX_VERSION=11.5.1
    - PYTORCH_ROCM_ARCH=gfx1151
    - OLLAMA_KEEP_ALIVE=30m
    - OLLAMA_NUM_PARALLEL=4
    - OLLAMA_FLASH_ATTENTION=1
    - OLLAMA_KV_CACHE_TYPE=q8_0
  volumes:
    - /var/lib/ollama:/root/.ollama
  ports:
    - "11434:11434"

OLLAMA_FLASH_ATTENTION=1 activates flash attention. OLLAMA_KV_CACHE_TYPE=q8_0 uses 8-bit KV cache, cutting the bandwidth needed per token in half. Free performance optimizations.

Monitoring (Grafana + Prometheus)

Grafana uses a named volume (grafana_data) which is NOT included in normal filesystem backups. That’s the reason I lost all my dashboards on the first attempt. The fix is an explicit backup of the named volume:

# On the old server:
docker run --rm -v grafana_data:/data:ro -v /tmp:/backup alpine \
  tar czf /backup/grafana_data.tar.gz -C /data .

# Transfer and restore on the new one:
docker run --rm -v grafana_data:/data -v /tmp:/backup alpine \
  sh -c "cd /data && tar xzf /backup/grafana_data.tar.gz"

Same thing for Portainer (portainer_data). Any volume defined in the volumes: block of a compose without a host path needs this treatment.

Cloudflare Tunnel

I use Cloudflare Tunnel to access all the services from outside the house without opening ports on the router. The migration was the easiest part: copy the tunnel’s JSON credentials file and the config.yml, update the IPs from .145 to .90, and bring the container up. The tunnel keeps the same ID, no need to recreate DNS.

The hostnames live in config.yml: portainer, grafana, plex, seerr, qbittorrent, syncthing, radarr, sonarr, bazarr, prowlarr, vault, gitea, kavita, and others. Everything accessible via https://<service>.example.com from anywhere.

Gitea (image registry)

Gitea acts as a private Docker registry on port 3007. The Frank FBI, Frank Mega, Frank Yomik and Mila projects have Docker images that are built and pushed to Gitea. For it to work, Docker’s daemon.json needs:

{
  "insecure-registries": ["192.168.0.90:3007"]
}

Gitea SSH gave me a problem during the migration: the old app.ini had SSH_LISTEN_PORT=22, but the container’s entrypoint also starts sshd on port 22. Conflict. Fix: GITEA__server__SSH_LISTEN_PORT=2222 as an environment variable in compose.

All 49 containers running

The migrated server runs 49 containers across 15 stacks. The media stack alone has 10 containers (Plex, Radarr, Sonarr, Bazarr, Prowlarr, qBittorrent, SABnzbd, Jackett, FlareSolverr, Seerr). The personal projects (Frank FBI, Frank Mega, Frank Yomik, Mila) add another 11. Monitoring with Grafana, Prometheus, node-exporter and cAdvisor. Utilities like Portainer, Vaultwarden, Syncthing, Organizr, Watchtower. Gitea as private Docker registry. Immich as self-hosted Google Photos. Kaizoku for manga with Kavita as the reader. Ollama with ROCm. And Bitcoin Core/Fulcrum indexing the blockchain off the NAS.

Backups: two layers

Layer 1: local btrfs snapshots (snapper)

/var lives on a 3.7TB btrfs partition. snapper creates automatic snapshots: 7 daily + 1 weekly. They’re crash-consistent, not application-consistent (postgres can be slightly inconsistent if there’s heavy writing during the snapshot).

To recover an accidentally deleted file:

sudo snapper -c var list
sudo cp /var/.snapshots/5/snapshot/opt/docker/media/radarr/appdata/config/radarr.db \
        /var/opt/docker/media/radarr/appdata/config/radarr.db

For a full stack rollback:

sudo docker compose -p media down
sudo snapper -c var undochange 7..0 /var/opt/docker/media
sudo docker compose -p media up -d

Layer 2: restic to the NAS (off-machine)

restic runs every night at 3am, doing incremental backups to /var/mnt/terachad/homelab-backups/. Retention: 7 daily + 4 weekly. Content-based deduplication, so Plex config (19GB) and Gitea repos (12GB) only transfer deltas.

Before restic runs, a pg_dump exports the postgres databases (Immich, Kaizoku). The dumps go to /tmp/homelab-db-dumps/ and are included in the backup.

What’s NOT included in the backup (re-downloadable): Bitcoin blockchain (785GB on the NAS), Docker images (re-pullable), Ollama models (re-downloadable), HuggingFace/EasyOCR caches, Plex transcoding scratch.

Large re-downloadable directories were converted to btrfs subvolumes so snapper ignores them: /var/lib/ollama and /var/opt/docker/bitcoin/fulcrum/fulc2_db.

Performance tuning

btrfs with zstd compression

Added compress=zstd:1 to the fstab for the /var partition. zstd level 1 has near-zero CPU overhead on NVMe and compresses Docker metadata, JSON configs and logs nicely. Incompressible data (SQLite, postgres) is automatically skipped by btrfs.

zram swap

With ~30GB of RAM available for the system (96GB go to VRAM), in-memory compressed swap helps. zram creates a ~15GB swap device (ram/2) with zstd compression, much faster than disk swap.

# /etc/systemd/zram-generator.conf
[zram0]
zram-size = ram / 2
compression-algorithm = zstd
swap-priority = 100

btrfs nodatacow on database directories

Copy-on-write + random database writes = write amplification. I disabled CoW on the directories that hold SQLite and postgres:

sudo chattr +C /var/opt/docker/gitea/data/gitea.db
sudo chattr +C /var/opt/docker/immich/db/
sudo chattr +C /var/opt/docker/media/radarr/appdata/config/

CPU in performance mode

On a headless server, there’s no point saving energy:

echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/energy_performance_preference

Persisted via systemd service cpu-epp.service.

Docker shutdown fix

A problem I discovered: Docker ships with KillMode=process, which means at system shutdown, systemd kills only dockerd and leaves all the containerd-shim processes (one per container, ~49 in my case) orphaned. systemd-shutdown then has to hunt them down one by one after the journal has already stopped, causing a silent multi-minute hang.

Fix:

# /etc/systemd/system/docker.service.d/shutdown.conf
[Service]
KillMode=control-group
TimeoutStopSec=30

The problems we ran into

This is the table of actual problems we hit during the migration. If you’re planning something similar, read it before starting:

What to tell Claude Code before you start

If I were redoing the migration from scratch, I’d give Claude Code these instructions on the very first message. In order of importance:

Tell it that SELinux is enforcing and that it should NEVER use :Z on any Docker volume, but rather security_opt: label:disable on every service. Tell it that /var/mnt/terachad/ is an NFS mount and that :Z should never appear on NFS paths. Tell it to always look at the original compose before rewriting and only change IPs, paths and container names, without inventing new volume layouts. Warn that named volumes need explicit backup (Grafana, Portainer). Explain that Plex runs on macvlan and needs default: {} in the networks. Inform that the GPU is gfx1151, not officially supported, and that it needs UMA 96GB in the BIOS + kernel TTM params + HSA_OVERRIDE_GFX_VERSION=11.5.1. And tell it that Bitcoin/Fulcrum don’t process environment variables, everything goes as an argument in command:.

Those instructions would have prevented 80% of the problems we hit.

Final server layout

/var/opt/docker/
├── bitcoin/          (bitcoind + fulcrum)
├── cloudflared/      (Cloudflare tunnel)
├── frank_fbi/        (email fraud analysis)
├── frank_mega/       (Mega clone)
├── frank_yomik/      (manga translation)
├── gitea/            (Docker registry)
├── immich/           (self-hosted Google Photos)
├── kaizoku/          (manga downloader + reader)
├── media/            (Plex + *arr stack)
├── mila/             (Discord bot)
├── monitor/          (Grafana + Prometheus)
├── ollama/           (local LLM with ROCm)
├── rip/              (HandBrake)
└── utils/            (Portainer, Vaultwarden, Syncthing, etc.)

/var/mnt/terachad/    (Synology NFS)
├── Bitcoin/data/     (blockchain, 785GB)
├── Downloads/        (torrents + nzbget)
├── Videos/           (Radarr movies + Sonarr series)
└── Ollama/models/    (model overflow if local disk fills up)

A warning about using AI to administer servers

I used Claude Code to speed up the migration. It created compose files, wrote backup scripts, configured the firewall, diagnosed SELinux problems. It worked well for my case: home server, only I use it, and I was reviewing every step.

But there are traps. Claude doesn’t know that :Z breaks SQLite unless you tell it. It doesn’t know that Fulcrum doesn’t accept env vars unless it’s already seen the Dockerfile. It will invent “better” volume layouts that break Plex because Plex stores absolute paths in its database.

If it were real production: don’t do this without review. Every compose file Claude generates, read it in full before applying. Every destructive command (rollback, delete, recreate), confirm manually. And have tested backups before you start. Claude is great for generating the first version and diagnosing errors, but the architecture decisions and the safety validations are yours.

The previous home server posts that may give additional context:

• My “Personal Netflix” with Docker Compose
• Accessing my Home Server with a real domain
• Protecting your Home Server with Cloudflare Zero Trust
• Installing Grafana on my Home Server
• Self-hosted Vaultwarden

And make sure to subscribe to my new newsletter The M.Akita Chronicles!

–

If I told you that you can run a full Rails application in production — with database, job queue, cache — on a single $12/month VPS without installing PostgreSQL, Redis, or any other external service, would you believe me?

Well, you can. That’s Rails 8 with SQLite and Kamal.

SQLite on Rails 8: now it’s for real

Rails 8 brought SQLite as a real production option, not as a development toy. And when I say “real”, I mean rails new generates everything ready to go: main database in SQLite, SolidQueue (jobs) in SQLite, SolidCache (cache) in SQLite, SolidCable (WebSocket) in SQLite.

Four services that used to require PostgreSQL + Redis + Memcached are now .sqlite3 files on disk.

WAL mode (Write-Ahead Logging) enables concurrent reads while a write is happening. For applications with tens of thousands of requests per day — which is the vast majority of Rails applications in the world — SQLite is more than enough. And read latency is measured in microseconds, not milliseconds. It’s local disk access, not a network round-trip.

# database.yml — production with SQLite
production:
  primary:
    <<: *default
    database: storage/production.sqlite3
  queue:
    <<: *default
    database: storage/production_queue.sqlite3
    migrations_paths: db/queue_migrate
  cache:
    <<: *default
    database: storage/production_cache.sqlite3
    migrations_paths: db/cache_migrate

Three files. No TCP connection. No pooling. No “is PostgreSQL accepting connections?”. Backup is a VACUUM INTO and that’s it — a single SQL command that produces an atomic copy. Restore? Paste the file back. Try doing that with a 2GB PostgreSQL dump.

Kamal: Docker Deploy Without Kubernetes

Kamal is the Rails 8 deployer. Think of it as “SSH + Docker, but smart”. No Kubernetes. No Helm charts. No 300-line YAML describing an Ingress Controller.

The entire configuration fits in one file:

# config/deploy.yml
service: minha-app
image: meuregistry/minha-app

servers:
  web:
    hosts:
      - 107.170.70.49
    options:
      network: minha-network
    volumes:
      - minha-app-storage:/rails/storage
      - minha-app-content:/rails/content

proxy:
  ssl: true
  host: app.meudominio.com

builder:
  arch: amd64

env:
  clear:
    RAILS_LOG_LEVEL: info
  secret:
    - RAILS_MASTER_KEY
    - DATABASE_URL

What Kamal does when you run kamal deploy:

1. Builds the Docker image locally (or in CI)
2. Pushes to the registry
3. SSHs into the server
4. Pulls the new image
5. Runs migrations
6. Restarts the container with zero downtime via kamal-proxy

The kamal-proxy is the secret: it acts as a reverse proxy in front of your containers. When the deploy happens, it brings up the new container, waits for it to be healthy, redirects traffic, and then tears down the old one. Your users never notice a thing.

Docker Volumes: The Secret of SQLite in Production

The critical point of SQLite in Docker is: the data can’t die with the container. Docker volumes solve this:

volumes:
  - minha-app-storage:/rails/storage

The storage/ directory contains all the SQLite databases. With the Docker volume, they persist across deploys. The container is ephemeral; the data is permanent.

But there’s a detail that trips up a lot of people: if you run two different services that share data through files (not database), you need a shared volume:

# Service A writes content
volumes:
  - content-compartilhado:/rails/content

# Service B reads content
volumes:
  - content-compartilhado:/rails/content

The same Docker volume mounted on two containers. Simple, it works, and you don’t need NFS, S3, or anything fancy. It’s a directory on the server’s disk.

Multiple Services on the Same VPS

A rarely discussed advantage of Kamal: you can run multiple applications on the same server. The kamal-proxy routes by hostname:

app.meudominio.com    → container-app
bot.meudominio.com    → container-bot

Each service has its own deploy.yml, its own Docker image, its own deploy cycle. But they all run on the same VPS, sharing the same Docker network for internal communication.

SSL is automatic via Let’s Encrypt — kamal-proxy handles that. Each hostname gets its certificate without you configuring anything.

Hooks: Deploy Automation

Kamal supports hooks at various points of the deploy cycle. The most useful is pre-deploy, which runs before the new container replaces the old one:

# .kamal/hooks/pre-deploy
#!/bin/sh
# Run migrations before the container goes live
ssh root@$KAMAL_HOSTS \
  "docker exec minha-app-web bin/rails db:migrate"

Other useful hooks:

• post-deploy — notify the team, clear cache
• pre-connect — check server health
• docker-setup — install dependencies on the host

The Dockerfile Rails Generates

Rails 8 generates an optimized production Dockerfile. Some highlights:

# Multi-stage build: large build stage, minimal runtime stage
FROM ruby:3.3-slim AS base

# Install only runtime dependencies
RUN apt-get install -y libsqlite3-0

FROM base AS build
# Here install everything to compile native gems
RUN apt-get install -y build-essential libsqlite3-dev

# Copy and install gems
COPY Gemfile* ./
RUN bundle install --without development test

# Final stage: only what's needed to run
FROM base
COPY --from=build /usr/local/bundle /usr/local/bundle
COPY . .
RUN bundle exec bootsnap precompile --gemfile app/ lib/

Multi-stage build means the final image has no compilers, no development headers, nothing unnecessary. The production image stays lean.

Secrets: No .env in Production

Kamal has a secrets system that reads from a local file (.kamal/secrets) and injects them as environment variables into the container. That file never goes to git:

# .kamal/secrets (gitignored)
RAILS_MASTER_KEY=abc123...
AWS_ACCESS_KEY_ID=AKIA...

In deploy.yml, you reference:

env:
  secret:
    - RAILS_MASTER_KEY
    - AWS_ACCESS_KEY_ID

Kamal reads from the local file and configures it on the server. Simple, secure, no HashiCorp Vault or AWS Secrets Manager (unless you want — Kamal supports adapters).

SQLite Backup: Absurdly Simple (If You Do It Right)

Want a backup? One SQL command:

VACUUM INTO '/tmp/backup/newsletter.sqlite3';

VACUUM INTO is atomic — it produces a consistent, defragmented copy of the database, even with writes happening. It runs while the application is serving requests normally. And the resulting copy is a complete, functional SQLite database — you can open it, query it, restore it.

Warning: copying the .sqlite3 file directly (cp, tar, rsync) from a database in use can corrupt the backup. If a write is in progress at the moment of the copy, you end up with a half-written file. SQLite has WAL (Write-Ahead Log) and journaling that protect the active database — but the raw copy doesn’t inherit that protection.

In practice, I automated this with a Rails job that runs every hour:

class BackupDatabaseJob < ApplicationJob
  def perform
    backup_dir = File.join(Rails.configuration.content_dir, "backups")
    FileUtils.mkdir_p(backup_dir)
    dest = File.join(backup_dir, "newsletter.sqlite3")
    ActiveRecord::Base.connection.execute("VACUUM INTO '#{dest}'")
  end
end

The backup goes to the shared content directory — the same one that’s already synced with rsync to my local machine. No extra backup agent, no external service, no volume snapshot. One SQL, one file, one rsync.

Compare that with pg_dump of a production PostgreSQL database with dozens of tables and constraints. The SQLite backup is one command that produces a working file. Restore? Copy it back and restart.

When NOT to Use SQLite

I’ll be honest: SQLite isn’t for everything.

• Multiple web servers writing to the same database? No. SQLite is single-writer. If you need horizontal scaling with multiple nodes, go PostgreSQL.
• Massive datasets (hundreds of GB)? PostgreSQL has better query planning and parallelism.
• Replication and high availability? PostgreSQL with streaming replication.

But the honest question is: how many applications really need that? The vast majority run on a single server and will never need more. And for those, SQLite + Kamal is the most productive combination out there today.

The Real Cost

Let’s do the math:

And the cognitive cost? Instead of debugging “why did Redis lose my jobs?”, “why is PostgreSQL rejecting connections?”, “why did the cert expire?”, you focus on what matters: the product.

Conclusion

SQLite + Kamal is not a downgrade. The truth is that most applications never needed the complexity we thought they needed. Rails 8 embraced that and delivered a deploy experience that is, without exaggeration, the simplest that ever existed in the Ruby ecosystem.

One VPS. One command. Zero external services. And an application that runs as fast as any enterprise setup with 15 services in docker-compose.

I had been too lazy to install Grafana back then, but I finally got around to it, and it was much easier than I expected, which is why I decided to share how I did it.

The idea is to spin up cAdvisor, which monitors container resources, and Prometheus, which scrapes system resource data, then consolidate the data from both into Grafana, a web application that lets you build graphical dashboards to monitor your systems. If you do it right, in the end you’ll have a screen like this:

Installation

Spinning this up on my home server is very simple. I do everything with Docker Compose, so:

# monitor-docker-compose.yml
networks:
  monitoring:
    driver: bridge

volumes:
  prometheus_data:
  grafana_data:

services:
  node-exporter:
    image: quay.io/prometheus/node-exporter:latest
    command:
      - --path.rootfs=/host
    pid: host
    restart: unless-stopped
    networks: [monitoring]
    volumes:
      - /:/host:ro,rslave
    security_opt: ["no-new-privileges:true"]
    read_only: true

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    restart: unless-stopped
    networks: [monitoring]
    ports: ["8080:8080"]   # optional: expose UI locally
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    security_opt: ["no-new-privileges:true"]
    read_only: true

  prometheus:
    image: prom/prometheus:latest
    restart: unless-stopped
    networks: [monitoring]
    volumes:
      - prometheus_data:/prometheus
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.retention.time=30d
      - --web.enable-lifecycle
    ports: ["9090:9090"]
    security_opt: ["no-new-privileges:true"]
    read_only: true

  grafana:
    image: grafana/grafana:latest
    restart: unless-stopped
    networks: [monitoring]
    depends_on: [prometheus]
    ports: ["3001:3000"]
    environment:
      GF_SECURITY_ADMIN_USER: admin
      GF_SECURITY_ADMIN_PASSWORD: change-me
      GF_USERS_ALLOW_SIGN_UP: "false"
    volumes:
      - grafana_data:/var/lib/grafana
      - ./provisioning/datasources:/etc/grafana/provisioning/datasources:ro
    security_opt: ["no-new-privileges:true"]

Pay attention to the local directory mappings. If you put /etc/grafana/prometheus.yml, for example, you have to change this line:

...
    volumes:
      - prometheus_data:/prometheus
      - /etc/grafana/prometheus.yml:/etc/prometheus/prometheus.yml:ro
...

Speaking of which, here’s the prometheus.yml:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs: [{ targets: ['prometheus:9090'] }]

  - job_name: 'node'
    static_configs: [{ targets: ['node-exporter:9100'] }]

  - job_name: 'cadvisor'
    static_configs: [{ targets: ['cadvisor:8080'] }]

And finally /etc/grafana/provisioning/datasources/datasource.yaml:

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

Be careful with port mappings too. The recommended approach is this:

ports: ["127.0.0.1:8080:8080"]

That’s because cAdvisor and Prometheus only need to be accessible to Grafana, inside the home server, and there’s no reason to bind on 0.0.0.0 and have them reachable from outside the server. That’s your call. Since I’m on a home LAN and there are no tunnels pointing to these services, it makes no difference to me.

If, like me, you already have a service sitting on port 8080, cAdvisor will fail. In that case, remap the ports and reconfigure Prometheus, or just remove the port line. You don’t need cAdvisor exposed externally.

Another detail: by default, Grafana comes up on port 3000, but I changed it to 3001 because I already had another service on that port. So once it’s up, I can access it at http://192.168.0.200:3001.

As I explained in the Cloudflared article, I route this address through Cloudflare to get a nice domain like https://grafana.fabioakita.dev to access it both locally and remotely, if I want. Go read those other articles afterward to understand how.

And that’s it, now just spin up the containers:

docker compose -f monitor-docker-compose.yml up -d

Done, super simple. I should have done this sooner.

As you can see in the Docker Compose file, when you open the site for the first time, the login is admin with password change-me. Obviously, the first thing to do is change the password:

Dashboards

Now we’re practically done. We can check whether Prometheus is working and integrated. Just go to the “Connections” menu and “Data Sources”:

Grafana can consolidate data from dozens of different sources, and Prometheus is only one of them. In a larger, more complex infrastructure, this list would be huge.

To build dashboards, you can start from scratch and assemble them widget by widget, but it’s easier to import dashboards that already exist. Just click the magnifying glass at the top, in the search box, and choose “Import dashboards”:

On that page you can either import a JSON file, copy and paste the JSON, or search by an ID already registered on grafana.com:

Right off the bat, I learned that there are 3 IDs that are useful for me:

• 1860 - Node Exporter Full - the screenshot at the beginning of this article, with several system metrics like CPU, memory, disk, network, etc. The most useful one for me.
• 13484 - Docker/cAdvisor Exporter - CPU and memory metrics, but broken down per container running on the server.
• 3662 - Prometheus 2.0 overview - a general health summary.

If you like these or other dashboards, just favorite them with a “pin” and you’re set.

For example, here’s the cAdvisor dashboard:

I haven’t explored other dashboards yet. If you have ID recommendations to import, drop them in the comments below!

With this, I have a quick and good-looking view of my home server’s resource usage, and I can spot anything weird happening right away. These services don’t seem to consume many resources, so I can leave them running alongside the other containers.

I have been working very hard the past few weeks, quite literally coding non-stop, 7 days a week. Remember that “stop complaining you don’t have time, what do you do from midnight to 6 AM?” thing? Yeah, won’t apply to me.

And it’s exactly when you’re sleep deprived and exhausted that you start to make mistakes. Even though your code runs, locally, you will shoot yourself in the foot in one of those 3 AM coding spree sessions.

The test suite was really my sidekick, co-pilot, bringing me back to my senses whenever the night got too dark.

I maintain my own GitLab custom server for all of my company’s client and internal projects. Just a bit of paranoia, to make sure my code is owned by me alone. And GitLab is a fantastic tool. Even more so because it has its own Continuous Integration companion.

Just add your .gitlab-ci.yml file to your project, create a new branch, push it and create a Merge Request and the CI kicks in, with multiple parallel jobs if you want. Even when I was forgetting to run my tests, the CI would not let me fail.

Keeping a CI up and running does require you to stop and maintain your specs though. Sometimes you will wonder, why does a test run on your local machine but keeps failing in the CI server?

That’s when it’s super useful to run the CI docker image locally, to iron out remaining environment-dependent quirks.

And you can do just that by running the GitLab CI Runner itself. It will pick up your project’s .gitlab-ci.yml and run it locally through docker. So whatever problem you’re seeing on the server will quite definitely happen locally as well. With the added benefit that you don’t need to wait in line in case there is a queue of jobs waiting to run (many developers working, not as many machines to chew the CI work immediately). And you won’t be the one adding useless jobs to the same queue.

And, if you’ve followed my recommendations and installed the awesome Arch Linux distro (or my personal favorite derivative, Manjaro Gnome) you can easily install the runner through AUR like this:

pacaur -S gitlab-runner

Otherwise you will have to check your particular distro repos or download the binary from here. For example:

wget https://gitlab-runner-downloads.s3.amazonaws.com/master/binaries/gitlab-runner-linux-amd64
sudo mv gitlab-runner-linux-amd64 /usr/local/bin
sudo chmod +x /usr/local/bin/gitlab-runner

Once you have it installed, let’s say you have a snippet like this in your .gitlab-ci.yml configuration:

backend:
  stage: test
  script:
    - ./bin/cc-test-reporter before-build
    - bundle exec rspec --exclude-pattern "**/features/**/*_spec.rb"
  after_script:
    - ./bin/cc-test-reporter after-build --exit-code $? || true

You can configure many test jobs to run in parallel. I recommend that you separate front-end related JS tests, Rspec/minitest back-end unit tests, Capybara-based feature tests, Brakeman related security checks, for instance. Each will run in parallel and you won’t have to wait for everything to run if you’re only interested in the JS tests, for example. So you have faster feedback and can start fixing immediately.

Locally, from your project root directory, just run:

gitlab-runner exec docker backend

Of course, this is assuming that you already have Docker installed and properly configured. If you don’t, make sure you read Arch Wiki’s excellent entry on Docker but it’s basically installing the Docker package and adding yourself to the docker group.

This is it! Super easy, barely an inconvenience, and runner will do everything your GitLab CI server is doing.

One caveat is that unfortunately, it seems like you don’t have the ability to save cache or artifacts between job runs (gems, npm packages, etc) so every time this local runner runs, it will start from scratch. This makes it less useful to run all the time, but it’s still faster than pushing to the server and waiting in line for your turn in a busy server.

Saved me a couple times already.

TL;DR: Google is actually doing a pretty good job being a counterbalancer so AWS doesn’t slack off. If you already know everything about AWS, I’d encourage you to test out Google Cloud. Possibly because of muscle memory, I would still be more comfortable with AWS, but now that I forced myself to suffer the learning process, I am pretty confident with Google Cloud and Kubernetes for most of my scenarios.

Full disclaimer that I am not an expert, so take what I say with a grain of salt. This is one of those subjects that I am super eager to talk about but I’m also very reluctant on the proper choice of words so you don’t get the wrong idea about the proposed solutions.

The goal of this exercise is mainly for me to store some snippets and thoughts for future reference. So keep in mind that this is also not a step-by-step tutorial. My first intention was to go that way, but then I realized that it would be almost like writing an entire book, so not this time.

To succeed with something like Google Cloud and Kubernetes you must be battle tested in infrastructure. If you never installed server-grade Linux boxes from scratch, if you never did server optimizations, if you’re not comfortable with bare server-side components, do not attempt a real production deployment. Your safest bet is still something like Heroku.

You have to be that kinda person that likes to tinker (as you’ve probably read me doing in previous blog posts).

I don’t know everything, but I know enough. So I only had to figure out which of the pieces would closely fit my needs. You must outline your needs before attempting to write your first YAML file. Planning is crucial.

First things first, this is what I wanted/needed:

• Scalable web application tier, where I could do both rolling updates (for zero downtime updates) and automatic and manual horizontal scaling of the servers.
• Mountable persistent storage with automatic snapshots/backups.
• Managed robust database (Postgresql) with automatic backups and easy replication to read-only instances.
• Managed solution to store secrets (such as Heroku’s ENV support). Never store production configuration in the source code.
• Docker images support without me having to build custom infrastructure to deploy.
• Static external IP addresses for integrations that required a fixed IP.
• SSL termination so I could connect to CloudFlare (CDN is mandatory, but not enough, in 2018 we need some level of DDoS protection).
• Enough security by default, so everything is - in theory - lockdown unless I decide to open it.
• High-availability in different data center regions and zones.

It’s easy to deploy a simple demo web application. But I didn’t want a demo, I wanted a production-grade solution for the long-run. Improvements to my implementation are super welcome, so feel free to comment down below.

Some of the problems for newcomers:

• The documentation is very extensive, and you will find almost everything - if you know what you are looking for. Also bear in mind that Azure and AWS also implement Kubernetes with some differences, so some documentation doesn’t apply to Google Cloud and vice-versa.
• There are many features in alpha, beta, and stable stages. The documentation kinda keeps up well, but most tutorials that are just a couple months old may not work as intended anymore (this one included - I am assuming Kubernetes 1.8.4-gke).
• There is a whole set of words that apply to concepts you already know but are called differently. Getting used to the vocabulary may get in the way at first.
• It feels like you’re playing with Lego. Lots of pieces that you can mix and match. It’s easy to mess up. This means that you can build a configuration tailored to your needs. But if you just copy and paste from tutorials you will get stuck.
• You can do almost everything through YAML files and the command line, but it’s not trivial to reuse the configuration (for production and staging environments, for example). There are 3rd party tools that deal with parameterizable and reusable YAML bits, but I’d do it all by hand first. Never, ever, try automated templates in infrastructure without knowing exactly what they are doing.
• You have 2 fat command line tools: gcloud and kubectl, and the confusing part is that they name some things differently even though they are the same “things”. At least, kubectl is close to docker, if you’re familiar with that.

Once again, this is NOT a step-by-step tutorial. I will annotate a few steps but not everything.

Scalable Web-Tier (the Web App itself)

The very first thing you must have is a fully 12-factor compliant web app.

Be it Ruby on Rails, Django, Laravel, Node.js or whatever. It must be a fully shared-nothing app, that does not depend on writing anything to the local filesystem. One where you can easily shut down and start up instances independently. No old-style session in local memory or in local files (I prefer to avoid session affinity). No uploads to the local filesystem (if you must, you will have to mount an external persistent storage), always prefer to send binary streams to managed storage services.

You must have a proper pipeline that outputs cache-busting through fingerprinting assets (and like it or not, Rails still has the best out-of-the-box solution in its Asset Pipeline). You don’t want to worry about manually busting caches in CDNs.

Instrument your app, add New Relic RPM, add Rollbar.

Again, this is 2018, you don’t want to deploy naive code with SQL (or any other input) injection, no unchecked eval around your code, no room for CSRF or XSS, etc. Go ahead, buy the license for Brakeman Pro and add it to your CI pipeline. I can wait …

As this is not a tutorial, I will assume you’re more than able to sign up to Google Cloud and find your way to set up a project, configure your region and zone.

It took me a while to wrap my head around the initial structure in Google Cloud:

• You start with a Project, which is the umbrella for everything your app needs.
• Then you create “clusters”. You can have a production or staging cluster, for example. Or a web cluster and a separated services cluster for non-web stuff, and so on.
• A cluster has a “cluster-master”, which is the controller of everything else (the gcloud and kubectl commands talk to its APIs).
• A cluster has many “node instances”, the proper “machines” (or, more accurately, VM instances).
• Each cluster also has at least one “node pool” (the “default-pool”), which is a set of node instances with the same configuration, the same “machine-type”.
• Finally, each node instance runs one or more “pods” which are lightweight containers like LXC. This is where your application actually is.

This is an example of creating a cluster:

gcloud container clusters create my-web-production \
--enable-cloud-logging \
--enable-cloud-monitoring \
--machine-type n1-standard-4 \
--enable-autoupgrade \
--enable-autoscaling --max-nodes=5 --min-nodes=2 \
--num-nodes 2

As I mentioned, it also creates a default-pool with a machine-type of n1-standard-4. Choose what combination of CPU/RAM you will need for your particular app beforehand. The type I chose has 4 vCPUs and 15GB of RAM.

By default, it starts with 3 nodes, so I chose 2 at first but auto-scalable to 5 (you can update it later if you need to, but make sure you have room for initial growth). And you can keep adding extra node-pools for differently sized node instances, let’s say, for Sidekiq workers to do heavy-duty background processing. Then you should create a separated Node Pool with a different machine-type for its set of node instances, for example:

gcloud container node-pools create large-pool \
--cluster=my-web-production \
--node-labels=pool=large \
--machine-type=n1-highcpu-8 \
--num-nodes 1

This other pool controls 1 node of type n1-highcpu-8 which has 8 vCPUs with 7.2 GB of RAM. More CPUs, less memory. You have a category of highmem which is less CPUs with a whole lot more memory. Again, know what you want beforehand.

The important bit here is the --node-labels this is how I will map the deployment to choose between Node Pools (in this case, between the default-pool and the large-pool).

Once you create a cluster, you must issue the following command to fetch its credentials:

gcloud container clusters get-credentials my-web-production

This sets the kubectl command as well. If you have more than one cluster (let’s say, one my-web-production and my-web-staging), you must be very careful to always get-credentials for the correct cluster first, otherwise, you may end up running a staging deployment on the production cluster.

Because this is confusing, I modified my ZSH PROMPT to always show which cluster I am dealing with. I adapted from zsh-kubectl-prompt:

As you will end up having multiple clusters in a big app, I highly recommend you add this PROMPT to your shell.

Now, how do you deploy your application to the pods within those fancy node instances?

You must have a Dockerfile in your application project repository to generate a Docker image. This is one example for a Ruby on Rails application:

FROM ruby:2.4.3
ENV RAILS_ENV production
ENV SECRET_KEY_BASE xpto
RUN curl -sL https://deb.nodesource.com/setup_8.x | bash -
RUN apt-get update && apt-get install -y nodejs postgresql-client cron htop vim
ADD Gemfile* /app/
WORKDIR /app
RUN gem update bundler --pre
RUN bundle install --without development test
RUN npm install
ADD . /app
RUN cp config/database.yml.prod.example config/database.yml && cp config/application.yml.example config/application.yml
RUN RAILS_GROUPS=assets bundle exec rake assets:precompile

From the Google Cloud Web Console, you will find a “Container Registry”, which is a Private Docker Registry.

You must add the remote URL to your local config like this:

git remote add gcloud https://source.developers.google.com/p/my-project/r/my-app

Now you can git push gcloud master. I recommend you also add triggers to tag your images. I add 2 triggers: one to tag it with latest and another to tag it with a random version number. You will need those later.

Once you add the registry repository as a remote on your git configuration (git remote add) and push to it, it should start building your Docker image with the proper tags you configured with the triggers.

Make sure your Ruby on Rails application doesn’t have anything in the initializers that require a connection to the database, as it’s not available. This is something you might get stuck with when your Docker build fails because of the assets:precompile task loaded an initializer that accidentally calls a Model - and that triggers ActiveRecord::Base to try to connect.

Also, make sure the Ruby version in the Dockerfile matches the one in Gemfile, otherwise it will also fail.

Notice the weird config/application.yml above? This is from figaro. I also recommend you using something to make it easy to configure ENV variables in your system. I don’t like Rails secrets, and it’s not exactly friendly to most deployment systems after Heroku made ENV vars ubiquitous. Stick to ENV vars. Kubernetes will also thank you for that.

Now, you can override any ENV variable from the Kubernetes Deployment YAML file. Now it’s a good time to show an example of that. You can name it deploy/web.yml or whatever suits your fancy and - of course - check it into your source code repository.

kind: Deployment
apiVersion: apps/v1beta1
metadata:
  name: web
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  minReadySeconds: 10
  replicas: 2
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - image: gcr.io/my-project/my-app:latest
          name: my-app
          imagePullPolicy: Always
          ports:
          - containerPort: 4001
          command: ["passenger", "start", "-p", "4001", "-e", "production",
          "--max-pool-size", "2", "--min-instances", "2", "--no-friendly-error-pages"
          "--max-request-queue-time", "10", "--max-request-time", "10",
          "--pool-idle-time", "0", "--memory-limit", "300"]
          env:
            - name: "RAILS_LOG_TO_STDOUT"
              value: "true"
            - name: "RAILS_ENV"
              value: "production"
            # ... obviously reduced the many ENV vars for brevity
            - name: "REDIS_URL"
              valueFrom:
                secretKeyRef:
                  name: my-env
                  key: REDIS_URL
            - name: "SMTP_USERNAME"
              valueFrom:
                secretKeyRef:
                  name: my-env
                  key: SMTP_USERNAME
            - name: "SMTP_PASSWORD"
              valueFrom:
                secretKeyRef:
                  name: my-env
                  key: SMTP_PASSWORD
            # ... this part below is mandatory for Cloud SQL
            - name: DB_HOST
              value: 127.0.0.1
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: cloudsql-db-credentials
                  key: password
            - name: DB_USER
              valueFrom:
                secretKeyRef:
                  name: cloudsql-db-credentials
                  key: username

        - image: gcr.io/cloudsql-docker/gce-proxy:latest
          name: cloudsql-proxy
          command: ["/cloud_sql_proxy", "--dir=/cloudsql",
                    "-instances=my-project:us-west1:my-db=tcp:5432",
                    "-credential_file=/secrets/cloudsql/credentials.json"]
          volumeMounts:
            - name: cloudsql-instance-credentials
              mountPath: /secrets/cloudsql
              readOnly: true
            - name: ssl-certs
              mountPath: /etc/ssl/certs
            - name: cloudsql
              mountPath: /cloudsql
      volumes:
        - name: cloudsql-instance-credentials
          secret:
            secretName: cloudsql-instance-credentials
        - name: ssl-certs
          hostPath:
            path: /etc/ssl/certs
        - name: cloudsql
          emptyDir:

There is a lot going on here. So let me break it down a bit:

• The kind, and apiVersion is important, you have to keep an eye on the documentation if those change. This is what is called a Deployment. There used to be a Replication Controller (you will find those in old tutorials), but it’s no longer in use. The recommendation is to use a ReplicaSet.
• Name things correctly, here you have metadata:name with web. Also pay close attention to spec:template:metadata:labels where I am labeling every pod having a label of app: web, you will need this to be able to select those pods later in the Service section down below.
• Then I have spec:strategy where we configure the Rolling Update, so if you have 10 pods, it will terminate one, boot up the new one and keep doing that, without never bringing everything down at once.
• spec:replicas declares how many Pods I want at once. You will have to manually calculate the machine-type of the node-pool then divide how many total CPUs/RAM you have by how much you need for each application instance.
• Remember the Docker image we generated above with the ’latest’ tag? You refer to it in spec:template:spec:containers:image
• I am using Passenger with production configuration (check out Phusion’s documentation, do not just copy this).
• In the spec:template:spec:containers:env section I can override the ENV vars with the real production secrets. And you will notice that I can hard-code values or use this strange contraption:

- name: "SMTP_USERNAME"
  valueFrom:
    secretKeyRef:
      name: my-env
      key: SMTP_USERNAME

Now, it’s referencing a “Secret” storage that I named “my-env”. And this is how you create your own:

kubectl create secret generic my-env \
--from-literal=REDIS_URL=redis://foo.com:18821 \
--from-literal=SMTP_USERNAME=foobar

Read the documentation as you can load text files instead of declaring everything from the command line.

As I said before, I’d rather use a managed service for a database. You can definitely load your own Docker image, but I really don’t recommend it. Same goes for other database-like services such as Redis, Mongo. If you’re from AWS, Google Cloud SQL is like RDS.

After you create your PostgreSQL instance you can’t access it directly from the web application. In the end, you have a boilerplate for a second Docker image, a “CloudSQL Proxy”.

For that to work you must first create a Service Account:

gcloud sql users create proxyuser host --instance=my-db --password=abcd1234

After you create the PostgreSQL instance it will prompt you to download a JSON credential, so be careful and save it somewhere safe. I don’t have to say that you must generate a strong secure password as well. Then you must create extra secrets:

kubectl create secret generic cloudsql-instance-credentials \
--from-file=credentials.json=/home/myself/downloads/my-db-12345.json

kubectl create secret generic cloudsql-db-credentials \
--from-literal=username=proxyuser --from-literal=password=abcd1234

These are referenced in this part of the Deployment:

- image: gcr.io/cloudsql-docker/gce-proxy:latest
  name: cloudsql-proxy
  command: ["/cloud_sql_proxy", "--dir=/cloudsql",
            "-instances=my-project:us-west1:my-db=tcp:5432",
            "-credential_file=/secrets/cloudsql/credentials.json"]
  volumeMounts:
    - name: cloudsql-instance-credentials
      mountPath: /secrets/cloudsql
      readOnly: true
    - name: ssl-certs
      mountPath: /etc/ssl/certs
    - name: cloudsql
      mountPath: /cloudsql
volumes:
- name: cloudsql-instance-credentials
  secret:
    secretName: cloudsql-instance-credentials
- name: ssl-certs
  hostPath:
    path: /etc/ssl/certs
- name: cloudsql
  emptyDir:

See that you must add the database name (“my-db” in this example) in the -instance clause in the command.

And by the way, the gce-proxy:latest refers to version 1.09 at the time when this post was published. But there already was a 1.11 version. That one gave me headaches, dropping connections and adding a super long timeout. So I went back to the 1.09 (latest) and everything worked as expected. So be aware! Not everything that is brand new is good. In infrastructure, you want to stick to stable.

You may also want the option to load a separated CloudSQL instance instead of having it in each pod, so the pods could connect to just one proxy. You may want to read this thread on the subject.

It seems that nothing is exposed to anything unless you say so. So we need to expose those pods through what’s called a Node Port Service. Let’s create a deploy/web-svc.yaml file as well:

apiVersion: v1
kind: Service
metadata:
  name: web-svc
spec:
  sessionAffinity: None
  ports:
  - port: 80
    targetPort: 4001
    protocol: TCP
  type: NodePort
  selector:
    app: web

This is why I highlighted the importance of the spec:template:metadata:labels, so we can use it here in the spec:selector to select the proper pods.

We can now deploy these 2 like this:

kubectl create -f deploy/web.yml
kubectl create -f deploy/web-svc.yml

And you can see the pods being created with kubectl get pods --watch.

The Load Balancer

Many tutorials will expose those pods directly through a different Service, called Load Balancer. I am not so sure how well this behaves under pressure and if it has SSL termination, etc. So I decided to go full blown with an Ingress Load Balancer using the NGINX Controller, instead.

First of all, I decided to create a separated node-pool for it, for example, like this:

gcloud container node-pools create web-load-balancer \
--cluster=my-web-production \
--node-labels=role=load-balancer \
--machine-type=g1-small \
--num-nodes 1 \
--max-nodes 3 --min-nodes=1 \
--enable-autoscaling 

As when we created the example large-pool here you must take care of adding --node-labels to make the controller be installed here instead of the default-pool. You will need to know the node instance name, we can do it like this:

$ gcloud compute instances list
NAME                                             ZONE        MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP      STATUS
gke-my-web-production-default-pool-123-123       us-west1-a  n1-standard-4               10.128.0.1   123.123.123.12   RUNNING
gke-my-web-production-large-pool-123-123         us-west1-a  n1-highcpu-8                10.128.0.2   50.50.50.50      RUNNING
gke-my-web-production-web-load-balancer-123-123  us-west1-a  g1-small                    10.128.0.3   70.70.70.70      RUNNING

Let’s save it like this for now:

export LB_INSTANCE_NAME=gke-my-web-production-web-load-balancer-123-123

You can manually reserve an external IP and give it a name like this:

gcloud compute addresses create ip-web-production \
        --ip-version=IPV4 \
        --global

For the sake of the example, let’s say that it generated a reserved IP “111.111.111.111”. Then let’s fetch it and save it for now like this:

export LB_ADDRESS_IP=$(gcloud compute addresses list | grep "ip-web-production" | awk '{print $3}')

Finally, let’s hook this address to the load balancer node instance:

export LB_INSTANCE_NAT=$(gcloud compute instances describe $LB_INSTANCE_NAME | grep -A3 networkInterfaces: | tail -n1 | awk -F': ' '{print $2}')
gcloud compute instances delete-access-config $LB_INSTANCE_NAME \
    --access-config-name "$LB_INSTANCE_NAT"
gcloud compute instances add-access-config $LB_INSTANCE_NAME \
    --access-config-name "$LB_INSTANCE_NAT" --address $LB_ADDRESS_IP

Once we do this, we can add the rest of the Ingress Deployment configuration. This will be kinda long but it’s mostly boilerplate. Let’s start by defining another web application that we will call default-http-backend that will be used to respond to HTTP requests in case our web pods are not available for some reason. Let’s call it deploy/default-web.yml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: default-http-backend
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: default-http-backend
    spec:
      terminationGracePeriodSeconds: 60
      containers:
      - name: default-http-backend
        # Any image is permissable as long as:
        # 1. It serves a 404 page at /
        # 2. It serves 200 on a /healthz endpoint
        image: gcr.io/google_containers/defaultbackend:1.0
        livenessProbe:
          httpGet:
            path: /healthz
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 30
          timeoutSeconds: 5
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 10m
            memory: 20Mi
          requests:
            cpu: 10m
            memory: 20Mi

No need to change anything here, and by now you may be familiar with the Deployment template. Again, you now know that you need to expose it through a NodePort, so let’s add a deploy/default-web-svc.yml:

kind: Service
apiVersion: v1
metadata:
  name: default-http-backend
spec:
  selector:
    app: default-http-backend
  ports:
    - protocol: TCP
      port: 80
      targetPort: 8080
  type: NodePort

Again, no need to change anything. The next 3 files are the important parts. First, we will create an NGINX Load Balancer, let’s call it deploy/nginx.yml:

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: nginx-ingress-controller
spec:
  replicas: 1
  template:
    metadata:
      labels:
        k8s-app: nginx-ingress-lb
    spec:
      # hostNetwork makes it possible to use ipv6 and to preserve the source IP correctly regardless of docker configuration
      # however, it is not a hard dependency of the nginx-ingress-controller itself and it may cause issues if port 10254 already is taken on the host
      # that said, since hostPort is broken on CNI (https://github.com/kubernetes/kubernetes/issues/31307) we have to use hostNetwork where CNI is used
      hostNetwork: true
      terminationGracePeriodSeconds: 60
      nodeSelector:
        role: load-balancer
      containers:
        - args:
            - /nginx-ingress-controller
            - "--default-backend-service=$(POD_NAMESPACE)/default-http-backend"
            - "--default-ssl-certificate=$(POD_NAMESPACE)/cloudflare-secret"
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          image: "gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.5"
          imagePullPolicy: Always
          livenessProbe:
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            timeoutSeconds: 5
          name: nginx-ingress-controller
          ports:
            - containerPort: 80
              name: http
              protocol: TCP
            - containerPort: 443
              name: https
              protocol: TCP
          volumeMounts:
            - mountPath: /etc/nginx-ssl/dhparam
              name: tls-dhparam-vol
      volumes:
        - name: tls-dhparam-vol
          secret:
            secretName: tls-dhparam

Notice the nodeSelector to make the node label we added when we created the new node-pool.

You may want to tinker with the labels, the number of replicas if you need to. But here you will notice that it mounts a volume that I named as tls-dhparam-vol. This is a Diffie Hellman Ephemeral Parameters. This is how we generate it:

sudo openssl dhparam -out ~/documents/dhparam.pem 2048

kubectl create secret generic tls-dhparam --from-file=/home/myself/documents/dhparam.pem

kubectl create secret generic tls-dhparam --from-file=/home/myself/documents/dhparam.pem

Also, notice that I am using version “0.9.0-beta_5” for the controller image. It works well, no problems so far. But keep an eye on release notes for newer versions as well and do your own testing.

Again, let’s expose this Ingress controller through the Load Balancer Service. Let’s call it deploy/nginx-svc.yml:

apiVersion: v1
kind: Service
metadata:
  name: nginx-ingress
spec:
  type: LoadBalancer
  loadBalancerIP: 111.111.111.111
  ports:
  - name: http
    port: 80
    targetPort: http
  - name: https
    port: 443
    targetPort: https
  selector:
    k8s-app: nginx-ingress-lb

Remember the static external IP we have reserved above and saved in the LB_INGRESS_IP ENV var? This is the one we must put in the spec:loadBalancerIP section. This is also the IP that you will add as an “A record” in your DNS service (let’s say, mapping your “www.my-app.com.br” on CloudFlare).

Finally, we can create the Ingress configuration itself, let’s create a deploy/ingress.yml like this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.org/ssl-services: "web-svc"
    kubernetes.io/ingress.global-static-ip-name: ip-web-production
    ingress.kubernetes.io/ssl-redirect: "true"
    ingress.kubernetes.io/rewrite-target: /
spec:
  tls:
    - hosts:
      - www.my-app.com.br
      secretName: cloudflare-secret
  rules:
    - host: www.my-app.com.br
      http:
        paths:
        - path: /
          backend:
            serviceName: web-svc
            servicePort: 80

Be careful with the annotations above that hooks everything up. It binds the NodePort service we created for the web pods with the nginx ingress controller and adds SSL termination through that spec:tls:secretName. How do you create that? First, you must purchase an SSL certificate - again, using CloudFlare as the example.

When you finish buying, the provider should give you the secret files to download (keep them safe! a public dropbox folder is not safe!). Then you have to add it to the infrastructure like this:

kubectl create secret tls cloudflare-secret \
--key ~/downloads/private.pem \
--cert ~/downloads/fullchain.pem

Now that we edited a whole bunch of files, we can deploy the entire load balancer stack:

kubectl create -f deploy/default-web.yml
kubectl create -f deploy/default-web-svc.yml
kubectl create -f deploy/nginx.yml
kubectl create -f deploy/nginx-svc.yml
kubectl create -f deploy/ingress.yml

This NGINX Ingress configuration is based off of Zihao Zhang’s blog post. There are also examples in the kubernetes incubator repository. You may want to check it out as well.

If you did everything right so far, https://www.my-app-com.br should load your web application. You may want to check for Time to First-Byte (TTFB). You can do it going through CloudFlare like this:

curl -vso /dev/null -w "Connect: %{time_connect} \n TTFB: %{time_starttransfer} \n Total time: %{time_total} \n" https://www.my-app.com.br

Or, if you’re having slow TTFB you can bypass CloudFlare doing this:

curl --resolve www.my-app.com.br:443:111.111.111.111 https://www.my-app.com.br -svo /dev/null -k -w "Connect: %{time_connect} \n TTFB: %{time_starttransfer} \n Total time: %{time_total} \n"

TTFB should be in the neighborhood of 1 second or less. Anything far and above could mean a problem in your application. You must check your node instance machine types, the number of workers loaded per pod, the CloudSQL proxy version, the NGINX controller version and so on. This is a trial and error procedure as far as I know. Sign up to services such as Loader or even Web Page Test for insight.

Rolling Updates

Now, that everything is up and running, how do we accomplish the Rolling Update I mentioned in the beginning? First you git push to the Container Registry repository and wait for the Docker image to build.

Remember that I said to let a trigger tag the image with a random version number? Let’s use it (you can see it from the Build History list in the Container Registry, from the Google Cloud console):

kubectl set image deployment web my-app=gcr.io/my-project/my-app:1238471234g123f534f543541gf5 --record

You must use the same name and image that is declared in the deploy/web.yml from above. This will start rolling out the update by adding a new pod, then terminating one pod and so on and so forth until all of them are updated, without downtime for your users.

Rolling updates must be carried out carefully. For example, if your new deployment requires a database migration, then you must add a maintenance window (meaning: do it when there is little to no traffic, such as in the middle of the night). So you can run the migrate command like this:

kubectl get pods # to get a pod name

kubectl exec -it my-web-12324-121312 /app/bin/rails db:migrate

# you can also bash to a pod like this, but remember that this is an ephemeral container, so files you edit and write there disappear on the next restart:

kubectl exec -it my-web-12324-121312 bash

To redeploy everything without resorting to rolling update you must do this:

kubectl delete -f deploy/web.yml && kubectl apply -f deploy/web.yml

You will find a more thorough explanation in Ta-Ching’s blog post.

Bonus: Auto Snapshots

One item I had in my “I wanted/needed” list, in the beginning, is the ability to have persistent mountable storage with automatic backups/snapshots. Google Cloud provides half of that for the time being. You can create persistent disks to mount in your pods but it doesn’t have a feature to automatically back it up. At least it does have APIs to manually snapshot it.

For this example, let’s create a new SSD disk and format it first:

gcloud compute disks create --size 500GB my-data --type pd-ssd

gcloud compute instances list

The last command is so we can copy the name of a node instance. Let’s say it’s gke-my-web-app-default-pool-123-123. We will attach the my-data disk to it:

gcloud compute instances attach-disk gke-my-web-app-default-pool-123-123 --disk my-data --device-name my-data

gcloud compute ssh gke-my-web-app-default-pool-123-123

The last command ssh’s in the instance. We can list the attached disks with sudo lsblk and you will see the 500GB disk, probably, as /dev/sdb, but make sure that’s correct because we will format it!

sudo mkfs.ext4 -m 0 -F -E lazy_itable_init=0,lazy_journal_init=0,discard /dev/sdb

Now we can exit from the SSH session and detach the disk:

gcloud compute instances detach-disk gke-my-web-app-default-pool-123-123 --disk my-data

You can mount this disk in your pods by adding the following to your deployment yaml:

spec:
  containers:
    - image: ...
      name: my-app
      volumeMounts:
        - name: my-data
          mountPath: /data
          # readOnly: true
   # ...
   volumes:
     - name: my-data
       gcePersistentDisk:
         pdName: my-data
         fsType: ext4

Now, let’s create a CronJob deployment file as deploy/auto-snapshot.yml:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: auto-snapshot
spec:
  schedule: "0 4 * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
          - name: auto-snapshot
            image: grugnog/google-cloud-auto-snapshot
            command: ["/opt/entrypoint.sh"]
            env:
            - name: "GOOGLE_CLOUD_PROJECT"
              value: "my-project"
            - name: "GOOGLE_APPLICATION_CREDENTIALS"
              value: "/credential/credential.json"
            volumeMounts:
              - mountPath: /credential
                name: editor-credential
          volumes:
            - name: editor-credential
              secret:
                secretName: editor-credential

As we already did before, you will need to create another Service Account with editor permissions in the “IAM & admin” section of the Google Cloud console, then download the JSON credential, and finally upload it like this:

kubectl create secret generic editor-credential \
--from-file=credential.json=/home/myself/download/my-project-1212121.json

Also notice that, as a normal cron job, there is a schedule parameter that you might want to change. In the example, “0 4 * * *” means that it will run the snapshot every day at 4 AM.

Check out the original repository of this solution for more details.

And this should be it for now!

As I said, in the beginning, this is not a complete procedure, just highlights of some of the important parts. If you’re new to Kubernetes you just read about Deployment, Service, Ingress, but you have ReplicaSet, DaemonSet, and much more to play with.

I think this is also already too long to add a multi-region High Availability setup explanation, so let’s leave it at that.

Any corrections or suggestions are more than welcome as I am still in the learning process, and there are a ton of things that I still don’t know myself.

Anyway, the website itself is super simple. A single page site. I chose to use Rails 5.1 as the site structure because it takes care of all the stuff I’d have to add manually in any other framework. Especially now that it brings native Yarn and Webpack support it’s a breeze to use by any competent front-end developer.

But even with the many built-in niceties, a full production setup still requires extra steps that most beginners will not know. So I decided to compile a small checklist of things you must take care of before deploying to production. It’s not an extensive and complete list for all use cases, but this should be enough for most small cases.

• Setting up the project
• Adding a CDN and configuring CORS support (to be able to load fonts for example)
• Adding Memcached

Setting up the project

One thing that most people forget, even experienced developers, is that the rails new command used to bootstrap the initial project structure accepts many option flags. Instead of having to manually tweak files later, you can start like this:

rails new your_project \
--database=postgresql \
--webpack=react \
--skip-action-cable \
--skip-coffee
--skip-turbolinks

If you’re building with React or another full-featured javascript framework, you will probably want to skip Turbolinks. Otherwise, if it’s a simple site, do use Turbolinks.

Start using PostgreSQL from the get-go, don’t use Sqlite3.

Skip Action Cable. Prefer a real solution such as Pusher.com. If you really need something done in-house, consider something like my solution with Elixir, the Ex Pusher Lite. Take this recommendation with a grain of salt, of course, for small things Action Cable is more than enough. I may write another post just elaborating on this point if people indicate they want to in the comments section.

Anyway, I digress. Make sure you have 2 boot files, first the canonical Procfile to be used by Heroku in production:

web: bin/rails server -p $PORT -b 0.0.0.0

Second, a Procfile.dev to be used only in your development environment:

web: ./bin/rails server
webpacker: ./bin/webpack-dev-server

This is how you fire up the webpack server that will compile your assets in real-time during development. You need to also remember to run these two dependency commands now:

yarn install
bundle install

Install javascript dependencies with yarn add [package] and that’s it! In production you don’t use the webpack server (which is why we don’t add it to the production Procfile), instead Heroku automatically detects the webpacker gem then it installs the nodejs buildpack, runs yarn install for you and when rails assets:precompile runs it will also execute yarn run which will pre-compile all the assets (javascript, stylesheets, images) with the proper fingerprinting for cache busting and everything else we are used to in the normal Rails Asset Pipeline.

So, for Heroku, you basically don’t have to do anything. And in a custom deployment server you just need to remember to run the assets:precompile task and have it do everything for you.

Adding a CDN and configuring CORS

This is really super easy. There is no reason why anyone wouldn’t add a CDN to any website. Please just do it.

Open your AWS Management Console and open CloudFront. From there click on “Create Distribution” and just follow the defaults in the Wizard. The requirement is that you MUST know the domain and sub-domain you want to point to (for example “www.theconf.club”) in the “Origin Domain Name” field.

The only customization you MUST do is to change the “Forward Headers” option to “Whitelist” in the “Default Cache Behavior Settings”. Then you need to add “Origin”, “Access-Control-Request-Headers”, and “Access-Control-Request-Method” to the Whitelist. And that’s it, now your distribution is CORS enabled.

It will take some time to create (it has to configure many data centers around the world), but you will end up with a URL representing your distribution, something like doz7rtw2u3wg4.cloudfront.net. I recommend you add it as a Heroku environment variable like this:

heroku config:set CDN_URL=doz7rtw2u3wg4.cloudfront.net

Now, edit your config/environments/production.rb and add the following:

config.action_controller.asset_host = ENV['CDN_URL'] if ENV['CDN_URL']

To actually use the CDN you must declare every asset you use across your view templates using Rails View Helpers such as image_tag, asset_path, javascript_pack_tag, stylesheet_pack_tag, stylesheet_link_tag, etc. The Rails bootstrap already creates layout templates with such helpers, you just need to follow them.

When webpack runs, it will generate all static, optimized and pre-compiled assets in public/packs with a manifest file declaring the full URL pointing to the CDN. For example, if I fetch the /app/public/packs/manifest.json from the Heroku dyno directly, I will get something like this:

{
  "Roboto-Bold.woff": "//doz7rtw2u3wg4.cloudfront.net/packs/Roboto-Bold-eed9aab5449cc9c8430d7d258108f602.woff",
  "Roboto-Bold.woff2": "//doz7rtw2u3wg4.cloudfront.net/packs/Roboto-Bold-c0f1e4a4fdfb8048c72e86aadb2a247d.woff2",
  "Roboto-Light.woff": "//doz7rtw2u3wg4.cloudfront.net/packs/Roboto-Light-ea36cd9a0e9eee97012a67b8a4570d7b.woff",
  "Roboto-Light.woff2": "//doz7rtw2u3wg4.cloudfront.net/packs/Roboto-Light-3c37aa69cd77e6a53a067170fa8fe2e9.woff2",
  "Roboto-Medium.woff": "//doz7rtw2u3wg4.cloudfront.net/packs/Roboto-Medium-cf4d60bc0b1d4b2314085919a00e1724.woff",
  "Roboto-Medium.woff2": "//doz7rtw2u3wg4.cloudfront.net/packs/Roboto-Medium-1561b424aaef2f704bbd89155b3ce514.woff2",
  "Roboto-Regular.woff": "//doz7rtw2u3wg4.cloudfront.net/packs/Roboto-Regular-3cf6adf61054c328b1b0ddcd8f9ce24d.woff",
  "Roboto-Regular.woff2": "//doz7rtw2u3wg4.cloudfront.net/packs/Roboto-Regular-5136cbe62a63604402f2fedb97f246f8.woff2",
  "Roboto-Thin.woff": "//doz7rtw2u3wg4.cloudfront.net/packs/Roboto-Thin-44b78f142603eb69f593ed4002ed7a4a.woff",
  "Roboto-Thin.woff2": "//doz7rtw2u3wg4.cloudfront.net/packs/Roboto-Thin-1f35e6a11d27d2e10d28946d42332dc5.woff2",
  "application.css": "//doz7rtw2u3wg4.cloudfront.net/packs/application-09b53ce9ca3dd595ee99.css",
  "application.css.map": "//doz7rtw2u3wg4.cloudfront.net/packs/application-09b53ce9ca3dd595ee99.css.map",
  "application.js": "//doz7rtw2u3wg4.cloudfront.net/packs/application-799300612ff6d6595198.js",
  "application.js.map": "//doz7rtw2u3wg4.cloudfront.net/packs/application-799300612ff6d6595198.js.map",
  "home_page.js": "//doz7rtw2u3wg4.cloudfront.net/packs/home_page-ff3b49407a1d01592ad5.js",
  "home_page.js.map": "//doz7rtw2u3wg4.cloudfront.net/packs/home_page-ff3b49407a1d01592ad5.js.map"
}

So, if for some reason I had to create a new CDN distribution, you have to remember to update the CDN_URL variable on Heroku and redeploy your app so it regenerates the assets and this manifest file. It will just use those URLs when rendering the final HTML.

When a user opens your site, it will receive the HTML with URLs pointing to the CDN. The very first time it won’t have any cached assets, so it will ask your site for them. Your site must return the assets WITH the correct CORS headers so the CDN caches them with the headers and forwards those headers to the browser. The browser needs those headers because it will open from domain www.theconf.club, for example, but fonts are loading from doz7rtw2u3wg4.cloudfront.net, so it would raise a security warning and not load the fonts because they are in different domains. Hence the CORS headers the font provides, indicating that they are safe to load.

For your Rails app to return the proper headers, you need to add the rack-cors gem to your Gemfile. Then you must add a new configuration file at config/initializers/rack-cors.rb with:

if defined? Rack::Cors
  Rails.configuration.middleware.insert_before 0, Rack::Cors do
    allow do
      origins %w[
        https://theconf.club
        http://theconf.club
        https://www.theconf.club
        http://www.theconf.club
        https://theconf.herokuapp.com
        http://theconf.herokuapp.com
      ]
      resource '/assets/*'
    end
  end
end

When you deploy, you know it’s working correctly when you Curl an asset and it returns the Access-* headers like this:

$ curl -I -s -X GET -H "Origin: www.theconf.club" http://www.theconf.club/packs/Roboto-Regular-5136cbe62a63604402f2fedb97f246f8.woff2
HTTP/1.1 200 OK
Server: Cowboy
Date: Wed, 28 Jun 2017 17:44:41 GMT
Connection: keep-alive
Access-Control-Allow-Origin: www.theconf.club
Access-Control-Allow-Methods: GET, POST, OPTIONS
Access-Control-Expose-Headers: 
Access-Control-Max-Age: 1728000
Access-Control-Allow-Credentials: true
Last-Modified: Wed, 28 Jun 2017 17:06:03 GMT
Content-Type: application/font-woff2
Cache-Control: public, max-age=2592000
Vary: Origin
Content-Length: 64832
Via: 1.1 vegur

And if everything above is already in place, you should be able to see the headers being forwarded through the CDN, like this:

$ curl -I -s -X GET -H "Origin: www.theconf.club" http://doz7rtw2u3wg4.cloudfront.net/packs/Roboto-Regular-5136cbe62a63604402f2fedb97f246f8.woff2
HTTP/1.1 200 OK
Content-Type: application/font-woff2
Content-Length: 64832
Connection: keep-alive
Server: Cowboy
Date: Wed, 28 Jun 2017 17:45:18 GMT
Access-Control-Allow-Origin: www.theconf.club
Access-Control-Allow-Methods: GET, POST, OPTIONS
Access-Control-Expose-Headers: 
Access-Control-Max-Age: 1728000
Access-Control-Allow-Credentials: true
Last-Modified: Wed, 28 Jun 2017 17:06:03 GMT
Cache-Control: public, max-age=2592000
Via: 1.1 vegur, 1.1 86e9abdb4c15b9d3a542f9b93245e87e.cloudfront.net (CloudFront)
Vary: Origin
X-Cache: Miss from cloudfront
X-Amz-Cf-Id: tVkZ41RRr66iBT6atWTO_oeTY_jG0zCBFuXU8bKyClZDQ8kl-hDegA==

A CDN is the secret sauce that allows any content-based website to scale way beyond what your server can provide. It’s a huge cost-saving and it also makes for a much smoother user experience for your users.

One last caveat. The many Rails View Helpers such as image_tag allow you to add the image file name without an extension in development and they will correctly find the image. But on the server it will fail to render the template this way. As a rule of thumb ALWAYS fill in the entire filename and extension, for example image_tag("logo.png") instead of just image_tag("logo").

You can see how this fails if you open a console in Heroku and check out how it fails to derive the full image URL:

$ heroku run rails c                                                                                                                  
Running rails c on ⬢ theconf... up, run.8271 (Hobby)
Loading production environment (Rails 5.1.2)

irb(main):001:0> ActionController::Base.helpers.asset_path("icon-goals")
Sprockets::Rails::Helper::AssetNotFound: The asset "icon-goals" is not present in the asset pipeline.
    from (irb):1

irb(main):002:0> ActionController::Base.helpers.asset_path("icon-goals.png")
=> "//d134ipy19a646x.cloudfront.net/assets/icon-goals-b969b3b7325d33ad85a88dbb5b894832909ed738eea9964b9cf535646b93674b.png"

Adding Memcached

Talking about caching is always a complicated thing, which is why many people avoid even trying. Even though you can go really crazy with super granular configurations such as using Russian Doll Caching, just adding cache in a few spots can greatly benefit you. And it’s super easy to boot.

The first thing to do is add Memcachier to your Heroku application. It has a free-tier and for most small apps it should be enough.

Configuration is also trivial. Start by adding the following gems to your Gemfile:

group :production do
  gem 'rack-cors', require: 'rack/cors'
  gem 'rack-cache'
  gem 'dalli'
  gem 'kgio'
  gem 'memcachier'
end

Now you must edit your config/environments/production.rb like this:

config.cache_store = :dalli_store

client = Dalli::Client.new((ENV["MEMCACHIER_SERVERS"] || "").split(","),
                           :username => ENV["MEMCACHIER_USERNAME"],
                           :password => ENV["MEMCACHIER_PASSWORD"],
                           :failover => true,
                           :socket_timeout => 1.5,
                           :socket_failure_delay => 0.2,
                           :value_max_bytes => 10485760)
config.action_dispatch.rack_cache = {
  :metastore    => client,
  :entitystore  => client
}

Now, let’s say you have a block in your template that requires a bunch of records from your database. But you know that those records barely change. What can you do? One alternative is to cache the ActiveRecord query entirely like this:

class HomePageController < ApplicationController
  def index
    @selected_proposals = Rails.cache.fetch('selected_proposals', expires_in: 1.day) do
      Proposal.includes(:authors).where(selected: true).to_a
    end
  end
end

The #to_a is necessary because ActiveRecord queries are lazy. The #to_a forces it to fetch and it will be cached. Next time, it will not touch the database for an entire day!

I could also have added a cache do ... end block in the template itself. There are many options. Point is that it’s not as difficult as most people would think.

What makes caching difficult is adding expiration logic. And the rule of thumb is: never try to manually expire any cache. Just change the lookup key for something else and let the old data just die (memcached will take care of getting rid of unused old data).

You really want to read the Rails Guides on Caching. It really is not as difficult as you think and you can cache only the few snippets you know are performance-sensitive and leave the other parts that are highly dynamic and you’re unsure how to properly cache.

But as it’s super cheap, just use it right now.

Adding SSL support

If you have any security-sensitive transactions going on (ex. purchases) you want to use SSL. Again, many people avoid it because it’s usually difficult to even understand how to properly get a certificate.

Let’s Encrypt made the process super trivial. Kudos to them! And even better: it’s free! So, you don’t have any excuses to not have SSL.

And over Heroku, it’s even easier!

ACM (Automated Certificate Management) is enabled by default for all applications created after March 21, 2017 that are running on Hobby or Professional dynos. The following steps are for applications currently running on Free dynos, and for applications created before that date.

On the Free Tier Dynos, this is what you do:

heroku certs:auto:enable

Check status with heroku certs:auto.

Done, there is no step 2.

We used to have to use the complicated gem letsencrypt-rails-heroku, but now it’s just too easy.

Conclusion

I believe this covers the very basics of stuff you should be doing before deploying your small app to Heroku. For larger apps there are many more concerns that are beyond the scope of this post.

If you remember about more tips and tricks, please share in the comments section below.

For starters, Erlang does not need multiple processes like Ruby (which is limited to one connection per process, or per thread if you’re using a threaded-server like Puma). One single Erlang process will take over the entire machine, if you need to. Internally it will keep one real thread per machine-core. And each thread will have its own Scheduler to manage as many micro-processes as you need. You can read all about it in my post titled “Yocto Services”.

Moreover, Erlang has built-in capabilities to form a cluster, where each Erlang instance acts as a peer-to-peer Node, without the need for a centralized coordinator. You can read all about it in my post about Nodes. The power of Erlang is in how “easy” it is to form reliable distributed systems.

You can fire up many Phoenix instances and from one of the instances, it can broadcast messages to Users subscribed in Channels even if their sockets are connected to different instances. It’s seamless and you don’t need to do anything special in your code. Phoenix, Elixir and Erlang are doing all the heavy lifting for you behind the scenes.

No Heroku for You :-(

Because you want to take advantage of this scalability and high availability feature for distributed systems (in the small example of a real-time chat system) you will need to have more control over your infrastructure. This requirement rules out the majority of Platform as a Service (PaaS) offerings out there, such as Heroku. Heroku’s model revolves around single, volatile processes in isolated containers. Those jailed processes (dynos) are not aware of other processes or the internal networking, so you can’t fire up Dynos and have them form a cluster because they won’t be able to find each other.

If you already know how to configure Linux related stuff: Postgresql, HAproxy, etc, go ahead directly to the Phoenix-specific section.

IaaS (DigitalOcean) to the rescue!

You want long lived processes in network reachable servers (either through private networking, VPN, or plain simple - insecure! - public networks).

In this example I want to walk you through a very simple deployment using DigitalOcean (you can choose any IaaS, such as AWS, Google Cloud, Azure or whatever you feel more comfortable with).

I have created 4 droplets (all using the smallest size of 512Mb of RAM):

• 1 Postgresql database (single point of failure: it’s not the focus of this article to build a highly available, replicated database setup);
• 1 HAProxy server (single point of failure: again, it’s not the focus to create a highly available load balancing scheme);
• 2 Phoenix servers - one in the NYC datacenter and another in the London datacenter, to demonstrate how easy it is for Erlang to form clusters even with geographically separated boxes.

Basic Ubuntu 16.04 configuration

• Goals: configure locale, assure unattended updates are up, upgrade packages, install and configure Elixir and Node.
• You should also do: change SSH to another port and install [fail2ban](https://www.digitalocean.com/community/tutorials/how-to-protect-ssh-with-fail2ban-on-ubuntu-14-04, disallow login through password).

You will want to read my post about configuring Ubuntu 16.04. To summarize, start by configuring proper UTF-8:

sudo locale-gen "en_US.UTF-8"
sudo dpkg-reconfigure locales
sudo update-locale LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8
echo 'LC_ALL=en_US.UTF-8' | sudo tee -a /etc/environment
echo 'LANG=en_US.UTF-8' | sudo tee -a /etc/environment

Make sure you add a proper user into the sudo group and from now on do not use the root user. I will create a user named pusher and I will explain in another post why. You should create a username that suits your application.

adduser pusher
usermod -aG sudo pusher

Now log out and log in again through this user. ssh pusher@server-ip-address. If you’re on a Mac copy the public key of your SSH certificate like this:

ssh-copy-id -i ~/.ssh/id_ed25519.pub pusher@server-ip-address

It creates the .ssh/authorized_keys if it doesn’t exist, sets the correct permission bits and appends your public key. You can do it manually, of course.

DigitalOcean’s droplets start without a swap file and I’d recommend adding one, especially if you want to start with the smaller boxes with less than 1GB of RAM:

sudo fallocate -l 2G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
sudo cp /etc/fstab /etc/fstab.bak
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab
sudo sysctl vm.swappiness=10
sudo sysctl vm.vfs_cache_pressure=50
echo 'vm.swappiness=10' | sudo tee -a /etc/sysctl.conf
echo 'vm.vfs_cache_pressure=50' | sudo tee -a /etc/sysctl.conf

Make sure you have unattended upgrades configured. You will want at least to have security updates automatically installed when available.

sudo apt install unattended-upgrades

Now, let’s install Elixir and Node (Phoenix needs Node.js):

wget https://packages.erlang-solutions.com/erlang-solutions_1.0_all.deb && sudo dpkg -i erlang-solutions_1.0_all.deb
curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash -
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install build-essential nodejs esl-erlang elixir erlang-eunit erlang-base-hipe
sudo npm install -g brunch
mix local.hex
mix archive.install https://github.com/phoenixframework/archives/raw/master/phoenix_new.ez # this is optional, install if you want to manually test phoenix in your box

Now you have an Elixir capable machine ready. Create a image snapshot over DigitalOcean, move it to the regions where you want to create your other droplets and use this image to create as many droplets as you need.

For this example, I created a second droplet in the London region, a third droplet for postgresql in the NYC1 region and a fourth droplet in the NYC3 region for HAProxy.

I will refer to their public IP addresses as “nyc-ip-address”, “lon-ip-address”, “pg-ip-address”, and “ha-ip-address”.

Basic PostgreSQL configuration

• Goal: basic configuration of Postgresql to allow the Phoenix servers to connect.
• To do: create a secondary role just to connect to the application database and another superuser role to create the database and migrate the schema. Also lock down the machine and configure SSH tunnels or another secure way, at least private networking, rather than allowing plain 5432 TCP port connections from the public Internet.

Now you can connect to ssh pusher@pg-ip-address and follow this:

sudo apt-get install postgresql postgresql-contrib

You should create a new role with the same name of the user you added above (“pusher” in our example):

$ sudo -u postgres createuser --interactive

Enter name of role to add: pusher
Shall the new role be a superuser? (y/n) y

$ sudo -u postgres createdb pusher

Postgresql expects to find a database with the same name as the role and the role should have the same name as the Linux user. Now you can use psql to set a password for this new role:

$ sudo -u postgres psql
\password pusher

Register a secure password, take note and let’s move on.

Postgresql comes locked down to external connections. One way to connect from the outside is to configure your servers to create an SSH tunnel to the database server and keep external TCP connections through port 5432 disavowed.

But for this example, we will just allow connections from the public Internet to the 5432 TCP port. Warning: this is VERY insecure!

Edit the /etc/postgresql/9.5/main/postgresql.conf and find the listen_addresses configuration line and allow it:

listen_addresses = '*'    # what IP address(es) to listen on;

This should bind the server to the TCP port. Now edit /etc/postgresql/9.5/main/pg_hba.conf and edit the end to look like this:

# IPv4 local connections:
host    all             all             127.0.0.1/32            trust
host    all             all             your-local-machine-ip-address/24        trust
host    all             all             nyc-ip-address/24       trust
host    all             all             lon-ip-address/24       trust
# IPv6 local connections:
host    all             all             ::1/128                 trust

Save the configuration file and restart the server:

sudo service postgresql restart

See what I did there? I only allowed connections coming from the public IPs of the Phoenix servers. This does not make the server secure, just a little bit less vulnerable. If you’re behind a DHCP/NAT based network, just Google for “what’s my IP” to see your public facing IP address - which is probably shared by many other users, remember you’re allowing connections from an insecure IP to your database server! Once you make initial tests, create your new schema, then you should remove that your-local-machine-ip-address/24 line from the configuration.

From your Phoenix application, you can edit your local config/prod.secret.exs file to look like this:

# Configure your database
config :your_app_name, ExPusherLite.Repo,
  adapter: Ecto.Adapters.Postgres,
  username: "pusher",
  password: "your-super-secure-pg-password",
  database: "your-app-database-name",
  hostname: "pg-ip-address",
  pool_size: 20

Replace the information for your server and database and now you can test it out like this:

MIX_ENV=prod iex -S mix phoenix.server

If you see a :econnrefused message from postgrex, then you’re in trouble. Re-check your configuration, restart the server and try again. If everything connects, you can run MIX_ENV=prod mix do ecto.create, ecto.migrate to prepare your database.

Finally, you will want to lock down the rest of your server with UFW, at the very least. UFW should come pre-installed in Ubuntu 16, so you can just do:

ufw allow 5432
ufw allow ssh
ufw enable

That’s it. And again, this does not make your server secure, it just makes it less insecure. There is a huge difference!

And by the way, if you’re a Docker fan:

DO NOT INSTALL A DATABASE INSIDE A DOCKER CONTAINER!

You have been warned!

Basic HAProxy configuration

• Goals: Provide a simple solution to load balance between our 2 Phoenix servers.
• To do: There is something wrong with session checking or something like that as sometimes I have to refresh my browser so I am not sent back to the login form in my application. Phoenix uses cookie-based sessions so I don’t think it is missing sessions.

Now let’s ssh pusher@ha-ip-address. This one is easy, let’s just install HAProxy:

sudo apt-get install haproxy

Edit /etc/haproxy/haproxy.cfg:

...
listen your-app-name
  bind 0.0.0.0:80
  mode http
  stats enable
  stats uri /haproxy?stats
  stats realm Strictly\ Private
  stats auth admin:some-secure-password-for-admin
  option forwardfor
  option http-server-close
  option httpclose
  balance roundrobin
  cookie SERVERID insert indirect nocache
  server us your_us_server_IP:8080 check cookie us1
  server uk your_uk_server_IP:8080 check cookie uk1

You can avoid the stats lines if you have other means of monitoring, otherwise set a secure password for the admin user. One very important part is the option http-server-close as explained in this other blog post, otherwise you may have trouble with Websockets.

For some reason I am having some trouble with my application after I login and it sets the session, sometimes I have to refresh to be sent to the correct page, not sure why yet and I believe it’s something in the HAProxy configuration. If anyone knows what it is, let me know in the comments section below. For now, I am just relying on Sticky sessions (server affinity) by making HAProxy write back a cookie with the server.

Now you can restart the server and enable UFW as well:

sudo service haproxy restart
sudo ufw allow http
sudo ufw allow https
sudo ufw allow ssh
sudo ufw enable

You can easily add SSL support to your application by configuring HAProxy (and not the Phoenix nodes). The DigitalOcean documentation is comprehensive, so just follow it. At the end, my haproxy.cfg looks like this:

global
   log /dev/log    local0
   log /dev/log    local1 notice
   chroot /var/lib/haproxy
   stats socket /run/haproxy/admin.sock mode 660 level admin
   stats timeout 30s
   user haproxy
   group haproxy
   daemon
   maxconn 2048
   tune.ssl.default-dh-param 2048

defaults
   log global
   mode http
   option httplog
   option dontlognull
   option redispatch
   option forwardfor
   option http-server-close
   timeout connect 5000
   timeout client  50000
   timeout server  50000

frontend www-http
   bind your_ha_proxy_IP:80
   reqadd X-Forwarded-Proto:\ http
   default_backend www-backend

frontend www-https
   bind your_ha_proxy_IP:443 ssl crt /etc/haproxy/certs/your_domain.pem
   reqadd X-Forwarded-Proto:\ https
   acl letsencrypt-acl path_beg /.well-known/acme-challenge/
   use_backend letsencrypt-backend if letsencrypt-acl
   default_backend www-backend

backend www-backend
   redirect scheme https if !{ ssl_fc }
   # setting session stickiness
   cookie SERVERID insert indirect nocache
   server us your_us_server_IP:8080 check cookie us1
   server uk your_uk_server_IP:8080 check cookie uk1

backend letsencrypt-backend
   server letsencrypt 127.0.0.1:54321

Finally, I will assume you have a DNS server/service somewhere where you can register the IP of this HAproxy server as an A record so you can access it by a full name such as “your-app-name.mydomain.com”.

Basic Phoenix Configuration

• Goal: configure the Phoenix app to be deployable. Configure the servers to have the necessary configuration files.
• To do: find out a way to cut down the super slow deployment times.

Finally, we have almost everything in place.

I will assume that you have a working Phoenix application already in place, otherwise create one from the many tutorials out there.

I have assembled this information from posts such as this very helpful one from Pivotal about an AWS-based deployment. In summary you must do a number of changes to your configuration.

When you’re developing your application, you will notice that whenever you run it, it delta-compiles what changed. The binary bits are in the _build/dev or _build/test in the form of .beam binaries (similar to what .class are for Java).

Different from Ruby or Python or PHP, you are not deploying source-code to production servers. It’s more akin to Java, where you must have everything compiled into binary bits and packaged into what’s called a release. It’s like a “.war” or “.ear” if you’re from Java.

To create this package people usually use “exrm”, but it’s being replaced by “distillery”, so we will use it.

Then, if you’re from Ruby you’re familiar with Capistrano. Or if you’re from Python, you know Capistrano’s clone, Fabric. Elixir has a similar tool (much simpler at this point), called “edeliver”. It’s your basic SSH automation tool.

You add them to mix.exs just like any other dependency:

...
def application do
  [mod: {ExPusherLite, []},
   applications: [..., :edeliver]]
end

defp deps do
  [...,
   {:edeliver, "~> 1.4.0"},
   {:distillery, "~> 1.0"}]
end
...

From the Pivotal blog post, the important thing to not forget is to edit this part in the config/prod.exs file:

http: [port: 8080],
url: [host: "your-app-name.yourdomain.com", port: 80],
...
config :phoenix, :serve_endpoints, true

You MUST hardcode the default PORT of the Phoenix web server and the allowed domain (remember the domain name you associated to your HAProxy server above? That one). And you MUST uncomment the :serve_endpoints, true line!

For edeliver to work you have to create a .deliver/config file like this:

USING_DISTILLERY=true

# change this to your app name:
APP="your-app-name"

# change this to your own servers IP and add as many as you want
US="nyc-ip-address"
UK="lon-ip-address"

# the user you created in your Ubuntu machines above
USER="pusher"

# which server do you want to build the first release?
BUILD_HOST=$US
BUILD_USER=$USER
BUILD_AT="/tmp/edeliver/$APP/builds"

# list the production servers declared above:
PRODUCTION_HOSTS="$US $UK"
PRODUCTION_USER=$USER
DELIVER_TO="/home/$USER"

# do not change here

LINK_VM_ARGS="/home/$USER/vm.args"

# For *Phoenix* projects, symlink prod.secret.exs to our tmp source
pre_erlang_get_and_update_deps() {
  local _prod_secret_path="/home/$USER/prod.secret.exs"
  if [ "$TARGET_MIX_ENV" = "prod" ]; then
    __sync_remote "
      ln -sfn '$_prod_secret_path' '$BUILD_AT/config/prod.secret.exs'

      cd '$BUILD_AT'

      mkdir -p priv/static

      mix deps.get

      npm install

      brunch build --production

      APP='$APP' MIX_ENV='$TARGET_MIX_ENV' $MIX_CMD phoenix.digest $SILENCE
    "
  fi
}

Remember the information we’ve been gathering since the beginning of this long recipe? These are the options you MUST change to your own. Just follow the comments in the file content above and add it to your git repository. By the way, your project is in a proper GIT repository, RIGHT??

If you like to use passphrase protected SSH private keys, then it’s going to be a huge pain to deploy because for each command, edeliver will issue an SSH command that will keep asking for you passphrase, a dozen times through everything. You’ve been warned! If you still don’t mind that, and you’re in a Mac you will have an extra trouble because the Terminal will not be able to create a prompt for you to input your passphrase. You must create an /usr/local/bin/ssh-askpass script:

#!/bin/bash
# Script: ssh-askpass
# Author: Mark Carver
# Created: 2011-09-14
# Licensed under GPL 3.0

# A ssh-askpass command for Mac OS X
# Based from author: Joseph Mocker, Sun Microsystems
# http://blogs.oracle.com/mock/entry/and_now_chicken_of_the

# To use this script:
#   Install this script running INSTALL as root
#
# If you plan on manually installing this script, please note that you will have
# to set the following variable for SSH to recognize where the script is located:
#   export SSH_ASKPASS="/path/to/ssh-askpass"

TITLE="${SSH_ASKPASS_TITLE:-SSH}";
TEXT="$(whoami)'s password:";
IFS=$(printf "\n");
CODE=("on GetCurrentApp()");
CODE=(${CODE[*]} "tell application \"System Events\" to get short name of first process whose frontmost is true");
CODE=(${CODE[*]} "end GetCurrentApp");
CODE=(${CODE[*]} "tell application GetCurrentApp()");
CODE=(${CODE[*]} "activate");
CODE=(${CODE[*]} "display dialog \"${@:-$TEXT}\" default answer \"\" with title \"${TITLE}\" with icon caution with hidden answer");
CODE=(${CODE[*]} "text returned of result");
CODE=(${CODE[*]} "end tell");
SCRIPT="/usr/bin/osascript"
for LINE in ${CODE[*]}; do
      SCRIPT="${SCRIPT} -e $(printf "%q" "${LINE}")";
done;
eval "${SCRIPT}";

Now do this:

sudo chmod +x /usr/local/bin/ssh-askpass
sudo ln -s /usr/local/bin/ssh-askpass /usr/X11R6/bin/ssh-askpass

Remember, Macs only. And now every time you try to deploy you will receive a number of graphical prompt windows asking for the SSH private key passphrase. It’s freaking annoying! And you must have XQuartz installed, by the way.

Now you must manually create 3 files in all Phoenix servers. Start with the your-app-name/vm.args:

-name us@nyc-ip-address
-setcookie @bCd&fG
-kernel inet_dist_listen_min 9100 inet_dist_listen_max 9155
-config /home/pusher/your-app-name.config
-smp auto

The /home/pusher/your-app-name is the directory where the release will be uncompressed after edeliver deploys the release tarball.

You must create this file in all Phoenix machines, by the way, changing the -name bit for the same name you declared in the .deliver/config file. The -setcookie should be any name, as long as it’s the same in all servers.

See the -config /home/pusher/your-app-name.config? Create that file with the following:

[{kernel,
  [
    {sync_nodes_optional, ['uk@lon-ip-address']},
    {sync_nodes_timeout, 30000}
  ]}
].

This is an Erlang source-code. On the NYC machine you must declare the London name, and vice-versa. If you have several machines, all of them but the one you’re in right now. Get it?

Finally, for the Phoenix app itself, you always have a config/prod.secret.exs that should never be git added to the repository, remember? This is where you put the Postgresql server information and random secret key to sign the session cookies:

use Mix.Config

config :your_app_name, YourAppName.Endpoint,
  secret_key_base: "..."

# Configure your database
config :your_app_name, YourAppName.Repo,
  adapter: Ecto.Adapters.Postgres,
  username: "pusher",
  password: "your-super-secure-pg-password",
  database: "your-app-database-name",
  hostname: "pg-ip-address",
  pool_size: 20

# if you have Guardian, for example:
config :guardian, Guardian,
  secret_key: "..."

How do you create a new random secret key? From your development machine just run: mix phoenix.gen.secret and copy the generated string into the file above.

So now you must have those 3 files in each Phoenix server, in the /home/pusher home folder:

~/your-app-name/vm.args
~/prod.secret.exs
~/your-app-name.config

As per distillery documentation, you have to set an environment variable to let it know about the vm.args file, and this is SUPER important otherwise it will generate a default one that doesn’t set the proper name and cookie, so the nodes will not find each other after booting up.

Using sudo, edit the /etc/environment and add the line:

RELEASE_CONFIG_DIR=/home/pusher/your-app-name
VMARGS_PATH=/home/pusher/your-app-name/vm.args

You have to do it in all the phoenix servers.

In your local development directory you still have to run this:

mix release.init

It will generate a standard rel/config.exs file that you must add to your git repository with the following changes to the bottom:

...
environment :prod do
  plugin Releases.Plugin.LinkConfig
  ...
end

release :your_app_name do
  set version: current_version(:your_app_name)

  set applications: [
    your_app_name: :permanent
  ]
end

The plugin should enable the release to find the local vm.args file in the server (this is super important, otherwise the remote commands such as start, stop, ping, etc will not work and the nodes will not load the correct information to boot up and form a cluster).

Finally, all set, you can issue this command:

$ mix edeliver build release production --skip-mix-clean --verbose

If it finishes without errors you will have a message like the following in the end:

...
==> Release successfully built!
    You can run it in one of the following ways:
      Interactive: _build/prod/rel/your-app-name/bin/your-app-name console
      Foreground: _build/prod/rel/your-app-name/bin/your-app-name foreground
      Daemon: _build/prod/rel/your-app-name/bin/your-app-name start
--> Copying release 0.0.1 to local release store
--> Copying your-app-name.tar.gz to release store

RELEASE BUILD OF YOUR-APP-NAME WAS SUCCESSFUL!

Now, this will take an absurdly long time to deploy. That’s because it will git clone the source code of your app, fetch all Elixir dependencies (every time!), it will have to compile everything, then it will run the super slow npm install (every time!), brunch your assets, create the so-called “release”, tar and gzip it, download it and SCP it to the other machines you configured.

In the .deliver/config file you set a BUILD_HOST option. This is the machine where all this process takes place, so you will want to have at least this machine be beefier than the others. As I am using small 512Mb droplets, the process takes forever.

The last command will download the generated tarball. It has to do it to make sure you have the NIFs compiled in the native environment where they run, because if you use a Mac, binaries from Macs won’t run on Linux.

Now we must upload and decompress this tarball in each server with the following command:

mix edeliver deploy release to production

The slower your network, the longer this will take, as it’s uploading a big tarfile across the public internet, so make sure you’re in a fast connection. And when it finishes, we can restart the servers (if you already had instances running):

mix edeliver stop production
mix edeliver start production

If you do everything right, the edeliver process finishes without any error and it leaves a daemon running in your server, like this:

/home/pusher/your-app-name/erts-8.2/bin/beam -- -root /home/pusher/your-app-name -progname home/pusher/your-app-name/releases/0.0.1/your-app-name.sh -- -home /home/pusher -- -boot /home/pusher/your-app-name/releases/0.0.1/your-app-name -config /home/pusher/your-app-name/running-config/sys.config -boot_var ERTS_LIB_DIR /home/pusher/your-app-name/erts-8.2/../lib -pa /home/pusher/your-app-name/lib/your-app-name-0.0.1/consolidated -name us@nyc-ip-address -setcookie ex-push&r-l!te -kernel inet_dist_listen_min 9100 inet_dist_listen_max 9155 -config /home/pusher/your-app-name.config -mode embedded -user Elixir.IEx.CLI -extra --no-halt +iex -- console

It will still take a long time, but it should be easier. So this is a pro-tip for you, Linux users. Follow this Gist for more details, you must emulate what’s run from the .deliver/config file’s bottom half.

Also notice that I ran the migrations manually, but you can do it using mix edeliver migrate.

Read their documentation for more commands and configurations.

Also, do not forget to enable UFW:

sudo ufw allow ssh
sudo ufw allow 8080
sudo ufw allow 4369
sudo ufw allow proto tcp from any to any port 9100:9155
sudo ufw default allow outgoing
sudo ufw enable

Debugging Production bugs

Right after I deployed, obviously it failed. And the problem is that the /home/pusher/your-app-name/log/erlang.log files (they are automatically rotated so you may find several files ending in a number), you won’t see much.

What I recommend you to do is to change the config/prod.exs file ONLY in your development machine and change the log setting to config :logger, level: :debug, use the same prod.secret.exs you edited in the servers above and run it locally with MIX_ENV=prod iex -S mix phoenix.server.

For example, in development mode I had a code in the controller that was checking the existence of an optional query string parameter like this:

if params["some_parameter"] do
 ...

That was working fine in development but crashing in production, so I had to change it to:

if Map.has_key?(params, "some_parameter") do
 ...

Another thing was that Guardian was working normally in development, but in production I had to declare its application in the mix.exs like this:

def application do
  [mod: {ExPusherLite, []},
   applications: [..., :guardian, :edeliver]]
end

I was getting :econnrefused errors because I forgot to run MIX_ENV=prod mix do ecto.create, ecto.migrate as I instructed above. Once I figured those out, my application was up and running through the http://your-app-name.yourdomain.com, HAProxy was correctly forwarding to the 8080 port in the servers and everything runs fine, including the WebSocket connections.

Conclusion

As I mentioned above, this kind of procedure makes me really miss an easy to deploy solution such as Heroku.

The only problem I am facing right now is that when I log in through Coherence’s sign in page, I am not redirected to the correct URI I am trying ("/admin" in my case), sometimes reloading after sign in works, sometimes it doesn’t. Sometimes I am inside a “/admin” page but when I click one of the links it sends me back to the sign in page even though I am already signed in. I am not sure if it’s a bug in Coherence, ExAdmin, Phoenix itself or an HAProxy misconfiguration. I will update this post if I find out.

Edeliver also takes an obscene amount of time to deploy. Even waiting for sprockets to process in a git push heroku master deploy feels way faster in comparison. And this is for a very bare-bone Phoenix app. Having to fetch everything (because Hex doesn’t keep a local global cache, all dependencies are statically vendored in the project directory). Having to run the super slow npm doesn’t help either.

I still need to research if there are faster options, but for now what I have “works”.

And more importantly, now I have a scalable cluster for real-time bi-directional WebSockets, which is the main reason one might want to use Phoenix in the first place.

If you want to build a “normal” website, keep it simple and do it in Rails, Django, Express or whatever is your web framework of choice. If you want real-time communications the easy way, I might have a better solution. Keep an eye on my blog for news to come soon! ;-)

In my previous post I forgot to mention minimal firewall and SSL configuration, but you can check it here.

One of the most difficult pieces to move away from is Slack. Don’t get me wrong, it’s a great tool and it gets the job done. The free-tier is really competent but I really want some of the paid features such as history of all conversations with search capabilities and uploads without limit.

The Cost Conundrum

But then you have to commit to pay USD 8/month per user (USD 6.67 if you pay ahead of time). If you’re a small team (around a dozen people) it’s well worth it and you should definitely pay and be done with it.

If you have more than 20 people then it starts to become a conundrum. On my GitLab cost calculations, some people argued that saving USD 15k yearly is not such a big deal, and they’re not wrong.

If you’re from outside of the US coastal cities, especially from countries in South America or Asia where currency can be more than 3 times weaker, then it quickly starts to become a big deal.

Apart from that, we are currently offloading a whole lot of our IPs and knowledge into the care of companies we don’t really know and where we don’t hold any real stakes.

If you’re a medium to big company it really starts to make sense to want to have more control over your own property. Of course, it costs some work and maintenance, so it’s not for everybody.

What I will argue is that if you have 150 people or more, and you’re still growing, it’s not such a big burden to jump into your own infrastructure.

You avoid paying around USD 12,000 a year or more, and instead you will pay USD 960 a year in Digital Ocean boxes and MMS plus a few hours every month in basic maintenance. That’s 12 times cheaper.

Again, you shouldn’t do it just for the cost savings as the maintenance burden can easily outweigh the savings if you or your team are not devops savvy and you don’t have anyone to keep monitoring it. For most companies, the conversation history is not so important and even the free-tier will be more than enough.

Cost is less relevant depending on your context, and there are many different uses for group chats. For example, if you’re an open source community, you would try Gitter instead.

There are also many other alternatives that you may want to try.

For the sake of this article, let’s assume you’re a small to medium company, in need of private rooms, many external users (for example, your clients), with at least 50 people. Then this can start to make sense. Any other scenario and we have to analyze it differently.

Why I chose Rocket.chat

With the cost reasoning out of the way, then it’s a matter of choosing which alternative to use.

I tried MatterMost first and I really wanted to like it. It has a very similar feature set to Slack and most importantly, it’s written in Go. So it’s both lightweight, very fast, and not so difficult to contribute. It uses MySQL or PostgreSQL which I like.

Unfortunately, it has an important show-stopper for my needs. You can create both public channels or private groups, but anyone can delete a private group. If you want permission control for that you must use their “Enterprise” offering and pay USD 20/year per user.

It’s still at least 3 times cheaper than Slack, so it may be a good option for you.

I really want to have full-control, not just make it cheaper. I could pay Slack already if I didn’t care, so I would not just pay an alternative and not have full control (including the ability to tweak and improve the entire code base).

Then, it took me back to Rocket.chat. Its feature set also rivals Slack, it’s good looking enough. But, it’s made in Meteor. Now, I don’t have anything against Meteor and I really think a Slack-clone is exactly the kind of use case where you could use Meteor to its full potential.

Technically I really think it’s a downside to be forced to use MongoDB (Meteor requires Mongo). For small installations I really prefer to have PostgreSQL. MongoDB is competent, but for medium to big installations there are caveats, and I will show you why below.

I strongly dislike the culture of writing software without minimal care such as having a reasonably complete test suite. You browse through the many packages that comprise Rocket.chat and several of them have no tests whatsoever. Some packages do have jasmine tests, but the majority is lacking.

This is one example I picked randomly, out of the few test files I found:

describe 'rocketchat:katex Client', ->

  it 'should exist', ->
    expect(RocketChat.katex).toBeDefined()

This is sad, sorry to say that, but it is. Not surprisingly I’ve seen katex issues while using the interface. And if you think this was just a poor choice, I picked another test file:

describe 'rocketchat:markdown Client', ->

  it 'should exist', ->
    expect(RocketChat.Markdown).toBeDefined()

Really?

So, the code itself is not so pretty, you have to keep this in mind. I’d much rather use Mattermost (which is much better structured and with enough tests), but it’s not entirely free, so it’s not an option for me. For my scenario, I’d rather have sloppy code (that minimally “works”) that I can tweak than code I can’t see, for this particular venture. Many inexperienced developers may argue about the necessity of having automated tests, but not having them also impacts this:

If we see how the project is evolving over time, we can see that interest is diminishing. The lack of good automated tests also makes it more difficult for new people to contribute without adding regressions, and makes contribution reviews more difficult and time consuming.

On the other hand the feature set is good enough as it is. There is only so much you can add to a Slack-clone. The upside is that if the code doesn’t change too much, I can tweak it more without having many things breaking all the time. It’s the complete opposite from the impressive contribution boost over at GitLab, for example.

Doing a few days of testing through the administration settings, user interface, having dozens of users doing real conversation, convinced me that even though the code quality is bad, the feature set is indeed there. I think I can live with that. Your mileage may vary though.

I am using Franz to open Slack, Rocket.chat, Hangouts, Messenger and any number of other communicators in one single app, which is great. And Rocket.chat does provide mobile apps based off PhoneGap. Again, it’s good, not great.

Many people have heavy bots or integrations into their Slack configuration. The alternatives such as Rocket.chat do support incoming and outgoing webhooks and most of the integrations should be possible (with some tweaking and hacking such as this Giphy example), so make a list of all important integrations and do some Google research on that before continuing.

Planning the Infrastructure

All that having been said, I’d recommend you try Rocket.chat’s demo server first to see if it has the minimal features you need. And also that you install a dry-run in a small Digital Ocean box with the intention of blowing it away later.

For a single-instance, stand-alone installation, temporary dry-run, I recommend you start by installing the MongoDB One-Click Application and follow this instruction to install everything in one machine.

This is the worst possible production environment, which is why I am being so repetitive in saying that you should destroy this box after you do your testing.

As I’ve said before, I am assuming a scenario where you have 50 or more people, with real clients, important communication going on. You don’t want to keep a simple installation like this for production usage.

For a minimal production installation you will need at the very least 4 boxes with 1 GB of RAM each. Now, why 4?

This is an important section for most of the beginner Javascript programmers out there playing with the so called “MEAN” stack (MongoDB, Express, Angularjs, Node.js). I’ve seen people deploy single-instance, stand-alone “MEAN” apps for production usage. And this is super bad.

MongoDB is NOT MEANT to function with just one single instance!

More than that:

DO NOT start an even number of MongoDB nodes, or at least install an Arbiter

So, the minimal setup for MongoDB is a Three-member Replica Set, no less!

The article “MongoDB Gotchas & How To Avoid Them” is a little bit old but you should be aware of several of the documented gotchas, including adding replica sets and avoiding waiting too long to start sharding your data.

The most important part of making Rocket.chat reliable is to make your MongoDB minimally reliable, and this is a hidden cost you must consider.

Installing a Three-Member Replica Set

For the sake of this article I will assume that you created 3 (three) MongoDB One-Click Applications on Digital Ocean’s, at the very least with 1 GB RAM boxes (2 GB RAM machines are a good choice if you’re in doubt, but never the 512 Mb) and another bare Ubuntu 14.04 box to be the web server. Create all machines in the same sub-region, all of them with private networking and backup enabled.

Take note of the public IPs and private IPs, for the sake of this article let’s say you have this:

Public IP     Private IP    Hostname

192.160.10.1  10.0.0.1      mongo1.yourdomain.com
192.160.10.2  10.0.0.2      mongo2.yourdomain.com
192.160.10.3  10.0.0.3      mongo3.yourdomain.com
192.160.10.4  10.0.0.4      yourdomain.com

In the commands and code snippets below, make sure to replace the fake IPs with your own private IPs.

As I said in my GitLab post, create a swap file, configure the locale, set it up for automatic security updates, and upgrade the Ubuntu packages.

The next first thing: configure the firewall in all machines (assuming ufw package is already installed):

sudo ufw reset
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow https
sudo ufw allow ssh
sudo ufw allow in on eth1 to any port 27017
sudo ufw enable

We are allowing MongoDB’s 27017 port only for machines in the same private networking (through eth1, the public IP goes through eth0). We’re also allowing port 22 for SSH and although I am not listing it here in this article you should also consider installing and configuring fail2ban. Digital Ocean also has a good post on UFW for more details.

Next thing to do on the MongoDB machines is to Disable Transparent Huge Pages (THP). For that create a file /etc/init.d/disable-transparent-hugepages, with this content:

#!/bin/sh
### BEGIN INIT INFO
# Provides:          disable-transparent-hugepages
# Required-Start:    $local_fs
# Required-Stop:
# X-Start-Before:    mongod mongodb-mms-automation-agent
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Disable Linux transparent huge pages
# Description:       Disable Linux transparent huge pages, to improve
#                    database performance.
### END INIT INFO

case $1 in
  start)
    if [ -d /sys/kernel/mm/transparent_hugepage ]; then
      thp_path=/sys/kernel/mm/transparent_hugepage
    elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then
      thp_path=/sys/kernel/mm/redhat_transparent_hugepage
    else
      return 0
    fi

    echo 'never' > ${thp_path}/enabled
    echo 'never' > ${thp_path}/defrag

    unset thp_path
    ;;
esac

Now you can register it as a service:

sudo chmod 755 /etc/init.d/disable-transparent-hugepages
sudo update-rc.d disable-transparent-hugepages defaults
sudo /etc/init.d/disable-transparent-hugepages start

Stop MongoDB in all 3 machines:

sudo service mongod stop

Now comes a complicated procedure that you should not do wrong. It’s only complicated because you have to do it in all 4 machines minding their correct private IP addresses. If you already have your own DNS server, you can skip this, otherwise follow through carefully.

You will have to edit both /etc/hostname and /etc/hosts. Each machine will have a particular name and they should be able to find the other machines by name. Let’s start by editing the /etc/hostname on each machine. Use the table in the beginning of this section, for example, in the machine with public IP 192.160.10.1 you must edit its hostname to be mongo1.yourdomain.com, and so on for all the machines.

Once you have it done, you have to replace the line 127.0.0.1 localhost in the /etc/hosts in each of them for the following:

127.0.0.1 localhost mongo1.yourdomain.com
10.0.0.1 mongo1.yourdomain.com
10.0.0.2 mongo2.yourdomain.com
10.0.0.3 mongo3.yourdomain.com
10.0.0.4 yourdomain.com

This snippet assumes you’re in the machine with private IP 10.0.0.1, of course. Make sure you change mongo1.yourdomain.com in the first line for the hostname of the machine you’re in. If this is too difficult for you, you definitely should not try to maintain your own infrastructure, so beware.

Finally, let’s edit the /etc/mongod.conf with these changes:

storage:
  dbPath: /mongodb-data
...
net:
  port: 27017
  bindIp: 0.0.0.0
...
replication:
  replSetName: rs0

This should be exactly the same in all machines. And you must create this /mongodb-data directory like this:

sudo mkdir /mongodb-data
sudo chown -R mongodb:mongodb /mongodb-data

Now go ahead on machine mongo1.yourdomain.com, start up with sudo service mongod start and fire up the mongo command, and you should see something like this:

# mongo
MongoDB shell version: 3.2.8
connecting to: test
>

Let’s now configure the replica set by issuing this command in the MongoDB command-line interface:

rs.initiate()
rs.add('mongo2.yourdomain.com')
rs.add('mongo3.yourdomain.com')

And you can now log in to the secondary machines and sudo service mongod start in all of them. They should all start joining the replica set and syncing.

You can check the status of the replication like this:

> rs.conf()

{
  "set" : "rs0",
  "date" : ISODate("2016-08-09T17:33:10.466Z"),
  "myState" : 1,
  "term" : NumberLong(-1),
  "heartbeatIntervalMillis" : NumberLong(2000),
  "members" : [
    {
      "_id" : 1,
      "name" : "mongo1.yourdomain.com:27017",
      "health" : 1,
      "state" : 1,
      "stateStr" : "PRIMARY",
      "uptime" : 11727,
      "optime" : Timestamp(1470763990, 3),
      "optimeDate" : ISODate("2016-08-09T17:33:10Z"),
      "electionTime" : Timestamp(1470752553, 1),
      "electionDate" : ISODate("2016-08-09T14:22:33Z"),
      "configVersion" : 6,
      "self" : true
    },
    {
      "_id" : 2,
      "name" : "mongo2.yourdomain.com:27017",
      "health" : 1,
      "state" : 2,
      "stateStr" : "SECONDARY",
      "uptime" : 11693,
      "optime" : Timestamp(1470763988, 4),
      "optimeDate" : ISODate("2016-08-09T17:33:08Z"),
      "lastHeartbeat" : ISODate("2016-08-09T17:33:10.026Z"),
      "lastHeartbeatRecv" : ISODate("2016-08-09T17:33:08.887Z"),
      "pingMs" : NumberLong(0),
      "syncingTo" : "mongo3.yourdomain.com:27017",
      "configVersion" : 6
    },
    {
      "_id" : 3,
      "name" : "mongo3.yourdomain.com:27017",
      "health" : 1,
      "state" : 2,
      "stateStr" : "SECONDARY",
      "uptime" : 11693,
      "optime" : Timestamp(1470763988, 4),
      "optimeDate" : ISODate("2016-08-09T17:33:08Z"),
      "lastHeartbeat" : ISODate("2016-08-09T17:33:09.994Z"),
      "lastHeartbeatRecv" : ISODate("2016-08-09T17:33:08.655Z"),
      "pingMs" : NumberLong(0),
      "syncingTo" : "mongo1.yourdomain.com:27017",
      "configVersion" : 6
    }
  ],
  "ok" : 1
}

Each box should be syncing data to another box, one of them will be marked “PRIMARY” and the others will be “SECONDARY”. If one of them “dies” for some reason, one of the others will be elected as PRIMARY until you add a replacement box. Read more details on this procedure in Digital Ocean’s post about implementing replication sets.

Adding your MongoDB environment to MMS

Another thing beginners will not realize is that it’s a good idea to monitor your services. At the very least you should install MongoDB Cloud Manager agents into your servers. Just set up an account at MongoDB Atlas.

It’s beyond the point of this article to explain Atlas, but you should have no trouble installing the agents with the Group ID and API Key provided. Then install the monitoring agent and have the Manager figure out your existing deployment and replica set.

The basic subscription for monitoring will cost you an extra USD 39 a month, but believe me when I say it’s worth it to add all your MongoDB deployments under MMS.

Installing a multi-instance Node.js service

Another mistake beginners in the MEAN stack make is to start up a single Node.js instance. Yes, Node.js’s asynchronous I/O nature makes it “concurrent enough” in 1 single thread. But you do want to maximize the rented machine so you should spin up at least one Node.js instance for each CPU core. A 1 GB RAM machine has just 1 core, but you can spin up at least 2 instances.

First of all, we must set up the yourdomain.com (192.160.10.4 or 10.0.0.4 in the example) to support Node.js, let’s do it:

sudo apt-get install -y npm curl graphicsmagick nginx git bc
sudo npm install -g n
sudo npm install -g forever
sudo npm install -g forever-service
sudo n 0.10.40

If you haven’t already, create a more restricted sudo user:

adduser rocket # add a password when asked
usermod -aG sudo rocket
su - rocket # and always ssh in with `ssh rocket@yourdomain.com`