The report checked out, and the fix is already available in v1.16.0.

Update to v1.16.0

If you use ai-jail, the update is the usual drill:

# Arch Linux (AUR)
yay -Syu ai-jail-bin

# Homebrew (macOS / Linux)
brew update && brew upgrade ai-jail

# crates.io
cargo install ai-jail --force

# mise
mise cache clear && mise upgrade github:akitaonrails/ai-jail

What changed in v1.16.0

Up through v1.15.x, ai-jail mounted the Docker socket inside the jail automatically whenever /var/run/docker.sock existed on the host. Read-write. No warning, no flag, no asking for your opinion. I documented this in the README as “favors usability,” and it was true: a coding agent often needs to run docker compose to bring up a test database, and the automatic passthrough saved some configuration.

Starting with v1.16.0, the behavior flipped:

• Socket passthrough is now off by default. It only goes in if you explicitly ask for it with the --docker flag or no_docker = false in .ai-jail.
• When you enable it and a host socket exists, ai-jail prints a launch warning spelling out that this amounts to giving host root to the process inside the jail.
• ai-jail status now shows Docker as disabled (default), so nobody thinks it got turned on by accident.
• In --lockdown and browser profile modes the socket never goes in, same as before.

A behavior change, yes, and a deliberate one. If your workflow depends on Docker inside the jail (mine does, in a few projects), opting in is one line in the project’s .ai-jail:

no_docker = false

The field name is ugly (no_docker = false to turn it on, I know), but old configs keep parsing the same way, and --no-docker / no_docker = true work as before. Only the default changed. The details are in the v1.16.0 release notes.

What issue #88 showed

mdindoffer’s report is the kind every maintainer wants to get: precise summary, a repro in a few lines, a fix proposal. The gist:

ai-jail was mounting the raw Docker socket inside the sandbox, read-write. The Docker daemon runs as root on the host. So an agent inside the jail could run this:

docker run --rm -v /:/host alpine sh -c 'cat /host/etc/shadow'

And that’s the ballgame. Read any file on the host, as root. It defeats the tmpfs $HOME, --mask, --deny-path, and Landlock in one move, because the action no longer happens inside the sandbox: it happens in the daemon, which lives outside and above any namespace bwrap created. The agent doesn’t even need to escape the jail when the jail has a door straight into the engine room.

In retrospect, shipping this on by default was a design mistake. I knew the passthrough was “dangerous” in the abstract, and I wrote as much in the README. What I hadn’t internalized: dangerous like this, with this default, is a vulnerability.

mdindoffer also proposed the definitive hardening path: instead of mounting the raw socket, put a filtered proxy in front of it (in the style of wollomatic/socket-proxy) that only accepts bind mounts from paths the agent can already write to inside the jail. It’s on the radar for a future release. To close the hole now, opt-in with an explicit warning fixes the default, which is where the problem lived.

The vulnerability class: docker.sock is root

This is not the first time I’ve run into this story, and I’d bet it isn’t yours either. “Whoever has access to the Docker socket has root on the host” is one of the classics of container security, documented by Docker itself on the daemon attack surface page: only trusted users should control the daemon, because Docker lets you share any host directory with a container, with no access restriction whatsoever.

The technical reason is simple. dockerd is a daemon that runs as root and obeys commands arriving through the API on the /var/run/docker.sock Unix socket. The docker CLI is just a client of that API. When you ask for docker run -v /:/host, the daemon is the one creating the container and mounting the host’s entire filesystem inside it, with full privilege. And a process inside a container runs as uid 0 by default, which the kernel sees as real uid 0 (barring user namespace remapping, which almost nobody turns on). The math checks out: write access to the socket equals root on the host. The docker group is passwordless sudo by another name.

The demo, on your machine

If you have Docker installed and your user in the docker group, reproduce it right now. No sudo, no exploiting any bug:

# confirm you're a regular user
$ id
uid=1000(akitaonrails) gid=1000(akitaonrails) groups=...,docker

# try reading /etc/shadow directly: denied, as expected
$ cat /etc/shadow
cat: /etc/shadow: Permission denied

# now ask the daemon to do it for you
$ docker run --rm -v /:/host alpine sh -c 'head -3 /host/etc/shadow'
root:$6$...:...
bin:!:...
daemon:!:...

And if you want the whole nine yards, a root shell on your own host:

$ docker run --rm -it -v /:/host alpine chroot /host /bin/bash
# id
uid=0(root) gid=0(root) groups=0(root)

No exploit, no 0-day. You used the official API, the documented way, and went from regular user to root in one command. That’s exactly what an agent inside ai-jail could do until today, even with every layer (bwrap, Landlock, seccomp, rlimits) turned on.

Why does this still exist?

If everyone has known about this for over a decade, why hasn’t anyone “fixed” it? Short answer: this is architecture, and architecture doesn’t get patched.

• The root daemon with an all-powerful API was the design that made Docker simple to operate. Granular per-request authorization exists in the form of authorization plugins, but it’s opt-in, annoying to configure, and I can count on one hand the setups I’ve seen using it.
• userns-remap has been around since Docker 1.10 and maps container root to an unprivileged user on the host. It ships turned off, because it breaks compatibility with images and volumes that assume uid 0.
• Rootless mode runs the whole daemon as your user, with the socket at $XDG_RUNTIME_DIR/docker.sock. It works, but it has network and storage limitations, and the entire internet of tutorials assumes the root daemon at the classic path.
• Podman was born rootless and daemonless precisely because of this criticism. There’s a whole section about it below.

Bottom line: this is here to stay. Every tool that mounts /var/run/docker.sock into an environment “for convenience” opens the same hole, knowingly or not. CI mounting the socket for image builds, remote IDEs, code-server, AI agent sandboxes (hi, me), web dashboard plugins. The fix always lives on the side of whoever builds the environment.

Fun fact: this is anything but new

If you’ve been following me for a while, this whole story should give you déjà vu. Back in 2023 I recorded [Akitando #139] - Understanding How Containers Work, where I explain what a container actually is: an ordinary Linux process, throttled by cgroups, fooled by namespaces, with its capabilities trimmed. No magic and no virtual machine. And look at what was already sitting in that episode’s link list: a walkthrough of privilege escalation via Docker, demonstrating the exact docker run -v /:/host trick. The hole issue #88 exploited inside ai-jail is the same one I was already pointing at in a 2023 video, and it was old news long before that.

Nobody knows this better than Red Hat. Podman was born there in 2018 as a direct answer to that architecture: no central daemon, rootless by default. Every container becomes a direct child of your user, via fork-exec, with no all-powerful process running as root brokering anything. The whole “docker.sock is root” problem doesn’t exist in that model, because there’s no docker.sock, no daemon, and no root.

And before you ask: yes, I run Podman for some things and I recommend it. But it’s no perfect solution either, and it’s worth understanding why.

What Podman does well:

• The right architecture. Daemonless and rootless from day zero. The entire vulnerability class from this article stops making sense.
• CLI compatibility. alias docker=podman covers the overwhelming majority of day-to-day commands. Build, run, push, pull: same commands, same flags.
• systemd integration. Quadlets are, in my opinion, the cleanest way to run a container as a service on Linux. Docker never came close.

Where it stumbles:

• The compatibility socket isn’t 100%. Podman offers a socket compatible with the Docker API (podman.socket), and plenty of tools work on top of it. But “plenty” isn’t “all”: Testcontainers, some IDE plugins, more exotic CI tooling trip over behavioral differences. It works until the day it doesn’t, and then you lose an afternoon debugging.
• Compose is a second-class citizen. The official docker compose does talk to the Podman socket, and podman-compose exists, but neither has the polish of the original pairing. Projects with a complicated compose file are where migrations usually get stuck.
• Rootless has a price. Rootless networking (slirp4netns, and pasta these days) has limits: no ping by default, weird source IPs, lower throughput. Images that assume uid 0 and volumes with wrong permissions need fiddling. Nothing fatal, but it’s friction.
• Docker Desktop is a product. On macOS and Windows, Docker Desktop delivers a polished experience that Podman Desktop is still catching up to. For a lot of people, that’s the only contact with containers they’ll ever have.

Add it all up and you get the answer to why the world stays on Docker: ecosystem inertia. Every tutorial, every CI pipeline, every example image, every docker run pasted from Stack Overflow assumes the root daemon at the classic path. Docker became the name of the category, the Kleenex of containers. Moving to Podman is technically easy and politically expensive: it’s you against the accumulated knowledge of the entire internet.

One detail that matters for sandbox users: mounting the rootless Podman socket inside a jail is far less catastrophic than mounting the Docker one. The equivalent “daemon” runs as your user, so a malicious agent would gain your privileges, not root. Still bad (it could overwrite your ~/.ssh, for one), but a whole different ballgame. Even so, ai-jail’s default stands: don’t mount any socket, from any runtime. Opt-in is opt-in.

Best practices so this doesn’t bite you

The list I apply and recommend:

1. Treat the docker group as passwordless sudo. Before adding any user or service to it, ask whether you’d give that thing unrestricted sudo. Same thing.
2. Never mount /var/run/docker.sock into untrusted environments. AI agents, CI jobs running code from a stranger’s pull request, third-party containers. Run grep -r docker.sock over your docker-compose files, CI manifests, and tool configs. It shows up in more places than you remember.
3. Need to expose it to something semi-trusted? Use a socket proxy. wollomatic/socket-proxy and Tecnativa’s docker-socket-proxy sit between the client and the daemon with an endpoint allowlist and blocking of arbitrary bind mounts. Read-only by default; you enable only what you need.
4. Prefer rootless whenever possible. Rootless Podman on Linux is the cleanest path; Docker’s own rootless mode is the second option. The daemon stops being root and this entire class of problem loses its bite.
5. In CI, build images without a privileged daemon. Kaniko and Buildah build images rootless, with no socket mounted in the job at all.
6. Can’t go rootless? Turn on userns-remap. It costs an afternoon of testing with your volumes and buys real isolation between container root and host root.
7. In ai-jail, let the default work for you. Docker off, and --docker only in projects where you trust the workload the way you’d trust a sudo. Rule of thumb: if you wouldn’t blindly hand sudo to the agent in that directory, don’t hand it --docker either.

Conclusion

A well-built sandbox loses much of its value with a back door left open for convenience. bwrap, Landlock, seccomp, and rlimits in ai-jail keep doing their jobs, but none of them can see what happens when the process inside politely asks the host’s root daemon to mount the whole filesystem into a container. A security layer you don’t audit becomes decoration.

My public thanks to @mdindoffer: clean report, minimal repro, correct severity, and a solution proposal on top. That’s how you report a vulnerability to an open source project.

If you want the full context of how I use sandboxes day to day, I wrote about it in How Do I Protect Myself From My Agents Deleting My Stuff?. The ai-jail story is in AI Agents: Locking Down Your System and in the Rust rewrite.

Now go run the upgrade.

On my YouTube channel I even used old games to teach computer science fundamentals. In this Akitando episode about Super Mario and old computers, I explain the 6502, memory maps, the PPU, hardware constraints, and why those games were programmed the way they were. If you never watched it, start there:

The problem is always the same: every few years I decide to set up a new emulation machine, download the main emulators again, and there I go repeating the same masochistic ritual. PCSX2, RPCS3, Eden, Azahar, Dolphin, RetroArch, Flycast, shadPS4, and so on. Each one with its quirks. Each one with its own particular way of wasting hours of your life.

In theory, setting all this up should be fun. In practice, it takes days. And that’s not an exaggeration. Dolphin still manages to be one of the worst because GameCube and Wii controller setups have no standard. RPCS3 requires per-game tuning. Eden needs DLC, updates and cheats in the right place. shadPS4 is a festival of trial and error. When I’d finally finish configuring everything, I didn’t feel like playing anything anymore. I just wanted to close the menus and go do something else.

I’ve been known to say that maybe the fun was the tuning itself. I don’t think that anymore. After repeating this process too many times, I’m done. I want to play the games, not fill out forms hidden inside emulator GUIs.

The problem was never Linux

I’ve solved this kind of thing in different ways in the past. Years ago I ran Linux on the host and a virtualized Windows with GPU passthrough to play inside the VM. At the time that made sense. I made a whole video about it:

But that was before Valve, Proton, modern Wine and the whole open source community pushed the Linux desktop to a much better place. Today you can avoid Windows in most cases. In the other video below I explain the evolution of CPU, GPU, OpenGL, Vulkan, and why we ended up here:

My gaming mini-PC with the RTX 4090 is still my main Steam machine, especially now that I’ve built a proper sim racing cockpit. On it I use EmuDeck, which was born to automate installing and configuring emulators on the Steam Deck, then grew Windows support, and helps cut the complexity of setting up this kind of environment. But my main desktop is also way too powerful to leave underused. It has an RTX 5090, and today I use that GPU mostly to test LLMs and for benchmarking, as I wrote in Testing Open Source and Commercial LLMs. Leaving it idle for gaming would be a waste.

Only on Linux I wanted something else. I didn’t want a black box doing everything for me without me knowing exactly what was being changed, especially on my main PC, which is my work machine. I wanted my own setup, handmade in the right sense: not clicking GUI by GUI, but understanding what’s being configured, keeping the files under my control, and being able to rebuild everything the way I like. I know the NixOS folks will jump in here to say “just use Nix” — I’ll explain why I ruled that out below. I also didn’t want to turn my work machine into a carnival of emulator packages, GTK themes, wrappers, launchers, weird runtimes and configs buried in ~/.config. Nor did I want dual boot or a VM. In 2026, the best answer for this, in my opinion, is Distrobox.

From what the project itself explains, Distrobox is a wrapper on top of podman, docker or lilipod to create containers tightly integrated with the host, with access to HOME, Wayland/X11, audio, USB devices, external storage and GPU. In other words: exactly the kind of pragmatic isolation I wanted. It’s not a security boundary, and the project site makes that clear. Don’t use it thinking about high-security sandboxing. Use it thinking about separating environments without paying the price of a VM.

And before anyone repeats the usual confusion: a container is not a virtual machine. I explain this calmly in this other episode:

The setup

The idea is simple: a vanilla Arch Linux inside a Distrobox called gaming, with NAS paths mounted read-only, Steam library accessible where it makes sense, and all emulators installed inside. Since the container has access to the NVIDIA GPU, audio, USB and other peripherals, I can separate work and gaming on the same machine without practical performance penalty.

The initial bootstrap looks roughly like this:

distrobox create --name gaming --image archlinux:latest --nvidia \
  --home /mnt/data/distrobox/gaming \
  --volume /mnt/data/steam:/mnt/data/steam \
  --volume /mnt/terachad/Emulators:/mnt/terachad/Emulators:ro

Even that first step has a catch. archlinux:latest inside Distrobox comes without [multilib] in pacman.conf, with a misconfigured sudoers, and with the old --nvidia problem: the host driver libs get bind-mounted read-only, so any install that depends on nvidia-utils breaks with file conflicts. Just that alone is the kind of headache that, a few years ago, would make me open half a dozen tabs, a bunch of terminals, and spend a whole night in trial and error.

This time I did it differently.

Claude Code as an infrastructure assistant

I’ve been getting more and more comfortable using Claude Code as my infrastructure assistant on my personal machines. I wouldn’t do this blindly on a client server. But on my own machine, in an environment I can destroy and rebuild however many times I want, it makes all the sense in the world. I wrote about this in Migrating my Home Server with Claude Code, when I used Claude to install and configure openSUSE MicroOS, Docker, NFS, firewall, hardening, and all the rest without me having to remember annoying shell commands.

So I thought: why not do the same thing here?

That’s exactly what I did. I started with a sequence of objective prompts, always asking for two things at once: do the work, and document enough for me to rebuild everything later. The most important history ended up in docs/distrobox-gaming-prompts.md.

A clarification worth making: those prompts on GitHub aren’t my raw messages, the way they came out in real time. After everything worked, I asked Claude itself to rewrite the prompts in a much more organized and detailed way, purely for documentation. The original prompts were much simpler and much less specific. I described the goal and let Claude figure out paths, config files, formats and commands on its own.

The first prompt created the box with --nvidia, a separate --home, and correct mounts. The second solved the three classic problems of Arch inside Distrobox: sudo, multilib, and the nvidia-utils dummy package. The third installed the gaming base, including a detail I would have easily forgotten if I was doing it by hand: pipewire-pulse, needed so several emulators don’t end up mute or out of sync.

The point isn’t the text itself. The point is the method. I don’t need to remember the exact order, the precise options, or where the docs for each detail are. I describe the goal and let the agent carry the piano. I stay in the tech-lead role: watching, reviewing, course-correcting when needed, and telling it to keep going.

The GitHub project

The setup lives in akitaonrails/distrobox-gaming, structured as an Ansible project. 17 roles cover everything from box creation, package bootstrap, config seeding, DLC installation, controller setup, to post-setup verification. The main playbooks are:

• site.yml: full setup from scratch
• reset-configs.yml: reset configs only (useful when you want to wipe tuning and redeploy)
• backup.yml / restore.yml: container snapshot and restore via podman commit
• refresh-shadps4.yml: standalone shadPS4 update
• install-xenia.yml: optional Xenia Manager install (Wine prefix)

Full walkthrough is in docs/rebuild-runbook.md. Package decisions (why AUR instead of Flatpak, for example) are in docs/distrobox-gaming-packages.md.

Why not Nix?

Someone will ask. “You’re recreating a reproducible environment. Why not use NixOS, or at least nix-shell/flakes?”

I considered and ruled it out. The emulator ecosystem I need lives on the AUR. AUR -bin packages are wrappers around AppImages that ship the official upstream binary without compiling anything. rpcs3-bin, pcsx2-latest-bin, eden-bin, duckstation-qt-bin, shadps4-bin — these binaries track upstream practically on release day. On Nixpkgs, many of these emulators are weeks or months behind release, and some don’t exist. Creating a Nix overlay for every emulator I need, maintaining it, and dealing with compatibility patches when upstream changes is work I don’t want to have.

There’s also the mental model. I don’t want to learn the Nix language, the derivation system and the evaluation model just to set up a personal gaming environment. I already know Ansible, roles are folders with YAML tasks, and debugging is ansible-playbook -vvv. When something goes wrong, I read the error, open the task that failed, and know exactly where to fix it. If I needed the same level of reproducibility at scale, across a cluster, with atomic rollback, then Nix would have an argument. For a distrobox with emulators on my personal machine, Ansible solves it with much less friction.

Flatpak was also tested and ruled out. Flatpak’s bwrap (bubblewrap) doesn’t nest properly inside Docker’s mount namespaces, so Flatpak apps simply don’t run inside distrobox. The alternative would be installing Flatpak on the host, but then you’re back to polluting the work machine with gaming packages. Plus, Flatpak versions of emulators lag well behind AUR — the AUR’s pcsx2-latest-bin tracks the 2.7.x AppImage, while Flathub still ships v2.6.3.

Platforms covered

The setup covers 12 platforms today:

RetroArch comes on top to cover the rest (NES, SNES, Genesis, N64, arcade, etc.), with 21 buildbot cores and 8 asset packs (databases, shaders, cheats, overlays, autoconfig).

Frontend is ES-DE. Custom system definitions live in ansible/group_vars/all/esde.yml, and each emulator comes with a wrapper that sets the right GPU environment variables (more on that in a sec).

GPU hardforce for NVIDIA

On a hybrid system with NVIDIA dGPU + AMD iGPU (my case on the desktop), Vulkan sometimes picks the iGPU on its own and you end up with an emulator running at 15 fps while the 5090 sits idle. The fix is to force NVIDIA’s Vulkan ICD in every launcher.

Each desktop entry rendered by Ansible exports VK_ICD_FILENAMES pointing to NVIDIA’s ICD, and the ES-DE wrapper does the same before launching any core. Combined with the --nvidia flag at box creation, which bind-mounts host drivers read-only, the result is predictable: Vulkan always on the NVIDIA dGPU, regardless of system state at runtime.

This mostly matters for hybrid setups. On a desktop with a single GPU, the ICD filtering is placebo, but it doesn’t hurt.

PS2: PCSX2 with per-game tuning

The Gran Turismo line is my declared addiction. Each version needs its own specific tweak. The PCSX2 setup today has sane global configs (Vulkan, 4x SSAA, widescreen 16:9, Xbox-style bindings, FXAA + PCRTC antiblur, 16x anisotropic, Automatic deinterlace) and per-game INIs for the titles I actually play:

• Gran Turismo 3 A-spec (SCUS-97102 + bundle PBPX-95503): widescreen pnach, optional retexture pack (disabled by default because NFS streaming causes flicker in car-showcase cutscenes)
• Gran Turismo 4 USA (SCUS-97328): Silentwarior112 HD HUD, full retexture pack, Silent’s trigger/cam patches, widescreen pnach
• Gran Turismo 4 Spec II (SCUS-97436, CRC 4CE521F2): special case — the vanilla GT4 HD packs are linked via symlink because Spec II has the same structure but a different CRC. Widescreen pnach renamed to match the CRC. Deinterlace mode 8 (Adaptive TFF), ShadeBoost tuning (+10 saturation, +3 brightness, +2 contrast) to fix pause-interlace ghosting and the anemic upscale saturation
• Enthusia Professional Racing (SLUS-20967): HD textures + widescreen
• Ridge Racer V (SLUS-20002): widescreen + no-interlace pnach

Silent’s and community pnach files are downloaded automatically by the pcsx2_textures role, which also deploys the cheats pack from the NAS and installs the Spec II HD packs via symlink. Widescreen is enabled per-CRC via gamesettings (PCSX2 identifies games by CRC before applying pnach).

On PCSX2 versioning: the AUR’s pcsx2-latest-bin tracks the 2.7.x line, which is where all the modern features live (texture replacement, FXAA, PCRTC antiblur). The default pcsx2 in Arch repos is still on 2.6.3. The gap is over 250 commits of fixes and new features.

PS3: RPCS3 with curated per-game configs

RPCS3 has an annoying detail: per-game configs live in ~/.config/rpcs3/custom_configs/config_<SERIAL>.yml. The file format has a version, and if you hand-write YAML with the wrong version, RPCS3 silently rejects it and falls back to defaults. That cost me time to figure out.

The rpcs3_per_game_configs role handles it like this:

1. Queries the RPCS3 community compatibility DB for recommended presets per title
2. Applies curated overrides on top for titles I know need specific tuning (typically GT5 and GT6)
3. Saves the YAML with the exact filename convention (config_<SERIAL>.yml, config_ prefix mandatory)

For the Gran Turismo line:

• GT5 (BCUS98114, BCES00569): Resolution Scale 300%, Shader Precision Ultra, Force High Precision Z, SPU XFloat Accurate, Multithreaded RSX. The combo kills the typical RSX dithering without breaking the GT-series safety config (WCB/RCB off).
• GT6 (BCES01893 + regional variants): special case, version pinned to v1.05. Patches 1.06+ regress with black surfaces on cars in the garage menu. The extract_ps3_dlc.py script has a per-title PATCH ceiling specifically to pin GT6 at 1.05 even when PSN serves newer patches. Also, Force CPU Blit is mandatory (without it, full-screen flicker). The trade-off is that the rear-view mirror stays permanently black — turning WCB on would restore the mirror but downgrade to native 720p. I’ll take no mirror.

More details on those two in docs/gt5-rpcs3.md and docs/gt6-rpcs3.md.

PS3 update checker

To avoid opening RPCS3’s GUI and clicking Download Game Updates game by game, I wrote a Python script: check_ps3_updates.py. It walks the games installed in dev_hdd0/game, parses PARAM.SFO to get the current version per title, queries the PSN update server (a0.ww.np.dl.playstation.net), and compares against the local patch cache. --list shows the diff; --download fetches what’s missing. --max-version respects per-title ceilings (e.g., 1.05 for GT6).

The first scan of my library found 51 of 72 games with updates available, 69 patches missing locally, ~24 GB in total. GT6 alone had 16 missing patches from 1.07 to 1.22, which I skipped for the reasons above.

Patches arrive as PSN packages type 0x0001 without encryption, so extract_ps3_dlc.py handles the install. Standard call:

distrobox enter gaming -- python3 scripts/extract_ps3_dlc.py \
  "/mnt/terachad/Emulators/EmuDeck/roms_heavy/ps3-DLC" \
  --dest "/mnt/data/distrobox/gaming/.config/rpcs3/dev_hdd0/game"

Tracking PS3 DLC versions manually, game by game through RPCS3, is impractical. It becomes an infinite to-do list, and you only find out there was a 1.22 patch when the game crashes.

PS1: DuckStation per-game

The Gran Turismo trilogy starts on PS1, and quality there only goes up if you enable the right things:

• Gran Turismo (SCUS-94949): DuckStation’s built-in WidescreenHack (no cheat exists for this game), PGXP enabled
• Gran Turismo 2 Simulation (SCUS-94455): widescreen cheat + 8 MB RAM cheat (fixes audio issues on crowded tracks), JINC2 filter
• Gran Turismo 2 Arcade (SCUS-94488): same treatment as Simulation

Global defaults: Vulkan, 8x upscale, JINC2 as the default filter.

GT2 cheats come from a widescreen-fixes repo and are linked automatically into DuckStation’s cheats folder.

Switch: Eden with DLC, update and cheats

On Switch, eden-bin is the emulator. The integration automates three things I used to do by hand:

1. DLC and update install: the install_dlcs role points to the dumps directory on the NAS, identifies .nsp/.nsz/.xci/.xcz files, and installs everything into ~/.local/share/eden/nand/user/Contents/registered/. If the dump is messy (patches and DLC mixed in a flat folder), the reorganize_switch_nsps.py script sorts everything by Title ID first.

2. Atmosphere cheats: Atmosphere-format cheats get symlinked into Eden’s load path (~/.local/share/eden/load/<TITLE_ID>/cheats/). The switch_cheats role handles this for every title covered by the NAS pack.

3. Update checker: check_switch_updates.py lists titles with updates available against what’s installed locally. Useful so I don’t miss important patches for games still getting updates.

A detail that burned me: QT_STYLE_OVERRIDE needs to be unset before launching eden-bin, otherwise it conflicts with Kvantum and breaks the UI. The wrapper does the unset before exec.

Xbox 360: Xenia Manager via Wine

Xbox 360 is still rough territory on Linux. There’s no decent native emulator. The best route today is Xenia running through Wine, managed by Xenia Manager.

This is opt-in in my setup. The install-xenia.yml playbook creates a dedicated Wine prefix, downloads Xenia Manager, and registers launchers. It’s not part of the main site.yml because not everyone wants Wine in the mix.

Project Forza Plus (Forza Motorsport 2/3/4)

Old Forza is still the king of previous-gen simcade. Running FM2/3/4 on Xenia requires the Project Forza Plus community mods: compatibility patches, performance mods, specific title-update installs. I documented the process in docs/project-forza.md.

Project Gotham Racing 3 and 4

PGR3 runs reasonably well on Xenia Canary. PGR4 needs a specific tweak: render_target_path_vulkan = "fsi" in the config, otherwise some races break with visual artifacts. Setup documented in docs/xbox360-pgr.md. PGR4 audio on NVIDIA still has an XMA decoding issue that produces intermittent garbage. There’s an open xenia-canary issue, no resolution so far.

Batch Title Updates

Title Updates for Xbox 360 games were distributed via Xbox Live, which has been shut down. The alternative is archive.org, which has the complete catalog preserved. I wrote scripts/download-xbox360-tus.py to automate:

distrobox enter gaming -- python3 scripts/download-xbox360-tus.py \
  --src /mnt/terachad/Emulators/EmuDeck/roms_heavy/xbox360 \
  --dest /mnt/terachad/Emulators/EmuDeck/roms_heavy/xbox360-updates \
  --dry-run

The script needs the internetarchive CLI authenticated (ia configure once, stores the token). It scans the games folder, matches against the archive.org manifest (cached locally), prioritizes by region (USA > World > Europe > Japan), and downloads via ia download --checksum with automatic retry. The resulting .zip files go to Xenia Manager via Manage → Install Content, which extracts into the correct directory (000B0000/).

Full setup in docs/xbox360-title-updates.md.

Steam in the distrobox

Since Proton became a first-class citizen on Linux, it makes sense to have Steam in the gaming box too. I added the steam package via Ansible (which pulls in multilib and 32-bit deps), mounted /mnt/data/steam read-write so the library persists outside the container, and created a host-side launcher.

In practice, this lets me run Steam games via Proton side-by-side with emulation, without switching environments. The mini-PC with the RTX 4090 is still the main Steam machine in the sim racing cockpit. The desktop is fallback and a test environment.

Shell and controllers

Minimal in-box shell

When I distrobox enter gaming, I want a decent prompt even for quick troubleshooting. I installed zsh + starship with a minimal config, zero exotic plugins. It’s not my full dev setup, it’s just “a prompt that shows path and git branch without embarrassing me”.

Dolphin and the modern-controller hell

Dolphin has always been the king of manual friction. If you use a modern controller, like my 8BitDo Ultimate 2, you have to remember how I like the GameCube mapping, how I adapt the Wii Remote, when to use a Nunchuk profile, when to switch to Classic Controller. I don’t have the patience to rebuild that by hand every time. Profiles are pre-built in config/emulator-overrides/dolphin/Profiles/ and copied by the seed_configs role. Details in docs/controller-hotkeys.md.

A few stumbles that stuck around

Not everything became magic. Emulation always has banana peels.

Flycast, for example, has an irritating trap. The emu.cfg file uses rend.* keys inside the [config] section. If you create a separate [rend] section, it looks right, but the emulator just ignores it and later rewrites everything with mediocre defaults. The fix became a dedicated wrapper at $DG_BOX_HOME/bin/flycast-hires:

$DG_BOX_HOME/bin/flycast-hires \
  -config config:pvr.rend=4 \
  -config config:rend.Resolution=2880 \
  -config config:rend.EmulateFramebuffer=no \
  -config config:rend.WideScreen=yes

Details in docs/flycast-resolution.md.

shadPS4 is still far from a closed case. The current setup is focused on Driveclub, with a mirrored config, XML patch and sys_modules links from firmware 11.00. And that wasn’t random: I really like Driveclub. It’s practically the only game still stuck on PS4 that I really wanted to get running properly. I’ve seen several YouTube videos showing the game working, but so far I haven’t been able to get it running satisfactorily on Linux myself. Documented in docs/driveclub-shadps4.md. The good part is that I no longer depend on muscle memory to remember how to launch it, with which patch, in which folder, with which modules. The bad part is that PS4 emulation is still PS4 emulation. No script makes upstream mature faster. If anyone knows how to close this setup on Linux, check the project on GitHub and send a PR.

Ridge Racer V has car-texture flickering documented in PCSX2 issues (#3639, #13729). The Hardware renderer is acceptable for a racing game where you’re not standing still admiring the model. Software renderer fixes it but kills performance. I chose to live with the flicker.

The real gain isn’t just automation

It’d be easy to sum this up as “look how cool, I used AI to automate shell scripts”. That’s not it.

The real gain is more mundane and more important: I stopped spending mental energy on manual work. I didn’t have to remember rare commands. I didn’t have to keep dozens of tabs of wiki, issue, forum, README. I didn’t have to leave half a dozen terminals scattered tailing logs while I try to remember which hidden GUI option the emulator insists on overwriting on the first boot. I started working in conversation.

I tell the agent the goal. It looks up the files, reads the logs, finds the right config format, compares defaults, proposes fixes, writes scripts, validates paths, checks UID/GID, verifies broken symlinks, generates wrappers, exports launchers. I’m still responsible for decisions, of course. But I stopped being a typist of rare commands.

That’s the point that interests me most about using coding agents on Linux. They dramatically reduce the entry friction. A lot of people bounce off the Linux desktop not because the system is incapable, but because the tuning curve has historically been too irritating. Having an assistant capable of reading docs, cross-referencing configs, proposing automation and executing under supervision changes that game.

If you want to reproduce

The repo was published for this:

git clone https://github.com/akitaonrails/distrobox-gaming.git
cd distrobox-gaming/ansible
ansible-galaxy collection install -r collections/requirements.yml
cp host_vars/localhost.yml.example host_vars/localhost.yml
$EDITOR host_vars/localhost.yml  # adjust paths for your machine
ansible-playbook site.yml

Xenia is opt-in:

ansible-playbook install-xenia.yml

Read the README.md and docs/rebuild-runbook.md first. I don’t distribute ROMs, BIOS, firmware or keys. The repo only detects, links and configures what you already have on your own machine.

If you prefer the Claude Code route

My recommendation is simple:

1. Start with the goal, not the command. “I want an Arch distrobox with NVIDIA GPU, separate home, ROMs read-only and Steam rw” beats dumping a half shell command without context.
2. Always ask it to document what it’s doing. If it works, promote the result to a script or role. If you don’t document, you just manufactured a throwaway hack.
3. Work in phases. Box creation. Bootstrap. Emulator config. Launcher export. Verification. That’s exactly how I broke down the problem.
4. Ask for objective checks. command -v, file existence, broken symlinks, UID/GID, ROM paths, audio, GPU. The best automation is the one that fails early.
5. Don’t use the agent as a command parrot. Use it as a technical assistant. You still review the decisions and tell it to adjust course.

For me, that was the gain. I went from zero to a much more complete emulation machine without having to customize everything by hand, GUI by GUI, one window at a time. And this time, I finished with energy left to do what I wanted from the start.

Play.

With the new Minisforum MS-S1 Max that I bought, I decided to do the migration properly. And I decided to use Claude Code from the start to speed up the process. It’s a home server, only I use it, so the risk of making a mistake is low. But if this were a real production server, I would never do this without rigorous human review at every step.

What follows is the migration writeup and a guide for anyone who wants to replicate it. If I ever have to rebuild from scratch, this post is the documentation.

Choosing the operating system

Why not Ubuntu Server again

I used Ubuntu Server on the NUC for convenience. But do-release-upgrade is Russian roulette. Every time Ubuntu releases a new version, the upgrade is a real risk of breaking things. Packages change, configs get overwritten, dependencies clash. For a server that has to be running constantly, that’s unacceptable.

Why not Arch Linux

I use Arch on the desktop and I like it. But Arch is a rolling release with no stability guarantees at all. For a desktop where I can stop and fix problems, fine. For a headless server running 49 Docker containers that has to work after every reboot, no.

Fedora CoreOS vs openSUSE MicroOS

The two modern options for a container server are Fedora CoreOS and openSUSE MicroOS. Both are immutable systems: the root filesystem is read-only, updates are atomic (they either apply in full or don’t apply at all), and rollback is instant.

The difference: Fedora CoreOS uses Ignition (declarative configuration before the first boot) and is designed to be provisioned automatically. MicroOS uses transactional-update and allows normal interactive use. For a home server where I want SSH and the ability to poke around manually, MicroOS fits better.

What makes MicroOS different

The immutable system concept changes how you operate the server:

Every package install or /etc edit goes through transactional-update, which creates a new btrfs snapshot, applies the change in that snapshot, and on the next reboot the system boots into the updated snapshot. If the change breaks something, you do transactional-update rollback and you’re back on the previous snapshot in seconds.

Updates are automatic and daily. The transactional-update.timer downloads patches, creates a snapshot, and rebootmgr reboots in a configured window (in my case, between 4am and 5:30am). If the update breaks the boot, GRUB automatically falls back to the previous snapshot.

SELinux is enforcing by default. That caused 90% of the problems during the migration, but it’s the right setting for security.

Initial setup

Hardware

• AMD Ryzen AI Max+ 395, 128GB LPDDR5X
• 96GB allocated as VRAM via BIOS (UMA Frame Buffer Size)
• 2TB NVMe (system + Docker)
• Wired 2.5Gbps network
• Synology DS1821+ NAS at 192.168.0.21 (NFS)

First steps

MicroOS install is standard. Then:

# Create user with a UID that matches the NAS (so NFS works without permission issues)
useradd -u 1026 -m akitaonrails
passwd akitaonrails

# Configure sudo (inside transactional-update shell)
sudo transactional-update shell
# inside: add akitaonrails to sudoers
exit
sudo reboot

Synology NFS

The Synology NAS exports /volume1/TERACHAD over NFS. The mount point on MicroOS is /var/mnt/terachad (not /mnt/, which lives on the immutable root).

In /etc/fstab (applied through transactional-update):

192.168.0.21:/volume1/TERACHAD /var/mnt/terachad nfs4 nfsvers=4.1,rsize=262144,wsize=262144,hard,_netdev 0 0

Details that matter: nfsvers=4.1 because 4.2 didn’t work with the Synology. rsize=262144,wsize=262144 (256KB buffers) was the biggest NFS performance improvement. hard instead of nofail so the mount keeps retrying indefinitely if the NAS disconnects temporarily.

GPU / ROCm

This step was a pain. The Radeon 8060S in the AI Max+ 395 is gfx1151, which ROCm doesn’t officially support. Three steps were needed, and all three are mandatory:

1. BIOS: set UMA Frame Buffer Size to 96GB
2. Kernel: add amdttm.pages_limit=25165824 amdttm.page_pool_size=25165824 to /etc/kernel/cmdline
3. Docker: use HSA_OVERRIDE_GFX_VERSION=11.5.1 in every ROCm container

Without step 2, ROCm only sees 15.5GB even with the BIOS allocation in place. The numbers are 96GB / 4KB (page size) = 25,165,824 pages.

sudo transactional-update shell
echo "amdttm.pages_limit=25165824 amdttm.page_pool_size=25165824" >> /etc/kernel/cmdline
exit
sudo sdbootutil update-all-entries  # OUTSIDE the transactional shell
sudo reboot

Verification:

cat /sys/class/drm/card1/device/mem_info_vram_total
# 103079215104 (96 * 1024^3)

Docker on MicroOS

sudo transactional-update --non-interactive pkg install docker
sudo reboot

sudo systemctl enable --now docker
sudo usermod -aG docker akitaonrails
# logout and login for the group to take effect

# Install standalone docker-compose (the openSUSE package doesn't include it)
sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-linux-x86_64" \
  -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
mkdir -p ~/.docker/cli-plugins
ln -s /usr/local/bin/docker-compose ~/.docker/cli-plugins/docker-compose

daemon.json

{
  "log-level": "warn",
  "log-driver": "local",
  "log-opts": {"max-size": "10m", "max-file": "5"},
  "selinux-enabled": true,
  "live-restore": true,
  "userland-proxy": false,
  "exec-opts": ["native.cgroupdriver=systemd"]
}

live-restore: true keeps containers alive across a Docker daemon restart. userland-proxy: false uses iptables directly instead of proxy processes (less overhead). selinux-enabled: true is mandatory on MicroOS.

SELinux and Docker: the biggest source of problems

This deserves an entire section because it was responsible for 90% of the bugs during the migration.

On MicroOS with SELinux enforcing, every container that writes to a host bind-mounted directory needs special handling. There are two approaches: the :Z suffix on volumes and the security_opt: label:disable option.

NEVER use :Z. Use security_opt: label:disable.

:Z tells Docker to relabel the host directory with the container’s SELinux context. Sounds like the right thing. In practice:

• SQLite databases break. The relabeling changes the file’s context and SQLite may refuse to open the WAL journal.
• NFS mounts silently ignore :Z. NFS doesn’t support SELinux xattrs. The kernel ignores the flag without error, but the container still doesn’t have permission.
• :ro,Z mounts try to relabel even when read-only, which fails on NFS and can corrupt the context on local paths.

The right solution for every container on this system:

services:
  myservice:
    security_opt:
      - label:disable     # disables SELinux enforcement for this container
    volumes:
      - ./data:/data      # NO :Z
      - ./config.yml:/etc/config.yml:ro  # NO :Z even on :ro

label:disable disables SELinux label enforcement for that container only, not for the entire system. Combined with Docker’s network and process isolation, it’s safe for a home server.

Migrating the stacks

All Docker stacks were reorganized into /var/opt/docker/<stack>/docker-compose.yml. On the old server, they were scattered across /home/akitaonrails/docker/, /home/akitaonrails/<service>/, with no pattern.

Substitutions applied across every compose file

Media stack (Plex, Radarr, Sonarr, etc.)

The media stack is the most complex. Plex needs its own LAN IP (macvlan) for direct streaming to work. The setup:

docker network create -d macvlan \
  --subnet=192.168.0.0/24 \
  --gateway=192.168.0.1 \
  -o parent=enp97s0 \
  plex_macvlan

In compose, Plex needs to be on two networks: the macvlan (for the IP 192.168.0.6) and the default bridge (so other containers like Seerr can talk to it):

plex:
  networks:
    plex_macvlan:
      ipv4_address: 192.168.0.6
      mac_address: "02:42:c0:a8:00:06"
    default: {}    # mandatory — without this, Seerr can't see Plex

A detail that almost broke me: Plex stores absolute paths in its database. If the container’s internal volume changed from /media to /data, Plex no longer finds anything. You have to use exactly the same mount target as the old compose.

Ollama with ROCm

New stack, didn’t exist on the previous server:

ollama:
  image: ollama/ollama:rocm
  container_name: ollama
  devices:
    - /dev/kfd:/dev/kfd
    - /dev/dri:/dev/dri
  security_opt:
    - seccomp:unconfined
    - label:disable
  group_add:
    - "485"   # render group GID
    - "488"   # video group GID
  environment:
    - HSA_OVERRIDE_GFX_VERSION=11.5.1
    - PYTORCH_ROCM_ARCH=gfx1151
    - OLLAMA_KEEP_ALIVE=30m
    - OLLAMA_NUM_PARALLEL=4
    - OLLAMA_FLASH_ATTENTION=1
    - OLLAMA_KV_CACHE_TYPE=q8_0
  volumes:
    - /var/lib/ollama:/root/.ollama
  ports:
    - "11434:11434"

OLLAMA_FLASH_ATTENTION=1 activates flash attention. OLLAMA_KV_CACHE_TYPE=q8_0 uses 8-bit KV cache, cutting the bandwidth needed per token in half. Free performance optimizations.

Monitoring (Grafana + Prometheus)

Grafana uses a named volume (grafana_data) which is NOT included in normal filesystem backups. That’s the reason I lost all my dashboards on the first attempt. The fix is an explicit backup of the named volume:

# On the old server:
docker run --rm -v grafana_data:/data:ro -v /tmp:/backup alpine \
  tar czf /backup/grafana_data.tar.gz -C /data .

# Transfer and restore on the new one:
docker run --rm -v grafana_data:/data -v /tmp:/backup alpine \
  sh -c "cd /data && tar xzf /backup/grafana_data.tar.gz"

Same thing for Portainer (portainer_data). Any volume defined in the volumes: block of a compose without a host path needs this treatment.

Cloudflare Tunnel

I use Cloudflare Tunnel to access all the services from outside the house without opening ports on the router. The migration was the easiest part: copy the tunnel’s JSON credentials file and the config.yml, update the IPs from .145 to .90, and bring the container up. The tunnel keeps the same ID, no need to recreate DNS.

The hostnames live in config.yml: portainer, grafana, plex, seerr, qbittorrent, syncthing, radarr, sonarr, bazarr, prowlarr, vault, gitea, kavita, and others. Everything accessible via https://<service>.example.com from anywhere.

Gitea (image registry)

Gitea acts as a private Docker registry on port 3007. The Frank FBI, Frank Mega, Frank Yomik and Mila projects have Docker images that are built and pushed to Gitea. For it to work, Docker’s daemon.json needs:

{
  "insecure-registries": ["192.168.0.90:3007"]
}

Gitea SSH gave me a problem during the migration: the old app.ini had SSH_LISTEN_PORT=22, but the container’s entrypoint also starts sshd on port 22. Conflict. Fix: GITEA__server__SSH_LISTEN_PORT=2222 as an environment variable in compose.

All 49 containers running

The migrated server runs 49 containers across 15 stacks. The media stack alone has 10 containers (Plex, Radarr, Sonarr, Bazarr, Prowlarr, qBittorrent, SABnzbd, Jackett, FlareSolverr, Seerr). The personal projects (Frank FBI, Frank Mega, Frank Yomik, Mila) add another 11. Monitoring with Grafana, Prometheus, node-exporter and cAdvisor. Utilities like Portainer, Vaultwarden, Syncthing, Organizr, Watchtower. Gitea as private Docker registry. Immich as self-hosted Google Photos. Kaizoku for manga with Kavita as the reader. Ollama with ROCm. And Bitcoin Core/Fulcrum indexing the blockchain off the NAS.

Backups: two layers

Layer 1: local btrfs snapshots (snapper)

/var lives on a 3.7TB btrfs partition. snapper creates automatic snapshots: 7 daily + 1 weekly. They’re crash-consistent, not application-consistent (postgres can be slightly inconsistent if there’s heavy writing during the snapshot).

To recover an accidentally deleted file:

sudo snapper -c var list
sudo cp /var/.snapshots/5/snapshot/opt/docker/media/radarr/appdata/config/radarr.db \
        /var/opt/docker/media/radarr/appdata/config/radarr.db

For a full stack rollback:

sudo docker compose -p media down
sudo snapper -c var undochange 7..0 /var/opt/docker/media
sudo docker compose -p media up -d

Layer 2: restic to the NAS (off-machine)

restic runs every night at 3am, doing incremental backups to /var/mnt/terachad/homelab-backups/. Retention: 7 daily + 4 weekly. Content-based deduplication, so Plex config (19GB) and Gitea repos (12GB) only transfer deltas.

Before restic runs, a pg_dump exports the postgres databases (Immich, Kaizoku). The dumps go to /tmp/homelab-db-dumps/ and are included in the backup.

What’s NOT included in the backup (re-downloadable): Bitcoin blockchain (785GB on the NAS), Docker images (re-pullable), Ollama models (re-downloadable), HuggingFace/EasyOCR caches, Plex transcoding scratch.

Large re-downloadable directories were converted to btrfs subvolumes so snapper ignores them: /var/lib/ollama and /var/opt/docker/bitcoin/fulcrum/fulc2_db.

Performance tuning

btrfs with zstd compression

Added compress=zstd:1 to the fstab for the /var partition. zstd level 1 has near-zero CPU overhead on NVMe and compresses Docker metadata, JSON configs and logs nicely. Incompressible data (SQLite, postgres) is automatically skipped by btrfs.

zram swap

With ~30GB of RAM available for the system (96GB go to VRAM), in-memory compressed swap helps. zram creates a ~15GB swap device (ram/2) with zstd compression, much faster than disk swap.

# /etc/systemd/zram-generator.conf
[zram0]
zram-size = ram / 2
compression-algorithm = zstd
swap-priority = 100

btrfs nodatacow on database directories

Copy-on-write + random database writes = write amplification. I disabled CoW on the directories that hold SQLite and postgres:

sudo chattr +C /var/opt/docker/gitea/data/gitea.db
sudo chattr +C /var/opt/docker/immich/db/
sudo chattr +C /var/opt/docker/media/radarr/appdata/config/

CPU in performance mode

On a headless server, there’s no point saving energy:

echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/energy_performance_preference

Persisted via systemd service cpu-epp.service.

Docker shutdown fix

A problem I discovered: Docker ships with KillMode=process, which means at system shutdown, systemd kills only dockerd and leaves all the containerd-shim processes (one per container, ~49 in my case) orphaned. systemd-shutdown then has to hunt them down one by one after the journal has already stopped, causing a silent multi-minute hang.

Fix:

# /etc/systemd/system/docker.service.d/shutdown.conf
[Service]
KillMode=control-group
TimeoutStopSec=30

The problems we ran into

This is the table of actual problems we hit during the migration. If you’re planning something similar, read it before starting:

What to tell Claude Code before you start

If I were redoing the migration from scratch, I’d give Claude Code these instructions on the very first message. In order of importance:

Tell it that SELinux is enforcing and that it should NEVER use :Z on any Docker volume, but rather security_opt: label:disable on every service. Tell it that /var/mnt/terachad/ is an NFS mount and that :Z should never appear on NFS paths. Tell it to always look at the original compose before rewriting and only change IPs, paths and container names, without inventing new volume layouts. Warn that named volumes need explicit backup (Grafana, Portainer). Explain that Plex runs on macvlan and needs default: {} in the networks. Inform that the GPU is gfx1151, not officially supported, and that it needs UMA 96GB in the BIOS + kernel TTM params + HSA_OVERRIDE_GFX_VERSION=11.5.1. And tell it that Bitcoin/Fulcrum don’t process environment variables, everything goes as an argument in command:.

Those instructions would have prevented 80% of the problems we hit.

Final server layout

/var/opt/docker/
├── bitcoin/          (bitcoind + fulcrum)
├── cloudflared/      (Cloudflare tunnel)
├── frank_fbi/        (email fraud analysis)
├── frank_mega/       (Mega clone)
├── frank_yomik/      (manga translation)
├── gitea/            (Docker registry)
├── immich/           (self-hosted Google Photos)
├── kaizoku/          (manga downloader + reader)
├── media/            (Plex + *arr stack)
├── mila/             (Discord bot)
├── monitor/          (Grafana + Prometheus)
├── ollama/           (local LLM with ROCm)
├── rip/              (HandBrake)
└── utils/            (Portainer, Vaultwarden, Syncthing, etc.)

/var/mnt/terachad/    (Synology NFS)
├── Bitcoin/data/     (blockchain, 785GB)
├── Downloads/        (torrents + nzbget)
├── Videos/           (Radarr movies + Sonarr series)
└── Ollama/models/    (model overflow if local disk fills up)

A warning about using AI to administer servers

I used Claude Code to speed up the migration. It created compose files, wrote backup scripts, configured the firewall, diagnosed SELinux problems. It worked well for my case: home server, only I use it, and I was reviewing every step.

But there are traps. Claude doesn’t know that :Z breaks SQLite unless you tell it. It doesn’t know that Fulcrum doesn’t accept env vars unless it’s already seen the Dockerfile. It will invent “better” volume layouts that break Plex because Plex stores absolute paths in its database.

If it were real production: don’t do this without review. Every compose file Claude generates, read it in full before applying. Every destructive command (rollback, delete, recreate), confirm manually. And have tested backups before you start. Claude is great for generating the first version and diagnosing errors, but the architecture decisions and the safety validations are yours.

The previous home server posts that may give additional context:

• My “Personal Netflix” with Docker Compose
• Accessing my Home Server with a real domain
• Protecting your Home Server with Cloudflare Zero Trust
• Installing Grafana on my Home Server
• Self-hosted Vaultwarden

More important: having a strong, random, exclusive and different password for EACH site, app or service you use. Never, ever, reuse a password. Why? Because every day one of these sites gets breached and passwords get leaked. If you search for yourself on Have I Been Pwned you’ll surely find accounts of yours that have already leaked. And if you share that same password across 10 other sites, someone has probably already gotten into your account by now.

Watch my video about security:

Being too lazy to use a password manager is just a dumb excuse from lazy people. And a programmer has zero right to use that excuse.

Omarchy 1Password

Anyone who has followed DHH since the early days when Macs were the best platform for developers remembers he was always a fan of 1Password, one of the apps that worked best on iPhones.

I migrated back to Android and Linux many years ago, in 2015. I used 1Password, but on Android the app was horrendous, ugly, and painful to use. That’s why I gave up on it and kept bouncing between several other apps. Google Authenticator, Lastpass (which I don’t recommend).

I know there are several other open source ones that people prefer, but I ended up settling on Bitwarden, which is BEAUTIFUL both on Android and as a browser extension. On top of that, it has open components and allows self-hosting.

More than that, it doesn’t make it hard to export all the data from your account and import it somewhere else. Unlike apps like Twilio Authy, which locks you out of everything and is why I stopped using it.

Bitwarden has a secure random password generator, supports 2FA TOTP, lets you store secure notes (mini texts) and a lot more. The apps integrate with phone biometrics, so it’s very convenient to use.

I still keep my Bitwarden inside Samsung’s Secure Vault, the Samsung Knox Vault. So there are 2 layers of security. And when I need to walk around somewhere suspicious, I don’t take my main phone, only a burner phone with nothing important logged in.

Look how beautiful it is. I like it and I’ve been using it for a few years now. My passwords range from 21 to 42 random characters, a different one for each site. I’ve updated all the passwords for everything more than once in the past few years, and all accounts have 2FA/TOTP registered inside Bitwarden too, which is how I managed to drop Authy.

Installing VaultWarden Self-Hosted

More important, the time has come to host my own Bitwarden server, vaultwarden, which is an open source alternative, code open on GitHub, so you can poke around yourself to learn or even contribute to it. Again, I bring everything up with Docker Compose on my Home Server, so I just needed to add this snippet to my file:

  vaultwarden:
    image: vaultwarden/server:latest
    container_name: vaultwarden
    restart: unless-stopped
    environment:
      DOMAIN: "https://bitwarden.fabioakita.dev"
      SIGNUPS_ALLOWED: "false"          # set to true only to create first user
      WEBSOCKET_ENABLED: "true"
      ADMIN_TOKEN: "iA...Q9"  # for /admin
      # SMTP (recommended)
      SMTP_HOST: "smtp.gmail.com"
      SMTP_PORT: "587"
      SMTP_SECURITY: "starttls"
      SMTP_USERNAME: "boss@akitaonrails.com"
      SMTP_PASSWORD: "xxx"
      SMTP_FROM: "boss@akitaonrails.com"
    volumes:
      - /home/akitaonrails/vaultwarden/data:/data
    ports:
      - "9999:80"
    healthcheck:
      test: ["CMD", "wget", "-q", "-O", "-", "http://localhost/health"]
      interval: 30s
      timeout: 5s
      retries: 5

Important:

• Create a long, random ADMIN_TOKEN. Use Bitwarden itself for that, keep it secret.
• Set SIGNUPS_ALLOWED to true only once, log into the local site and create your new account. Then change it back to false in the file and restart the container. That way nobody else will be able to create any new accounts.
• It’s recommended to configure a GMAIL SMTP to send notification emails. For that, you need to create an app password in Google.

To create a password exclusively for vaultwarden, go to Google App Passwords and generate a new exclusive secret password:

It will only show you that password once. Copy and paste it straight into the docker compose yaml. With email properly configured, every time someone logs in, it sends an email notification. That’s good to make sure nobody shady is messing with your server:

Now I just need to bring up the container:

docker compose -f utils-docker-compose.yml up -d

Routing the Cloudflared tunnel

As I already showed in the Cloudflared post now I want to be able to access it remotely, via secure tunnel, using a public URL like https://bitwarden.fabioakita.dev. To do that we have to edit that /etc/cloudflared/config.yml file and add this to the end of it:

  - hostname: bitwarden.fabioakita.dev
    service: http://192.168.0.200:9999
    originRequest:
      httpHostHeader: bitwarden.fabioakita.dev
      http2Origin: true
  - service: http_status:404

Then we have to restart the service and also add the route manually:

sudo cloudflared tunnel route dns homelab bitwarden.fabioakita.dev
sudo systemctl restart cloudflared

And done! It might take up to 5 minutes for DNS to sync and all that, but then just hop into your browser and you’ll see this:

If that SIGNUPS_ALLOWED is true there will also be a link to “Create new account”. In my case I already did that and turned it off, so there’s only login.

Importing your Data

Before moving on, let’s open the Bitwarden extension - which is still logged into the cloud version at bitwarden.com - and export all my data:

Just authenticate, then on the bottom icon bar you’ll find “Settings”, then “Vault Options” and finally “Export vault”. Leave the default JSON format (it will be an encrypted JSON) and download that file somewhere safe.

Now we can log back into our self-hosted version at https://bitwarden.fabioakita.dev and go through the same menu, but choose “Import”:

And done! Now my self-hosted version is identical to the cloud version I had been using until now.

Logging into the new self-hosted server

Now just log out of both the mobile App and the browser extension. Then on the login screen you’ll see the following option:

On the web extension the window doesn’t let you see that option, use the mouse scroll and scroll to the bottom of the window and you’ll see that small option. Choose “self-hosted” and a window will open to type the URL https://bitwarden.fabioakita.dev and done!

From here on the usage is exactly the same as before. The interface is the same, the options are the same. Configure biometrics, PIN, lock timeout and everything else the way you like it.

Do the same on the mobile app and done!

On this new account we created on the self-hosted server, don’t forget to enable 2FA too, which is the “Authenticator App” option under “Settings/Security”:

For all other accounts, we can register the 2FA inside Bitwarden. But for the master account of Bitwarden/Vaultwarden itself, we need to use a second app just for that.

I don’t know if it’s the best, but I’ve been enjoying using Aegis Authenticator on my Android:

It’s open source, the code is on GitHub, so it seems minimally trustworthy. I also keep it inside the Samsung Knox Vault. I only need it when I have to re-login into Bitwarden/Vaultwarden. To log into Google, Steam or any other account, the 2FA for each one lives inside Vaultwarden itself:

Conclusion

Now all that’s left is to log in with my old account at bitwarden.com, which is the “official” cloud version, and delete my account, to make sure nobody else has access.

And there, PERMANENTLY DELETED! My accounts no longer exist on some random server I don’t control, even if they were encrypted.

I’m pretty happy with this solution. Now my Vaultwarden is in a container, the home server data gets automatic backups to my NAS, everything is encrypted and access only happens via Cloudflare’s secure tunnel. The risk is very low and nobody has a chance of getting access to my data, because it’s no longer in the “cloud”.

My data is mine alone.

I had been too lazy to install Grafana back then, but I finally got around to it, and it was much easier than I expected, which is why I decided to share how I did it.

The idea is to spin up cAdvisor, which monitors container resources, and Prometheus, which scrapes system resource data, then consolidate the data from both into Grafana, a web application that lets you build graphical dashboards to monitor your systems. If you do it right, in the end you’ll have a screen like this:

Installation

Spinning this up on my home server is very simple. I do everything with Docker Compose, so:

# monitor-docker-compose.yml
networks:
  monitoring:
    driver: bridge

volumes:
  prometheus_data:
  grafana_data:

services:
  node-exporter:
    image: quay.io/prometheus/node-exporter:latest
    command:
      - --path.rootfs=/host
    pid: host
    restart: unless-stopped
    networks: [monitoring]
    volumes:
      - /:/host:ro,rslave
    security_opt: ["no-new-privileges:true"]
    read_only: true

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    restart: unless-stopped
    networks: [monitoring]
    ports: ["8080:8080"]   # optional: expose UI locally
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    security_opt: ["no-new-privileges:true"]
    read_only: true

  prometheus:
    image: prom/prometheus:latest
    restart: unless-stopped
    networks: [monitoring]
    volumes:
      - prometheus_data:/prometheus
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.retention.time=30d
      - --web.enable-lifecycle
    ports: ["9090:9090"]
    security_opt: ["no-new-privileges:true"]
    read_only: true

  grafana:
    image: grafana/grafana:latest
    restart: unless-stopped
    networks: [monitoring]
    depends_on: [prometheus]
    ports: ["3001:3000"]
    environment:
      GF_SECURITY_ADMIN_USER: admin
      GF_SECURITY_ADMIN_PASSWORD: change-me
      GF_USERS_ALLOW_SIGN_UP: "false"
    volumes:
      - grafana_data:/var/lib/grafana
      - ./provisioning/datasources:/etc/grafana/provisioning/datasources:ro
    security_opt: ["no-new-privileges:true"]

Pay attention to the local directory mappings. If you put /etc/grafana/prometheus.yml, for example, you have to change this line:

...
    volumes:
      - prometheus_data:/prometheus
      - /etc/grafana/prometheus.yml:/etc/prometheus/prometheus.yml:ro
...

Speaking of which, here’s the prometheus.yml:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs: [{ targets: ['prometheus:9090'] }]

  - job_name: 'node'
    static_configs: [{ targets: ['node-exporter:9100'] }]

  - job_name: 'cadvisor'
    static_configs: [{ targets: ['cadvisor:8080'] }]

And finally /etc/grafana/provisioning/datasources/datasource.yaml:

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

Be careful with port mappings too. The recommended approach is this:

ports: ["127.0.0.1:8080:8080"]

That’s because cAdvisor and Prometheus only need to be accessible to Grafana, inside the home server, and there’s no reason to bind on 0.0.0.0 and have them reachable from outside the server. That’s your call. Since I’m on a home LAN and there are no tunnels pointing to these services, it makes no difference to me.

If, like me, you already have a service sitting on port 8080, cAdvisor will fail. In that case, remap the ports and reconfigure Prometheus, or just remove the port line. You don’t need cAdvisor exposed externally.

Another detail: by default, Grafana comes up on port 3000, but I changed it to 3001 because I already had another service on that port. So once it’s up, I can access it at http://192.168.0.200:3001.

As I explained in the Cloudflared article, I route this address through Cloudflare to get a nice domain like https://grafana.fabioakita.dev to access it both locally and remotely, if I want. Go read those other articles afterward to understand how.

And that’s it, now just spin up the containers:

docker compose -f monitor-docker-compose.yml up -d

Done, super simple. I should have done this sooner.

As you can see in the Docker Compose file, when you open the site for the first time, the login is admin with password change-me. Obviously, the first thing to do is change the password:

Dashboards

Now we’re practically done. We can check whether Prometheus is working and integrated. Just go to the “Connections” menu and “Data Sources”:

Grafana can consolidate data from dozens of different sources, and Prometheus is only one of them. In a larger, more complex infrastructure, this list would be huge.

To build dashboards, you can start from scratch and assemble them widget by widget, but it’s easier to import dashboards that already exist. Just click the magnifying glass at the top, in the search box, and choose “Import dashboards”:

On that page you can either import a JSON file, copy and paste the JSON, or search by an ID already registered on grafana.com:

Right off the bat, I learned that there are 3 IDs that are useful for me:

• 1860 - Node Exporter Full - the screenshot at the beginning of this article, with several system metrics like CPU, memory, disk, network, etc. The most useful one for me.
• 13484 - Docker/cAdvisor Exporter - CPU and memory metrics, but broken down per container running on the server.
• 3662 - Prometheus 2.0 overview - a general health summary.

If you like these or other dashboards, just favorite them with a “pin” and you’re set.

For example, here’s the cAdvisor dashboard:

I haven’t explored other dashboards yet. If you have ID recommendations to import, drop them in the comments below!

With this, I have a quick and good-looking view of my home server’s resource usage, and I can spot anything weird happening right away. These services don’t seem to consume many resources, so I can leave them running alongside the other containers.

I have a nice Synology DS1821+ that also supports running Docker, but I prefer to keep the NAS doing only NAS duties and avoid risking my data by running other stuff on it. Having a single server that does everything isn’t a “problem” per se, but since I can, I prefer to separate things.

This Intel NUC runs a headless Ubuntu Server LTS (no desktop manager), which I access only via SSH (as I explained in the previous post about SSH). On it I run things like Portainer, Plex, Sonarr, Bazarr, qBitTorrent and much more.

The way it’s set up, I just added an entry to my desktop’s /etc/hosts like this:

192.168.0.100 synology.lan
192.168.0.200 plex.lan

From now on all my domains and IP addresses will be fake, obviously.

Anyway, this way I can open my browser and go to http://plex.lan:32400 to access my Plex, for example. But this has three disadvantages:

1. I can only use these hostnames on my desktop; from my laptop I’d need to edit the hosts file too. On top of that, I need to know the ports for each container.
2. I have no SSL for anything, so all authentication goes in plain text. It’s not a huge problem because the traffic is only internal on my local LAN. I haven’t been accessing it from outside the house.
3. If I ever want to access it from outside, none of this works.

There are dozens of ways to solve this problem, solutions involving WireGuard, or commercial solutions like TailScale and so on. But there’s a reasonably simple solution that solves all the problems at once: Cloudflare Tunnels.

The idea is to create secure tunnels via Cloudflare to access my services on my internal LAN WITHOUT having to expose ports outside my router or anything like that. My network stays isolated and unchanged.

1. Registering a Domain

The first recommendation is to register a real domain. Doing this lets you have real SSL certificates that Cloudflare itself manages and renews automatically.

I have a few domains registered and I’m not going to reveal which one I use for my Home Server. I’ll use this empty fabioakita.dev as an example just for this post.

Domains are cheap, around USD 12 per year, depending on the name you choose. Pick one that’s short and easy to type.

2. Installing Cloudflared

With the domain created, now we need to install the cloudflared service on the Home Server. It’s the one that connects to Cloudflare’s servers and opens the tunnel into my local server. Click the link for installation instructions, but on an Ubuntu Server 22.04 it goes like this:

# Add cloudflare gpg key
sudo mkdir -p --mode=0755 /usr/share/keyrings
curl -fsSL https://pkg.cloudflare.com/cloudflare-main.gpg | sudo tee /usr/share/keyrings/cloudflare-main.gpg >/dev/null

# Add this repo to your apt repositories
echo 'deb [signed-by=/usr/share/keyrings/cloudflare-main.gpg] https://pkg.cloudflare.com/cloudflared jammy main' | sudo tee /etc/apt/sources.list.d/cloudflared.list

# install cloudflared
sudo apt-get update && sudo apt-get install cloudflared

Now we need to log in:

# login
sudo cloudflared tunnel login

# add the homelab tunnel
sudo cloudflared tunnel create homelab

# create the config directory
sudo mkdir -p /etc/cloudflared

Use the same credentials you use to log in to the Cloudflare website. If everything goes well, at the end it will create a file at /root/.cloudflared/xxxxx-yyyy-zzzz...json, which holds your credentials, just like an SSH private key, so don’t throw it on a USB stick or some cloud drive. Keep it very safe. Let’s move it:

sudo mv /root/.cloudflared/*.json /etc/cloudflared
sudo chmod 600 /etc/cloudflared/*.json

Finally, we need to create a configuration file /etc/cloudflared/config.yml with all the hostnames we want for each of the containers, pointing to the correct ports. Here’s a snippet of my config, modified with fake IPs and the fake domain:

# /etc/cloudflared/config.yml
tunnel: homelab
credentials-file: /etc/cloudflared/xxxx-yyyy-....json
ingress:
  ...
  - hostname: plex.fabioakita.dev
    service: http://192.168.0.254:32400
    originRequest:
      httpHostHeader: plex.fabioakita.dev
  - hostname: immich.fabioakita.dev
    service: http://192.168.0.200:2283
    originRequest:
      httpHostHeader: immich.fabioakita.dev
  - hostname: overseer.fabioakita.dev
    service: http://192.168.0.200:5055
    originRequest:
      httpHostHeader: overseer.fabioakita.dev
  - hostname: syncthing.fabioakita.dev
    service: http://192.168.0.200:8384
    originRequest:
      httpHostHeader: qbittorrent.fabioakita.dev
  - hostname: radarr.fabioakita.dev
    service: http://192.168.0.200:7878
    originRequest:
      httpHostHeader: radarr.fabioakita.dev
  - hostname: sonarr.fabioakita.dev
    service: http://192.168.0.200:8989
    originRequest:
      httpHostHeader: sonarr.fabioakita.dev
  - hostname: bazarr.fabioakita.dev
    service: http://192.168.0.200:6767
    originRequest:
      httpHostHeader: bazarr.fabioakita.dev
  - hostname: prowlarr.fabioakita.dev
    service: http://192.168.0.200:9696
    originRequest:
      httpHostHeader: prowlarr.fabioakita.dev
  - hostname: portainer.fabioakita.dev
    service: http://192.168.0.200:9000
    originRequest:
      httpHostHeader: portainer.fabioakita.dev
  - hostname: grafana.fabioakita.dev
    service: http://192.168.0.200:3001
    originRequest:
      httpHostHeader: grafana.fabioakita.dev
  - service: http_status:404 # this has to be the LAST ONE

As you can see, it’s an extremely simple file. I think anyone can understand what it does. Modify it for your own needs.

After saving the file and going back to the command line, you have to manually run this command for each hostname:

sudo cloudflared tunnel route dns homelab plex.fabioakita.dev

But that’s a lot of work with this many hostnames, so I found it easier to create a script /etc/cloudflared/add.sh like this:

# run only once
TUN=homelab
CONF=/etc/cloudflared/config.yml
grep -E '^[[:space:]]*-[[:space:]]*hostname:' "$CONF" \
| awk '{print $3}' | tr -d '"' | tr -d "'" \
| while read -r h; do
    echo "adding $h"
    sudo cloudflared tunnel route dns "$TUN" "$h"
  done

Now just run: sudo sh /etc/cloudflared/add.sh, only once! So make sure your config file is correct first!

Finally, we can configure the service to restart by itself on every boot:

sudo cloudflared --config /etc/cloudflared/config.yml service install

sudo systemctl enable --now cloudflared

3. Accessing your services via the tunnel

If nothing went wrong, you’re done! You can go straight to your browser and test it:

How cool is that. Notice in the URL bar I typed https://immich.fabioakita.dev and it just worked. By the way, if you didn’t know it, Immich works identically to Google Photos, except entirely under your control! It has a smartphone app and syncs nicely.

More importantly, notice it’s on HTTPS, which was provided by Cloudflare for my entire domain. When I log in, it will go through an encrypted tunnel. And more than that: even if I’m on 5G, away from home, on my phone, I’ll be able to access my photos using this address!

From my phone it opens a secure tunnel to Cloudflare, and that cloudflared service on my home server also has a secure tunnel open to Cloudflare, so Cloudflare bridges the two tunnels and done! I have remote access without exposing my server directly to the internet! The best of both worlds!

Don’t forget to configure all your services with strong passwords (I generate them via Bitwarden, but any service works, 1Password or whatever):

4. Bridge to vlans

I don’t remember exactly why, but my Plex container alone gave me more trouble. That’s because it uses a different VLAN than the others. Again, I don’t remember why I did this at the time, but it looks like this:

  plex:
    #image: plexinc/pms-docker:latest
    image: linuxserver/plex:latest
    restart: unless-stopped
    environment:
      - TZ=America/Sao_Paulo
      - PUID=1026
      - PGID=1026
      - VA_DRIVER=IHD
      - PLEX_HW_TRANS_MAX=16
      - PLEX_MEDIA_SERVER_OPTS=--network-host
    volumes:
      - /config:/config
      - /data:/data
      - /transcode:/transcode
      - /Videos:/media
    devices:
      - /dev/dri:/dev/dri
      - /dev/bus/usb:/dev/bus/usb
    healthcheck:
      test: curl --connect-timeout 15 --silent --show-error --fail http://localhost:32400/identity
      interval: 1m00s
      timeout: 15s
      retries: 3
      start_period: 1m00s
    mem_limit: 4g
    cpus: 5.0
    networks:
      plex_macvlan:
        ipv4_address: 192.168.0.254
        mac_address: "xx:xx:xx:xx:xx:xx"

networks:
  plex_macvlan:
    external: true
    name: plex_macvlan

Because of this, cloudflared can’t route to that container. The way to fix it is by creating a bridge from that vlan to my ethernet.

sudo ip link add macvlan0 link enp3s0 type macvlan mode bridge
sudo ip addr add 192.168.0.200/24 dev macvlan0   # free IP in your LAN
sudo ip link set macvlan0 up

Obviously, replace enp3s0 with the correct name of your ethernet interface. Check with ip addr or ip link, for example. Just by running the commands above, if I go to the browser and type https://plex.fabioakita.dev it should already work. To make this permanent, we need systemd:

# /etc/systemd/network/80-macvlan0.netdev
[NetDev]
Name=macvlan0
Kind=macvlan

[MACVLAN]
Mode=bridge

# /etc/systemd/network/80-macvlan0.network
[Match]
Name=macvlan0

[Network]
Address=192.168.0.200/24
Gateway=192.168.0.1
DNS=192.168.0.1

And now, just restart the services:

sudo systemctl restart systemd-networkd
ip addr show macvlan0

The “correct” thing would be to have my Plex configured with network_mode: host, but, again, I don’t remember why it wasn’t, and I wasn’t in the mood to unconfigure everything just to find out. In case anyone has similar setups, this is how you bypass it.

Speaking of Plex, I had heard that you have to be careful with streaming via cloudflared, I don’t know if there are limits or extra costs. I found this article that talks more about it but I didn’t stop to look into it deeply. The only extra thing I did, on the Cloudflare dashboard, was add a rule to not do CACHING on the plex subdomain, to avoid additional costs.

Conclusion

As you can see, it’s extremely easy to have a home server, isolate everything with Docker, and then expose your containers via a secure tunnel for remote access. None of this means instant security. Make sure your containers are always up to date using services like Watchtower, make sure each service has strong, random, unique passwords - don’t use the same password for everything. Make sure your firewall is well configured and active, even on a LAN.

Many modern Wi-Fi routers let you create a separate VLAN for guest Wi-Fi. That way guests can use your Wi-Fi but can’t see your computers, which will be logically isolated on another VLAN. Check your router’s documentation. I use a TP-Link BE1900 Tri-Band Wi-Fi 7, but for most people it’s too expensive and overkill. Do your research. And don’t bother asking me about other models: I haven’t tested all of them to know the differences.

If you saw names like Radarr, Sonarr and don’t know what they mean, read the Servarr Wiki, which consolidates all the documentation for these services in one place. It’s the infrastructure that lets you have your own “Personal Netflix”.

Also, don’t ask me why I use Plex and not Jellyfin. It’s a personal choice, pick what’s best for you. I like Plex, I’ve used Jellyfin, and I don’t plan to switch.

I hate the idea of paying subscriptions for everything and owning nothing. I think it’s money thrown away. I prefer to OWN my own things under MY CONTROL. It’s more expensive, it’s more work. Too much convenience means you have control over nothing and always depend on others.

I hate depending on others.

But you don’t need to use these menus. Let me explain.

On a regular Arch Linux or Ubuntu, an amateur would install ruby or python doing something like this:

# arch
yay -S ruby python3
# ubuntu
apt install ruby python3

This is the WRONG way.

These commands will install the latest version of each language, and every time you update the system with yay -Syu or apt upgrade, they’ll pull in even newer versions.

The thing is, if you develop real projects, you don’t want the versions to change.

The version deployed on the cloud server and the version on your local machine MUST BE THE SAME.

If you’re using Ruby v3.0.7 and Node.js 22.19.0, you can’t be held hostage to the fact that when you update your system, out of nowhere Ruby 3.4.5 and Node 24.7.0 show up. Everything will start breaking and you’ll be super confused without knowing why. This is the most basic mistake a junior developer makes.

To avoid this, every project you’re working on MUST BE LOCKED TO FIXED VERSIONS.

The easiest way to do this in 2025 is using MISE. As I said, it already comes installed and activated on Omarchy, but if you’re using another distro just install Mise manually:

yay -S mise

# .bashrc
eval $(mise activate bash)

Adapt this for ZSH, Fish or whatever shell you’re using, read the documentation

Now navigate to your project directory, for example:

cd ~/Projects/AkitaOnRails.com

And lock the versions:

mise use ruby@3.2.3
mise use node@14.21.3

This will create a .tool-versions file that you should add to your repository’s git:

❯ cat .tool-versions
ruby 3.2.3
nodejs 14.21.3

❯ git add .tool-versions

And done, every time you cd into the directory, Mise will activate exactly the right version for that project. Look:

AkitaOnRails.com [ master][$!?⇡][ v24.7.0][💎 v3.2.3]
❯ ruby -v
ruby 3.2.3 (2024-01-18 revision 52bb2ac0a6) [x86_64-linux]

AkitaOnRails.com [ master][$!?⇡][ v24.7.0][💎 v3.2.3]
❯ which ruby
/home/akitaonrails/.local/share/mise/installs/ruby/3.2.3/bin/ruby

Notice that, if you’re using the Starship prompt I explained in the previous post, with a preset like Pure Prompt, it shows the version of each language in the prompt so it’s easy to see if you’re on the wrong version.

In this case, if you read carefully, you’ve probably already noticed that YES, I’M ON THE WRONG VERSION of Node.js. Look how the prompt says I’m using Node 24.7.0 but the .tool-versions file asks for 14.21.3.

The reason it’s “missing” is because this machine was recently installed, but the project is old.

So let’s check:

AkitaOnRails.com [ master][$!?⇡][ v24.7.0][💎 v3.2.3]
❯ mise list
Tool    Version            Source                                              Requested
go      1.25.0             ~/.config/mise/config.toml                          latest
node    14.21.3 (missing)  /mnt/data/Projects/AkitaOnRails.com/.tool-versions  14.21.3
node    22.18.0
python  3.13.7
ruby    3.2.3              /mnt/data/Projects/AkitaOnRails.com/.tool-versions  3.2.3
ruby    3.4.5

See how in the mise list output it says 14.21.3 is “missing”. So let’s install it:

AkitaOnRails.com [ master][$!?⇡][ v24.7.0][💎 v3.2.3]
❯ mise use node@14.21.3
mise /mnt/data/Projects/AkitaOnRails.com/.tool-versions tools: node@14.21.3

AkitaOnRails.com [ master][$!?⇡][ v14.21.3][💎 v3.2.3]
❯

With the mise use command we can ask it to install and use a particular version. If it’s not already installed, it will install it. Notice how after that the Starship prompt changed to reflect that we’re now using 14.21.3. That’s the right way.

If you’re not necessarily in a project directory with locked versions, you can still run commands using a specific version, like this:

mise exec ruby@3.2.1 -- ruby ...

Say you want to generate a Rails project on a specific Ruby version:

mise exec ruby@3.2.3 -- rails new todo-exercise

The same goes for generators in Node, like Next.js or any other framework.

I haven’t checked whether LazyVim on Omarchy comes configured this way, but in case it doesn’t, for NeoVim to support Mise you need to add this:

-- Prepend mise shims to PATH
vim.env.PATH = vim.env.HOME .. "/.local/share/mise/shims:" .. vim.env.PATH

Just put this in ~/.config/nvim/lua/config/mise.lua and it should load the correct path. To integrate with other editors, check this documentation. There are TONS of advanced options and it’s worth reading about them in the Dev-Tools section, but the basics are what I listed here.

And databases??

The same goes for databases. Every project should be locked to a specific version identical to what’s running on the production server and should never depend on operating system packages. You can use mise for this but the most recommended option is to use Docker. Every project MUST have a docker-compose.yml to spin up only the databases. You don’t need to spin up containers for languages, for that it’s better to use Mise.

If you don’t understand Docker, I made videos teaching it:

Adding up Mise + Docker Compose + LazyVim, that’s everything you need to be a productive web developer, while also using your machine’s resources efficiently. Every developer NEEDS to learn this combo of options. Omarchy already brings it all pre-installed, which is why it’s been my recommendation for every beginner.

Don’t worry: Arch Linux (with archinstall) is as easy to install as an Ubuntu or Linux Mint, and Omarchy’s Hyprland is much prettier and smoother than MacOS. The best of both worlds. Learn it today!

The Problem

Before, to contribute to the blog you had to:

1. Install Hugo Extended
2. Install Go
3. Install Ruby
4. Set up all the dependencies
5. Generate the post index manually

This created barriers for new contributors and could cause conflicts with other installations on the system. A nightmare.

The Solution: Docker

Now all you need is Docker and Docker Compose. Everything else runs inside isolated containers.

Quick Setup

# Clone the repository
git clone https://github.com/akitaonrails/akitaonrails.github.io.git
cd akitaonrails.github.io

# Start the environment
./scripts/dev.sh start

# Open the blog
open http://localhost:1313

Done! The blog is running and you can start contributing.

Useful Commands

The project includes scripts that make development easier:

# See all available commands
./scripts/dev.sh help

# Create a new post
./scripts/dev.sh new-post "Título do Meu Post"

# View server logs
./scripts/dev.sh logs

# Generate the post index
./scripts/dev.sh generate-index

# Stop the environment
./scripts/dev.sh stop

Creating a Post

Creating a new post is dead simple:

./scripts/dev.sh new-post "Como Contribuir no Blog do AkitaOnRails"

The script automatically:

• Creates the directory structure with the current date
• Generates an index.md file with basic front matter
• The Hugo server reloads automatically

Structure of a Post

The created file will have this structure:

---
title: "Título do Post"
date: "2025-08-29"
draft: false
description: "Descrição do post"
tags: [tag1, tag2]
categories: [categoria]
---

Conteúdo do post aqui...

Advantages of Docker

For Contributors

• Fast setup: Docker is the only requirement
• Isolated environment: Stays out of the way of your local system
• Hot reload: Changes show up automatically
• Reproducible: Works the same way for everyone

For the Project

• Fewer issues: Consistent environment
• More contributors: Lower barrier to entry
• Maintenance: Easy to update dependencies

How It Works

The Docker environment includes:

• Hugo Extended: Static site generator
• Go: For Hugo modules
• Ruby: For the index generation script
• Volumes: For hot reload of files

The docker-compose.yml maps the local directories into the container, so when you edit a file, Hugo detects the change and reloads automatically.

Local Development (Alternative)

If you prefer to skip Docker, you can still install everything locally:

# Generate the index
cd content && ruby generate_index.rb

# Run the server
hugo server --logLevel debug --disableFastRender -p 1313

But Docker is way more practical!

Next Steps

1. Test the environment: Run ./scripts/dev.sh start
2. Create a post: Use ./scripts/dev.sh new-post
3. Explore the code: See how to customize layouts
4. Contribute: Open a Pull Request

Tip: The Docker environment automatically saves your changes and reloads the server. Just edit the files and watch the changes in real time! 🚀

More than a year ago, when I made my last videos about A.I. and on the podcasts, I always said I ran everything on my own machine. But I never detailed how. So today I fixed that with several blog posts and little projects on GitHub with Dockerfiles so you can run it on your own machine, without configuring anything difficult.

I finally made a Dockerfile for ComfyUI, which is the best graphical interface for editing image generation workflows. If you’ve ever worked with Davinci Resolve Fusion or Blender Geometry Nodes and the like, workflows are basically node configurations to process your images, audio, video and pipe them through processing steps, not just A.I. ones. It’s like programming visually. That’s why it’s a much more complicated tool than the FramePack I showed in the previous post. You don’t just click and go.

Recently everyone went hysterical with the ChatGPT updates and how it “OH MY GOD” can generate Ghibli anime style images, and they kept spamming the world with idiotic anime-style memes. Online hysteria is really annoying. But as I said, as impressive as it is, that’s all it is: a meme generator. You can’t use it professionally.

I’m tired of trying to upload my own photos, or even generic photos, fictional characters, sometimes drawings of mine, and randomly ChatGPT just refuses to generate the image. Claiming privacy, security, copyright and whatever else it can invent as an excuse. You can’t rely on this for work. Anyone who’s a professional should have full control over the process, and that’s what I’m going to teach today.

To start, clone my GitHub repository and follow the README instructions or do this:

git clone https://github.com/akitaonrails/ComfyUI-Docker-CUDA-preloaded.git
cd ComfyUI-Docker-CUDA-preloaded
docker compose build

Since it’s more complex, this time I had to put together a Docker Compose. So to bring it up, after the build finishes, just do this:

docker compose up

Now it’s going to take a while, because by default it will download almost 500GB of pre-trained models so it can do almost anything image-related. Yes, to play with A.I. locally you need infrastructure. Otherwise, rent a beefy machine with lots of storage on a RunPod or similar and run it remotely. You can download fewer models, I’ll explain that next.

On Docker Compose. I recommend avoiding the “-d” option to bring it up as a daemon, because it’s good to reserve a terminal to keep watching the outputs (you can run docker compose logs --follow if you want, but it’s easier directly).

ComfyUI is a more complex program than average, much more so than something like FramePack. It supports installing “extensions” (plugins, addons, call them what you want). And honestly on its own it doesn’t do much, you need to install several extensions for it to start being really useful. Every now and then one of these extensions doesn’t handle errors properly and blows up in the console, something like this:

It’s always useful to keep an eye on the console to find out if it tried to load a file that doesn’t exist. Then you can grab the file name and search Google, download it and put it in the right place. Or if it’s a really buggy extension that’s better off removed from the “custom_nodes” directory, and so on. Without watching the console, it doesn’t show up in the web interface and you’ll be left in the dark.

A lot of the models and extensions I based on the Ultimate ComfyUI version 3 configuration recommended by Aitrepreneur, again. Watch this video of his to get all the details and everything that’s possible to do. He has the scripts for Linux, Windows, and the workflows in his paid Patreon version and, of course, I won’t share the workflows because they’re not mine. In theory, my config should be able to do what he teaches.

Aitrepreneur’s V3 workflow is really badass. But I can’t share it because it’s closed material on his Patreon, but here’s a screenshot of my ComfyUI with all of his stuff loaded:

But the config itself, I redid it entirely to work in this Docker. You can load any other open workflow you find on a Reddit or wherever.

Here’s what I did:

Models

In the project there has to be a “models” directory, if there isn’t one, create it, because docker compose will map it inside the container. When it starts, it’ll run the file init_models.sh. This script will load a configuration file you can edit later: models.conf. It’s huge and a piece of it looks like this:

...
[CONTROLNET]
https://huggingface.co/XLabs-AI/flux-controlnet-collections/resolve/main/flux-canny-controlnet-v3.safetensors
https://huggingface.co/XLabs-AI/flux-controlnet-collections/resolve/main/flux-depth-controlnet-v3.safetensors
https://huggingface.co/brad-twinkl/controlnet-union-sdxl-1.0-promax/resolve/main/diffusion_pytorch_model.safetensors|controlnet-union-sdxl-1.0-promax.safetensors
https://huggingface.co/lllyasviel/ControlNet-v1-1/resolve/main/control_v11p_sd15_canny.pth
https://huggingface.co/lllyasviel/ControlNet-v1-1/resolve/main/control_v11f1p_sd15_depth.pth
https://huggingface.co/lllyasviel/ControlNet-v1-1/resolve/main/control_v11p_sd15_openpose.pth
https://huggingface.co/Aitrepreneur/FLX/resolve/main/Shakker-LabsFLUX1-dev-ControlNet-Union-Pro.safetensors

[STYLE_MODELS]
https://huggingface.co/Aitrepreneur/FLX/resolve/main/flux1-redux-dev.safetensors

[SAMS]
https://huggingface.co/Aitrepreneur/FLX/resolve/main/sam_vit_b_01ec64.pth

Each section maps to a sub-directory, so “[SAMS]” will map to “models/sams” and download the binaries to the right places. There are some special ones in the “[GIT_REPOS]” section, for those that can be cloned via git, or “[CUSTOM]” for files you need to download later inside those cloned projects, for example. I don’t yet know if it can be simplified, but I picked this up from Aitrepreneur’s config.

The ideal is to always keep this list organized with models you find later. If you want to contribute, that would also be cool. The reason is that everything is scattered across all kinds of sites out there, you download manually, then move to the directory inside “models”. Tomorrow, if you reinstall your machine, how are you supposed to remember all the sites? There are dozens of available models.

In the next post I’ll try to summarize what the hell “checkpoints”, “diffusion models”, “loras”, “controlnet” and all the rest are. But for now, the file extension doesn’t say much. “.pt” for example only means it’s a file PyTorch can open, but it doesn’t tell you whether it’s a clip or a text encoder, for example. Same thing with “.safetensors”, it can be anything, and ComfyUI needs each file to be in the correct sub-directory in order to use it, you can’t just go around moving files randomly.

The “init_models.sh” script runs every time Docker is restarted and is smart enough to only download URLs that aren’t already downloaded in the “models” directory. That’s why it always runs when the container comes up, in the entrypoint. That’s one of the advantages of my config: it has automatic maintenance.

Extensions and Dependencies

I made a second script called init_extensions.sh, which also runs at the container’s entrypoint and which you shouldn’t touch, but it loads another configuration file called extensions.conf and that one, yes, you can mess with and add new extensions later. A piece of it looks like this:

[EXTENSIONS]
https://github.com/ltdrdata/ComfyUI-Manager.git
https://codeberg.org/Gourieff/comfyui-reactor-node.git
https://github.com/ssitu/ComfyUI_UltimateSDUpscale.git
https://github.com/kijai/ComfyUI-LivePortraitKJ.git
https://github.com/PowerHouseMan/ComfyUI-AdvancedLivePortrait.git
...

Here I declare all the extensions that add new functionality on top of ComfyUI, in particular new NODES, which is why they live in the “custom_nodes” directory, which is a volume external to Docker and therefore persistent, meaning every new extension survives container restarts.

But there’s still a BIG catch: Python Dependencies. A nightmare.

The Docker image build does a bunch of pip install and since I’m using Virtual ENV, everything gets installed in /venv inside the container, as it should.

However, the extensions that get installed outside the build - every time the container restarts - tend to be Python code and also have more dependencies. Extensions are GitHub projects and may be changing right this minute. If they change and you do a git pull to update, you need to run pip install -r requirements.txt. But if you have dozens of extensions, running pip install for each one on every Docker restart takes a long time to do almost nothing. As I said, a nightmare.

My solution - to brag a bit - fixes a good chunk of this maintenance. When the init_extensions.sh script runs, if an extension from the list is already installed, it’ll skip it. Then it’ll write to the ./custom_nodes/.last_commit/*.commit directory the latest commit of each extension. When Docker restarts, it’ll do a git fetch and check whether the latest commit hash has changed.

Only if it has changed will it run pip install for that extension that got an update, and skip all the others that haven’t changed since the last restart, saving A LOT of time and avoiding having to manually update things. Since I’m using VENV, everything will stay persisted in the “/venv” volume. Yes, it’s a VOLUME, because it changes after the build with each extension update.

This way, everything in the image gets copied into that volume, and everything the extensions install on the side also goes into that volume. Every time you restart, it all comes back the way it was. That was the solution I found and so far it seems to work just fine.

But it’s only a matter of time until some poorly made extension installs dependencies that break other dependencies, or an actual case I went through: there was an extension that crashes with no error message if you’re using Python 3.11, the Zoe Depth Map from the comfyui_controlnet_aux extension, one of the packages that seems to be popular.

I hadn’t realized I’d started using a Docker base image of ubuntu22 instead of ubuntu24, I was afraid of using something too new and breaking things, but I ended up using something too old and it broke just the same. Managing versioning is a pain. Anyway, I changed the Dockerfile to ubuntu24.

But in the external VENV volume now there were a bunch of dependencies pinned to Python 3.11. No problem, just delete the volume:

docker compose down # make sure it's stopped
docker volume rm comfyui-venv
rm ./custom_nodes/.last_commits/*.commit

And done, the VENV will start over empty, fill up with the correct Python 3.12 dependencies from the image, and when the container starts my init_extensions.sh script won’t find “.last_commits” and will force a pip install -r requirements.txt of all the extensions, now with Python 3.12. Super clean and reliable.

If even so you get an error, then it might be a bug in a specific extension, just go to its directory in “./custom_nodes” and delete it manually before restarting the container.

I think that’s the least complicated thing you can do in this case. This docker wasn’t built to run on its own on a server, it’s meant to run on your own local machine, where you’re monitoring the terminal output anyway.

Oh yes, if a new version of ComfyUI comes out then you have to rebuild the image and delete the VENV volume. The image is static, don’t forget. Inside, the Dockerfile does a new git clone and grabs the newest commit. Then it’ll install newer python packages, so you need to wipe the VENV volume and have the extensions reinstall their dependencies afterwards too. It’s a huge pain, but that’s just how it works.

Conclusion

With all that done, with a simple docker compose up I can have a stable ComfyUI environment isolated from the rest of my system, without dirtying anything with thousands of dubious dependencies. When I’m done I just do docker compose down or “ctrl-c” in its console and that’s it, my system stays clean.

On top of that, directories like “models” and their 500GB of models stay separate, you can back them up elsewhere if you need to. And if you want to bring it up on another machine, just take only those directories and bring up Docker Compose there and it has to come up exactly the same way, zero-setup. That was my goal in Dockerizing ComfyUI, FramePack, Video2K: to have everything portable, isolated and clean to bring up on any machine later without having to run sketchy installation scripts directly on my machine.

On my X I already showed my ComfyUI working. I loaded some random workflow I found on the Web and look at this:

In this workflow I upload a reference image and I can write a prompt for how I want to transform it:

With ComfyUI I have control over every step of the process, for example, this workflow can separate things like Depth Map and Edge Contour from the original image, to use as context before making the final image, ensuring a much more precise result:

And here’s a result I already got using the “waiNSFIllustrious” model that this workflow suggests, which is a model trained to generate anime-style images (there are dozens, for various different styles).

Switching the model to Mistoon, based on Stable Diffusion (SDXL), here’s another result, with the same prompt, in the same workflow:

And I can keep tuning dozens of parameters inside this workflow, swap vaes, text encoders, controlnets and much more until I get exactly the result I want. That’s the power of ComfyUI and its Custom Nodes and Workflows. That’s what I’ll try to explain the basics of in the next post.

Since I was already on a roll packaging this kind of tool in Docker, I decided to make another little project with a Dockerfile to run on my Manjaro Linux with an RTX 4090. The good thing about Aitrepreneur is that he has a Patreon you can subscribe to in order to contribute (it’s cheap) and he always uploads scripts to run on RunPod in case you don’t have a beefy machine or GPU, or .BAT scripts to run on your Windows.

My little project is on this GitHub and using it is very simple:

git clone https://github.com/akitaonrails/FramePack-Docker-CUDA.git
cd FramePack-Docker-CUDA
mkdir outputs
mkdir hf_download

# Build the image
docker build -t framepack-torch26-cu124:latest .

# Run mapping the directories outside:
docker run -it --rm --gpus all -p 7860:7860 \
  -v ./outputs:/app/outputs \
  -v ./hf_download:/app/hf_download \
  framepack-torch26-cu124:latest

This will build the Docker image and run it. And the first time you can see it will download a TON of models, HunyuanVideo, Flux and more. Get ready for over 30GB. But it’s only the first time because I map the download directory outside, so if you restart the container it will already have everything next time.

When it’s done, just go to http://localhost:7860 and you’ll see this super simple interface:

The controls are very simple:

• Disable the “Use TeaCache” option which makes it faster but then you risk getting those A.I. defects like hands with extra fingers and stuff like that.
• Total Video length can go up to 2 minutes, but only do that if you really have patience because it takes A LOT of time! I think it’s better to start by testing with short 5 to 10 second videos first.
• If your GPU has little VRAM you’ll be forced to mess with “GPU Inference Preserved Memory (GB) (larger means slower)”. In my case with 24GB of VRAM I left it as is, but look how it consumes resources while running:

Yes! 100% of the GPU and almost 100% of the VRAM, pulling more than 350W from the wall! (screw the environment! lol) The CPU stays chill, because this process is made to really MASSACRE the GPU. If you don’t have a good GPU, rent a machine on RunPod like I said before.

The 10-second video at the start of the post is a photo of one of my favorite action figures in my collection, Link from Breath of the Wild. And this is something that got me excited. This is the original photo that served as the basis for FramePack to make the video:

Making videos of photos of people is dull and tedious, but now that I’m learning to model 3D to print on my Bambulab, imagine then being able to make it animated too?

To animate in Blender and render also takes a lot of processing and time. But with this I can quickly prototype an animation and see how it looks, to decide if it’s worth making a better version in Blender afterwards. It opens up many possibilities for experimentation!

As impressive as this seems, it doesn’t do JUST ANYTHING, I had several experiments that went wrong and the movement it makes is quite limited, after all it only has a single photo to use as a reference and you can’t make movements that are too abrupt, there are limits. Even so it’s very fun to play with. The fact that it can identify more than just human beings is a big “plus” in my opinion, because that’s what got me more interested.

OH YES! Remember the other blog post I published today about doing UPSCALE TO 4K?? You can use it here too, take a video that FramePack generated that you really liked and upscale it to 4K to have even more definition! Then it’s at quality you can even use in video editing.

2-minute videos must take hours to make. 30 seconds took almost 1 hour. The best thing is to stick with 5-second videos to play and experiment, just that already takes nearly half an hour. Here are some more examples taken from photos of items in my collection (the ones you used to see in the background of my videos). Have fun!

This is an Ultraseven over 30cm tall that I found and brought directly from a used goods shop in Tokyo last year, one of my favorites especially because I watched it a lot in the early 80s when I was a kid.

Speaking of nostalgia, this is another awesome item in my collection, a super detailed Jiraya made by Iron Studios. And animated it actually turned out pretty cool. What did you think?

But now my FAVORITE, those who follow me on Instagram have seen that for the past few weeks I’ve been dedicating myself to improving my 3D techniques and also drawing and I made a concept of Mandalorian + Judge Dredd:

.png?disposition=attachment&locale=en)

Check out my other drawings on my Instagram afterwards, but anyway, I needed to know if FramePack was capable of animating drawings too and YES, IT CAN!! LOOK HOW B4D4SS!!! This one really impressed me, it gives a new dimension to my stuff!

Yes, all of this is quite heavy, but the advantage is that I can experiment INFINITELY, because everything runs locally. On someone’s commercial product in the “cloud”, I’d have to pay some subscription and they’d limit me, it would be impossible to keep experimenting whenever I wanted. Now I can do everything I want, whenever I want, however I want, without anyone or anything bothering me.

By the way, if you can, watch that “Boba Dredd” clip in full screen on a big monitor. Did you notice how sharp it is? That’s because I ran it through Video2K and upscaled to 4x the resolution that FramePack gave me. Combining these tools you can do REALLY interesting things, let me know later if you managed to use it and what cool stuff you made!

For collectors, the problem is that a lot of anime from the 90s and earlier never came out, and never will, on Blu-Ray (1080p) or UHD (4K). Only the most famous ones get the “remaster” treatment (taking the original master tapes and recapturing them at higher resolution). So a ton of old anime is stuck in the DVD era (480p) or VHS (480i).

Not every old anime can be upscaled, it can end up looking too “washed out”. Anyone who downloads torrents has probably seen badly done upscales. But I wanted to be able to test it myself, on my local machine, with my old files. Here’s a small example clip:

It’s a piece of the 1st episode of the legendary Rurouni Kenshin from 1996. Yes, today we have a remake, which they’re redoing from scratch. But for those of us who feel nostalgic about the original, there’s no replacement. The DVD quality isn’t horrendous, you can still watch it just fine today as is. But I’m curious to see how it looks in 4K.

I didn’t do any long research, but a year ago I had stumbled onto Real ERSGAN. That project has been discontinued for about 4 years, so I don’t know what the most modern option is today. But their pretrained models are still useful.

There are still projects that use this model, the one I know of is Video2K. On Windows it apparently even has a nice little graphical interface, but the advantage for Linux is that it runs in Docker. An example would be like this:

docker run --rm --gpus all \
  -v "$PWD/videos_in":/input \
  -v "$PWD/videos_out":/output \
  ghcr.io/k4yt3x/video2x:6.4.0 \
  -i /input/old_anime.mp4 -o /output/old_anime_4K.mkv \
  -p realesrgan -s 4 --realesrgan-model realesr-animevideov3

If you have your NVIDIA set up with the Container Toolkit, CUDA and all that (ask ChatGPT), Docker can map the device straight into the container. This makes it much easier to keep my Linux from getting dirty with dependencies and loose configs scattered around. I don’t do anything of this kind outside of containers anymore.

But I ran into a problem. If you’re using an AMD GPU, just doing this should already work, because this Docker image actually uses Vulkan and not CUDA. In my case I have a primary integrated AMD GPU and a secondary NVIDIA RTX 4090. I tried passing it directly:

❯ docker run --gpus all --privileged \
  --device=/dev/nvidia0 \
  --device=/dev/nvidiactl \
  --device=/dev/nvidia-uvm \
  --device=/dev/nvidia-modeset \
  --runtime=nvidia \
  -e VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/nvidia_icd.json \
  -e VK_LAYER_PATH=/usr/share/vulkan/explicit_layer.d \
  -v "$HOME/Downloads/Video2K/videos_in":/input \
  -v "$HOME/Downloads/Video2K/videos_out":/output \
  --rm -it \
  ghcr.io/k4yt3x/video2x:6.4.0 \
  -i "/input/(B-A)Rurouni_Kenshin_-_01_(A5D3CB57).mkv" \
  -o "/output/(B-A)Rurouni_Kenshin_-_01_(A5D3CB57)_upscaled.mkv" \
  -p realesrgan -s 4 --realesrgan-model realesr-animevideov3

But it just won’t work. If I only say “–gpus all” it grabs the iGPU by default. Upscaling a video ends up consuming 100% of my CPU, all 32 threads, but my NVIDIA GPU just sits there watching, doing nothing. If I try to pass “/dev/dri/card0” or “/dev/nvidia0”, it errors out and can’t even start.

So from here on the solution is only for those who have NVIDIA:

CUDA Dockerfile

This time I decided not to be lazy and organize everything into a little project on Github. Clone it directly from there Batch Anime Upscaler Video2K Docker CUDA. Just do this:

git clone git@github.com:akitaonrails/Batch-Anime-Upscaler-Video2K-Docker-CUDA.git
cd Batch-Anime-Upscaler-Video2K-Docker-CUDA
mkdir input # put your old videos here
mkdir output

docker build --build-arg HOST_UID=$(id -u) --build-arg HOST_GID=$(id -g) -t anime-upscaler:latest .

If nothing goes wrong, this will build a Docker image with everything you need, then it’s just a matter of placing your videos in the subdirectory I mentioned, or any other you want, and configuring the run command:

 docker run --gpus all --rm \
  -v "./input":/input \
  -v "./output":/output \
  -e HOST_UID=$(id -u) -e HOST_GID=$(id -g) \
  anime-upscaler:latest

And that’s it. It takes a while. With my Ryzen 7950X3D plus RTX 4090, it processes at a rate of around 13 frames per second. So less than half the playback speed. If a video is 20 minutes long, it’ll take more than 40 minutes to process each one. It’s a good idea to set aside plenty of time, and that’s also why this is best reserved for videos you really want to watch.

The original Video2K script supports several parameters, which I left exposed as environment variables in the Dockerfile:

MODEL="${MODEL:-realesr-animevideov3}"
SCALE="${SCALE:-4}"
TILE="${TILE:-0}"
DENOISE="${DENOISE:-1.0}"
NUM_PROC="${NUM_PROC:-1}"

Then you can change them like this:

docker run --gpus all --rm \
  -e DENOISE=0.5 \
  -v ... \
  anime-upscaler:latest

As you can see, by default we use the pretrained model “realesr-animevideov3”, which is less detailed than the 6 billion parameter model but is much more stable between frames, which matters more for video. Most of the other models it supports, like “RealESRGAN_x4plus” or “ESRGAN_x4”, are meant for photos and images and not for video. Video demands temporal coherence, which is quite hard.

At most it’s worth trying the “realesr-general-x4v3” model, which they say is a bit lighter, in case your machine is struggling too much to run the default. But then you’ll have to modify the Dockerfile to download that model and rebuild:

# === OPTIONAL: Preload additional Real-ESRGAN models ===
# Uncomment any models you want available in the container at build time

# Anime 6-block model (better for anime stills, line art)
# RUN wget -q https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/RealESRGAN_x4plus_anime_6B.pth \
#     -O /opt/Real-ESRGAN/weights/RealESRGAN_x4plus_anime_6B.pth

# Original RealESRGAN 4x general-purpose model
# RUN wget -q https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/RealESRGAN_x4plus.pth \
#     -O /opt/Real-ESRGAN/weights/RealESRGAN_x4plus.pth

# RealESRGAN 2x model (lighter upscale)
# RUN wget -q https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/RealESRGAN_x2plus.pth \
#     -O /opt/Real-ESRGAN/weights/RealESRGAN_x2plus.pth

# ESRNet (non-GAN model, ultra-smooth)
# RUN wget -q https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/RealESRNet_x4plus.pth \
#     -O /opt/Real-ESRGAN/weights/RealESRNet_x4plus.pth

# General-purpose lightweight video-friendly model (with denoise control)
# RUN wget -q https://github.com/xinntao/Real-ESRGAN/releases/download/v0.3.0/realesr-general-x4v3.pth \
#     -O /opt/Real-ESRGAN/weights/realesr-general-x4v3.pth

# Anime video model (default in your setup)
# RUN wget -q https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesr-animevideov3.pth \
#     -O /opt/Real-ESRGAN/weights/realesr-animevideov3.pth

# Original ESRGAN (2018) model
# RUN wget -q https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/ESRGAN_x4.pth \
#     -O /opt/Real-ESRGAN/weights/ESRGAN_x4.pth

I haven’t tested all of them, but for anyone who wants to, I think those are the links. Change it in the Dockerfile, run the build, change the MODEL variable, and test on your videos.

What’s worth tweaking are things like DENOISE, which is set to “1.0”, the maximum, but that can leave the image too “stretched”. It’s worth trying smaller values, like “0.5”. It varies case by case, only by testing.

Another factor is the scale, SCALE, where the default is “4x”, which turns 1080p into 4K or 720p into 1440p. But sometimes you don’t need to go that high, and there’s no point because the original image already has very little detail anyway. Maybe 2x is enough. The script supports 2x, 3x, 4x but it caps at 4K resolution, so there’s no point trying to turn a video that’s already 4K into 8K, for example.

My RTX 4090 and my Ryzen 9 are quite powerful. Processing just one video, I don’t see them breaking a sweat at all:

Clearly you could run more videos in parallel. You have to be careful when it’s unbalanced. If the CPU climbs close to 100% just to separate the frames, having spare GPU capacity doesn’t help. With both having headroom, you can make the script spin up more than 1 process on the GPU and send frames in parallel with the “-e NUM_PROC=2” option. Look at how it now uses more of the GPU and the CPU still has room to spare. Maybe 3 is the maximum, but you have to test, it’s not constant the whole time.

This can dramatically reduce the total time. You can choose between increasing the GPU processes via the script or having two separate video directories and spinning up two parallel containers of this program to run at the same time, whichever works best for you.

Final Result

Let’s see how that first clip turned out with 4x upscale and 1.0 denoise:

The difference is HUGE, but whether it ended up “GOOD” is going to be VERY subjective. Personally I didn’t find the original bad to watch. Maybe tweaking DENOISE down a bit, or SCALE to 2x instead of 4x, would already be good enough, I don’t know.

For comparison, here’s another clip in the original old version:

And again, how it looks after the 4x upscale:

At least it’s working as it should, from here on it’s a matter of doing some tuning. If anyone has more tips or knows of newer open source upscaling models or projects, send them in the comments below!

Before someone asks, why not use NVIDIA’s DLSS 3, which they use to upscale game resolution? Answer: because they don’t expose that functionality as an API you can access, it’s only for their proprietary drivers in game profiles. They turn it on and tune it case by case. There’s no general API in the CUDA lib.

Post-publication Update

I decided to test the RealERSGAN_x4plus_anime_6B model, which is much heavier than the realesr-animevideov3 I uploaded to the repository.

Change the Dockerfile and add this after the wget for the other model:

RUN wget -q https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth \
     -O /opt/Real-ESRGAN/weights/RealESRGAN_x4plus_anime_6B.pth

This model can’t be run in parallel even on my RTX 4090, which stays constantly at 100%. If before it ran at around 13 frames per second, with this one it only runs at 3 frames per second. It’s 4x heavier and only for really beefy GPUs. If you don’t have something like that, stick with “realesr-animevideov3”, or with the even lighter (lower quality) “realesr-general-x4v3”. These models depend heavily on GPU power and especially on the amount of VRAM.

To run it, add “-e MODEL=RealESRGAN_x4plus_anime_6B” to the “docker run” command.

See how the same clip from before turned out using this model:

For having spent 4x more of my machine’s resources to process it, I didn’t find the final result that different, and in some parts I think it even has more defects. So that’s it, the result will vary, you have to test. It depends a lot on the quality of the original anime, but just because a bigger and theoretically “better” model exists doesn’t mean every result will turn out better.

I went poking around GitHub and there is an Awesome Image Colorization page with a bunch of research papers. Very useful for researchers but totally useless for me, since I am not going to build one from scratch lol. There were links to projects like ChromaGAN, but that one has been discontinued for about 5 years. And several other projects I saw were also discontinued ages ago. I do not know why there is so little open source still actively maintained.

Luckily I bumped into a promising one: DDColor.

Let’s clone the repo. And to keep going, I prefer to do everything inside Docker. Downloading Python dependencies always messes up my system and leaves a bunch of garbage behind, so the best thing is to isolate it all. Another thing, you need to download the binaries of the pre-trained models. Just for that I will use a bit of Python because of the “modelscope” lib that handles it:

git clone https://github.com/piddnad/DDColor.git
cd DDColor

mkdir modelscope

# create a new venv inside the DDColor project
python -m venv venv

venv/bin/pip3 install modelscope

venv/bin/python3 -c "from modelscope.hub.snapshot_download import snapshot_download; snapshot_download('damo/cv_ddcolor_image-colorization', cache_dir='./modelscope')"

Good Python practice is to do everything inside a VENV, right? Anyway, this should download the model binary, which weighs almost 900MB. Not huge.

Now we need a Dockerfile:

# Use NVIDIA CUDA base image with cuDNN 8 and Python support
FROM nvidia/cuda:11.8.0-cudnn8-runtime-ubuntu22.04

# Install Python and system dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
    python3 python3-pip git libgl1 libglib2.0-0 && \
    apt-get clean && rm -rf /var/lib/apt/lists/*

# Set working directory
WORKDIR /app

# Copy DDColor source code into the image
COPY . /app

# Install PyTorch (CUDA 11.8 compatible) and matching torchvision/torchaudio
RUN pip3 install torch==2.2.0 torchvision==0.17.0 torchaudio==2.2.0 \
    --index-url https://download.pytorch.org/whl/cu118

# Install Python dependencies, skipping dlib (not needed for inference)
RUN sed '/dlib/d' requirements.txt > temp-req.txt && pip3 install -r temp-req.txt && rm temp-req.txt

# Optional: install modelscope if you want to run snapshot_download manually inside the container
RUN pip3 install modelscope

Too lazy to open a pull request for this, if anyone wants to, send it over there.

Now we need to put a black and white image somewhere:

mkdir input
mkdir results
mv ~/Downloads/bw.jpg input/

Since my girlfriend’s photos are family and private, obviously I am not going to share them, so I grabbed a random one from Google Images:

Now we drop it in “./input” and run Docker with this command:

docker run --rm --gpus all \
  -v "$PWD/input":/workspace/input_images:ro \
  -v "$PWD/results":/workspace/results:rw \
  -v "$PWD/modelscope":/app/modelscope:ro \
  ddcolor:latest \
  python3 infer.py \
    --model_path /app/modelscope/damo/cv_ddcolor_image-colorization/pytorch_model.pt \
    --input /workspace/input_images \
    --output /workspace/results

Pay attention to the directory mapping. And note the “–gpus” option, which only works in my case because I have an RTX 4090 installed here. I have no idea if it is the same with AMD, but since the image is based on cuDNN/CUDA, I would say no. Ask ChatGPT.

If everything is right and it runs successfully, it will take just 1 second and produce this output in the terminal:

==========
== CUDA ==
==========

CUDA Version 11.8.0

Container image Copyright (c) 2016-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

This container image and its contents are governed by the NVIDIA Deep Learning Container License.
By pulling and using the container, you accept the terms and conditions of this license:
https://developer.nvidia.com/ngc/nvidia-deep-learning-container-license

A copy of this license is made available in this container at /NGC-DL-CONTAINER-LICENSE for your convenience.

Output path: /workspace/results
100%|██████████| 1/1 [00:00<00:00,  4.09it/s]

And here is the result that shows up in the ./results directory:

Yes, I was impressed. Even with the image being from the Web, with watermark and all, it did everything properly and worked!!

The only problem left is being able to use a reference image to try to pull the colors. That is a harder problem and I asked ChatGPT, which answered me this:

No native support for a reference image:

The DDColor model, both in the official version hosted on ModelScope and in the piddnad/DDColor GitHub repo, was designed only for automatic colorization of black and white images, with no option to accept a reference image as input. The official description itself makes it clear that the model takes a single B&W image and returns a colored image.

The model’s forward method expects only one tensor, the B&W image converted into artificial RGB (actually the luminance expanded into 3 channels). In other words, there is no place in the model to pass a second image as a color reference.

This is also evident in the original predict.py from the Cog library, which only accepts the image parameter as input — with no support for reference, ref_image or anything of the kind.

How to solve the problem: two possible approaches

Since DDColor does not natively support a reference image to guide the colors, we have two alternatives:

🔁 1. Use a model specialized in reference-based colorization

There are models developed specifically for this, such as:

• Deep Exemplar-based Colorization (Zhang et al., CVPR 2018)

• ChromaGAN (2019)

• Exemplar-Transformer Colorization (Yin et al., ACM MM 2021)

• UniColor (Huang et al., 2022)

These models accept two images:

• The B&W image (main input)

• A colored image as a color reference

These models map semantic similarities (e.g., “skin here should have the same skin color as over there”) and do a transfer based on the image content, getting much more detailed and specific results. However, this would require:

• Downloading different weights

• Completely changing the architecture used

• Adapting the pipeline and inference

🎯 2. Use global color transfer after the DDColor output (our current approach)

The practical and lightweight solution we used was: ➡️ Run DDColor normally to generate the colored image automatically ➡️ Then globally adjust the colors of the result to look more like the reference image.

This adjustment is done with the Reinhard et al. (2001) algorithm:

1. We convert both the output image and the reference to the Lab color space.

2. We compute the mean and standard deviation of the a and b channels (the ones that represent color).

3. We adjust the result to have the same statistical distribution of color as the reference image.

4. This tends to leave the image with the same overall “mood” or tone (warmer, cooler, more vibrant).

This method is fast, lightweight, and works without altering the DDColor model.

Continuing

I used the modification it suggested to the predict.py file:

import os
import cv2
import numpy as np
import torch
import torch.nn.functional as F
from cog import BasePredictor, Input, Path
from basicsr.archs.ddcolor_arch import DDColor

def color_transfer(reference_bgr: np.ndarray, target_bgr: np.ndarray) -> np.ndarray:
    """Transfer color palette of reference image to target image using Lab mean/std."""
    # Convert BGR images to Lab color space (float32 for precision)
    ref_lab = cv2.cvtColor(reference_bgr, cv2.COLOR_BGR2LAB).astype("float32")
    tgt_lab = cv2.cvtColor(target_bgr, cv2.COLOR_BGR2LAB).astype("float32")
    # Split channels
    L_ref, a_ref, b_ref = cv2.split(ref_lab)
    L_tgt, a_tgt, b_tgt = cv2.split(tgt_lab)
    # Compute mean and std for reference and target (a and b channels)
    a_ref_mean, a_ref_std = a_ref.mean(), a_ref.std()
    b_ref_mean, b_ref_std = b_ref.mean(), b_ref.std()
    a_tgt_mean, a_tgt_std = a_tgt.mean(), a_tgt.std()
    b_tgt_mean, b_tgt_std = b_tgt.mean(), b_tgt.std()
    # Subtract target means, scale by reference/target std, add reference means
    # (Avoid division by zero in case of zero std)
    if a_tgt_std > 1e-6:
        a_tgt = ((a_tgt - a_tgt_mean) * (a_ref_std / a_tgt_std)) + a_ref_mean
    else:
        a_tgt = a_tgt - a_tgt_mean + a_ref_mean
    if b_tgt_std > 1e-6:
        b_tgt = ((b_tgt - b_tgt_mean) * (b_ref_std / b_tgt_std)) + b_ref_mean
    else:
        b_tgt = b_tgt - b_tgt_mean + b_ref_mean
    # Clip values to valid Lab range [0,255] after transfer
    a_tgt = np.clip(a_tgt, 0, 255)
    b_tgt = np.clip(b_tgt, 0, 255)
    # Merge channels back (use original L from target to preserve brightness)
    merged_lab = cv2.merge([L_tgt, a_tgt, b_tgt]).astype("uint8")
    # Convert back to BGR color space
    result_bgr = cv2.cvtColor(merged_lab, cv2.COLOR_LAB2BGR)
    return result_bgr

class ImageColorizationPipeline:
    """Helper pipeline to load DDColor model and process images."""
    def __init__(self, model_path: str, input_size: int = 256, model_size: str = "large"):
        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
        # Choose encoder backbone based on model size
        encoder_name = "convnext-t" if model_size == "tiny" else "convnext-l"
        # Initialize DDColor model
        self.model = DDColor(
            encoder_name=encoder_name,
            decoder_name="MultiScaleColorDecoder",
            input_size=[input_size, input_size],
            num_output_channels=2,    # model predicts ab channels
            last_norm="Spectral",
            do_normalize=False,
            num_queries=100,
            num_scales=3,
            dec_layers=9,
        ).to(self.device)
        # Load weights
        state = torch.load(model_path, map_location="cpu")
        # Some checkpoints store weights under 'params' key
        self.model.load_state_dict(state.get("params", state), strict=False)
        self.model.eval()
        self.input_size = input_size  # store for processing

    @torch.no_grad()
    def process(self, img_bgr: np.ndarray) -> np.ndarray:
        """Colorize a BGR image (numpy array) using the loaded DDColor model."""
        # Preserve original resolution L channel
        orig_h, orig_w = img_bgr.shape[:2]
        # Convert to Lab and extract L channel at original size
        img_lab = cv2.cvtColor(img_bgr, cv2.COLOR_BGR2LAB).astype("float32")
        orig_L = img_lab[:, :, 0:1]  # shape (H, W, 1)
        # Prepare grayscale input at model resolution
        # Resize input to model expected size
        inp = cv2.resize(img_bgr, (self.input_size, self.input_size))
        inp_lab = cv2.cvtColor(inp, cv2.COLOR_BGR2LAB).astype("float32")
        L = inp_lab[:, :, 0:1]
        # Create grayscale Lab image by zeroing AB channels
        gray_lab = np.concatenate([L, np.zeros_like(L), np.zeros_like(L)], axis=2)
        # Convert back to RGB (now it's a gray RGB image of size input_size)
        gray_rgb = cv2.cvtColor(gray_lab.astype("uint8"), cv2.COLOR_LAB2RGB)
        # Prepare tensor and run model
        tensor = torch.from_numpy(gray_rgb.transpose(2, 0, 1)).float().unsqueeze(0).to(self.device)
        out_ab = self.model(tensor)  # output is (1,2,H_out,W_out) in Lab AB
        out_ab = out_ab.cpu().float()
        # Resize output AB to original image size
        out_ab_resized = F.interpolate(out_ab, size=(orig_h, orig_w), mode="bilinear", align_corners=False)
        out_ab_resized = out_ab_resized[0].numpy().transpose(1, 2, 0)  # (H, W, 2)
        # Combine original L and predicted AB, then convert to BGR
        out_lab = np.concatenate([orig_L, out_ab_resized], axis=2).astype("uint8")
        out_bgr = cv2.cvtColor(out_lab, cv2.COLOR_LAB2BGR)
        return out_bgr

class Predictor(BasePredictor):
    def setup(self):
        """Load models into memory for efficient multiple predictions."""
        # Determine paths for large and tiny model weights
        large_model_path = "checkpoints/ddcolor_modelscope.pth"
        tiny_model_path  = "checkpoints/ddcolor_paper_tiny.pth"
        # (Ensure the above files exist. If not, download from ModelScope or HuggingFace as described.)
        # Initialize pipelines for large and tiny models
        self.colorizer_large = ImageColorizationPipeline(model_path=large_model_path, input_size=512, model_size="large")
        self.colorizer_tiny  = ImageColorizationPipeline(model_path=tiny_model_path,  input_size=512, model_size="tiny")

    def predict(
        self,
        image: Path = Input(description="Grayscale input image."),
        reference: Path = Input(description="Optional reference image to guide colors", default=None),
        model_size: str = Input(description="Model size to use (large = highest quality, tiny = faster).",
                                choices=["large", "tiny"], default="large"),
    ) -> Path:
        """Run a single prediction. Colorizes the image, optionally using a reference image for color bias."""
        # Read input image
        img = cv2.imread(str(image))
        if img is None:
            raise ValueError("Failed to load input image")
        # Select model pipeline
        pipeline = self.colorizer_tiny if model_size == "tiny" else self.colorizer_large
        # Colorize the image using DDColor
        output_bgr = pipeline.process(img)
        # If a reference image is provided, transfer its color style to the output
        if reference is not None:
            ref_img = cv2.imread(str(reference))
            if ref_img is None:
                raise ValueError("Failed to load reference image")
            # Only apply transfer if reference is a color image
            if ref_img.shape[2] == 3:
                output_bgr = color_transfer(ref_img, output_bgr)
        # Save result
        out_path = "/tmp/out.png"
        cv2.imwrite(out_path, output_bgr)
        return Path(out_path)

With this modification I can run with a new parameter, note the “–ref” option that exists now and place your image in the right spot:

❯ docker run --rm --gpus all \
  -v "$PWD/input":/workspace/input_images:ro \
  -v "$PWD/results":/workspace/results:rw \
  -v "$PWD/ref_images":/workspace/ref_images:ro \
  -v "$PWD/modelscope":/app/modelscope:ro \
  -v "$PWD/infer.py":/app/infer.py:ro \
  ddcolor:latest \
  python3 /app/infer.py \
    --model_path /app/modelscope/damo/cv_ddcolor_image-colorization/pytorch_model.pt \
    --input /workspace/input_images \
    --ref /workspace/ref_images/images.jpg \
    --output /workspace/results

I passed in another colored reference photo but saw no difference in the result. Once again, I asked ChatGPT:

Why didn’t you notice a difference in the result?

The global color transfer only changes the “overall palette” — and not the colors of each object.

If the reference image has soft tones similar to what DDColor would already apply automatically, the final effect will look identical.

But if you use as reference a heavily saturated image (for example, an orange sky, or a deep blue environment), you will see DDColor’s output visually pulled toward that style.

Conclusion

I followed the suggestion and downloaded another totally unrelated image, colored, super saturated with vibrant colors to see if it made a difference. This is the reference image I passed to Docker:

And here is the new result applying it to the previous image in post-processing (this step is no longer A.I., it really is just image post-processing).

Compare it with the other re-colorized photo, I will put it down here again so you can compare:

Got it? Now you have to keep testing with different references to see how they influence the final result, but indeed, it manages to apply the Reinhard algorithm and do a global color transfer. As the name says, it is “global”, not “per object”, so it is hard to control only parts of the image — it acts on the whole image. But in theory I think you could do a manual colorization in Photoshop with saturated colors, near where you want to influence, to try to nudge it, but I have not tried that yet. Later, if anyone does, drop it in the comments.

But that is it. Now at least I can have fun grabbing old family photos and re-colorizing them.

Every video from my channel, including the original source files, is there. That alone is terabytes. My whole Steam library is there, around 4 terabytes. All my old games, retro games too, including Xbox 360 and PS3 ISOs. That’s another 4 terabytes. My entire Ultra HD (4K BluRay) disc collection I ripped (made a backup), a few more terabytes. Right now I’m already using more than 50 terabytes.

Before anyone jumps in with opinions, yes, this isn’t for everyone. We’re talking about 8 drives of 10.9 TB, plus an upgrade of 2 drives of 20 TB, plus another 2 drives of 20 TB for backup in case one of them fails, plus 2 NVMEs of 1 TB each just for cache, plus a 10 Gbps network card. On top of that I keep several online copies of part of this content on Google Drive or Dropbox, and a full backup on Amazon Glacier, for the rare occasion my house catches fire, for example. I’m always prepared for any catastrophe.

Synology is one of the best home NAS brands. Yes, you can go way more sophisticated too, just build a real server, with CPUs like AMD EPYC ThreadRipper, using systems like TrueNAS. But that’s too much for me, I don’t feel comfortable maintaining it. Like I said, everyone has to know their own limits.

1o HD de 20TB terminou de se integrar ao RAID! Levou 2 dias inteiros!! Colocando o 2o HD ... de 4 😅 pic.twitter.com/Rv6dqBQr05

— Akitando.com (@AkitaOnRails) March 27, 2024

“Why not just keep everything online??” For most people that really is the easiest path. Like I said, my case is unusual. You do NOT want to have more than 50 TERAbytes online. My local wired network is 10 Gbps, my fiber internet is a measly 0.5 Gbps (500 Mbps). Streaming high quality video over the internet is horribly slow. I want everything in real time, and for that it has to be a local wired network.

That said, I built a second mini-computer, using an Intel NUC, connected to the same wired network as the NAS. It’s a Core i7, 32GB of RAM, 2.5Gbps ethernet, with the single purpose of being an empty Ubuntu server (it could be any Linux, I just picked the easiest one) whose job is to run Docker containers.

Even this little PC is overkill, way more than necessary. Many will comment that a Raspberry Pi 5 would be enough, and they’re not wrong. The best solution is the one you can afford and feel comfortable maintaining. Don’t try to copy anyone’s setup exactly, study what fits you best. And don’t waste time bikeshedding other people’s setups, build your own.

Anyway, I decided to share with you all the docker-compose files I’m using to bring up the docker containers on this server. Here’s the GitHub repository.

WARNING: don’t forget you need to edit the directories in the scripts. It’s configured to access my NAS network mount point, which is /mnt/terachad. Fix it to point to the right places on your machine.

WARNING 2: there are a lot of details I’m not covering in this article. This Reddit forum has more details and more discussion. Depending on your specific doubts or problems, it may be answered there.

So you don’t get lost in the middle of so many services, here’s the full picture of what we’re going to build, from the request all the way to watching on the TV, including the protection layers that keep malware and wrong downloads out of your library:

The idea is this: you request in one place (Seerr), the system finds it (Prowlarr and the indexers), downloads it with protection (qBittorrent and SABnzbd, plus the security layers), organizes it on the NAS (with subtitles from Bazarr) and you watch it on any screen (Plex and Navidrome). Let me start explaining the main ones one by one.

Portainer and Utilities

Here’s the snippet (see the full thing in the repository):

volumes:
  portainer_data:

services:
  portainer:
    image: portainer/portainer-ce:latest
    restart: unless-stopped
    volumes:
      - portainer_data:/data
      - /var/run/docker.sock:/var/run/docker.sock
    ports:
      - '9000:9000'

Portainer is a visual manager for the resources managed by the Docker service on your machine. Its purpose is more for me to visually “glance” and check if any container has a problem, or to test some quick little config tweak before editing the docker compose file.

In the utils-docker-compose.yml file there are other services like Organizr that I installed but haven’t stopped to configure yet, nor decided if I really want to keep. There’s Librespeed, which works like those internet speed tests, but for the local network. There’s Watchtower, which monitors image updates for all Docker containers, downloads them and automatically restarts with new versions, to keep all containers always up to date.

And to reach your local services from outside home, there are easy VPN options like Tailscale or ZeroTier, with little configuration. Yes, yes, before someone runs to the comments, you can set up OpenVPN from scratch, but trust me, these options are infinitely more “plug and play” and possibly safer, because everyone installs it, but few have the discipline to keep it maintained and updated all the time. These days I actually ended up moving to a Cloudflare tunnel to expose my services, but to get started, a Tailscale does the job really well.

MakeMKV and HandBrake

To build your personal movie library, legitimately, the correct way is to own BluRay or UHD discs and “rip” them. If that’s your case, I recommend buying disc reader models like the Pioneer BDR-XS07S. You have to be VERY careful because not every reader works.

Check the MakeMKV forum for the most recommended models. Some models got firmware updates that stop them from being used to rip BluRay. You need to find models that either weren’t updated, or that aren’t hard to firmware-downgrade. The ideal is to buy models everyone already knows work, and for that you have to dig through forums like that one.

MakeMKV, as the name says, “Makes MKV video files”, which is the good old Matroska video envelope format, one of the most versatile (because it supports multiple streams not just of audio, but of subtitles too).

Crash course: one thing is the codec that does the encoding of the video streams and audio streams. For example, H.265 or AV1 are video codecs. MP3 or AAC are audio codecs. Now we need to package these two streams (or more, if there are multiple dubs, for example) into an “envelope”, a “container”, a “file format”, like “.mp4” or “.mov” files. Matroska is a file envelope format that’s very flexible, allows multiple streams of everything, including subtitles, which is why it’s so widely used for sharing over torrent.

With this container, you can access the app’s graphical interface through the browser (it’s like a remote Windows client, like VNC). Just have the Blu-Ray reader connected via USB to your computer and it should find it. It will use the libredrive library to decrypt the Blu-Ray data streams, which is why the firmware versions being compatible (already cracked) matters.

Many people don’t know that we call everything “Blu-Ray” but in reality Blu-Ray is for movies encoded in Full HD, which is 1080p. The 4K versions (2160p) come as UHD, which is Ultra HD. The Blu-Ray case is usually blue, the UHD ones are black, you want the UHD ones whenever possible, of course. And I have a huge collection:

"onde eu tô gastando tanto espaço??"

Boa parte é fazendo Rip (backup) de todos os meus UHD (BluRay 4K). E isso porque estou re-encodando em bitrate menor depois de ripar kkkk pic.twitter.com/T1yPa6EIcV

— Akitando.com (@AkitaOnRails) March 26, 2024

The advantage of an app like MakeMKV is being able to decrypt the video stream and copy it bit by bit exactly as it is on the UHD, the best quality possible. This is what’s called REMUX (remuxing), which is extracting the stream from the container and putting it in another container without re-encoding. You’ve probably seen it in some torrent files, it’s the best quality. But it means it will use the same space as on the disc, a range of 50 to 80 GIGAbytes per movie. Every dozen movies will eat up almost 1 terabyte. Depending on how many movies you want to have, your free space will run out very fast.

To minimize this, we can recode or “re-encode”. It’s like when we take a RAW photo and re-encode it to a lower quality format, like PNG or even JPEG. If you know where to look, you’ll find the compression artifacts, but overall, it’s worth it. In my case, for example, encoding to H.265 10-bit, QP 15 Constant Quality.

The best option is to have an NVIDIA graphics card, with hardware video encoding support via NVENC, and use the best video encoding software: Handbrake. In my case, I pick the pre-configured Matroska H.265 MKV 4K60fps profile and switch it to use the NVENC encoder on my RTX 4090, which speeds up encoding several times over.

In docker compose, just like MakeMKV, I can bring up this app for browser access, like this:

  handbrake:
    restart: unless-stopped
    image: jlesage/handbrake
    ports:
      - "5801:5800"
    environment:
      - AUTOMATED_CONVERSION=0
      - HANDBRAKE_GUI=1
      - DARK_MODE=1
      - TZ=America/Sao_Paulo
    volumes:
      - "/home/akitaonrails/handbrake:/config:rw"
      - "/home/akitaonrails:/storage:ro"
      - "/mnt/terachad/Videos/BluRay:/watch:rw"
      - "/mnt/terachad/Videos/BluRayOptimized:/output:rw"
    devices:
      - /dev/dri:/dev/dri

Your PC has at least an Intel graphics card with hardware encoding support via QSV, or better, an NVIDIA one with NVENC support. In both cases, we give the container access to that by mapping the /dev/dri device. Without it, encoding goes through the CPU, and it takes orders of magnitude longer.

Keep in mind that software encoding is slower but has better quality than Intel QSV, which, despite being fast, always gives me inferior video quality. NVENC is the best of both worlds: excellent quality and very fast.

WARNING: in my case, I want the best quality possible, that’s why I back up UHDs, that’s why even over torrent I download BR-DISC, which is a UHD backup, and that’s absurdly heavy as I explained. So I manually redo the encoding to the Matroska H.265 profile, as I also explained.

Many people don’t want this hassle, so be sure to read the Reddit forum, which explains how to configure the next services to not download huge files like this. I won’t show it here, because it’s not my case.

QBitTorrent

No matter how much Blu-Ray and UHD you own or can buy, a lot of stuff doesn’t even exist on physical disc. In that case, your only option is BitTorrent. And yes, piracy is illegal, I am not encouraging piracy blah blah blah…

Every country has its own law, go research it. I’ll assume everyone knows what they’re doing and it’s not my problem. No point commenting about this, I won’t go on about it. The web is already stuffed with material about it, go to Google and search yourself.

The first thing we want is to install QBitTorrent, probably the best and most complete app to download torrents, which also has a web interface and works via Docker:

  qbittorrent:
    image: linuxserver/qbittorrent:latest
    restart: unless-stopped
    environment:
      - TZ=America/Sao_Paulo
      - WEBUI_PORT=8080
      - PUID=1000
      - PGID=1000
    volumes:
      - /home/akitaonrails/qbittorrent/config:/config
      - /mnt/terachad/Downloads/torrents:/downloads
    ports:
      - '8080:8080'
      - '62609:62609'

It’s important to remember to generate a random port in the QBitTorrent settings, as shown in the image below. Any port between 1025 and 65535 should work, but your network might block some port range in that interval, so you have to test. Search web forums, and be careful because a lot of online tips are already outdated.

You need to understand the minimum about networking, understand whether you’re behind a NAT, how to map external ports, whether you have an active firewall blocking ports, etc.

If the torrent isn’t downloading, “it might be” this. I put it in quotes because it’s only a possibility. So click “random” to pick another port, edit the docker compose config to map the new port and restart the container.

For some reason the saving management comes all set to Manual, it’s important to change it to automatic:

Another detail is to tell QBitTorrent to re-announce downloads to the trackers when the port changes. For that, enable the “Reannounce to all trackers when IP or port changed” option, otherwise downloads might not restart.

Now the most important setting in this whole section, and one I learned the hard way: block executables at download time. One fine day I found nine “completed” torrents in my library whose only content was a Windows executable of almost 1 GB, no video at all. And worse, the release name was squeaky clean, stuff like Reacher S04E07 1080p WEB H264-CAKES or Rick and Morty S09E10 ... EDITH.scr. You look at the name and swear it’s the episode. The .exe or .scr was hidden in the file inside.

A name filter doesn’t catch this, because the name is clean. The defense that actually works is blocking the extension at write time. In QBitTorrent, go to Options, Downloads, and enable “Excluded file names” with this list:

*.exe
*.scr
*.bat
*.cmd
*.msi
*.lnk
*.com
*.vbs
*.pif

With that, the malicious file simply never gets written to disk. The torrent “completes” with nothing useful, Sonarr or Radarr notice the import failed, dump that release into the blocklist and go looking for another one on their own. The malware never touches the disk, and you don’t even pass the garbage forward by seeding it. This is the first layer of defense, and the most effective.

Finally, how do I search for things to download? I can manually go to Google or DuckDuckGo or straight to sites like the PirateBays of the world and look for “.torrent” files or magnet links. To understand what that is, watch my video about Cryptography in Practice, where I explain everything about Torrent too.

QBitTorrent supports search plugins. It comes with a few enabled. In the bottom right corner, there’s the “Search plugins” button. We want to add the plugin that covers everything: the Jackett service.

Notice it wants a host named “jackett” enabled on port 9117, and for that we have this snippet in the docker compose:

  jackett:
    image: ghcr.io/linuxserver/jackett:latest
    restart: unless-stopped
    environment:
      - TZ=America/Sao_Paulo
      - PUID=1000
      - PGID=1000
    volumes:
      - /home/akitaonrails/jackett/downloads:/downloads
      - /home/akitaonrails/jackett/config:/config
    ports:
      - '9117:9117'

It brings up a service where we can access the web interface like this:

It’s a service that consolidates dozens of torrent search sites in one place, can test whether they’re online and, more importantly, provides an API key in the top right corner, which we should copy and edit into the QBitTorrent plugin config. With that you can search directly, getting results like this:

Notice in the right column that lists where these results were pulled from: it’s coming from the Jackett service.

An important heads-up, because this changed over time: Jackett here is for THAT, the manual searches you do yourself in QBitTorrent. Don’t fall for the temptation of wiring Jackett straight into Sonarr and Radarr. The automated part is going to be managed by Prowlarr, which I explain later, and it’s Prowlarr that feeds the services’ indexers. I keep Jackett running only for manual searches and, at most, as a bridge for some tracker Prowlarr doesn’t support natively, like RuTracker, which Prowlarr can consume through Jackett.

Radarr (Movies)

Searching manually works, but there are better ways: let a service do the searching and manage the download for you, and to download movies, we can use the Radarr service:

For that, we have this snippet in the docker compose:

  radarr:
    image: ghcr.io/linuxserver/radarr:latest
    restart: unless-stopped
    depends_on:
      - qbittorrent
    environment:
      - TZ=America/Sao_Paulo
      - PUID=1000
      - PGID=1000
    volumes:
      - /home/akitaonrails/radarr/appdata/config:/config
      - /mnt/terachad/Downloads/torrents:/downloads
      - /mnt/terachad/Videos/radarr/movies:/movies
      - /mnt/terachad/Videos/radarr/anime:/anime
    ports:
      - '7878:7878'

Notice I mapped different directories for western movies and anime movies, that’s a preference of mine, you can organize it however you want. As I warned before, don’t copy and paste exactly what I’m showing here: adjust the directories for your machine.

The main thing is that once it’s started, we can navigate the web interface to the “Settings” option. There you’ll find “Download Clients” and we can register the QBitTorrent web service:

I didn’t show it, but in the QBitTorrent settings there’s a place to set the admin password to access its API via web. Don’t forget to set it there and use the same password here. That way Radarr can control the movie downloads directly with QBitTorrent, without you having to get involved.

There’s a setting in Radarr I recommend doing right away, because it ships with a silly hole: by default, EVERY quality definition comes with a minimum size of zero. That lets a tiny fake file passing itself off as 4K slip through. I’ve been a victim: a Superman (2025) WEBDL-2160p.iso of 598 MB came in as if it were the movie and stayed showing in Plex for months. Inside the ISO was a .exe of 597 MB in disguise.

The fix is to go to Settings, Quality, and set a size floor on each quality definition. I use roughly this, in MB per minute of video:

• 720p (HDTV/WEBDL/WEBRip/Bluray): 3
• 1080p (HDTV/WEBDL/WEBRip): 5, Bluray 1080p: 8, Remux 1080p: 25
• 2160p (HDTV/WEBDL/WEBRip): 10, Bluray 2160p: 15, Remux 2160p: 50

It’s low enough to accept a good lean x265 encode, and high enough to refuse a “4K” under 1 GB, which can only be a trap. One annoying detail: Radarr has a bug where saving everything at once doesn’t persist the minimum size, so you have to save quality by quality, one at a time.

Sonarr (TV Shows)

Same thing as Radarr, but for TV shows, we have the Sonarr service:

This service is fantastic. It finds your shows, cross-referencing online databases with details like seasons, each episode’s name and lots of other metadata. So it can not only download old episodes, but also keeps a schedule to download shows that are still airing. Every week a new episode shows up on its own:

And same as Radarr: in Settings and Download Clients, I can register my QBitTorrent, then Sonarr handles downloading the episodes:

And to bring up the service, in the docker compose we have this snippet:

  sonarr:
    image: ghcr.io/linuxserver/sonarr:latest
    restart: unless-stopped
    depends_on:
      - qbittorrent
    environment:
      - TZ=America/Sao_Paulo
      - PUID=1000
      - PGID=1000
    volumes:
      - /home/akitaonrails/sonarr/appdata/config:/config
      - /mnt/terachad/Videos/sonarr:/tv
      - /mnt/terachad/Downloads/torrents:/downloads
    ports:
      - '8989:8989'

Same as Radarr, I mapped different folders for anime and non-anime, which is a matter of your organization preference.

Both in Sonarr and Radarr, there’s a setting that became mandatory for me after I started getting caught by traps: a Release Profile that blocks release names known to be booby-trapped. In Settings, Profiles, I keep a “Must Not Contain” list with terms like these:

BROADCAST
FULL HD
.scr
.exe
SODAPOP
REACHERSON

Each of these has a story. BROADCAST, SODAPOP and REACHERSON are groups that pack split RARs with a Sample/ folder that jams the import (I caught almost 60 GB of junk from a fake Welcome to Derry in one go). FULL HD is the naming signature of the ones carrying .exe. And .scr/.exe in the name is literal. A heads-up for anyone copying this: Radarr, under the hood, uses the same ignored field as Sonarr, even though the interface calls it “Must Not Contain”, so configure the same list in both.

There’s still a class no filter catches on its own: pre-air fakes. A Reacher S04E07 shows up with zero seeds for an episode that only airs next week. Sonarr doesn’t reject a release dated before the air date, so I clean those out by hand. The good news is that, with the QBitTorrent executable block from above, even when one of those downloads, it comes in empty and gets discarded automatically.

Downloads that get stuck

A classic everyday problem: the torrent gets stuck forever in “downloading metadata” or “stalled”, with zero peers, and Sonarr never gives up on it on its own. I’ve had an episode sitting behind a dead magnet for five days while the good release passed by, waiting for a turn that never came.

The solution I set up is a simple script running once a day (a cron or a systemd timer, doesn’t matter). It queries the Sonarr, Radarr and Lidarr queue via the API, finds the items stuck in “downloading metadata” or “stalled with no connections” for more than 72 hours, and removes them with a single command:

DELETE /api/v3/queue/{id}?removeFromClient=true&blocklist=true

That deletes the dead download, dumps the release into the blocklist so it won’t grab it again, and the app goes after another source automatically. The trick is being careful to NOT touch a download that’s merely slow (under 72h) nor an item waiting for a manual import, otherwise you delete good stuff by mistake. The first time I ran it, it cleared out nineteen zombie downloads in one shot.

Prowlarr (indexer)

For both Radarr and Sonarr to know where to pull the show and movie torrents from, they need to search public indexers or trackers (or private ones, if you have good contacts, I don’t, don’t even ask). For that we have Prowlarr. This is the most important service of all, because if this one doesn’t work, nothing downloads properly.

It took me a while to reach a good configuration, but today I have a simple rule that solves it: Prowlarr is the only place that manages indexers. Sonarr, Radarr and Lidarr should NOT have any indexer registered directly, only the ones Prowlarr syncs. Every indexer I had registered by hand inside Radarr was a source of headaches, including a “Lime Torrents” that was exactly the one that served me that Superman ISO malware. I deleted them all and left only Prowlarr feeding the three.

Notice it lists dozens of indexers, or trackers. It’s similar to Jackett. The difference is that Jackett is for manual searches you’ll do yourself, straight from QBitTorrent. Prowlarr’s indexers, on the other hand, get configured straight into Sonarr and Radarr. Here’s where the confusion starts, let’s understand it:

Both in Sonarr and Radarr, besides Download Clients, where we can configure QBitTorrent, there’s also this “Indexers” option, where we can configure which trackers to search for the media.

Think of a tracker like a public mini-Google, it’s where people register their torrents when they want to share. Torrents aren’t magic and don’t appear out of nowhere. Someone has to create a record somewhere. But there isn’t just “one” place: it’s a distributed network and any participant can put up a tracker if they want. It’s as if you could put up your own miniature Google, just to register torrent files.

It’s a pain to keep managing this. Some small trackers decide to go offline, new ones appear, configs get changed, and we have to keep managing this manually, both in Sonarr and Radarr. That’s what Prowlarr is for: to manage the trackers for you. Let’s understand how.

In the Prowlarr settings, we start by configuring where to find our Sonarr and Radarr services:

For example, this is my Sonarr config. Notice the hostname is “sonarr” and that works because in my docker compose I bring all services up on the same VLAN, and by default docker registers the service name configured in the YAML file as the hostname. If you don’t understand Docker, be sure to watch my videos about containers.

We also need to configure access to QBitTorrent:

This other one, SABnzbd, uses another, older download protocol, based on USENET newsreaders. Yes, that old Usenet. Those who know, know. I keep torrent as the primary, but I keep SABnzbd configured as a second source, because a lot of content only exists in those older groups. If you’re going to use it, apply the same extension protection as QBitTorrent: in the sabnzbd.ini file, in the [misc] section, set unwanted_extensions = exe, scr, bat, cmd, msi, lnk, com, vbs, pif and action_on_unwanted_extensions = 2, which tells it to abort the whole download if one of those shows up. That way Sonarr and Radarr see the failure and look for another source.

Finally, many trackers today implement some kind of bot protection using Cloudflare captcha, you know that thing that goes “are you a human?”. To get past that, we need the FlareSolverr service, which we can bring up in the docker compose like this:

  flaresolverr:
    image: ghcr.io/flaresolverr/flaresolverr:latest
    restart: unless-stopped
    volumes:
      - /home/akitaonrails/flaresolverr/config:/config
    ports:
      - '8191:8191'

In Prowlarr it enters as an “indexer proxy”. You register FlareSolverr pointing to http://flaresolverr:8191/, create a tag for it, and mark only the indexers that sit behind Cloudflare with that tag. Those then start solving the captcha under the hood, and the others don’t even need to go through it:

We’re not done yet. On the main screen, you go enabling the indexers that make sense for you. Don’t enable everything: it gets heavy and, worse, some indexers serve more traps than good content. Check each one’s tags. Some indexers are porn-only, some are only Russian or Chinese content, those I skip.

The ones that serve me well today are EZTV and 1337x (both behind Cloudflare, so they depend on FlareSolverr), YTS for movies, and AnimeTosho plus Nyaa for anime. In total I’m left with somewhere between 18 and 21 active indexers per app. And more important than enabling the good ones is disabling the bad ones: I turned off LimeTorrents and TorrentDownload, because, together, they generated more than two hundred downloads and almost no usable import, on top of being a recurring source of malware and fakes. Fewer good indexers is worth more than a pile of bad ones.

At the end, it’s good to click “Test All Indexers” to check which ones are still active or not. And finally, we can click “Sync App Indexers”, which will register all the indexers we know are working into Sonarr and Radarr. Get it? Prowlarr will configure both with the most up-to-date indexers, and remove the ones that are no longer active. We don’t need to configure each of them manually.

It’s important to understand this, because otherwise you’ll see Sonarr and Radarr not managing to download anything and won’t know why: but it’s because the indexers are wrong or obsolete or insufficient. Spend a good amount of time reading the Prowlarr documentation.

Plex (Private Netflix)

To watch everything you download, there are several options. If it’s just for you, you can just download everything to an external drive or something, and watch directly from your PC or laptop. Depending on your Smart TV or game console, you can also connect your drive or flash drive via USB and watch directly.

If you want something a bit more sophisticated, and have a spare PC or Raspberry Pi, you can install Sonarr, Radarr, and share the downloads directory over the network, using protocols like CIFS, which is for Windows shared folders. Again, many Smart TVs and consoles support mounting network shared folders.

But if you want something friendlier and more sophisticated, you can literally have your own “Private Netflix”, with a graphical interface similar to any streaming platform, showing thumbnails, descriptions, movie or show details, a video player and all that, and even suggestions of what to watch, movie suggestions based on what you’re watching, actor info and everything else.

There are several projects that do this. One of them, which the community likes more for being 100% open, is Jellyfin (derived from the old Emby). I recommend testing it, it has everything most people need. Just map the Sonarr and Radarr download folders and it will index everything and organize it for you. Then, from any PC, just access the web interface and it all just works.

But I personally prefer Plex, which is older, more closed (which is why some people don’t like it), but because it offers paid plans with exclusive features, the level of polish is clearly better. The interface is much prettier, and looks more “professional”. Jellyfin, as good as it is, still has that look like a backend programmer made it, with a lot of ill will toward UX and design.

On top of that, to watch on a Smart TV or Android TV, you need an app on the Google Play Store. Jellyfin doesn’t have an app for everything, some TVs, I don’t remember if LG or Samsung, don’t even have the app available. So you have to side-load. Not friendly. Plex, on the other hand, is available in way more places, so it tends to be more “plug and play”, similar to downloading the Netflix or Amazon Prime app.

Plex also has features that “just work”, trivial things you expect in a player, like a “skip intro” button or “resume from where you stopped” or “skip to the next episode” or “search and change subtitle”. That’s what interfaces like these offer you, so the usability really is a lot like watching Netflix. You don’t get that with a generic PC player like a VLC.

Bringing it up is easy, here’s my docker compose snippet:

  plex:
    image: plexinc/pms-docker:latest
    restart: unless-stopped
    environment:
      - TZ=America/Sao_Paulo
      - PUID=1000
      - PGID=1000
      - VA_DRIVER=IHD
      - PLEX_HW_TRANS_MAX=16
    volumes:
      - /home/akitaonrails/plex/config:/config
      - /home/akitaonrails/plex/data:/data
      - /home/akitaonrails/plex/transcode:/transcode
      - /mnt/terachad/Videos:/media
    devices:
      - /dev/dri:/dev/dri
      - /dev/bus/usb:/dev/bus/usb
    network_mode: host

There’s a small detail many people don’t understand: Plex does transcoding of the videos. That is, it will re-encode the video before sending it to the app or browser that will play it, and that’s heavy, it’s video stream conversion. If your server isn’t strong enough, your video will stutter when playing.

In reality, it’s more complicated than that. There’s a protocol called “Direct Play” or “Direct Stream”. There’s an article that explains this on the Plex site. If the PC, SmartTV, console, etc. support the codec of the video in question, in theory Plex will just pass the bits straight through, without trying to convert anything. Understand: not every device can play any video codec.

Most content nowadays is encoded in H.264 or H.265 or even AV1, which is newer and more efficient. The problem, let’s say you want to play it on a really old PC or a PS4. They’ll support H.264, maybe H.265 if you’re lucky, but certainly not AV1. So it would be super slow to try to play using only the CPU. The right thing is to have hardware instructions to play AV1, especially on an old CPU.

So Plex asks the device: “do you support AV1?” The player will say “nope, I’m weak.” So Plex will transcode from AV1 to H.264, for example. All of this is transparent to you.

But then you’ll notice that when playing the video it stutters, pauses all the time. It could be a bandwidth problem on your network, or it could be a problem of your Plex server being too weak, or of it not being configured for hardware transcoding, and brute-forcing it using only the CPU.

Watch my video about compression, where I explain codecs, but in short, the right thing is that whatever little PC you’re using for your Plex, it should have at least a minimum of hardware video encoding capability. Most have at least H.264 capability, like the Intel QSV instructions, check if your CPU has that support. It only won’t have it if it’s too old, like something more than 10 years back. If it’s a strong server, with an NVidia GPU, it’ll have things like NVENC, which supports H.265. If it’s a modern Intel CPU, it’ll support AV1.

In any case, on Linux, these instructions should be exposed on the /dev/dri device, which is why we need to map it into the docker container as I did above. Check that this device exists.

If everything is correct, in Plex you’ll have this option to configure the “Transcoder”:

We want to make sure the quality is set to “Make my CPU hurt” and that the “Use hardware acceleration when available” option is enabled too, so it will use /dev/dri to accelerate transcoding. And that makes a BRUTAL difference in speed. If you’re having stuttering playback, it’s very likely the transcoding.

When the device, your smartphone or your TV, supports the exact codec used in the video file we want to watch, in theory, Plex can pass the stream straight through, without transcoding, which is the ideal world, because then there’s no processing at all. That’s Direct Play. But for that the video files need to be in a certain codec, in a certain configuration. But even in a supported codec, if your TV is weak and doesn’t support 4K, and asks Plex for Full HD (1080p), then Plex will transcode from H.264 4K to H.264 1080p, for example. That’s why I say it’s more complicated than it seems.

IMPORTANT DETAIL: Recently I got confused playing some of my videos. I have the best hardware, both for the Plex server, with hardware acceleration, and an Android TV to play it which is my NVIDIA Shield Pro, on a wired network, etc. But some videos kept stuttering. And in the end the culprit wasn’t hardware, or config, or any of that, it was the damn subtitle:

Anyone who’s had to download subtitles has probably seen the most common format, which is .SRT (SubRip). Some who ventured further have seen other formats, like SSA (SubStation Alpha), TTML (Timed Text Markup Language), VTT (Web Video Text Track), VOBSUB, etc.

Yes, there’s a LOT more than SRT. In particular, if you do like me and rip UHDs and extract the subtitle tracks from the disc itself, apps like MakeMKV can download them in VOBSUB format (which is ok) or the cursed PGS (Presentation Graphic Stream Subtitle Format).

Most subtitle tracks are made to be sent to the player in parallel, and separate, from the video track. Just like the audio track goes separately. The player receives a video track, an audio track, and plays the two in sync. Same with the subtitle track, in the case of SRT or SSA or TTML files.

But PGS is different: it has graphical capabilities! I’m guessing here, but if you’ve seen colored, animated subtitles, that move or position themselves at absolute spots on screen (like sitting exactly on top of a billboard in the video, for example), then there has to be a step to rasterize and render that graphical subtitle ON TOP of the video stream. That’s a heavier encoding step, meaning it’ll be HEAVY to play in real time.

The solution: use Plex to search for a new subtitle, in SRT format, and ignore the PGS subtitle. Just doing that stopped the stuttering when playing.

Seerr (formerly Overseerr)

With just Sonarr and Radarr, plus Prowlarr’s indexers, it’s already enough to search and download everything. But there’s another app that makes discovering new stuff to download a lot easier. It’s called Seerr, and it’s the evolution of what used to be Overseerr. The Overseerr project merged with Jellyseerr, the fork that served Jellyfin, and the two became one: Seerr, which now serves Plex, Jellyfin and Emby in the same place. If you still hear people talk about “Overseerr” out there, this is it.

It goes to free online databases (like the IMDBs of the world) and keeps an eye on everything new coming out, everything that’s “trending”. If I want to find something new, it’s a good one. Even things I already know I want, I just search in Seerr and click “Request”:

I configure the integration with Sonarr and Radarr in the settings, so it knows who to send the request to. It all works integrated. I just need to use the Seerr interface to request things and I can forget about Sonarr and Radarr running in the background.

Bazarr (subtitles)

I don’t need it, but my girlfriend sometimes prefers watching with Portuguese subtitles. And it’s a pain because there isn’t always one, or what exists is low quality. By the way, let me warn you that both official and unofficial subtitles, I’m tired of seeing wrong translations. My fun is finding the errors in the subtitle. If you depend on subtitles, know that many important meanings are actually wrong. Always prefer the original.

Back in the day you had to go manually to sites like OpenSubtitles.org, download the “.SRT” file, rename it the same as the video file name and keep testing manually to see if it was in sync or not.

There are two ways to solve this. One is through Plex itself, which has the option to search subtitles in various languages and keep testing them straight from the player interface. It helps a lot because it skips the step of having to sit at the PC, manually downloading files. As I explained in the previous section, in my case, since the UHD backup subtitles were extracted in PGS format, I need to search in SRT format so it doesn’t stutter when playing.

Another way is to try downloading the subtitles automatically beforehand. This isn’t perfect because there are several versions of subtitles, some made for DVD, others for BluRay, some even for the more popular “CAM” versions, which are torrents of videos recorded straight in the theater with a smartphone, the crappiest version of all.

It’s very important to rename the video files in the right format, with the metadata well organized, as explained in the Reddit forum I mentioned at the start of the article. That helps Bazarr match the right subtitle to the right file. And a heads-up: Brazilian Portuguese is one of the languages with the fewest subtitles. It’s easier to find them in French, Italian or Russian than in Portuguese.

To make this easier we have the Bazarr service:

In the Bazarr config, we point it to our local Sonarr and Radarr services, so it finds out every time a new show episode or a new movie appears, and schedules a subtitle search:

The main thing is to configure the Providers, which are the sites the subtitles come from. This is where the difference between “finds nothing” and “finds almost everything” lives. The five that give me the best results are: opensubtitlescom, tvsubtitles, yifysubtitles (movies only), animetosho (for anime, no account needed) and gestdown (an Addic7ed proxy for English, also no account). One gotcha that cost me hours: the OpenSubtitles.com provider wants your username, not your email. If you put the email, it errors out with “you cannot consume this service”, and still keeps you grounded for 12 hours even after fixing it.

To bring up the service, in the docker compose, we have this snippet:

  bazarr:
    image: ghcr.io/linuxserver/bazarr:latest
    restart: unless-stopped
    environment:
      - TZ=America/Sao_Paulo
      - PUID=1000
      - PGID=1000
    volumes:
      - /mnt/terachad/Videos/radarr:/movies
      - /mnt/terachad/Videos/sonarr:/tv
      - /home/akitaonrails/bazarr/appdata/config:/config
    ports:
      - '6767:6767'

Here we need to map the radarr and sonarr folders so it knows where to write the subtitles. One detail that trips a lot of people up: map the exact same subpaths that Radarr and Sonarr use (in my case, /movies and /anime), otherwise Bazarr keeps complaining that it “can’t find the video file for analysis” across the whole library.

The last step, and the one that finally made Bazarr useful for me, is the language profile. I created a “Brazilian Portuguese with English fallback” profile: it lists pb first and en after, with the cutoff on pb. In practice, Bazarr searches for both, downloads whichever shows up first, and considers the item resolved once a Portuguese subtitle exists. If pt-BR never appears, English stays as the backup.

And the trick, which cost me dearly: every item needs a language profile assigned. Bazarr silently ignores any movie or show without a profile. When I went to check, there were 140 shows and 201 movies with no profile at all, silently ignored ever since they were added to the library. Marking the profile as default only covers what you add from then on; for the library that already exists, you have to assign the profile in bulk, all at once, to everything already there.

There’s also a sibling problem to this one, but on the audio side. Every now and then Sonarr or Radarr download a release whose only audio is a foreign dub: an Italian one, a French one, a Russian one. And no subtitle saves wrong audio. For that I use a Custom Format in both, the “foreign-language-only” one, which recognizes foreign-dub markers (ITALIAN, FRENCH, RUS, GERMAN and so on) and gives a score of -500, which makes the release get refused at search time. A detail I learned the hard way: a Russian release came with the whole title in Cyrillic, without any “RUS” in Latin letters, and slipped right past. So the filter also needs to catch any Cyrillic character.

After I fixed the providers, moved the entire library onto that profile and turned on this audio filter, Bazarr stopped being the most useless service in my stack and became one that resolves most cases on its own. When something’s still missing, then yes I hunt for it by hand through Plex, but today that’s the exception. Just to reiterate what I said before: Brazilian Portuguese subtitles are among the scarcest out there, so even with everything set up right, sometimes there just isn’t one.

Lidarr (Music)

I’m not much of a music person. I have a few MP3s I downloaded decades ago and, day to day, I play the Spotify Top 50 playlist and that’s fine. But the same logic as Sonarr and Radarr applies to music, so I ended up setting it up too: the service is called Lidarr.

It catalogs your artists, cross-references discography databases and goes after the missing albums, marking each artist as “Monitored” to look for new releases on its own, the same way Sonarr does with show episodes. In the screenshot above you can see my library with everything in “Lossless”, which is the quality profile I configured.

In the docker compose it’s the same pattern as the others:

  lidarr:
    image: ghcr.io/linuxserver/lidarr:latest
    restart: unless-stopped
    depends_on:
      - qbittorrent
    environment:
      - TZ=America/Sao_Paulo
      - PUID=1000
      - PGID=1000
    volumes:
      - /home/akitaonrails/lidarr/appdata/config:/config
      - /mnt/terachad/Music:/music
      - /mnt/terachad/Downloads/torrents:/downloads
    ports:
      - '8686:8686'

The configuration is identical in spirit to Radarr and Sonarr. In Settings and Download Clients I register the same QBitTorrent, but using its own category called lidarr, so it doesn’t mix the music downloads with the movie and show ones. The library root folder points to /music, and the quality profile can be left open (the “Any”, which grabs whatever shows up) or locked to “Lossless” when you only want FLAC, which is my case.

And the best part: I don’t configure the indexers by hand. Prowlarr, which I explained above, also registers Lidarr as an application and syncs the music-category indexers on its own. It’s the same “Sync App Indexers” it does for Sonarr and Radarr, now for music too.

To listen, Plex itself already plays music: just create a Music-type library pointing to that folder, and on the phone the Plexamp app is great just for this. And, the same way I have Kavita as “Plex for manga”, for music you can also use Navidrome, an open source streaming server like a private Spotify, which indexes the same /music folder (read-only) and has several compatible phone apps. Lidarr downloads and organizes, Navidrome or Plex serve.

Other Services

In my repository there are other Docker Compose files for my local infra. I won’t explain them in detail, but just so you know:

To block banners, ads, trackers and even malware sites across the whole network, I use filtered DNS. For a long time I ran a local Pi-Hole for that, which is a home DNS server with built-in blocking. It works great, but it needs maintenance. These days I switched to a hosted service, NextDNS, which does the same ad, tracker and malware blocking, except I don’t have to maintain any server, and it comes with encrypted DNS out of the box.

And why does encrypted DNS matter? A DNS query, by default, is an open text protocol. Your provider knows exactly everything you browse. I prefer not to hand them that data for free. With NextDNS the queries go out encrypted and still pass through the blocklist I choose. It doesn’t replace an anti-virus, but it makes it much harder for third parties to build a fingerprint of my browsing habits, and it’s a lot less work than the Pi-Hole I used to run.

Now, in the kaizoku-docker-compose.yml file there’s another service like Sonarr for TV, Radarr for movies, which is Kaizoku for manga. If you read manga online, you’ve probably seen many sites getting shut down over copyright, intellectual property and all that. Eventually, the online manga you read will disappear. With Kaizoku we can preserve them locally too.

Kaizoku is only for downloading. To read, we need a reader, the same way we have Plex for video, we have Kavita Reader to read the downloaded manga. Just like Plex, on the first setup you just point it to the folder where Kaizoku downloads everything, then Kavita handles organizing the library, downloading covers and other metadata. It’s literally Plex for Manga.

I haven’t explored this service much yet. Either there are still things to refine in it, or I just don’t know how to configure it right, but not everything downloads covers correctly. See how it doesn’t look nice. If you have tips about Kavita and Kaizoku, be sure to share them below.

For the actual reading, it’s a lot like a Kindle Web or other online manga sites. You can read page by page, or “scroll” the whole chapter, it remembers where you stopped and basic stuff like that. It could be a good project to contribute to, if you feel like practicing with open source.

Conclusion

I’m very happy with the current state of file-sharing services. Don’t waste too much time overthinking. Just install them and have fun. The code for everything is open, if you’re a programmer, go dig through the code and learn something new, and who knows, even contribute.

I’m not into apocalyptic theories, conspiracy theories, or anything like that. I just consider that whatever I don’t have physical control over, I don’t have. If what “is mine” is under a third party’s control, then “it isn’t mine”. That’s the principle. I want everything that’s mine under my sole and exclusive control, period. It doesn’t matter if it’s less convenient, it doesn’t matter if it’s more expensive. It’s a matter of principle: mine is mine and that’s final.

Besides, on the streaming platforms, not everything exists. A lot of stuff from past decades was simply erased from memory. Many don’t even remember anymore, especially those who didn’t see it back then. I feel nostalgia for TV shows, movies, that don’t exist online. Many don’t even exist on physical media. There are movies that after VHS never even came out on DVD. There are DVDs that never came out as Blu-Ray. There’s stuff on physical disc that will never come to streaming. There’s stuff on streaming that will be erased soon, and many won’t even notice.

What’s mine, is mine. They can erase as much as they want. What’s mine, no one erases. If you value what’s yours, start exploring these options. It doesn’t have to be a giant setup like mine. Any spare external drive is enough to start. Start small, grow little by little. I’ve been collecting this stuff for decades! I didn’t start today.

I have been working very hard the past few weeks, quite literally coding non-stop, 7 days a week. Remember that “stop complaining you don’t have time, what do you do from midnight to 6 AM?” thing? Yeah, won’t apply to me.

And it’s exactly when you’re sleep deprived and exhausted that you start to make mistakes. Even though your code runs, locally, you will shoot yourself in the foot in one of those 3 AM coding spree sessions.

The test suite was really my sidekick, co-pilot, bringing me back to my senses whenever the night got too dark.

I maintain my own GitLab custom server for all of my company’s client and internal projects. Just a bit of paranoia, to make sure my code is owned by me alone. And GitLab is a fantastic tool. Even more so because it has its own Continuous Integration companion.

Just add your .gitlab-ci.yml file to your project, create a new branch, push it and create a Merge Request and the CI kicks in, with multiple parallel jobs if you want. Even when I was forgetting to run my tests, the CI would not let me fail.

Keeping a CI up and running does require you to stop and maintain your specs though. Sometimes you will wonder, why does a test run on your local machine but keeps failing in the CI server?

That’s when it’s super useful to run the CI docker image locally, to iron out remaining environment-dependent quirks.

And you can do just that by running the GitLab CI Runner itself. It will pick up your project’s .gitlab-ci.yml and run it locally through docker. So whatever problem you’re seeing on the server will quite definitely happen locally as well. With the added benefit that you don’t need to wait in line in case there is a queue of jobs waiting to run (many developers working, not as many machines to chew the CI work immediately). And you won’t be the one adding useless jobs to the same queue.

And, if you’ve followed my recommendations and installed the awesome Arch Linux distro (or my personal favorite derivative, Manjaro Gnome) you can easily install the runner through AUR like this:

pacaur -S gitlab-runner

Otherwise you will have to check your particular distro repos or download the binary from here. For example:

wget https://gitlab-runner-downloads.s3.amazonaws.com/master/binaries/gitlab-runner-linux-amd64
sudo mv gitlab-runner-linux-amd64 /usr/local/bin
sudo chmod +x /usr/local/bin/gitlab-runner

Once you have it installed, let’s say you have a snippet like this in your .gitlab-ci.yml configuration:

backend:
  stage: test
  script:
    - ./bin/cc-test-reporter before-build
    - bundle exec rspec --exclude-pattern "**/features/**/*_spec.rb"
  after_script:
    - ./bin/cc-test-reporter after-build --exit-code $? || true

You can configure many test jobs to run in parallel. I recommend that you separate front-end related JS tests, Rspec/minitest back-end unit tests, Capybara-based feature tests, Brakeman related security checks, for instance. Each will run in parallel and you won’t have to wait for everything to run if you’re only interested in the JS tests, for example. So you have faster feedback and can start fixing immediately.

Locally, from your project root directory, just run:

gitlab-runner exec docker backend

Of course, this is assuming that you already have Docker installed and properly configured. If you don’t, make sure you read Arch Wiki’s excellent entry on Docker but it’s basically installing the Docker package and adding yourself to the docker group.

This is it! Super easy, barely an inconvenience, and runner will do everything your GitLab CI server is doing.

One caveat is that unfortunately, it seems like you don’t have the ability to save cache or artifacts between job runs (gems, npm packages, etc) so every time this local runner runs, it will start from scratch. This makes it less useful to run all the time, but it’s still faster than pushing to the server and waiting in line for your turn in a busy server.

Saved me a couple times already.

TL;DR: Google is actually doing a pretty good job being a counterbalancer so AWS doesn’t slack off. If you already know everything about AWS, I’d encourage you to test out Google Cloud. Possibly because of muscle memory, I would still be more comfortable with AWS, but now that I forced myself to suffer the learning process, I am pretty confident with Google Cloud and Kubernetes for most of my scenarios.

Full disclaimer that I am not an expert, so take what I say with a grain of salt. This is one of those subjects that I am super eager to talk about but I’m also very reluctant on the proper choice of words so you don’t get the wrong idea about the proposed solutions.

The goal of this exercise is mainly for me to store some snippets and thoughts for future reference. So keep in mind that this is also not a step-by-step tutorial. My first intention was to go that way, but then I realized that it would be almost like writing an entire book, so not this time.

To succeed with something like Google Cloud and Kubernetes you must be battle tested in infrastructure. If you never installed server-grade Linux boxes from scratch, if you never did server optimizations, if you’re not comfortable with bare server-side components, do not attempt a real production deployment. Your safest bet is still something like Heroku.

You have to be that kinda person that likes to tinker (as you’ve probably read me doing in previous blog posts).

I don’t know everything, but I know enough. So I only had to figure out which of the pieces would closely fit my needs. You must outline your needs before attempting to write your first YAML file. Planning is crucial.

First things first, this is what I wanted/needed:

• Scalable web application tier, where I could do both rolling updates (for zero downtime updates) and automatic and manual horizontal scaling of the servers.
• Mountable persistent storage with automatic snapshots/backups.
• Managed robust database (Postgresql) with automatic backups and easy replication to read-only instances.
• Managed solution to store secrets (such as Heroku’s ENV support). Never store production configuration in the source code.
• Docker images support without me having to build custom infrastructure to deploy.
• Static external IP addresses for integrations that required a fixed IP.
• SSL termination so I could connect to CloudFlare (CDN is mandatory, but not enough, in 2018 we need some level of DDoS protection).
• Enough security by default, so everything is - in theory - lockdown unless I decide to open it.
• High-availability in different data center regions and zones.

It’s easy to deploy a simple demo web application. But I didn’t want a demo, I wanted a production-grade solution for the long-run. Improvements to my implementation are super welcome, so feel free to comment down below.

Some of the problems for newcomers:

• The documentation is very extensive, and you will find almost everything - if you know what you are looking for. Also bear in mind that Azure and AWS also implement Kubernetes with some differences, so some documentation doesn’t apply to Google Cloud and vice-versa.
• There are many features in alpha, beta, and stable stages. The documentation kinda keeps up well, but most tutorials that are just a couple months old may not work as intended anymore (this one included - I am assuming Kubernetes 1.8.4-gke).
• There is a whole set of words that apply to concepts you already know but are called differently. Getting used to the vocabulary may get in the way at first.
• It feels like you’re playing with Lego. Lots of pieces that you can mix and match. It’s easy to mess up. This means that you can build a configuration tailored to your needs. But if you just copy and paste from tutorials you will get stuck.
• You can do almost everything through YAML files and the command line, but it’s not trivial to reuse the configuration (for production and staging environments, for example). There are 3rd party tools that deal with parameterizable and reusable YAML bits, but I’d do it all by hand first. Never, ever, try automated templates in infrastructure without knowing exactly what they are doing.
• You have 2 fat command line tools: gcloud and kubectl, and the confusing part is that they name some things differently even though they are the same “things”. At least, kubectl is close to docker, if you’re familiar with that.

Once again, this is NOT a step-by-step tutorial. I will annotate a few steps but not everything.

Scalable Web-Tier (the Web App itself)

The very first thing you must have is a fully 12-factor compliant web app.

Be it Ruby on Rails, Django, Laravel, Node.js or whatever. It must be a fully shared-nothing app, that does not depend on writing anything to the local filesystem. One where you can easily shut down and start up instances independently. No old-style session in local memory or in local files (I prefer to avoid session affinity). No uploads to the local filesystem (if you must, you will have to mount an external persistent storage), always prefer to send binary streams to managed storage services.

You must have a proper pipeline that outputs cache-busting through fingerprinting assets (and like it or not, Rails still has the best out-of-the-box solution in its Asset Pipeline). You don’t want to worry about manually busting caches in CDNs.

Instrument your app, add New Relic RPM, add Rollbar.

Again, this is 2018, you don’t want to deploy naive code with SQL (or any other input) injection, no unchecked eval around your code, no room for CSRF or XSS, etc. Go ahead, buy the license for Brakeman Pro and add it to your CI pipeline. I can wait …

As this is not a tutorial, I will assume you’re more than able to sign up to Google Cloud and find your way to set up a project, configure your region and zone.

It took me a while to wrap my head around the initial structure in Google Cloud:

• You start with a Project, which is the umbrella for everything your app needs.
• Then you create “clusters”. You can have a production or staging cluster, for example. Or a web cluster and a separated services cluster for non-web stuff, and so on.
• A cluster has a “cluster-master”, which is the controller of everything else (the gcloud and kubectl commands talk to its APIs).
• A cluster has many “node instances”, the proper “machines” (or, more accurately, VM instances).
• Each cluster also has at least one “node pool” (the “default-pool”), which is a set of node instances with the same configuration, the same “machine-type”.
• Finally, each node instance runs one or more “pods” which are lightweight containers like LXC. This is where your application actually is.

This is an example of creating a cluster:

gcloud container clusters create my-web-production \
--enable-cloud-logging \
--enable-cloud-monitoring \
--machine-type n1-standard-4 \
--enable-autoupgrade \
--enable-autoscaling --max-nodes=5 --min-nodes=2 \
--num-nodes 2

As I mentioned, it also creates a default-pool with a machine-type of n1-standard-4. Choose what combination of CPU/RAM you will need for your particular app beforehand. The type I chose has 4 vCPUs and 15GB of RAM.

By default, it starts with 3 nodes, so I chose 2 at first but auto-scalable to 5 (you can update it later if you need to, but make sure you have room for initial growth). And you can keep adding extra node-pools for differently sized node instances, let’s say, for Sidekiq workers to do heavy-duty background processing. Then you should create a separated Node Pool with a different machine-type for its set of node instances, for example:

gcloud container node-pools create large-pool \
--cluster=my-web-production \
--node-labels=pool=large \
--machine-type=n1-highcpu-8 \
--num-nodes 1

This other pool controls 1 node of type n1-highcpu-8 which has 8 vCPUs with 7.2 GB of RAM. More CPUs, less memory. You have a category of highmem which is less CPUs with a whole lot more memory. Again, know what you want beforehand.

The important bit here is the --node-labels this is how I will map the deployment to choose between Node Pools (in this case, between the default-pool and the large-pool).

Once you create a cluster, you must issue the following command to fetch its credentials:

gcloud container clusters get-credentials my-web-production

This sets the kubectl command as well. If you have more than one cluster (let’s say, one my-web-production and my-web-staging), you must be very careful to always get-credentials for the correct cluster first, otherwise, you may end up running a staging deployment on the production cluster.

Because this is confusing, I modified my ZSH PROMPT to always show which cluster I am dealing with. I adapted from zsh-kubectl-prompt:

As you will end up having multiple clusters in a big app, I highly recommend you add this PROMPT to your shell.

Now, how do you deploy your application to the pods within those fancy node instances?

You must have a Dockerfile in your application project repository to generate a Docker image. This is one example for a Ruby on Rails application:

FROM ruby:2.4.3
ENV RAILS_ENV production
ENV SECRET_KEY_BASE xpto
RUN curl -sL https://deb.nodesource.com/setup_8.x | bash -
RUN apt-get update && apt-get install -y nodejs postgresql-client cron htop vim
ADD Gemfile* /app/
WORKDIR /app
RUN gem update bundler --pre
RUN bundle install --without development test
RUN npm install
ADD . /app
RUN cp config/database.yml.prod.example config/database.yml && cp config/application.yml.example config/application.yml
RUN RAILS_GROUPS=assets bundle exec rake assets:precompile

From the Google Cloud Web Console, you will find a “Container Registry”, which is a Private Docker Registry.

You must add the remote URL to your local config like this:

git remote add gcloud https://source.developers.google.com/p/my-project/r/my-app

Now you can git push gcloud master. I recommend you also add triggers to tag your images. I add 2 triggers: one to tag it with latest and another to tag it with a random version number. You will need those later.

Once you add the registry repository as a remote on your git configuration (git remote add) and push to it, it should start building your Docker image with the proper tags you configured with the triggers.

Make sure your Ruby on Rails application doesn’t have anything in the initializers that require a connection to the database, as it’s not available. This is something you might get stuck with when your Docker build fails because of the assets:precompile task loaded an initializer that accidentally calls a Model - and that triggers ActiveRecord::Base to try to connect.

Also, make sure the Ruby version in the Dockerfile matches the one in Gemfile, otherwise it will also fail.

Notice the weird config/application.yml above? This is from figaro. I also recommend you using something to make it easy to configure ENV variables in your system. I don’t like Rails secrets, and it’s not exactly friendly to most deployment systems after Heroku made ENV vars ubiquitous. Stick to ENV vars. Kubernetes will also thank you for that.

Now, you can override any ENV variable from the Kubernetes Deployment YAML file. Now it’s a good time to show an example of that. You can name it deploy/web.yml or whatever suits your fancy and - of course - check it into your source code repository.

kind: Deployment
apiVersion: apps/v1beta1
metadata:
  name: web
spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
  minReadySeconds: 10
  replicas: 2
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - image: gcr.io/my-project/my-app:latest
          name: my-app
          imagePullPolicy: Always
          ports:
          - containerPort: 4001
          command: ["passenger", "start", "-p", "4001", "-e", "production",
          "--max-pool-size", "2", "--min-instances", "2", "--no-friendly-error-pages"
          "--max-request-queue-time", "10", "--max-request-time", "10",
          "--pool-idle-time", "0", "--memory-limit", "300"]
          env:
            - name: "RAILS_LOG_TO_STDOUT"
              value: "true"
            - name: "RAILS_ENV"
              value: "production"
            # ... obviously reduced the many ENV vars for brevity
            - name: "REDIS_URL"
              valueFrom:
                secretKeyRef:
                  name: my-env
                  key: REDIS_URL
            - name: "SMTP_USERNAME"
              valueFrom:
                secretKeyRef:
                  name: my-env
                  key: SMTP_USERNAME
            - name: "SMTP_PASSWORD"
              valueFrom:
                secretKeyRef:
                  name: my-env
                  key: SMTP_PASSWORD
            # ... this part below is mandatory for Cloud SQL
            - name: DB_HOST
              value: 127.0.0.1
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: cloudsql-db-credentials
                  key: password
            - name: DB_USER
              valueFrom:
                secretKeyRef:
                  name: cloudsql-db-credentials
                  key: username

        - image: gcr.io/cloudsql-docker/gce-proxy:latest
          name: cloudsql-proxy
          command: ["/cloud_sql_proxy", "--dir=/cloudsql",
                    "-instances=my-project:us-west1:my-db=tcp:5432",
                    "-credential_file=/secrets/cloudsql/credentials.json"]
          volumeMounts:
            - name: cloudsql-instance-credentials
              mountPath: /secrets/cloudsql
              readOnly: true
            - name: ssl-certs
              mountPath: /etc/ssl/certs
            - name: cloudsql
              mountPath: /cloudsql
      volumes:
        - name: cloudsql-instance-credentials
          secret:
            secretName: cloudsql-instance-credentials
        - name: ssl-certs
          hostPath:
            path: /etc/ssl/certs
        - name: cloudsql
          emptyDir: