Underneath this whole story there’s a decision almost nobody makes consciously: the filesystem. The Synology used btrfs all along. The QNAP uses ZFS. And your Linux machine, most likely, came from the factory with ext4 without anyone asking you anything. That’s what this is about.

Where the two came from

ext4 is the end of a dynasty of over three decades. ext (extended filesystem) was born with Linux itself, ext2 became the standard in the 90s, ext3 added a journal in 2001 so you wouldn’t need an epic fsck after every power loss, and ext4 arrived in 2008 with extents, delayed allocation and volumes up to 1 exbibyte. The design goal was always the same: be the default filesystem, fast, simple and predictable. It’s excellent at that. And it stopped there.

btrfs was born in 2007, at Oracle, with Chris Mason, with a completely different goal: be Linux’s answer to ZFS. Instead of writing data over data, it’s copy-on-write: every write goes to a new block and the pointers are updated cascading up the tree. That single decision is the mother of everything that matters in it.

What each one delivers, in a list:

ext4:

• journaling (consistent metadata after a crash)
• extents and delayed allocation (good performance)
• mature, stable, everywhere, every tool works

btrfs:

• copy-on-write on everything
• data and metadata checksums on every block
• instant, cheap snapshots
• independent subvolumes inside the same volume
• transparent compression (zstd, lzo, zlib)
• native RAID and scrub (verification sweep)
• snapshot send/receive for incremental backup
• reflinks (copies that don’t duplicate blocks)

The list already answers the title’s question. Let’s go piece by piece.

Even NTFS and APFS have passed ext4

And if the comparison with btrfs seems unfair, look sideways. NTFS was born in 1993, with Windows NT, already with metadata journaling and ACLs, and back in the 2000s it gained a log of every change on the volume plus shadow copies (Windows’ “Previous Versions”). APFS, which Apple launched in 2017 made for flash, is copy-on-write like btrfs: native snapshots (it’s what Time Machine uses for local restores), instant file clones that share blocks, volumes sharing the space of the same container, and per-file encryption.

Read the ext4 list again and compare: no snapshots, no data checksums, no CoW, no compression. Windows’ and Mac’s filesystems, which nobody considers examples of sophistication, protect your data better than Linux’s default, whose last big feature came out in 2008. btrfs is what puts Linux in the same generation as the other two, instead of a decade behind.

Why btrfs is far superior

What copy-on-write is, for those who never stopped to think about it

In classic filesystems, writing is destructive by definition: updating a block means writing the new content over the old one, in the same place on disk. If the machine dies mid-write, the file ends up half old, half new. The ext3/ext4 journal covers that for metadata (the directory structure doesn’t corrupt), but your data itself is left to luck.

Copy-on-write inverts the order: nothing is ever overwritten. Writing to a file allocates a new block, writes the new content there, and only when everything is safely on disk does the filesystem flip the pointers to the new version, atomically. At any instant, you have either the whole old version or the whole new version. Never a mix of the two.

What that buys in practice:

• Nearly free snapshots. The old blocks already persist by CoW’s nature; a snapshot is just the filesystem holding the root pointer of that instant, pure accounting.
• Power loss doesn’t corrupt. An interrupted write leaves the previous version intact, because it was never touched. For anyone who has lost a VM image or a database to a mid-write crash, this is worth the whole filesystem.
• Verification becomes possible. Since each block is written once and never touched again, keeping its checksum forever makes sense. It’s the base of the corruption protection I detail further down.

The cost exists and I pay it knowingly: random-write-heavy files (VM disks, databases) fragment and duplicate blocks, because each rewrite becomes a new block. For those cases you disable CoW per directory with chattr +C, which must be set before the file is created. In my layout, the VM image made with the right +C stayed healthy; the one made without became the textbook fragmentation case.

Snapshots are the feature that changes your life

A btrfs snapshot is an instant photo of a subvolume, and it costs almost nothing, because CoW already keeps the old blocks while someone references them. Creating a snapshot is a seconds operation, not hours. In practice: before every upgrade, the system takes a photo. If the upgrade breaks, you don’t “uninstall packages”: you boot back into the previous photo and you’re done. The boot recovery section, further down, shows this happening for real.

Subvolumes are the right way to organize the disk

A subvolume is an independent namespace inside the same filesystem, with its own snapshot policy. And it has a killer property: snapshots are not recursive. A child subvolume inside another is automatically left out of the parent’s snapshot.

I use this for everything. On my desktop, the cache directories that can be re-downloaded (mise, rustup, npm, opencode, AI caches) are becoming nested subvolumes inside @home: they stay in the same place for any program, but out of every snapshot. That separation alone takes 165 GiB out of snapshot scope, and the big caches (Hugging Face, yay, uv, pip) already live outside the system volume, with symlinks pointing to them: minus 148 GiB of snapshot, part already migrated and part in progress. The snapshot stays small, the restore stays fast, and whatever can be re-downloaded from the internet never enters the photo.

The real cost, measured on the new QNAP (ZFS, but the CoW math is the same): 27 hourly snapshots of a folder with almost 80 TB, and nearly all of them consume megabytes, because between one hour and the next almost nothing changed. Cheap snapshots are what let this routine exist without guilt.

Reflink deduplication

First, the caveat that matters: btrfs has no native online deduplication like ZFS (which keeps a dedup table in RAM and charges dearly for it). What it has are two practical things: reflink and offline dedup. Reflink is a copy that shares blocks: cp --reflink=auto creates a new file that points to the same blocks as the original, instant and free, and each one starts diverging only where written. In the era of dozens of checkouts, worktrees and similar builds, that saves a mountain of blocks. For retroactive dedup there are userspace tools like duperemove and bees.

Real corruption protection

Every btrfs block, data or metadata, carries a checksum. When the filesystem reads and the checksum doesn’t match, it knows the block rotted, and if there’s a second copy (RAID or metadata DUP) it self-heals. ext4 checks no data at all: bit rot passes silently until you open the file years later.

Two pieces of my configuration show what that buys in practice, and what ext4 simply doesn’t offer. The first is metadata in DUP mode. On ext4, each structure on the volume exists only once: a well-placed bad sector corrupts the directory table and the whole drive stops mounting. On btrfs, even with a single drive, every metadata block exists duplicated. The bad sector keeps existing, but the checksum flags the rotten block on read and the second copy answers in its place, without you noticing. It’s the difference between losing the volume and noting a warning in the log.

The second is the monthly scrub. ext4 never re-reads anything to check health: if a block rots, you find out the day someone tries to open that specific file, maybe years later, when there’s no fixing it anymore. The scrub is a scheduled sweep that re-reads the whole volume validating checksums, and finds the rot while it’s still small and recoverable. When the btrfs device stats counter leaves zero, it’s the siren to replace the drive before the disaster, not after.

Transparent compression, measured

Every mount on my desktop uses compress=zstd:1. Level 1 costs a few percentage points of one CPU core, and btrfs detects what doesn’t compress (model blob, video) and writes it raw, at no cost. The result, measured this week on my two new drives:

That’s 630 GB of free space on 2.2 TB: 15% more drive, paying nothing. On the initial migration the gain was even bigger, 45% on the system volume, because there was more text and less binary. Notice the table’s details: programs and libraries compress well, Docker layers compress even better (39%), text logs compress 4.5x, and AI model weights compress nothing, which is expected and costs nothing. Source code, which is what we produce the most in the AI era, compresses beautifully.

Synology uses btrfs, QNAP uses ZFS

Being fair to ZFS: for a big NAS it’s superior, and it’s what I chose for the QNAP. Native checksums and self-healing, mature RAIDZ, excellent send/receive. The price is complexity and resources: ZFS wants RAM for the ARC cache (my QNAP has 7 GB for a 174 TB pool, and I feel the lack), its online deduplication would need about 190 GB of RAM just for the table on my volume, and its provisioning model gave me a free scare when the snapshot was refused with 20 TB free (the thick vs thin story is in the NAS article).

btrfs is the right balance for a consumer machine. Light, native in the kernel, does almost everything ZFS does without demanding a storage administrator. Desktop, laptop, homelab, small or medium NAS: btrfs. Synology proved it in practice, twice: the btrfs volume survived readable with two drives out, and its snapshots saved 11 albums I deleted by mistake in the middle of the cleanup, byte by byte.

Big, serious NAS, with RAM to spare: ZFS. Everything else: btrfs. And for a personal Linux machine, it’s not even a discussion.

Boot recovery via snapshot

The feature that has saved me for real, told on both machines.

On the desktop, Omarchy configures snapper to take a snapshot before every update and limine-snapper-sync publishes each snapshot as a boot entry in Limine. New kernel won’t come up? In the boot menu you pick yesterday’s snapshot and the machine comes back as if nothing happened. On my machine the scheduled routine runs via Timeshift on cron (every boot and hourly), with snapper covering the update points: the way back is the same mechanism. (The gotcha: each snapshot entry carries the kernel image along, so a small ESP fills up fast. My 1 GB one already refused a kernel update for lack of space; the system stayed bootable on the old kernel, which is the right behavior.)

On the server, which runs openSUSE MicroOS, it’s even stronger: the whole system is transactional btrfs, and every update is a new snapshot. Last week, kernel 7.2.9 took down the NVMe controller silently, twice, in the middle of the nightly backup. The fix was a transactional-update rollback and a reboot back to 7.2.6, which ran the whole load (754 GiB of backup reads with concurrent download) without a hiccup. A kernel regression became a scheduled reboot, not a recovery all-nighter. One documented caveat: /etc lives inside the snapshot, so the rollback undoes configuration changes along with it. After any rollback, check your units again.

Maintenance is not optional (and the deleted file that won’t go away)

A modern CoW filesystem delivers a lot, but it charges two routines ext4 never asked for.

The first is the scrub, which already appeared here: re-read everything and check checksums, monthly, on all machines. It’s bit-rot detection in action, and without it the checksum only warns you about the rot when you try to read the file. The second point is subtler: a snapshot pins the blocks of deleted files. When you delete a file inside a snapshotted subvolume, the space doesn’t come back, because the block stays referenced by the photo. That’s exactly what happened on my NAS: the 2.14 TB cleanup I ran on the Synology only freed space for real after the old snapshots were removed; until then, the blocks stayed pinned.

The treatment is simple routine. Retention has to be automatic, with snapper or Timeshift deleting the old snapshots per policy (on the QNAP, the Snapshot Manager’s retention policy plays the same role). Space that doesn’t come back after deleting old snapshots is usually over-allocated chunks: a btrfs balance start -dusage=50 returns the little-used pieces to the free pool (in my house I don’t blindly schedule balance, I monitor the unallocated space and act when it gets tight). And on the restic side, forget alone only removes the snapshot’s reference: the repository space is only reclaimed with prune along, forget --prune.

House cleaning in the AI coding era

Now the part almost nobody connects with filesystems: the agent era produces garbage at industrial speed. I know because I measured.

• Rust is the biggest disk polluter in the dev world. Every build leaves behind 100 to 180 MB test binaries, and they accumulate with each edit. On my desktop, the target/ directories added up to 920 GiB. ai-memory alone accounted for 704 GiB. The hygiene tool I was using is cargo-sweep, weekly. A reader pointed out that the project declares itself unmaintained, and its own README points to cargo-clean-all, which works differently: instead of pruning old artifacts inside one target/, it finds every Rust project under a directory and deletes the whole target/ of the ones you filter, for example cargo clean-all --keep-days 7 ~ to clean only what hasn’t been compiled in a week.
• Docker is right behind. On my machine I found 1.26 TB of build cache plus 938 GB of images. A single cleanup recovered about 600 GB. On the server, the weekly cleanup collects 0.6 to 4.5 GB every Sunday.
• AI and package caches bloat quietly. The Hugging Face cache had 86 GB until I deleted three dead models and it dropped to 21. Pacman had 22 GB of outdated packages. yay, uv and pip, another 125 GB.
• And agents create temporary directories without asking permission. A complex project with several branches means worktrees, and each release worktree has its own CARGO_TARGET_DIR full of test binaries. On ai-memory that has already blown up the disk, and it became a mandatory rule in the project’s AGENTS.md: after every release, cargo clean in each created worktree and removal of finished worktrees, no exceptions.

The solution is to automate the sweeping. My minimal example, a user timer that runs on the 1st and 15th at dawn, with Nice and idle I/O:

• deletes target/ directories of projects idle for more than 14 days
• cleans the yay cache
• prunes only Docker build cache older than 30 days, never images or volumes

And the part that became standard in my house: instruct the agents to clean up what they create. Three real rules from my projects, for you to copy:

In the desktop’s AGENTS.md: “Never run broad cleanups like docker system prune without explicit user authorization after a read-only inventory. Cleanup uses an exact list of IDs; idle, old or unreferenced does not mean disposable.”

In the homelab’s CLAUDE.md: “Before finishing any task, remove the temporary things the task created, and only those. Never delete what you didn’t create or can’t attribute to the task. When in doubt, list it for the owner.”

In ai-memory’s AGENTS.md: “target/ grows without limit: every edit to a shared crate leaves behind a copy of each 100-180 MB test binary (seen at 157 GiB). Weekly cargo sweep --time 7, and mandatory cargo clean after every release.”

And the real backup: restic

A snapshot lives on the same drive, so a snapshot is not a backup: if the drive dies, the snapshot dies with it. The last item in the kit is restic, which for me is the best backup system on Linux today.

restic is a single open source binary, written in Go, that does deduplicated, encrypted and versioned backup to practically any destination: another directory, another NAS via SFTP or REST, S3, Glacier. The central trick is content-defined chunking: it slices each file into variable-size blocks defined by the content, hashes each block and only writes to the repository the blocks that don’t exist there yet. That’s why the delta is cheap: today’s backup carries the blocks that changed since yesterday, not the files that changed. Editing a 100 GB database in the middle costs a few megabytes of backup, not 100 GB again. Each run becomes a repository snapshot, a restore point you browse with restic mount as if it were a filesystem, and restic check audits the whole repository’s integrity whenever you ask.

Configuration matters more than the tool, so here’s ours, mistakes included. On the desktop, the repository lives on the NAS and retention is keep-daily 7, keep-weekly 4, keep-monthly 6, keep-yearly 1. On the server, backup runs daily at dawn to the repository on the QNAP, covering /etc, /usr/local/bin, /home and the container volumes, with the password off the machine and pg_dump before the databases, because a filesystem snapshot doesn’t guarantee consistency for a database that’s writing. The first full load was 729 GiB in under an hour, and restic crossed unharmed, on retries, the window in which the NFS mount went read-only by mistake during the migration.

The most important part of the configuration is what does not go into the backup. Our exclusions, all with a reason:

• AI model caches (Immich, Hugging Face, EasyOCR): if lost, download again;
• Prometheus metrics: history that regenerates itself;
• the Fulcrum database: reindexes from bitcoind;
• the decision classifiers’ weights: re-downloadable by revision hash (and this exclusion’s glob goes inside quotes inside the bash array, because unquoted it becomes a backup source instead of an exclusion);
• node_modules, target/, cargo registry, rustup toolchains: pure build artifacts.

And one retention lesson to close. When we shortened the server policy to “yesterday, 2 dailies, 2 weeklies and 1 monthly”, the correct translation to flags is --keep-daily 3 --keep-weekly 3 --keep-monthly 2, not 2/2/1: restic’s levels overlap, and yesterday’s snapshot is also the newest of this week and this month, so each number needs to be one more than the older points you want to keep. The detail caught us off guard: the script had a literal 2/2/1, and with it the monthly level would never hold anything older than last night. In practice, the intention was to keep 5 restore points, and 2/2/1 delivers only 3: yesterday, plus one daily, plus one weekly. We erred, measured, corrected.

A humility note from the disaster: the old repositories were left on the Synology by my decision (almost 2 TB of abandoned history, because the content was already re-covered on the new QNAP). A backup repository ages too, and revisiting its scope is part of the hygiene.

Conclusion: in the AI era, disk is not a detail

Agents coding all day change the storage math at both ends. On the volume end: builds, caches, worktrees and models fill the disk much faster than any human habit ever did, and without a cleanup routine you find out when it blows up. On the safety end: the more generated code, the more you need a cheap way back, because not every agent change is a good one.

btrfs covers both. Transparent compression gives back 15% of disk for free, and source code is exactly the kind of data that compresses the most. Snapshots and subvolumes give you instant rollback on the system and isolation for what doesn’t need protection. Checksums and scrub watch the health of blocks that ext4 simply ignores. ext4 had its 15 years of well-deserved reign. In 2026, choosing it on a new Linux is giving up protection you’ll need, sooner or later. Choose btrfs and set up the cleanup today.

At that moment I knew I was a hiccup away from losing everything. This post is the post-mortem of what happened, and the first part was written in the heat of the disaster, with the rescue copy still running.

Update of October 11: the copy finished and the ending was a happy one. No file was lost by accident, both NASes are healthy, each with its own RAID-6. The full recovery is in the second half of this post. The first part stays as it was published, untouched.

I’ve been using a NAS for a long time

I’m not new to this. I made an entire video mini-series on storage and filesystems (in Portuguese), and I recommend watching it before asking me “what’s a good beginner NAS?”. There is no such thing.

I’ve used external storage since the second half of the 2000s, when I was a Mac user and Drobo existed. The company is gone, but I had three of their DAS units before migrating to Synology, first on an entry-level model and then on the mid-tier DS1821+, which I’ve been using for more than five years. I’ve written about it here, setting up NFS on Linux and accessing it over iSCSI.

It’s been very valuable to me, and you don’t get to question why I need that much space. It doesn’t matter: everyone decides what they keep on theirs.

When I was producing YouTube videos, I edited 4K footage straight from the NAS, thanks to the NVMe cache and 10 GbE networking. I got used to big storage at 10 gigabits and I can’t go back.

The array was organized in SHR, Synology’s hybrid RAID, which in practice behaves like RAID-5 (more on that later): it survives the loss of one drive without losing any data. I always thought that was enough.

Hard drives die. Always

Make no mistake: hard drives go bad. Never, ever trust a lone drive forgotten in a closet. It will fail, and you will lose everything on it.

The best public data on this comes from Backblaze, which runs hundreds of thousands of drives and publishes failure rates every quarter. In the Q2 2026 report, with 354,415 drives monitored, the annualized rate was 1.73%, and the lifetime rate for the whole fleet stands at 1.41% per year.

That sounds small. Translated to human scale:

• One drive on its own, for 5 years: the chance of it dying in that period is close to 7%. That’s roughly the chance of rolling one specific number on a 14-sided die. Would you bet every family photo on that?
• One drive on its own, for 10 years: about 13%, one in eight.
• Eight drives, like my NAS, for one year: the chance of at least one failing is about 11%. In five years, more than 40%.

And those numbers are from a datacenter, with controlled temperature, clean power and drives spinning all the time. A drawer drive that sits idle for years and gets knocked around in a move doesn’t have statistics that pretty.

Memory cards are worse, and there isn’t even an equivalent statistic to consult. What exists is torture testing: the Great MicroSD Card Survey tested 216 cards over a year of continuous writing, and 52 of them, one in four, had already died.

Genuine cards started throwing errors, on average, after about 2,500 rewrite cycles; counterfeit cards, after 700. A memory card is transport media. Keeping the only copy of anything on a microSD is asking to lose it.

The practical conclusion is the same as in the mini-series: important data needs redundancy. A parity array exists precisely so that the death of one drive, which is a matter of time, doesn’t take anything with it.

What happened

With all that, I should have been at ease with my setup. I wasn’t.

It started the day before yesterday, Sunday. For a while I’d been watching the volume approach the ceiling.

At first it was 8 drives of 10 TB. Over the years I swapped them, one by one, for 20 TB drives (DSM shows binary units; on the label they’re 12 and 22 TB). It’s a slow process: each swap takes days, because the array has to rebuild and redistribute the data.

I decided to swap one more, the seventh. Synology supports hot-swap, so it’s simple: pull the old drive, slot in the new one, add it to the array and let DSM handle the rebuild.

On Sunday night it looked like this: rebuild starting, all eight drives healthy, 108 TB allocated out of 121.8 TB.

On Monday, at 8 a.m., the beep. Drive 1, the new one, with 13 hours of use, marked critical:

And drive 8, one of the old ones, with 22,095 hours powered on (two and a half years), also critical, with a read error at 08:05:

The pool’s message left no room: “Unrecoverable errors have occurred because drive errors occurred after storage pool degradation. Please back up your data immediately.”

Why two drives is the end

In RAID-5, if two drives go down, redundancy is zero, and in theory the whole array goes with them. It’s an unrecoverable situation.

Anyone who watched my playlist knows why. An array like this doesn’t store “files,” one per drive. A file is a collection of blocks, and the blocks are spread across all the drives, along with the parity. Losing a drive is losing a piece of every file. It doesn’t matter that the other seven are healthy, because they only have parts.

Parity is there to recompute the missing piece when one drive disappears. With two missing, the math doesn’t close.

And there’s a cruel detail in the rebuild: to recreate the new drive, the array has to read every other drive end to end. That’s more than 100 TB of continuous reading. If some old drive had a bad sector hiding in a corner nobody had read in a long time, that’s when it shows up. That’s what happened with drive 8.

Where do you copy 90 TB to?

The only way out, at that point, was to thank the heavens that the volumes were still accessible, with the files showing up, and use the window to copy everything out immediately. I couldn’t touch anything else.

The problem: copy about 90 TB to where? Nobody has an empty 100 TB NAS lying around at home.

Normally I’d buy this kind of hardware in the United States, on a tourist trip, and bring it back with me. It’s much cheaper. Buying expensive electronics in Brazil isn’t worth it: the country has charged import taxes worse than Trump’s tariffs for decades, and the final price usually goes past double the American list price.

Except I had to choose. What is it worth to lose 90 TB of data I spent years collecting and that is nearly impossible to recover from anywhere else? It’s one of those situations where the answer is “it’s priceless,” and so, whatever the price, I pay.

I found a storage reseller close to home, Controle Net. Good people, very quick to respond, and they sent me a proposal for equipment with same-day delivery.

Their first suggestion was a QNAP TS-832PX. I researched it before accepting. It has the same 8 bays and ships with 10 GbE, but the processor is a 1.7 GHz ARM Cortex-A57 (Annapurna Labs AL324) with 4 GB of RAM, half the 8 GB minimum that QuTS hero asks for. For RAID-6 on ZFS, with checksums and double parity on every block, that’s weak hardware, as I explain further down. And it would be a downgrade from the DS1821+ I already had, which runs on a Ryzen V1500B.

I asked for another option, and after some back and forth I settled on a QNAP TS-873A, 8 bays, with the same Ryzen V1500B as my Synology, 8 GB of RAM, and 10 Toshiba MG11 24 TB enterprise-class drives.

This is why you have to do your own research and know, objectively, what you want to do with the equipment. The reseller didn’t act in bad faith: they suggested an 8-bay NAS to someone who asked for an 8-bay NAS. The one who knew it would run ZFS in RAID-6 was me. If I had accepted the first proposal in a panic, I’d have paid a lot for a worse machine than the one I was replacing.

R$ 185,320.00. Eye-watering.

What it would cost in the United States

I checked today’s American list prices. About the exchange rate: the dollar is at R$ 4.97 right now, but that’s the effect of Sunday’s first-round election results, which knocked the rate down more than 4% on Monday. Through Friday it closed at R$ 5.22, and that’s the fair rate to compare with, because that’s the rate the reseller’s stock was built at:

At R$ 5.22, about R$ 60 thousand. I paid R$ 185 thousand, more than three times that.

The comparison isn’t perfect: the Brazilian proposal includes installation, training and six years of technical support, and the equipment was on my desk by the end of the same day. But it gives the measure of the Brazil effect.

If the rule of thumb is “double the American price,” normal would be about R$ 120 thousand. The extra R$ 65 thousand is, in my own rough estimate, the sum of services, local stock of enterprise drives and my urgency. Whoever buys on a Monday morning with the array dying is in no position to haggle.

And drives are expensive everywhere

To make it worse, this is the worst possible moment to buy hard drives. The AI bubble is sucking up production.

I thought Seagate had left the consumer market. I checked, and that’s not quite it: it hasn’t left, but in February 87% of its hard drive sales were already nearline drives, for datacenters, and WD gets 89% of its revenue from cloud customers and only 5% from retail. WD’s CEO said the company is “pretty much sold out for calendar year 2026.”

Seagate has said it won’t expand production capacity; growth comes only from higher-density drives. For someone buying drives at home, the practical effect is similar to them having left: you get what the hyperscalers didn’t want, at whatever price the seller asks.

QNAP instead of another Synology

After spending the morning and afternoon between negotiation, paperwork and payment, by the end of the day I had the QNAP powered on next to the Synology.

Choosing QNAP went beyond availability. Synology has been making questionable business decisions.

In 2025 it launched the Plus line requiring its own-brand drives or drives it certified.

The own-brand ones are Seagate, Toshiba and WD drives rebadged with custom firmware, and anyone using something else was left without pool creation, health monitoring, deduplication or firmware updates. After the outcry, it walked that back in DSM 7.3 and allowed HDDs and SATA SSDs from other brands.

Except M.2 pools still require the compatibility list. The uncertainty is left hanging: what stops them from doing it again?

Setting up the QNAP is very easy. QuTS hero, their ZFS-based system, is as simple to use as Synology’s DSM.

Fixing my mistake: RAID-6 and ZFS

This time I knew I had to configure RAID-6, which survives two drives failing at the same time. The downside is less total capacity, because the equivalent of two drives becomes parity instead of one. The upside is that I shouldn’t have to go through an emergency like this again.

One of QNAP’s strengths is supporting ZFS out of the box, well integrated and easy to use.

For those who don’t know it: ZFS was born at Sun Microsystems, for Solaris, in the mid-2000s, and today lives on as OpenZFS on Linux and FreeBSD. It merges into one system what used to be three layers, the RAID, the volume manager and the filesystem. It keeps a checksum of every block, so it notices when a drive silently returns corrupted data, and repairs it on its own from parity. And it never overwrites data in place: every write goes to a new block, which makes snapshots practically free.

Its reputation as a RAM hog comes from two things. The first is the ARC, its cache, which by design takes up almost all the machine’s free memory. That scares anyone looking at the graph, but it’s cache: it gives the memory back when another process asks. The second is deduplication, which keeps a huge table with the hash of every block and needs it in RAM or it becomes unusably slow. The old rule of “1 GB of RAM per TB of disk” comes from there, and only applies to people who turn deduplication on.

Even without it, ZFS is heavier than ext4 or a simple Btrfs. Computing checksums, compression and double parity for every block costs CPU, and with little memory the cache shrinks and everything gets slow. On weak hardware, like an entry-level NAS with an ARM processor and 2 GB of RAM, it suffers.

QNAP asks for at least 8 GB for QuTS hero. Mine came with 8 GB and a quad-core Ryzen, which works for my use, which is storing large files and serving them over the network. Deduplication stays off.

I bought 10 drives for an 8-bay NAS, so two are left in the drawer as spares, for the day one fails.

For my simple setup, three settings:

• 5% over-provisioning. It’s a slice of the pool the system reserves and never hands out for data. ZFS always writes to new space and gets slow when the pool fills up near the limit; this slack keeps performance stable. On SSD pools it also extends drive life, but these are hard drives, so the minimum is enough.
• 5% guaranteed snapshot space. It’s the reserve for snapshots, the “photos” of the state of your files that let you go back after an accident or ransomware. Guaranteeing the space means the snapshots keep existing even if I fill up the rest. My data changes little (it’s mostly cold archive), so snapshots take almost nothing and 5% is plenty.
• Alert at 85%. The system warns me when the pool passes 85% usage, with room for me to act before hitting the ceiling. It was exactly the lack of room that had me swapping drives in a hurry on the Synology.

Adding the two reserves, about 13 TB are set aside and 112.9 TB are left for use.

A detail of my case: I’m copying the 90 TB with snapshots turned off, and I’ll only enable the schedule after the copy finishes. A snapshot protects a file’s previous state, and on an empty NAS receiving its first load there is no previous state worth protecting. The source of truth is still the Synology.

And there’s a practical cost. I stopped and restarted the job several times, deleted folders that had gone in by mistake and copied again. With snapshots on, every deleted thing would keep taking up space inside some snapshot, and I’d be filling the pool with the migration’s own garbage. When everything is on this side and checked, then yes: scheduled snapshots, to protect me from deleting what I shouldn’t and from ransomware.

After that I created one giant “thick” partition. In ZFS you can create a “thin” folder, which only takes up what’s inside it and can promise more space than exists, or a “thick” one, which reserves the full size at creation. Since it’s a pool with one owner and one use, thick is more predictable: the space is mine and nothing competes for it.

Finally, I created the shares and installed the HBS 3 app.

Copying 90 TB

HBS 3 (Hybrid Backup Sync) is QNAP’s app for backup and synchronization. It connects to another NAS, to rsync, FTP, SMB or the cloud.

I connected to the Synology’s SMB share and told it to sync everything.

I had to stop and restart a few times, tweaking the job settings, because it was absurdly slow. And the file count wouldn’t stop growing:

865 TB on a 108 TB NAS, and that’s when I understood what was going on.

I excluded two kinds of path from the job:

• The #snapshot directories. On Synology, when you check “make snapshot visible,” each shared folder gets a #snapshot subfolder with a full view of the folder at each point in time. Inside the Synology that takes no space, because a snapshot only stores the difference. Seen from outside, over SMB, every snapshot looks like a full copy of everything, and HBS was trying to copy them all.
• The restic backup directories. restic stores everything in hundreds of thousands of pack files of a few megabytes each. Network copying is fast with large files and suffers with small ones, because each one costs an open, metadata and a close. And those backups I’ll redo from scratch on the new NAS.

With that, throughput rose to an average of 400 MB/s.

That’s less than half of what a 10 GbE network delivers, but it’s more than three times the gigabit most people have at home. Both sides really are at 10 gigabits:

Why doesn’t it reach 1,000 MB/s? My main hypothesis is drive 8. The source array is degraded, and every read that goes through the affected pieces has to be recomputed from parity, with a drive that on top of that returns errors and retries. It makes sense that the Synology can’t go faster than this.

There’s a second factor, which my own screenshots give away: the Synology has jumbo frames on (MTU 9000), and the QNAP image shows the default MTU of 1500. That screenshot is from earlier; I’ve already changed the QNAP to 9000 as well.

Why it matters: MTU is the maximum size of each packet that travels on the network. At the default 1500 bytes, filling a 10-gigabit link means processing more than 800 thousand packets per second, and each packet costs CPU on both sides, for the header, the checksum and the interrupt. With jumbo frames each packet carries six times more data, so it’s one sixth of the packets for the same volume, and more processor is left for moving files. It only works if everything in the path matches: both cards and the switch. If one side stays at 1500, the connection negotiates down to the smaller value and the other side’s jumbo frames do nothing.

At 400 MB/s, 90 TB takes about two and a half days, and I prefer two and a half guaranteed days.

What Synology explained

In parallel, I opened a ticket with Synology support. After some back and forth, I sent the logs. They responded fast, the same day, and I’ll give them that. The engineer’s explanation, summarized:

• Drive 8 had been logging media errors for over a month. A media error is when the NAS tells the drive to read a sector and the drive answers that it couldn’t, even after several retries. It’s always an internal drive failure, and the drive should have been replaced.
• It was drive 8’s poor health that took it down during the rebuild of drive 1.
• In practice I lost two drives, and the pool should have crashed completely. It didn’t because of how SHR works.

SHR isn’t a single RAID-5. To use drives of different sizes without wasting space, it slices the drives and builds several smaller RAIDs, one per size band, and joins everything into one volume. The engineer sent this example diagram:

In my case, the pool is made of two partitions. In the first, the rebuild of drive 1 was far enough along and the failure of drive 8 was mild enough that both kept working. In the second, both are missing. That’s why the pool didn’t crash entirely and I can still see the files.

His recommendation was the one I was already following: copy everything out before anything else, because there’s no telling whether the situation gets worse or whether it’s fixable. After the backup he can even try forcing drive 8 back into the pool so the rebuild continues, but there’s no guarantee the drive will go back in, or that it will survive the process. In his words, copying and recreating the pool from scratch is probably the quickest way forward.

Classic RAID-5, by the way, requires all drives to be the same size, and to grow capacity I’d have to replace all of them at once. SHR is what let me swap one at a time over the years. Ironically, it’s also what saved me from losing everything at once.

The two mistakes

The way I understood it, there are two mistakes, one mine and one theirs, and that’s where the whole disaster lives.

Mine: I underestimated RAID-6, which on Synology is called SHR-2. With two parity drives, drive 8 failing in the middle of the rebuild would have been just a scare. I traded safety for 20 TB of space and spent five years thinking I’d made a good deal.

Theirs: from the logs, the software knew drive 8 had been failing for at least a month, and I never got a single notification. I have email notifications configured. And they work: when the critical situation blew up, I got a flood of emails. This one, about drive 8’s bad sectors, arrived at one in the afternoon on Monday, five hours after the disaster:

Before that, nothing. No warning about drive 8 until it failed catastrophically.

If I had received one, I would never have tried to swap drive 1 before replacing drive 8. The whole situation was avoidable, and it was the result of a human mistake plus some software bug.

That’s the other reason I didn’t buy another Synology. I have a principle: if the horse throws me, I put the horse down. I can’t trust it again.

Where I am now

So far I’ve copied about 20 TB of the original 90 TB, and HBS 3 keeps working. I still don’t know whether I lost any data permanently. I hope not, but the most likely outcome is that I lost what was in the RAID slice the engineer described as crashed.

I have offsite backup on AWS Glacier of what matters most, but not of everything.

A lot of people ask why I don’t use S3 or some other cloud storage for everything. I return the question: think for ten seconds. How long does it take to transfer 90 TB over your internet connection? Do the math:

• With a solid 1 gigabit of upload, never dropping for a second: more than 8 days.
• With 500 megabits: almost 17 days.
• With 100 megabits, which is good upload for many homes: 83 days.

And that’s to upload. When disaster hits you need to download it all back, with the clock running. Not counting the rent: 90 TB on standard S3 costs about US$ 2 thousand a month at list price. Cloud is great for the critical subset, which is how I use it. For the whole archive, physics and the invoice won’t allow it.

Lessons

I was put in an impossible situation on a Monday morning, I acted right away as best I could, and I’ll probably come out of this mostly unscathed. We’ll see, HBS 3 is still syncing. What I take from it:

1. Single parity isn’t enough for big drives. With 20 TB drives, a rebuild takes days and reads everything. That’s the window in which the second drive tends to show up. RAID-6, SHR-2 or RAIDZ2.
2. Don’t rely only on the vendor’s notifications. Look at SMART and the drive logs before starting any swap. I didn’t, because I trusted I’d be warned.
3. Replace the sick drive first, then the small one. A capacity upgrade on an array with a suspect drive is roulette.
4. Leave slack. I was swapping drives because the volume was at the ceiling. An 85% alert exists for that.
5. Have somewhere to copy to. My way out cost R$ 185 thousand because I had no plan B. A second NAS, even a modest one, with what matters, would have bought me time to shop calmly.
6. RAID is not backup. I know that, I teach that, and I still only had offsite for part of it.

The copy is done and I already know the size of the damage: zero. The full recovery comes next.

Six days later: the end of the story

When I published the first part, I had about 20 TB copied and no certainty. The outcome: I finished evacuating the ~91 TB, verified file by file, and lost nothing by accident. Not even what was on the fallen RAID slice, because it stayed readable until the end.

Who operated the recovery were my two agents. The desktop agent controls my PC and both NASes over SSH, and the homelab agent takes care of the server where my 57 containers run. The two coordinate by message: one leaves a report in the other’s inbox, and I follow the conversation.

)

This part matters because nothing that follows was me typing commands. I made the decisions; they executed, measured and screwed up badly a few times, as you’ll see.

HBS 3 was quietly betraying me

In the first part I told how HBS 3, pulling the Synology shares over CIFS, reached 400 MB/s after the adjustments. What I didn’t know: speed was the smallest of its problems.

Two defects showed up when the agent checked the source inventory against what HBS reported:

• It didn’t go where it had no permission, and told nobody. HBS mounts the Synology as my user. The Bitcoin/data directory, 901 GB with my node’s wallet and chainstate, belongs to uid 101 and is closed to anyone else. CIFS couldn’t get in there, and HBS reported success without recording the omission. A “green” backup that leaves the entire Bitcoin node behind.
• It skipped files with “unsupported” names. 113 files, recorded only on a detail screen nobody opens during a days-long copy.

A backup tool that reports success while omitting what it couldn’t read is worse than one that fails screaming. Verify the backup against an independent inventory of the source, never against the tool’s own log.

We abandoned HBS. The agent opened SSH on both NASes, enabled the rsync server on the QNAP, and the Synology started pushing straight to its rsyncd, with nothing going through the PC. Each job’s wrapper: rsync -a --numeric-ids -W --partial --timeout=600, attempts with retry, and never --delete. The --numeric-ids preserved the original uids with no chown afterwards, --partial turned any interruption into a continuation, and the absence of --delete made the destination always a superset: re-sharding or killing a job never cost data.

The bottleneck wasn’t the broken drive

Some context was missing from the first part. This Synology was born with eight consumer-line Seagate IronWolfs, and over the years I had been swapping them one by one for 20 TB IronWolf Pros. The disaster happened exactly during one of those upgrades: I swapped drive 1 for the new Pro and, during the rebuild, drive 8 failed. One single old drive was left in the array: drive 2.

And it, not drive 8, was the evacuation bottleneck. Measured under load:

• Drive 2 (the old consumer one, with 37 thousand hours of use): 91% utilization, delivering 27 MB/s, each I/O taking 4 to 15 times longer than its siblings.
• The six Pros: between 34% and 45% utilization, sitting idle waiting for it.

In RAID-5 (and in SHR), a sequential read spans the whole stripe, and the stripe only completes when the slowest member answers. One slow drive caps the whole array: because of it, the array couldn’t go past about 200 MB/s while six good drives had room to spare. And its SMART was clean, zero reallocated sectors. Consumer drives usually ship without time-limited error recovery (ERC/TLER) active, so the drive retries a marginal sector for seconds, internally, without accounting for anything. The Synology’s NVMe cache diverted part of the reads from the platters and helped mask the symptom.

The drive that fails and the drive that slows you down don’t have to be the same, and SMART “PASSED” says nothing about latency. A slow array is diagnosed with per-drive utilization and response time, not with SMART.

Parallelizing rsync is science, not guesswork

First instinct: more streams. Wrong, twice.

The first time, a bug of ours: a “clean” job relaunch didn’t kill anything, because the Synology’s busybox ps truncates the command line and the verification grep didn’t find the old processes. Result: 32 rsyncs running into 7 destinations, four to five writers per tree, the QNAP with its receive buffers clogged, and the cable transmitting 516 MB/s while the array read 140. After the cleanup (enumerating /proc/*/cmdline, never trusting ps again), useful throughput rose from 140 to 181 MB/s while total load collapsed.

The second time, measurement inverted common sense: with large-file trees, fewer streams were faster, because the average size of each read request tripled and the array did long sequential runs instead of jumping around.

But small-file trees are another sport. TERACHAD/Emulators has 21 TB in 3.4 million files averaging 6 MB: latency-bound work (stat, creation, metadata), not bandwidth-bound. And there aggressive parallelization worked, with one detail: the right shard is chosen by file count, not by size. A single directory, ROMS_FINAL/xbox-hdd-ready, held 995 thousand of ROMS_FINAL’s 1.59 million files, averaging 1.7 MB per file. Isolating it from the multi-gigabyte ISOs:

Between 2.8 and 3.2 times the single stream’s average throughput, on the same machine, on the same network. And there’s a beautiful reason: with the array missing a member, reads of data that doesn’t live on drive 2 don’t need it, so the extra concurrency let the other drives work while it choked.

“More parallelism” and “less parallelism” are both wrong answers as a general rule. Bandwidth-bound work splits little; latency-bound work splits a lot, and by file count. Measure before choosing.

Copying is half the work; proving is the other half

Every job that finished went through a dry-run of the same rsync, with the same excludes: zero files needing transfer means an identical destination by size and date. And even so it almost went wrong, because the verification inherited the copy’s excludes: a tree excluded from the transfer stayed excluded from its own proof. Re-running the check without excludes turned up 952 files and 191 GB missing in Downloads (exactly the batches I had set aside as “only existing copy”) and the old restic repositories, which a similar directory name on the QNAP pretended to cover: the repositories’ internal ID proved they were different ones, not copies.

There was source drift too: a tree verified as complete received 16 GB of new checkpoints on the Synology after the check, and only the final re-verification caught it. Verification has an expiration date.

And a conscious decision, recorded so nobody accuses me of hiding it: the old backup histories (about 2 TB of restic snapshots and a duplicity set from 2025) were left behind by my choice, because the new repositories on the QNAP already cover the same content from now on. Deliberate loss of history, not of data.

Closing the books: 17 rsync jobs, all exit 0, all verified, about 5.9 million byte-identical files, and the Synology shutdown only happened after every tree was proven.

)

Meanwhile, in the homelab

On the other side of the house, the marvin server, which mounts the NAS over NFS and runs the 57 containers, almost died standing up. Dead NAS plus “hard” NFS mounts plus 57 containers restarting on their own made an infinite queue of uninterruptible I/O: the machine answered ping, but sshd couldn’t complete a handshake. And every insistent SSH attempt only made it worse, because OpenSSH punishes a repetitive source by extending the ban. We knocked on the door and the door added a lock.

The way out was to stop knocking: I paused the torrents through the qBittorrent API (which kept responding), went to the physical console, disabled Docker, rebooted, and the server came back. A watchdog I didn’t even remember existed had, on its own, taken down the 14 NAS-dependent containers before any database corrupted: bitcoind’s LevelDB opened clean on the first try.

The migration itself was elegant: fstab pointed at the QNAP and the old mount points became symlinks into the new one. No docker-compose changed. Then came the QNAP NFS gotcha: permission is per client. The desktop was allowed without squash, but marvin’s writes came in as uid 1001, blocking everything that writes to the NAS. The first fix attempt still flipped the mount to read-only in the middle of the first restic backup, which survived on retries. Final configuration: read and write, no squash for anyone, tested with touch and ls -ln from each client, because “it worked on the desktop” proves nothing about the server.

)

After that it was resurrection tree by tree: each service only came up when its tree was verified on the QNAP. In two days, 57 of 57 containers back, and a brand-new restic backup running daily to the QNAP.

The new kernel took down the NVMe

In the middle of the recovery, a free scare from the server. Seven minutes after the nightly backup started, the NVMe controller vanished silently: every binary took SIGBUS, Docker and sshd died, and nothing remained recorded anywhere, because the log lives on the device that died. Not even a kernel panic: just silence.

The same workload had run fine hours earlier on kernel 7.2.6 and died on 7.2.9. Disabling APST (the NVMe’s power management) didn’t fix it: a second identical crash, a 7.2.9 regression confirmed by elimination. MicroOS is transactional and snapshot-based, so the rollback was a reboot into the 7.2.6 lineage, not a reinstall. Automatic updates were deliberately left broken on that lineage so nobody goes back to 7.2.9 unsupervised, and testing 7.2.8 waits until I can sit next to it.

A transactional operating system turns a kernel regression into a reboot. Without system snapshots, that night would have cost a lot more.

The thick provisioning scare

On the QNAP, the whole copy ran with no snapshots at all: in a bulk migration, snapshots are dead weight. With the copy done, I went to enable the ZFS snapshot schedule and got a dry error: “pool free space is not enough”. With 20 TB free.

)

Both statements were true about different things. The CHAD folder had been created as thick: out of the 100 TB quota, ZFS reserves the full 100 TB, and to snapshot a dataset with a reservation, the pool has to guarantee space to overwrite every referenced block while the snapshot pins the old copies. The math: 80 TB written plus 100 TB reserved, against 126 TB usable in the pool. It doesn’t close. And df lies on both sides, because it measures the folder against the quota, not against the pool.

The fix was converting the folder to thin, which QuTS hero does in-place, hidden on the folder’s Management screen, not in the edit dialog where “Thick” looks like static text. In thin, the number becomes a ceiling instead of a reservation: the 10 TB reservation evaporated, the pool ended up with 39 TB free, and the first snapshot came out consuming 334 KB over 79.9 TB referenced. ZFS needed almost nothing; the reservation was what demanded a 100 TB guarantee to authorize.

)

Now the QNAP has a snapshot schedule with a retention policy, as it should have had since day one:

)

)

Two footnotes for anyone building one of these. If your thin folder has an 80% alert, raise the quota for aesthetic slack, because the alert measures against the quota and not against the pool. And this QNAP’s real bottleneck is RAM: 7 GB for a 174 TB pool. QuTS hero’s official minimum is 8 GB, and sensible practice at this scale is 16 and up. The size of the ARC cache governs metadata speed, and that’s why the whole-tree sweeps during the migration were slow. A RAM upgrade is worth more than any fine-tuning.

The Synology is born again

With everything verified on the QNAP, the Synology could finally be operated on. I swapped drives 2 and 8 for new IronWolf Pros, deleted the wounded pool and recreated it from scratch, this time in SHR-2, the two-parity-drive mode, their RAID-6. The new pool came out with 100 TB, all drives healthy:

)

It goes back to work as a replica target, and no longer as a single point of failure. The QNAP stays as primary, in RAIDZ2, with 87 TB of the 126 TB used and snapshots scheduled:

)

What I take from the second half

1. Distrust silent success. HBS reported green while leaving 901 GB behind. A backup is verified against a source inventory.
2. SMART doesn’t measure slowness. The drive that nearly killed the evacuation was “healthy.” Per-drive utilization and latency tell the truth.
3. Parallelism is calibrated by measuring. Bandwidth-bound splits little, latency-bound splits a lot, and the shard is chosen by file count.
4. Verification doesn’t inherit excludes. If the proof uses the same filters as the copy, it only reaffirms the copy’s assumptions. And verify again immediately before destroying the source.
5. NFS permission is per client. Test writes from every machine that will use it.
6. Don’t hammer a dying sshd. Each attempt extends the ban. One patient attempt, or the physical console.
7. The dangerous repair comes last. The array stayed degraded and slow until the last byte was proven somewhere else; only then did rebuilding stop costing anything.

I came out of this better protected than I went in: two NASes, each with its own RAID-6, scheduled snapshots on the QNAP, daily restic, and Glacier for what’s critical. The disaster cost me a QNAP, two drives and a week. It could have cost 90 TB.

First things first: this isn’t an evangelism piece or a day-trading pitch. Quite the opposite. As I write this, on April 1, 2026, Bitcoin is around US$ 68k and close to R$ 391k, below the 2025 peaks. Plenty of people look at that and either panic or start fantasizing about leveraged trades. I think both reactions are wrong. There’s a “super cycle” thesis floating around based on institutional demand, spot ETFs and the lagged halving effect. Maybe. Maybe not. What I do know is that short-term candles don’t change the part I actually care about: infrastructure. If you need leverage to “speed up your gains,” you’re probably just speeding up your chances of getting liquidated.

For me, the useful question isn’t “is it going up tomorrow?” The useful question is: “if I want to store and move Bitcoin without outsourcing everything to an exchange, a web wallet and a public API, how do I set that up properly at home?”

The real problem: too much convenience costs too much privacy

Most people’s default flow is simple: buy on an exchange, leave the balance sitting there, or install some random wallet on the phone and call it done. It works. It also concentrates risk and leaks metadata everywhere.

If you leave a balance on an exchange, you have custody risk. If you use a desktop wallet pointed at a public server, you have privacy risk. If you use a hardware wallet casually, bought second-hand on Mercado Livre, you have supply chain risk. Mix all that with hurry, and it gets worse.

That’s why I ended up at a combination that, for someone technical who wants to run their own infra, feels pretty solid:

• Coldcard for cold storage
• Sparrow Wallet on Linux as the desktop wallet and transaction coordinator
• Fulcrum on the home server as a private Electrum server
• bitcoind on the same server as a real full node, validating the chain and broadcasting without depending on third parties

It’s not the easiest path. But that’s exactly the point. Real security rarely comes from the easiest path.

The concepts that confuse beginners

Before getting into the stack, it’s worth aligning on four terms that usually get tossed around like everyone already knows them:

In plain language, the flow looks like this:

1. Sparrow, on the desktop, builds the transaction.
2. That transaction becomes a PSBT.
3. The PSBT goes to the Coldcard via microSD.
4. The Coldcard signs it offline.
5. The signed file goes back to Sparrow.
6. Sparrow broadcasts through your own server, not through someone else’s public infrastructure.

That’s what people mean by “airgapped workflow.” It’s not magic. It’s just disciplined separation of roles.

Coldcard: cold signer, offline, the right kind of annoying

I use Coldcard as cold storage. The reason is simple: it was designed from day one as a Bitcoin-only device, with a heavy focus on airgapped operation through microSD. That alone eliminates an entire category of “conveniences” that many people find practical, but that I’d rather not have anywhere near my keys.

In practice, the Coldcard holds the most important part of the system: the private key. It doesn’t need to know about a server, Electrum, public API, exchange, or any of that. Its job is one thing: sign transactions offline.

That decoupling is great for two reasons:

• The desktop can be convenient without becoming a single point of failure.
• The signer stays isolated even if your main machine has problems.

And here’s a warning I really want to put in mental all-caps:

Never buy a hardware wallet second-hand. Ever.

This isn’t an exaggeration. You have no way to actually know what happened to that device before it reached your hands. It could have a pre-generated seed, tampered firmware, swapped components, repackaged box, compromised supply chain, or simply some dumb trick waiting for you to let your guard down. Hardware wallet is one of those categories where saving R$ 300 buying used is insanity. Always buy from the manufacturer’s official site or from a reseller officially authorized by the manufacturer. And even then, check seals, provenance and firmware.

Can you do something similar with an old phone?

You can. But I’d treat it as a study or budget alternative, not as an obvious substitute for a Coldcard.

The most serious path for that today is AirGap Vault, which was specifically designed to use an old smartphone as an offline signer over QR codes, keeping the device off the network. The idea is good, and for many people it might be the right entry point.

But there are trade-offs:

• An old smartphone wasn’t designed as a dedicated hardware wallet
• The device’s prior history matters
• An aged battery, bad screen and abandoned Android are real problems
• The threat model is less clear than on a dedicated device

So my view is simple: can you use it? Yes. Would I recommend it as the main solution for storing meaningful wealth? No. For that I still prefer dedicated hardware bought from the right source.

Sparrow Wallet: the best desktop piece in this puzzle

On Linux, I use Sparrow Wallet. For me, today, it’s one of the best pieces of software in this ecosystem.

What I like about it:

• works very well on Linux desktop
• supports hardware wallets properly
• understands PSBT without drama
• makes it crystal clear what’s happening in a transaction
• it’s great as a watch-only wallet

In my flow, Sparrow does three things:

1. Holds the watch-only wallet.
2. Builds the transaction with outputs and fees.
3. Receives the signature back from the Coldcard and broadcasts it.

That separation is elegant. The desktop becomes the coordinator. The signer stays cold.

Why Coldcard + Sparrow works so well

This combo is good because each piece does what it does best:

• the Coldcard protects the key
• Sparrow organizes the human use of the wallet
• the server handles the infrastructure

A lot of wallets try to do everything. I prefer this modular design. It’s less “magic,” more explicit, and easier to reason about without lying to yourself.

If I’m at the desktop, I want visibility. If I’m at the signer, I want isolation. If I’m at the server, I want validation and a local index. That division is clean.

The Sparrow problem when you don’t run your own infra

Now comes the important detail. Sparrow alone doesn’t solve privacy.

If you install it, open it and just use public servers, the people on the other end learn quite a lot about your wallet: your address set, xpubs or derivations, balance, history, query behavior, broadcast. It’s not custody, but it’s still exposure.

That’s the hole Fulcrum fills.

Fulcrum: the home’s private Electrum server

Fulcrum is an Electrum server. Instead of letting Sparrow ask things of a third-party public server, it asks my own server.

In practice, that means:

• local balance lookups
• local history
• local address discovery
• local broadcast

In other words: the desktop wallet stops “phoning home” to the world every time you open the program.

In my current setup, Sparrow points at a Fulcrum running on the home server on the LAN, with port 50001 on the internal network and 50002 with TLS.

And why Fulcrum isn’t enough on its own

Because Fulcrum doesn’t replace a full node. It indexes on top of a full node.

The thing actually validating blocks, consensus rules, scripts, transactions and the chain is bitcoind. Fulcrum sits in front of it as an indexing layer, because plain Bitcoin Core wasn’t built to serve a desktop wallet with that kind of fast querying.

So the correct architecture is:

Coldcard (offline signer)
        ^
        | microSD / PSBT
        v
Sparrow Wallet (desktop watch-only + coordinator)
        |
        v
Fulcrum (private Electrum server)
        |
        v
bitcoind (full node)

What I actually brought up on the home server

On my home server, the stack lives in a dedicated Docker Compose folder and is made of two containers:

• bitcoin-bitcoind
• bitcoin-fulcrum

The compose is simple. And that’s good. Sensitive infra doesn’t gain anything by getting clever in YAML.

The main design is this:

services:
  bitcoin:
    image: lncm/bitcoind:v28.0
    container_name: bitcoin-bitcoind
    user: "${BITCOIN_UID}:${BITCOIN_GID}"
    restart: always
    security_opt:
      - label:disable
    volumes:
      - /srv/bitcoin/data:/data/.bitcoin
    ports:
      - "8333:8333"
    stop_grace_period: 5m
    healthcheck:
      test: ["CMD", "bitcoin-cli", "-datadir=/data/.bitcoin", "ping"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 60s

  fulcrum:
    image: cculianu/fulcrum:latest
    container_name: bitcoin-fulcrum
    restart: always
    security_opt:
      - label:disable
    volumes:
      - /srv/bitcoin/fulcrum:/data
      - /srv/bitcoin/data:/bitcoin:ro
    command: ["Fulcrum", "/data/fulcrum.conf"]
    ports:
      - "50001:50001"
      - "50002:50002"
    depends_on:
      bitcoin:
        condition: service_healthy

In my case, the restriction of 50001 to LAN happens at the host’s network layer. The YAML above is the skeleton of the stack, not the entire firewall policy.

The most important parts of this:

• restart: always because this is a long-running service
• explicit volume so state isn’t lost
• user: "${BITCOIN_UID}:${BITCOIN_GID}" because the persistent directory needs to match the storage’s real ownership, so I’d rather pin UID/GID explicitly than trust the image’s default
• the RPC isn’t published on the host; it stays on the Compose internal network, which is all Fulcrum needs
• the healthcheck uses Bitcoin Core’s own local .cookie, so there’s no need to spread a fixed password through commands
• Fulcrum mounts the node’s datadir as read-only just to authenticate via the .cookie without inventing parallel credentials
• in fulcrum.conf, that becomes a simple configuration: talk to bitcoin:8332 and read the mounted .cookie, instead of repeating credentials in plaintext
• security_opt: label:disable because on this host with MicroOS, SELinux and sensitive bind mounts, I preferred the pragmatic route of disarming this specific friction rather than wasting time fighting labels on a volume that’s already being handled in a controlled way
• depends_on with service_healthy so Fulcrum only comes up after bitcoind’s RPC is responding
• stop_grace_period: 5m because bitcoind needs real time to flush state on a graceful shutdown

The final version

Today, the design I want to keep is this: bitcoind with txindex, dbcache=1024, persistent volume, 5-minute graceful stop, .cookie authentication, and Fulcrum in front serving Sparrow over LAN or TLS.

The current stack looks like this:

I’m not interested in turning this into a spectacle. The point is simpler: the final infrastructure has to be boring, predictable and stable.

The tunings that actually matter

There’s no magic here. There are a few parameters that make a real difference and a bunch of stuff that just decorates compose files.

stop_grace_period: 5m exists because bitcoind isn’t a disposable stateless API container. It maintains chainstate, indexes and an in-memory cache. If you don’t give the process time to finish properly, you create unnecessary work for the next start.

user: "${BITCOIN_UID}:${BITCOIN_GID}" is there for a much less glamorous and much more important reason: persistent storage with the wrong permissions is an excellent way to break a working service. So I’d rather align the container with the volume’s actual ownership instead of leaving that implicit.

dbcache=1024 is the spot I find most reasonable for a domestic node that’s always on. Big enough not to suffer constant I/O, small enough that every restart isn’t a labor.

txindex=1 I keep because I want the complete node, not a minimalist install just to claim “it runs Bitcoin.” If the goal here is operational autonomy, I’d rather have the full index.

rpcworkqueue=512 and rpcthreads=16 are the kind of tweak that makes sense when you know you’ll have Fulcrum querying the node all day and you want some headroom.

On the Fulcrum side, the main parameters are:

• db_mem = 8192
• db_max_open_files = -1
• bitcoind_clients = 8
• worker_threads = 0
• peering = false

Again: nothing esoteric. Just enough cache, reasonable parallelism and absolutely no announcing this server as a public service.

In my current bitcoin.conf, the important core ended up like this:

server=1
txindex=1
prune=0
rpcbind=0.0.0.0
rpcallowip=172.0.0.0/8
rpcthreads=16
rpcworkqueue=512
dbcache=1024
maxmempool=512

All of this makes sense on a server with decent RAM and fast NVMe. But the detail that matters most is still the clean shutdown. Wallet infrastructure has no room for “we’ll deal with it later” thinking.

The actual size of all this

This is another point a lot of people underestimate.

If you look at older Bitcoin Core documentation, you’ll find numbers like 350 GB of disk for a node with default config. That’s outdated. More current data on the size of the blockchain points to something around 725.82 GB on March 11, 2026, and that’s just the raw chain, without the extra indexes that many technical folks will want to keep.

And here comes the catch: the stack I’m describing isn’t “a bare Bitcoin Core just to claim you run a node.” It’s bitcoind with txindex, plus Fulcrum, plus headroom for rebuild, logs, snapshots and normal network growth.

So to put together something similar today, I’d think like this:

• below 1 TB: I wouldn’t even start
• 1 TB: pragmatic minimum
• 2 TB: comfortable range
• above that: if you want long-term headroom, snapshots and less operational anxiety

And here’s the most important observation of all in self-hosting: don’t assume persistence, mount and backup are right just because the YAML looks clean. Verify.

Another thing I wouldn’t forget on a btrfs host: put Fulcrum’s database (fulc2_db) on a separate subvolume. The reason is mundane. That directory grows, changes constantly and has nothing to do with generic automated snapshots of /var. If you mix everything, you end up dragging a large rebuildable index along with system snapshots, burning space and making maintenance more annoying than it needed to be. The Fulcrum index isn’t sensitive configuration. It’s heavy, volatile, rebuildable data. I treat it exactly like that.

Hardening: what I’ve already applied

This is where the difference between “ran on my laptop” and “I’d trust this to operate my wallet” shows up.

In the current state of the stack, the points I consider important ended up like this:

• Bitcoin Core’s RPC no longer relies on unnecessary host exposure; Fulcrum talks to bitcoind over the Docker internal network, which is what actually matters
• 50001 is restricted to internal LAN use
• 50002 is available with TLS, which is the right move when you need to leave plaintext behind
• shutdown is graceful, with stop_grace_period: 5m, so bitcoind has time to flush state instead of dying any old way
• the storage mount isn’t on a “we’ll see later” basis; there’s a mount check before Docker comes up, precisely to avoid silent drift

Each of those items exists for a very concrete reason.

Pulling the RPC off the host’s surface reduces attack at zero cost. Fulcrum is already in the same Compose and can already talk to the service by its internal name. There’s no real gain in leaving that port exposed where it doesn’t need to be exposed.

Separating 50001 and 50002 also helps keep the house in order. Within a controlled LAN, plaintext is acceptable. Outside of that, the minimum reasonable thing is TLS. Mixing the two scenarios usually turns into a mess.

stop_grace_period: 5m looks like a container detail, but it isn’t. Anyone who’s ever had a database, an index or a blockchain node killed without grace knows how that turns into hours of work later. A stateful service needs a decent stop.

And the mount check is one of those annoying things that saves you from yourself. The YAML can look beautiful. If the storage didn’t mount and the service came up writing where it shouldn’t, you’ve just manufactured a really irritating problem.

There’s also one detail I really like in this final version of the stack: Fulcrum authenticates to bitcoind through the .cookie file, not through a fixed plaintext password. That’s interesting for two reasons:

• you don’t need to leave a static credential showing up in compose, inspect, healthcheck or documentation
• the authentication is more aligned with the way Bitcoin Core already knows how to operate locally

In practical terms, that reduces accidental leakage of operational secrets. It’s not a magic solution to everything, but it’s much better than spreading rpcuser and rpcpassword across files, logs and commands.

The only kind of hardening I try to avoid here is the one that’s overly performative in YAML and loose in operation. I’d rather have less “stage engineering” and more basic discipline:

• minimum network
• minimum secrets
• minimum privilege
• clean shutdown
• verified storage
• separate subvolume for large rebuildable data, like the Fulcrum index

And, again, document everything. Good infrastructure isn’t the kind that just works today. It’s the kind that keeps working when you come back to it six months later.

Why this improves transactions on your side

When I build a transaction in Sparrow and sign it on the Coldcard, the chain of trust is much better defined:

• the private key never touches the internet
• the desktop wallet doesn’t have to trust a public server
• the broadcast can come out of my own node
• the address history doesn’t need to land on a third-party Electrum server

This doesn’t make anything invulnerable. There’s still a risk of malware on the desktop, badly stored seeds, human error, social engineering and physical disaster. But the design becomes much more coherent.

What about Lightning? Especially in Brazil?

There I separate things.

Reserves and larger-value transactions I treat one way. Day-to-day spending I treat another.

For day-to-day spending, especially in Brazil, I think it’s operationally dumb to carry a lot of balance in a hot wallet. Lightning wallets and spending apps have to be almost like a “pocket wallet”: just enough for daily life.

That goes double if you use a hybrid or custodial solution like RedotPay. I get why it’s interesting for Brazilians: a Hong Kong company, international focus, a reasonably practical bridge between crypto and card spending. For travel, online shopping and life outside the Brazilian banking axis, it makes sense. But I’d never treat it as a place to store wealth. That’s a spending tool, not a vault.

Same logic for Bitrefill Brasil. I think the service is interesting precisely because it solves a real pain in Brazil: turning sats into concrete utility without selling your full position or depending on banking integration all the time. Gift cards, top-ups, small expenses. As a use tool, it makes a lot of sense.

For a Lightning wallet on the phone, I’d look at first:

• Phoenix for people who want something very good and simple
• Breez for people who want a great payments experience
• ZEUS if you’re more technical and you eventually plan to operate your own Lightning node

All of them, in my head, fit into the “pocket wallet” category. Small balance. Daily use. Don’t turn a phone app into a retirement vault.

Recent news that reinforces this reasoning

I’m not building this kind of stack because I think it’s pretty. I’m building it because outsourcing too much goes wrong too often.

Two recent examples:

• the Bybit hack in 2025 showed, again, the basic risk of leaving meaningful custody at an exchange
• the Coinbase customer data leak in 2025 showed the other side of the problem: even when custody isn’t the immediate concern, your identity, balance and history become an attack surface

A stack like Coldcard + Sparrow + Fulcrum + full node doesn’t eliminate every risk in the world. But it avoids two very real classes of problem:

• losing custody sovereignty
• handing over wallet and transaction privacy on a silver platter to third parties

So is it worth it?

For most people, honestly, probably not in the first month. It’s labor-intensive, has a learning curve, and demands discipline.

But for programmers, engineers and any technical person who wants to learn not to depend on someone else’s service all the time, I think it’s an excellent exercise.

You learn about:

• separation of concerns
• persistence and state
• graceful shutdown
• observability
• secret isolation
• the trade-off between convenience and security

And all of that is valuable beyond Bitcoin.

In the end, that’s what interests me most about this stack. It isn’t about preaching “hyperbitcoinization” or posing as a price prophet. It’s about building a system at home that I can trust more because I’m the one who installed it, measured it, broke it, fixed it and documented it.

Is it work? Yes.

But that kind of work teaches exactly what modern software tries to make you forget: depending less on others is more work upfront, but it usually buys a lot more control in the long run.

With the new Minisforum MS-S1 Max that I bought, I decided to do the migration properly. And I decided to use Claude Code from the start to speed up the process. It’s a home server, only I use it, so the risk of making a mistake is low. But if this were a real production server, I would never do this without rigorous human review at every step.

What follows is the migration writeup and a guide for anyone who wants to replicate it. If I ever have to rebuild from scratch, this post is the documentation.

Choosing the operating system

Why not Ubuntu Server again

I used Ubuntu Server on the NUC for convenience. But do-release-upgrade is Russian roulette. Every time Ubuntu releases a new version, the upgrade is a real risk of breaking things. Packages change, configs get overwritten, dependencies clash. For a server that has to be running constantly, that’s unacceptable.

Why not Arch Linux

I use Arch on the desktop and I like it. But Arch is a rolling release with no stability guarantees at all. For a desktop where I can stop and fix problems, fine. For a headless server running 49 Docker containers that has to work after every reboot, no.

Fedora CoreOS vs openSUSE MicroOS

The two modern options for a container server are Fedora CoreOS and openSUSE MicroOS. Both are immutable systems: the root filesystem is read-only, updates are atomic (they either apply in full or don’t apply at all), and rollback is instant.

The difference: Fedora CoreOS uses Ignition (declarative configuration before the first boot) and is designed to be provisioned automatically. MicroOS uses transactional-update and allows normal interactive use. For a home server where I want SSH and the ability to poke around manually, MicroOS fits better.

What makes MicroOS different

The immutable system concept changes how you operate the server:

Every package install or /etc edit goes through transactional-update, which creates a new btrfs snapshot, applies the change in that snapshot, and on the next reboot the system boots into the updated snapshot. If the change breaks something, you do transactional-update rollback and you’re back on the previous snapshot in seconds.

Updates are automatic and daily. The transactional-update.timer downloads patches, creates a snapshot, and rebootmgr reboots in a configured window (in my case, between 4am and 5:30am). If the update breaks the boot, GRUB automatically falls back to the previous snapshot.

SELinux is enforcing by default. That caused 90% of the problems during the migration, but it’s the right setting for security.

Initial setup

Hardware

• AMD Ryzen AI Max+ 395, 128GB LPDDR5X
• 96GB allocated as VRAM via BIOS (UMA Frame Buffer Size)
• 2TB NVMe (system + Docker)
• Wired 2.5Gbps network
• Synology DS1821+ NAS at 192.168.0.21 (NFS)

First steps

MicroOS install is standard. Then:

# Create user with a UID that matches the NAS (so NFS works without permission issues)
useradd -u 1026 -m akitaonrails
passwd akitaonrails

# Configure sudo (inside transactional-update shell)
sudo transactional-update shell
# inside: add akitaonrails to sudoers
exit
sudo reboot

Synology NFS

The Synology NAS exports /volume1/TERACHAD over NFS. The mount point on MicroOS is /var/mnt/terachad (not /mnt/, which lives on the immutable root).

In /etc/fstab (applied through transactional-update):

192.168.0.21:/volume1/TERACHAD /var/mnt/terachad nfs4 nfsvers=4.1,rsize=262144,wsize=262144,hard,_netdev 0 0

Details that matter: nfsvers=4.1 because 4.2 didn’t work with the Synology. rsize=262144,wsize=262144 (256KB buffers) was the biggest NFS performance improvement. hard instead of nofail so the mount keeps retrying indefinitely if the NAS disconnects temporarily.

GPU / ROCm

This step was a pain. The Radeon 8060S in the AI Max+ 395 is gfx1151, which ROCm doesn’t officially support. Three steps were needed, and all three are mandatory:

1. BIOS: set UMA Frame Buffer Size to 96GB
2. Kernel: add amdttm.pages_limit=25165824 amdttm.page_pool_size=25165824 to /etc/kernel/cmdline
3. Docker: use HSA_OVERRIDE_GFX_VERSION=11.5.1 in every ROCm container

Without step 2, ROCm only sees 15.5GB even with the BIOS allocation in place. The numbers are 96GB / 4KB (page size) = 25,165,824 pages.

sudo transactional-update shell
echo "amdttm.pages_limit=25165824 amdttm.page_pool_size=25165824" >> /etc/kernel/cmdline
exit
sudo sdbootutil update-all-entries  # OUTSIDE the transactional shell
sudo reboot

Verification:

cat /sys/class/drm/card1/device/mem_info_vram_total
# 103079215104 (96 * 1024^3)

Docker on MicroOS

sudo transactional-update --non-interactive pkg install docker
sudo reboot

sudo systemctl enable --now docker
sudo usermod -aG docker akitaonrails
# logout and login for the group to take effect

# Install standalone docker-compose (the openSUSE package doesn't include it)
sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-linux-x86_64" \
  -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
mkdir -p ~/.docker/cli-plugins
ln -s /usr/local/bin/docker-compose ~/.docker/cli-plugins/docker-compose

daemon.json

{
  "log-level": "warn",
  "log-driver": "local",
  "log-opts": {"max-size": "10m", "max-file": "5"},
  "selinux-enabled": true,
  "live-restore": true,
  "userland-proxy": false,
  "exec-opts": ["native.cgroupdriver=systemd"]
}

live-restore: true keeps containers alive across a Docker daemon restart. userland-proxy: false uses iptables directly instead of proxy processes (less overhead). selinux-enabled: true is mandatory on MicroOS.

SELinux and Docker: the biggest source of problems

This deserves an entire section because it was responsible for 90% of the bugs during the migration.

On MicroOS with SELinux enforcing, every container that writes to a host bind-mounted directory needs special handling. There are two approaches: the :Z suffix on volumes and the security_opt: label:disable option.

NEVER use :Z. Use security_opt: label:disable.

:Z tells Docker to relabel the host directory with the container’s SELinux context. Sounds like the right thing. In practice:

• SQLite databases break. The relabeling changes the file’s context and SQLite may refuse to open the WAL journal.
• NFS mounts silently ignore :Z. NFS doesn’t support SELinux xattrs. The kernel ignores the flag without error, but the container still doesn’t have permission.
• :ro,Z mounts try to relabel even when read-only, which fails on NFS and can corrupt the context on local paths.

The right solution for every container on this system:

services:
  myservice:
    security_opt:
      - label:disable     # disables SELinux enforcement for this container
    volumes:
      - ./data:/data      # NO :Z
      - ./config.yml:/etc/config.yml:ro  # NO :Z even on :ro

label:disable disables SELinux label enforcement for that container only, not for the entire system. Combined with Docker’s network and process isolation, it’s safe for a home server.

Migrating the stacks

All Docker stacks were reorganized into /var/opt/docker/<stack>/docker-compose.yml. On the old server, they were scattered across /home/akitaonrails/docker/, /home/akitaonrails/<service>/, with no pattern.

Substitutions applied across every compose file

Media stack (Plex, Radarr, Sonarr, etc.)

The media stack is the most complex. Plex needs its own LAN IP (macvlan) for direct streaming to work. The setup:

docker network create -d macvlan \
  --subnet=192.168.0.0/24 \
  --gateway=192.168.0.1 \
  -o parent=enp97s0 \
  plex_macvlan

In compose, Plex needs to be on two networks: the macvlan (for the IP 192.168.0.6) and the default bridge (so other containers like Seerr can talk to it):

plex:
  networks:
    plex_macvlan:
      ipv4_address: 192.168.0.6
      mac_address: "02:42:c0:a8:00:06"
    default: {}    # mandatory — without this, Seerr can't see Plex

A detail that almost broke me: Plex stores absolute paths in its database. If the container’s internal volume changed from /media to /data, Plex no longer finds anything. You have to use exactly the same mount target as the old compose.

Ollama with ROCm

New stack, didn’t exist on the previous server:

ollama:
  image: ollama/ollama:rocm
  container_name: ollama
  devices:
    - /dev/kfd:/dev/kfd
    - /dev/dri:/dev/dri
  security_opt:
    - seccomp:unconfined
    - label:disable
  group_add:
    - "485"   # render group GID
    - "488"   # video group GID
  environment:
    - HSA_OVERRIDE_GFX_VERSION=11.5.1
    - PYTORCH_ROCM_ARCH=gfx1151
    - OLLAMA_KEEP_ALIVE=30m
    - OLLAMA_NUM_PARALLEL=4
    - OLLAMA_FLASH_ATTENTION=1
    - OLLAMA_KV_CACHE_TYPE=q8_0
  volumes:
    - /var/lib/ollama:/root/.ollama
  ports:
    - "11434:11434"

OLLAMA_FLASH_ATTENTION=1 activates flash attention. OLLAMA_KV_CACHE_TYPE=q8_0 uses 8-bit KV cache, cutting the bandwidth needed per token in half. Free performance optimizations.

Monitoring (Grafana + Prometheus)

Grafana uses a named volume (grafana_data) which is NOT included in normal filesystem backups. That’s the reason I lost all my dashboards on the first attempt. The fix is an explicit backup of the named volume:

# On the old server:
docker run --rm -v grafana_data:/data:ro -v /tmp:/backup alpine \
  tar czf /backup/grafana_data.tar.gz -C /data .

# Transfer and restore on the new one:
docker run --rm -v grafana_data:/data -v /tmp:/backup alpine \
  sh -c "cd /data && tar xzf /backup/grafana_data.tar.gz"

Same thing for Portainer (portainer_data). Any volume defined in the volumes: block of a compose without a host path needs this treatment.

Cloudflare Tunnel

I use Cloudflare Tunnel to access all the services from outside the house without opening ports on the router. The migration was the easiest part: copy the tunnel’s JSON credentials file and the config.yml, update the IPs from .145 to .90, and bring the container up. The tunnel keeps the same ID, no need to recreate DNS.

The hostnames live in config.yml: portainer, grafana, plex, seerr, qbittorrent, syncthing, radarr, sonarr, bazarr, prowlarr, vault, gitea, kavita, and others. Everything accessible via https://<service>.example.com from anywhere.

Gitea (image registry)

Gitea acts as a private Docker registry on port 3007. The Frank FBI, Frank Mega, Frank Yomik and Mila projects have Docker images that are built and pushed to Gitea. For it to work, Docker’s daemon.json needs:

{
  "insecure-registries": ["192.168.0.90:3007"]
}

Gitea SSH gave me a problem during the migration: the old app.ini had SSH_LISTEN_PORT=22, but the container’s entrypoint also starts sshd on port 22. Conflict. Fix: GITEA__server__SSH_LISTEN_PORT=2222 as an environment variable in compose.

All 49 containers running

The migrated server runs 49 containers across 15 stacks. The media stack alone has 10 containers (Plex, Radarr, Sonarr, Bazarr, Prowlarr, qBittorrent, SABnzbd, Jackett, FlareSolverr, Seerr). The personal projects (Frank FBI, Frank Mega, Frank Yomik, Mila) add another 11. Monitoring with Grafana, Prometheus, node-exporter and cAdvisor. Utilities like Portainer, Vaultwarden, Syncthing, Organizr, Watchtower. Gitea as private Docker registry. Immich as self-hosted Google Photos. Kaizoku for manga with Kavita as the reader. Ollama with ROCm. And Bitcoin Core/Fulcrum indexing the blockchain off the NAS.

Backups: two layers

Layer 1: local btrfs snapshots (snapper)

/var lives on a 3.7TB btrfs partition. snapper creates automatic snapshots: 7 daily + 1 weekly. They’re crash-consistent, not application-consistent (postgres can be slightly inconsistent if there’s heavy writing during the snapshot).

To recover an accidentally deleted file:

sudo snapper -c var list
sudo cp /var/.snapshots/5/snapshot/opt/docker/media/radarr/appdata/config/radarr.db \
        /var/opt/docker/media/radarr/appdata/config/radarr.db

For a full stack rollback:

sudo docker compose -p media down
sudo snapper -c var undochange 7..0 /var/opt/docker/media
sudo docker compose -p media up -d

Layer 2: restic to the NAS (off-machine)

restic runs every night at 3am, doing incremental backups to /var/mnt/terachad/homelab-backups/. Retention: 7 daily + 4 weekly. Content-based deduplication, so Plex config (19GB) and Gitea repos (12GB) only transfer deltas.

Before restic runs, a pg_dump exports the postgres databases (Immich, Kaizoku). The dumps go to /tmp/homelab-db-dumps/ and are included in the backup.

What’s NOT included in the backup (re-downloadable): Bitcoin blockchain (785GB on the NAS), Docker images (re-pullable), Ollama models (re-downloadable), HuggingFace/EasyOCR caches, Plex transcoding scratch.

Large re-downloadable directories were converted to btrfs subvolumes so snapper ignores them: /var/lib/ollama and /var/opt/docker/bitcoin/fulcrum/fulc2_db.

Performance tuning

btrfs with zstd compression

Added compress=zstd:1 to the fstab for the /var partition. zstd level 1 has near-zero CPU overhead on NVMe and compresses Docker metadata, JSON configs and logs nicely. Incompressible data (SQLite, postgres) is automatically skipped by btrfs.

zram swap

With ~30GB of RAM available for the system (96GB go to VRAM), in-memory compressed swap helps. zram creates a ~15GB swap device (ram/2) with zstd compression, much faster than disk swap.

# /etc/systemd/zram-generator.conf
[zram0]
zram-size = ram / 2
compression-algorithm = zstd
swap-priority = 100

btrfs nodatacow on database directories

Copy-on-write + random database writes = write amplification. I disabled CoW on the directories that hold SQLite and postgres:

sudo chattr +C /var/opt/docker/gitea/data/gitea.db
sudo chattr +C /var/opt/docker/immich/db/
sudo chattr +C /var/opt/docker/media/radarr/appdata/config/

CPU in performance mode

On a headless server, there’s no point saving energy:

echo performance | sudo tee /sys/devices/system/cpu/cpu*/cpufreq/energy_performance_preference

Persisted via systemd service cpu-epp.service.

Docker shutdown fix

A problem I discovered: Docker ships with KillMode=process, which means at system shutdown, systemd kills only dockerd and leaves all the containerd-shim processes (one per container, ~49 in my case) orphaned. systemd-shutdown then has to hunt them down one by one after the journal has already stopped, causing a silent multi-minute hang.

Fix:

# /etc/systemd/system/docker.service.d/shutdown.conf
[Service]
KillMode=control-group
TimeoutStopSec=30

The problems we ran into

This is the table of actual problems we hit during the migration. If you’re planning something similar, read it before starting:

What to tell Claude Code before you start

If I were redoing the migration from scratch, I’d give Claude Code these instructions on the very first message. In order of importance:

Tell it that SELinux is enforcing and that it should NEVER use :Z on any Docker volume, but rather security_opt: label:disable on every service. Tell it that /var/mnt/terachad/ is an NFS mount and that :Z should never appear on NFS paths. Tell it to always look at the original compose before rewriting and only change IPs, paths and container names, without inventing new volume layouts. Warn that named volumes need explicit backup (Grafana, Portainer). Explain that Plex runs on macvlan and needs default: {} in the networks. Inform that the GPU is gfx1151, not officially supported, and that it needs UMA 96GB in the BIOS + kernel TTM params + HSA_OVERRIDE_GFX_VERSION=11.5.1. And tell it that Bitcoin/Fulcrum don’t process environment variables, everything goes as an argument in command:.

Those instructions would have prevented 80% of the problems we hit.

Final server layout

/var/opt/docker/
├── bitcoin/          (bitcoind + fulcrum)
├── cloudflared/      (Cloudflare tunnel)
├── frank_fbi/        (email fraud analysis)
├── frank_mega/       (Mega clone)
├── frank_yomik/      (manga translation)
├── gitea/            (Docker registry)
├── immich/           (self-hosted Google Photos)
├── kaizoku/          (manga downloader + reader)
├── media/            (Plex + *arr stack)
├── mila/             (Discord bot)
├── monitor/          (Grafana + Prometheus)
├── ollama/           (local LLM with ROCm)
├── rip/              (HandBrake)
└── utils/            (Portainer, Vaultwarden, Syncthing, etc.)

/var/mnt/terachad/    (Synology NFS)
├── Bitcoin/data/     (blockchain, 785GB)
├── Downloads/        (torrents + nzbget)
├── Videos/           (Radarr movies + Sonarr series)
└── Ollama/models/    (model overflow if local disk fills up)

A warning about using AI to administer servers

I used Claude Code to speed up the migration. It created compose files, wrote backup scripts, configured the firewall, diagnosed SELinux problems. It worked well for my case: home server, only I use it, and I was reviewing every step.

But there are traps. Claude doesn’t know that :Z breaks SQLite unless you tell it. It doesn’t know that Fulcrum doesn’t accept env vars unless it’s already seen the Dockerfile. It will invent “better” volume layouts that break Plex because Plex stores absolute paths in its database.

If it were real production: don’t do this without review. Every compose file Claude generates, read it in full before applying. Every destructive command (rollback, delete, recreate), confirm manually. And have tested backups before you start. Claude is great for generating the first version and diagnosing errors, but the architecture decisions and the safety validations are yours.

The previous home server posts that may give additional context:

• My “Personal Netflix” with Docker Compose
• Accessing my Home Server with a real domain
• Protecting your Home Server with Cloudflare Zero Trust
• Installing Grafana on my Home Server
• Self-hosted Vaultwarden

So I did what any programmer would do: sat down, wrote a spec document, and with the help of Claude Code (Claude Opus 4.6) I built FrankMega, a simplified, self-hosted Mega clone, in Rails 8, in a single day of work. 21 commits, 3 hours of active development, 210 tests, zero external dependencies beyond SQLite.

And before the smart guy shows up in the comments with “ah, but Vibe Coding is just prompt engineering, anyone can do it” – hold on. This post exists exactly to show that it isn’t.

From Idea to Code: the IDEA.md

Every project starts with a document. Mine was docs/IDEA.md – 56 lines describing what I wanted:

• Single file upload, with a shareable link
• Download counter (default 5, configurable)
• Automatic expiration within 24 hours
• Full authentication: password + 2FA + passkeys
• No public registration page – invites only
• Admin creates invites, manages users, bans abusers
• Aggressive rate limiting, automatic IP banning
• Deploy via Docker + Cloudflare Tunnel on my home server
• Stack: Rails 8, SQLite, Tailwind CSS, Hotwire/Stimulus

I deliberately added at the end: “Suggest important features that may be relevant for a service like this.” Because I know a spec document never covers everything. And that’s where the interesting part begins.

The Big Bang: Commit #1

The first commit (e277226) at 12:17 delivered 207 files, 6,901 lines of code. In a single commit. Yes, I know it’s controversial, but when you’re building from scratch with AI, the first commit is necessarily huge.

What came in that initial commit:

• Full authentication with has_secure_password (bcrypt), TOTP via rotp, passkeys via webauthn
• Invitation system with unique codes and expiration
• Full admin panel (Users, Invitations, Files, MIME Types)
• Upload with Active Storage + drag-and-drop via Stimulus
• Download links with 24-byte hashes (SecureRandom.urlsafe_base64(24))
• Rate limiting via Rack::Attack on every public endpoint
• Automatic IP banning with a Ban model and async job
• Dark/light theme with Tailwind CSS inspired by Mega.nz
• 73 tests (Minitest + FactoryBot)
• Docker configuration with Puma + Thruster
• Solid Queue/Cache/Cable – zero Redis

Got it? All of that in commit #1. The IDEA.md became functional code in a single shot. But here’s the point: the first commit was not production-ready. Not even close.

The Reality: 20 Iteration Commits

After the Big Bang came 20 commits over 3 hours. And this is where the truth about software development lives, the one no “Vibe Coding” tutorial will tell you.

Phase 1: “Works on my machine” (12:40 - 12:46)

Four commits in 6 minutes. You know what happened? I went to build the Docker image and it broke. Missing libssl-dev. Fixed it. Then the CSP (Content Security Policy) from secure_headers clashed with the inline scripts from Rails’ importmap. Fixed it. Changed the port from 3000 to 3100 to avoid colliding with other services on my server.

Look at commit 389ebe8:

# CSP is now handled by the Rails built-in, not by secure_headers
# Because secure_headers doesn't support nonces for importmap inline scripts
config.csp = SecureHeaders::OPT_OUT

This kind of gem incompatibility you only discover in the moment. No LLM will warn you about this up front because it’s a specific combination of versions.

Phase 2: The Security Commit – 22 Issues Fixed (13:10)

This is the most important commit of the entire project: 4a854a6. 35 files, 612 lines, 22 security issues fixed. I’ll go into detail because this is where toy code separates from production code.

What the initial commit got wrong:

CRITICAL – Encryption keys with hardcoded fallbacks:

# BEFORE (dangerous):
config.active_record.encryption.primary_key = ENV.fetch("KEY", "test-primary-key")

# AFTER (fail-fast):
if Rails.env.production?
  config.active_record.encryption.primary_key = ENV.fetch("ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY")
end

If the ENV var didn’t exist in production, the app would happily run with the hardcoded key. Anyone reading the source code could decrypt every user’s OTP secrets. Now in production, if the variable is missing, the app crashes on boot. That’s the correct behavior.

CRITICAL – Replay Attack on OTP:

# BEFORE:
def verify_otp(code)
  totp = ROTP::TOTP.new(otp_secret)
  totp.verify(code, drift_behind: 30, drift_ahead: 30).present?
end

# AFTER:
def verify_otp(code)
  totp = ROTP::TOTP.new(otp_secret)
  timestamp = totp.verify(code, drift_behind: 30, drift_ahead: 30)
  return false unless timestamp

  # Prevent replay: reject if this code's timestamp was already used
  if last_otp_at.present? && Time.at(timestamp) <= last_otp_at
    return false
  end

  update_column(:last_otp_at, Time.at(timestamp))
  true
end

Every TOTP tutorial online shows verify(code).present? and stops there. The problem: a TOTP code is valid for 30 seconds. If I intercept your code (shoulder surfing, camera, clipboard), I can use it multiple times in that window. last_otp_at guarantees real one-time use.

HIGH – Race Condition on the Download Counter:

# BEFORE (race condition):
def increment_download!
  increment!(:download_count)
end

# AFTER (atomic):
def increment_download!
  self.class.where(id: id)
      .where("download_count < max_downloads")
      .where("expires_at > ?", Time.current)
      .update_all("download_count = download_count + 1") == 1
end

increment! does: (1) read the value, (2) add 1 in Ruby, (3) write it back. Two simultaneous requests can both read 4, both write 5, and the user gets two downloads when they should only have had one. The atomic UPDATE ... WHERE in SQL guarantees only one request wins. And the == 1 returns success/failure in a single operation, with no lock.

This is the classic TOCTOU bug (Time of Check, Time of Use). You’ll never see it in local development with one browser. It only shows up in production with concurrent requests.

HIGH – Open Redirect after login:

# AFTER:
def safe_redirect_url?(url)
  uri = URI.parse(url)
  uri.host.nil? || uri.host == request.host
rescue URI::InvalidURIError
  false
end

Without this, an attacker could send a link like frankmega.com/session?return_to=https://evil.com and after login the user would be redirected to the malicious site.

MEDIUM – Predictable CSP nonce:

# BEFORE:
config.content_security_policy_nonce_generator = ->(request) { request.session.id.to_s }

# AFTER:
config.content_security_policy_nonce_generator = ->(_request) { SecureRandom.base64(16) }

Session ID doesn’t change between requests. If the nonce is predictable, an attacker can inject scripts with the correct nonce. It has to be random per request.

And more: Cloudflare IPv6 added to trusted proxies, minimum 12 characters for passwords, server-side MIME detection via Marcel, file size verified from the tempfile (not from the client header that can be forged), protection against deleting/banning/demoting the last admin.

The test count jumped from 73 to 109 with this single commit.

How Rails Makes Security Easier

Rails makes implementing security MUCH easier than doing it from scratch. Look at the comparison:

Rails 8 in particular brought rate_limit built into the controller:

class SessionsController < ApplicationController
  rate_limit to: 10, within: 3.minutes, only: :create,
    with: -> { redirect_to new_session_path, alert: t("flash.sessions.create.rate_limit") }
end

And authentication? Rails 8 now ships with a built-in authentication generator. You no longer need Devise for basic stuff. FrankMega’s app/controllers/concerns/authentication.rb is based on that and extends it with 2FA, passkeys and invitation-only registration.

File Sharing Is More Than Downloads

One thing lots of people don’t realize: a file sharing service has a huge attack surface. It goes way beyond “save file, generate link”. Look at everything you need to consider:

Filename Sanitization

def sanitize_filename(name, content_type = nil)
  sanitized = File.basename(name.to_s)                          # Strip path traversal
  sanitized = sanitized.encode("UTF-8", invalid: :replace,      # Handle invalid UTF-8
                                undef: :replace, replace: "")
  sanitized = sanitized.gsub(/[\x00-\x1f\x7f\/\\:*?"<>|]/, "") # Control chars + unsafe chars
  sanitized = sanitized.sub(/\A\.+/, "")                        # Leading dots (hidden files)
  sanitized = sanitized.gsub(/\s+/, " ").strip                  # Collapse whitespace

  # Windows reserved device names
  base_without_ext = sanitized.sub(/\.[^.]*\z/, "")
  if base_without_ext.match?(/\A(CON|PRN|AUX|NUL|COM[1-9]|LPT[1-9])\z/i)
    sanitized = "_#{sanitized}"
  end

  sanitized = strip_extension_junk(sanitized, content_type)
  sanitized = truncate_filename(sanitized, 255)
  sanitized.presence || "unnamed_file"
end

Every line exists for a reason:

• File.basename blocks path traversal (../../etc/passwd becomes passwd)
• Stripping control chars blocks null byte injection and terminal escape injection
• Windows reserved names (CON, NUL, AUX, LPT1-9) cause problems if storage is accessed from Windows
• Truncating to 255 bytes (not characters!) while preserving the extension – a 4-byte emoji counts as 4 toward the ext4/NTFS limit
• strip_extension_junk handles names that come from URLs with glued-on parameters: photo.jpg_1280x720+quality=80 becomes photo.jpg

And this is server-side. There’s also client-side validation in the Stimulus controller to reject before upload:

isInvalidFilename(name) {
  if (new Blob([name]).size > 255) return true
  if (/[\x00-\x1f\x7f/:*?"<>|\\]/.test(name)) return true
  return false
}

Server-Side MIME Type Detection

@shared_file.content_type = Marcel::MimeType.for(uploaded.tempfile, name: uploaded.original_filename)
@shared_file.file_size = uploaded.tempfile.size

Don’t trust the Content-Type the browser sends. Marcel inspects the file’s magic bytes to determine the real type. And the file size comes from the tempfile on disk, not from the Content-Length header which can be forged.

Quotas and Disk Usage

Each user has a disk quota (5 GB default, admin can customize):

def can_upload?(file_size)
  grace = Rails.application.config.x.security.disk_quota_grace_bytes
  (storage_used + file_size) <= (disk_quota + grace)
end

Without a quota, a single malicious user fills up the server disk. With the 10% grace buffer, an upload that slightly exceeds the limit is accepted so the user isn’t frustrated in the edge case.

Automatic Cleanup

class CleanupExpiredFilesJob < ApplicationJob
  queue_as :default

  def perform
    SharedFile.inactive.find_each do |shared_file|
      shared_file.file.purge if shared_file.file.attached?
      shared_file.destroy
    end
  end
end

Runs every 15 minutes via Solid Queue. If nobody downloaded in 24 hours, gone. If it hit the download limit, gone. Without this, the server disk fills up and the service dies.

Automatic IP Banning

class InvalidHashAccessJob < ApplicationJob
  queue_as :default

  def perform(ip_address)
    security = Rails.application.config.x.security
    return unless security.enable_banning

    cache_key = "invalid_hash:#{ip_address}"
    count = Rails.cache.increment(cache_key, 1, expires_in: 1.hour) || 1

    if count >= security.max_invalid_hash_attempts
      Ban.ban!(ip_address, reason: "Repeated invalid download hash access",
               duration: security.ban_duration)
      Rails.cache.delete(cache_key)
    end
  end
end

Tried 3 invalid hashes? Banned for 1 hour. That stops hash enumeration. With 24 bytes of entropy (192 bits, 2^192 possibilities) brute force is computationally infeasible, but the extra protection costs nothing.

Layered Rate Limiting

Rack::Attack operates at the middleware layer (before Rails processes the request):

# Login: 5 attempts per minute per IP
Rack::Attack.throttle("logins/ip", limit: 5, period: 1.minute) do |req|
  req.ip if req.path == "/session" && req.post?
end

# Login: 5 attempts per minute per email
Rack::Attack.throttle("logins/email", limit: 5, period: 1.minute) do |req|
  if req.path == "/session" && req.post?
    req.params.dig("email_address")&.to_s&.downcase&.strip
  end
end

# Downloads: 60 views and 30 downloads per minute
Rack::Attack.throttle("downloads_get/ip", limit: 60, period: 1.minute) do |req|
  req.ip if req.path.start_with?("/d/") && req.get?
end

# General: 300 requests per 5 minutes (except assets)
Rack::Attack.throttle("requests/ip", limit: 300, period: 5.minutes) do |req|
  req.ip unless req.path.start_with?("/assets")
end

Seven different throttles for specific scenarios. Plus Rails 8’s rate_limit in the controllers. Banned IPs never even hit Rails – they’re rejected at the middleware blocklist with status 403.

And the limits are configurable: 1x in production, 10x in development so you don’t lock yourself out during testing. Everything centralized in config/initializers/security.rb.

The Download Saga: 5 Commits in 21 Minutes

This part is fun because it shows real debugging. Between 14:44 and 15:05 I made 5 commits trying to get downloads working with Turbo Drive:

1. cc3c23d – Added data-turbo=false on the download button. Didn’t work.
2. 5224ed2 – Moved data-turbo=false to the <form>. Didn’t work.
3. 98b0d4f – Dropped redirect_to rails_blob_path and used send_file directly. Partially worked.
4. 1fdf5f9 – Switched from POST to GET link. Worked, but… bots can burn downloads with GET.
5. 4c74b27 – Went back to POST. Added URL artifact cleanup in filenames.

The root cause: Turbo Drive intercepts navigation aggressively, and downloading a file via redirect_to (which generates a chain of 302 redirects from Active Storage) confuses Turbo. The final fix: send_file straight from disk with disposition: "attachment" via POST with data: { turbo: false } on the form.

This kind of problem an LLM won’t solve on its own. You need to test in a real browser, see what happens, try, fail, adjust. It’s pure iterative development.

The Gap Between the Idea and the Final Product

Look at what the IDEA.md didn’t mention and that emerged during development:

1. Per-user disk quotas – without these one user fills the server disk
2. Full I18n (EN + PT-BR) – 66 files, ~250 strings extracted
3. Terms of Service with mandatory acceptance
4. User self-deletion – privacy requirement
5. Blocked downloads for banned users – their links return 410 Gone
6. Client-side upload validation – checks size, quota and filename before upload
7. Inline file previews – images, video and audio on the download page without consuming downloads
8. Styled error pages – instead of plain text “Not Found”, branded pages with i18n

The IDEA.md said “use activeadmin or administrate”. In practice I built a custom admin panel with Tailwind because administrate had compatibility issues with Rails 8.1. The IDEA.md mentioned “progress bar if possible”. In practice I used drag-and-drop with a file preview via Stimulus, which is better than a progress bar.

Nobody can predict every feature on day 1. You discover requirements as you build. And each new feature brings edge cases that need handling.

The Numbers

At the end of the project:

• 21 commits in a ~3-hour session
• 210 tests (Minitest + FactoryBot), 513 assertions, zero failures
• ~3,100 lines of application code (models, controllers, views, JS, CSS, configs)
• ~1,965 lines of tests (including dedicated security tests)
• 9 models, 20 controllers, 36 views, 9 Stimulus controllers
• 24 distinct security measures across 7 layers
• CI: SimpleCov, RuboCop (zero offenses), Brakeman (zero warnings), bundler-audit (zero vulnerabilities)
• Zero external dependencies beyond SQLite. No Redis, no PostgreSQL, no external queue service

The Invitation-Only Server

FrankMega has no public signup. The flow is:

1. On first run, with no users, the setup screen appears to create the admin
2. The /setup route only exists while User.count.zero? – afterward it vanishes entirely:

constraints(->(request) { User.count.zero? }) do
  get "setup", to: "setup#new"
  post "setup", to: "setup#create"
end

3. The admin creates invites in the admin panel (with expiration)
4. Each invite generates a unique 16-byte code
5. Each code can only be used once (with_lock + transaction to prevent a race condition)
6. After signup, the invite is marked as “used”

This is perfect for a personal/family service. Full control over who gets in.

Deploy: Docker + Cloudflare Tunnel

To wrap up, the deploy tutorial in case you want to run this on your own Docker home server. It’s not complicated but there are details that matter.

The Dockerfile

Multi-stage build in 3 stages:

# Stage 1: Base (runtime)
FROM ruby:3.4.8-slim AS base
RUN apt-get install -y curl libjemalloc2 libvips sqlite3
# jemalloc for better memory management

# Stage 2: Build (throwaway)
FROM base AS build
RUN apt-get install -y build-essential git libssl-dev
# Bundle install + asset precompilation with dummy env vars

# Stage 3: Final
# Copies gems and app, runs as non-root user (rails:rails, UID 1000)
CMD ["./bin/thrust", "./bin/rails", "server"]

bin/thrust is Thruster (from Basecamp) – an HTTP proxy in front of Puma that handles gzip/brotli compression, asset caching, X-Sendfile acceleration. It listens on port 80 and proxies to Puma on 3000.

The entrypoint runs db:prepare and db:seed automatically at boot:

#!/bin/bash -e
if [ "${@: -2:1}" == "./bin/rails" ] && [ "${@: -1:1}" == "server" ]; then
  ./bin/rails db:prepare
  ./bin/rails db:seed
fi
exec "${@}"

docker-compose.yml

Two services:

services:
  web:
    image: akitaonrails/frankmega:latest
    ports:
      - "3100:80"
    volumes:
      - /home/seuuser/frankmega/uploads:/rails/storage/uploads
      - /home/seuuser/frankmega/db:/rails/storage
    environment:
      SECRET_KEY_BASE: ${SECRET_KEY_BASE}
      HOST: ${HOST}
      # ... around 15 more variables
    restart: unless-stopped

  tunnel:
    image: cloudflare/cloudflared:latest
    command: tunnel run
    environment:
      - TUNNEL_TOKEN=${TUNNEL_TOKEN}
    depends_on:
      - web
    restart: unless-stopped

The cloudflare/cloudflared sidecar creates an outbound tunnel to Cloudflare’s edge. No need to open ports on your home server’s firewall. Cloudflare handles SSL termination, DDoS protection, and routes traffic to the container.

Step by Step

1. Create the Tunnel on Cloudflare:

• Cloudflare Zero Trust Dashboard > Networks > Tunnels > Create
• Type: Cloudflared
• Copy the TUNNEL_TOKEN
• Configure the hostname: frankmega.yourdomain.com pointing to http://web:80

2. Generate the secrets:

# SECRET_KEY_BASE
openssl rand -hex 64

# Encryption keys (3 separate values)
openssl rand -hex 32  # PRIMARY_KEY
openssl rand -hex 32  # DETERMINISTIC_KEY
openssl rand -hex 32  # KEY_DERIVATION_SALT

3. Configure .env:

SECRET_KEY_BASE=<generated above>
RAILS_MASTER_KEY=<from config/master.key>
HOST=frankmega.seudominio.com
WEBAUTHN_ORIGIN=https://frankmega.seudominio.com
WEBAUTHN_RP_ID=frankmega.seudominio.com
ACTIVE_RECORD_ENCRYPTION_PRIMARY_KEY=<generated>
ACTIVE_RECORD_ENCRYPTION_DETERMINISTIC_KEY=<generated>
ACTIVE_RECORD_ENCRYPTION_KEY_DERIVATION_SALT=<generated>
TUNNEL_TOKEN=<from Cloudflare>
FORCE_SSL=true
APP_LOCALE=pt-BR

4. Bring it up:

docker compose pull
docker compose up -d

5. Visit https://frankmega.yourdomain.com, create the admin account, start using it.

Important Caveats

• FORCE_SSL=false to test locally at http://localhost:3100 without Cloudflare in front, otherwise you hit a redirect loop
• The WebAuthn origin has to match exactly, including https://. If you get it wrong, passkeys silently stop working
• Cloudflare IPs are hardcoded in the initializer (because the cloudflare-rails gem isn’t compatible with Rails 8.1). If Cloudflare changes the ranges, you need to rebuild the image
• Encryption keys are permanent. If you swap them after users configure 2FA, the OTP secrets become unreadable and users lose access
• Cloudflare free has a 100 MB upload limit. On the free plan, files above that don’t make it through the tunnel. The app allows up to 1 GB but Cloudflare is the bottleneck
• The container runs as UID 1000. If you use bind mounts instead of named volumes, the directory has to be owned by UID 1000
• There are 4 separate SQLite databases in production (app, cache, queue, cable), all in the db_data volume. Backup means copying the entire volume

The Real Point About “Vibe Coding”

Claude Code is absurdly productive. I wouldn’t have done all of this in 3 hours without it. Remember when you’d bang your head against the wall for a week just to get login working? Yeah.

The I18n commit alone (66 files, ~250 strings extracted for two languages) would take a full day by hand.

But here’s the point nobody wants to hear: the LLM did not make the security decisions. I asked it to run an audit and it found the problems, but I had to know how to ask. And I had to know which questions to ask. “Review security” is a vague instruction. “Check if there’s a race condition on the download counter increment” is an instruction only someone who knows what TOCTOU is can give.

Those 56 lines of IDEA.md carry decades of web development experience. I knew to ask for rate limiting because I’ve seen services taken down by bots. I knew to ask for IP banning because I’ve dealt with abuse. I knew to ask for an atomic download counter because I understand concurrency.

If a beginner wrote the same IDEA.md, they probably wouldn’t have half of those concerns. And the LLM wouldn’t spontaneously suggest them. The result would be a service that’s functional, pretty, and completely insecure.

The experienced programmer’s value in Vibe Coding isn’t writing code. It’s knowing what to ask for and reviewing what was generated. Experience is the filter between “works” and “works in production”.

Those 210 tests and 24 security measures didn’t come from prompt engineering. They came from knowing what to test and why.

The code is open source under AGPL-3.0: github.com/akitaonrails/frank_mega. Deploy it on your server, poke around the code, learn from the commits. And if you find more security flaws, let me know.

Remember:

AI is the mirror of your own competence.

More important: having a strong, random, exclusive and different password for EACH site, app or service you use. Never, ever, reuse a password. Why? Because every day one of these sites gets breached and passwords get leaked. If you search for yourself on Have I Been Pwned you’ll surely find accounts of yours that have already leaked. And if you share that same password across 10 other sites, someone has probably already gotten into your account by now.

Watch my video about security:

Being too lazy to use a password manager is just a dumb excuse from lazy people. And a programmer has zero right to use that excuse.

Omarchy 1Password

Anyone who has followed DHH since the early days when Macs were the best platform for developers remembers he was always a fan of 1Password, one of the apps that worked best on iPhones.

I migrated back to Android and Linux many years ago, in 2015. I used 1Password, but on Android the app was horrendous, ugly, and painful to use. That’s why I gave up on it and kept bouncing between several other apps. Google Authenticator, Lastpass (which I don’t recommend).

I know there are several other open source ones that people prefer, but I ended up settling on Bitwarden, which is BEAUTIFUL both on Android and as a browser extension. On top of that, it has open components and allows self-hosting.

More than that, it doesn’t make it hard to export all the data from your account and import it somewhere else. Unlike apps like Twilio Authy, which locks you out of everything and is why I stopped using it.

Bitwarden has a secure random password generator, supports 2FA TOTP, lets you store secure notes (mini texts) and a lot more. The apps integrate with phone biometrics, so it’s very convenient to use.

I still keep my Bitwarden inside Samsung’s Secure Vault, the Samsung Knox Vault. So there are 2 layers of security. And when I need to walk around somewhere suspicious, I don’t take my main phone, only a burner phone with nothing important logged in.

Look how beautiful it is. I like it and I’ve been using it for a few years now. My passwords range from 21 to 42 random characters, a different one for each site. I’ve updated all the passwords for everything more than once in the past few years, and all accounts have 2FA/TOTP registered inside Bitwarden too, which is how I managed to drop Authy.

Installing VaultWarden Self-Hosted

More important, the time has come to host my own Bitwarden server, vaultwarden, which is an open source alternative, code open on GitHub, so you can poke around yourself to learn or even contribute to it. Again, I bring everything up with Docker Compose on my Home Server, so I just needed to add this snippet to my file:

  vaultwarden:
    image: vaultwarden/server:latest
    container_name: vaultwarden
    restart: unless-stopped
    environment:
      DOMAIN: "https://bitwarden.fabioakita.dev"
      SIGNUPS_ALLOWED: "false"          # set to true only to create first user
      WEBSOCKET_ENABLED: "true"
      ADMIN_TOKEN: "iA...Q9"  # for /admin
      # SMTP (recommended)
      SMTP_HOST: "smtp.gmail.com"
      SMTP_PORT: "587"
      SMTP_SECURITY: "starttls"
      SMTP_USERNAME: "boss@akitaonrails.com"
      SMTP_PASSWORD: "xxx"
      SMTP_FROM: "boss@akitaonrails.com"
    volumes:
      - /home/akitaonrails/vaultwarden/data:/data
    ports:
      - "9999:80"
    healthcheck:
      test: ["CMD", "wget", "-q", "-O", "-", "http://localhost/health"]
      interval: 30s
      timeout: 5s
      retries: 5

Important:

• Create a long, random ADMIN_TOKEN. Use Bitwarden itself for that, keep it secret.
• Set SIGNUPS_ALLOWED to true only once, log into the local site and create your new account. Then change it back to false in the file and restart the container. That way nobody else will be able to create any new accounts.
• It’s recommended to configure a GMAIL SMTP to send notification emails. For that, you need to create an app password in Google.

To create a password exclusively for vaultwarden, go to Google App Passwords and generate a new exclusive secret password:

It will only show you that password once. Copy and paste it straight into the docker compose yaml. With email properly configured, every time someone logs in, it sends an email notification. That’s good to make sure nobody shady is messing with your server:

Now I just need to bring up the container:

docker compose -f utils-docker-compose.yml up -d

Routing the Cloudflared tunnel

As I already showed in the Cloudflared post now I want to be able to access it remotely, via secure tunnel, using a public URL like https://bitwarden.fabioakita.dev. To do that we have to edit that /etc/cloudflared/config.yml file and add this to the end of it:

  - hostname: bitwarden.fabioakita.dev
    service: http://192.168.0.200:9999
    originRequest:
      httpHostHeader: bitwarden.fabioakita.dev
      http2Origin: true
  - service: http_status:404

Then we have to restart the service and also add the route manually:

sudo cloudflared tunnel route dns homelab bitwarden.fabioakita.dev
sudo systemctl restart cloudflared

And done! It might take up to 5 minutes for DNS to sync and all that, but then just hop into your browser and you’ll see this:

If that SIGNUPS_ALLOWED is true there will also be a link to “Create new account”. In my case I already did that and turned it off, so there’s only login.

Importing your Data

Before moving on, let’s open the Bitwarden extension - which is still logged into the cloud version at bitwarden.com - and export all my data:

Just authenticate, then on the bottom icon bar you’ll find “Settings”, then “Vault Options” and finally “Export vault”. Leave the default JSON format (it will be an encrypted JSON) and download that file somewhere safe.

Now we can log back into our self-hosted version at https://bitwarden.fabioakita.dev and go through the same menu, but choose “Import”:

And done! Now my self-hosted version is identical to the cloud version I had been using until now.

Logging into the new self-hosted server

Now just log out of both the mobile App and the browser extension. Then on the login screen you’ll see the following option:

On the web extension the window doesn’t let you see that option, use the mouse scroll and scroll to the bottom of the window and you’ll see that small option. Choose “self-hosted” and a window will open to type the URL https://bitwarden.fabioakita.dev and done!

From here on the usage is exactly the same as before. The interface is the same, the options are the same. Configure biometrics, PIN, lock timeout and everything else the way you like it.

Do the same on the mobile app and done!

On this new account we created on the self-hosted server, don’t forget to enable 2FA too, which is the “Authenticator App” option under “Settings/Security”:

For all other accounts, we can register the 2FA inside Bitwarden. But for the master account of Bitwarden/Vaultwarden itself, we need to use a second app just for that.

I don’t know if it’s the best, but I’ve been enjoying using Aegis Authenticator on my Android:

It’s open source, the code is on GitHub, so it seems minimally trustworthy. I also keep it inside the Samsung Knox Vault. I only need it when I have to re-login into Bitwarden/Vaultwarden. To log into Google, Steam or any other account, the 2FA for each one lives inside Vaultwarden itself:

Conclusion

Now all that’s left is to log in with my old account at bitwarden.com, which is the “official” cloud version, and delete my account, to make sure nobody else has access.

And there, PERMANENTLY DELETED! My accounts no longer exist on some random server I don’t control, even if they were encrypted.

I’m pretty happy with this solution. Now my Vaultwarden is in a container, the home server data gets automatic backups to my NAS, everything is encrypted and access only happens via Cloudflare’s secure tunnel. The risk is very low and nobody has a chance of getting access to my data, because it’s no longer in the “cloud”.

My data is mine alone.

I had been too lazy to install Grafana back then, but I finally got around to it, and it was much easier than I expected, which is why I decided to share how I did it.

The idea is to spin up cAdvisor, which monitors container resources, and Prometheus, which scrapes system resource data, then consolidate the data from both into Grafana, a web application that lets you build graphical dashboards to monitor your systems. If you do it right, in the end you’ll have a screen like this:

Installation

Spinning this up on my home server is very simple. I do everything with Docker Compose, so:

# monitor-docker-compose.yml
networks:
  monitoring:
    driver: bridge

volumes:
  prometheus_data:
  grafana_data:

services:
  node-exporter:
    image: quay.io/prometheus/node-exporter:latest
    command:
      - --path.rootfs=/host
    pid: host
    restart: unless-stopped
    networks: [monitoring]
    volumes:
      - /:/host:ro,rslave
    security_opt: ["no-new-privileges:true"]
    read_only: true

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    restart: unless-stopped
    networks: [monitoring]
    ports: ["8080:8080"]   # optional: expose UI locally
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    security_opt: ["no-new-privileges:true"]
    read_only: true

  prometheus:
    image: prom/prometheus:latest
    restart: unless-stopped
    networks: [monitoring]
    volumes:
      - prometheus_data:/prometheus
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.retention.time=30d
      - --web.enable-lifecycle
    ports: ["9090:9090"]
    security_opt: ["no-new-privileges:true"]
    read_only: true

  grafana:
    image: grafana/grafana:latest
    restart: unless-stopped
    networks: [monitoring]
    depends_on: [prometheus]
    ports: ["3001:3000"]
    environment:
      GF_SECURITY_ADMIN_USER: admin
      GF_SECURITY_ADMIN_PASSWORD: change-me
      GF_USERS_ALLOW_SIGN_UP: "false"
    volumes:
      - grafana_data:/var/lib/grafana
      - ./provisioning/datasources:/etc/grafana/provisioning/datasources:ro
    security_opt: ["no-new-privileges:true"]

Pay attention to the local directory mappings. If you put /etc/grafana/prometheus.yml, for example, you have to change this line:

...
    volumes:
      - prometheus_data:/prometheus
      - /etc/grafana/prometheus.yml:/etc/prometheus/prometheus.yml:ro
...

Speaking of which, here’s the prometheus.yml:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs: [{ targets: ['prometheus:9090'] }]

  - job_name: 'node'
    static_configs: [{ targets: ['node-exporter:9100'] }]

  - job_name: 'cadvisor'
    static_configs: [{ targets: ['cadvisor:8080'] }]

And finally /etc/grafana/provisioning/datasources/datasource.yaml:

apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

Be careful with port mappings too. The recommended approach is this:

ports: ["127.0.0.1:8080:8080"]

That’s because cAdvisor and Prometheus only need to be accessible to Grafana, inside the home server, and there’s no reason to bind on 0.0.0.0 and have them reachable from outside the server. That’s your call. Since I’m on a home LAN and there are no tunnels pointing to these services, it makes no difference to me.

If, like me, you already have a service sitting on port 8080, cAdvisor will fail. In that case, remap the ports and reconfigure Prometheus, or just remove the port line. You don’t need cAdvisor exposed externally.

Another detail: by default, Grafana comes up on port 3000, but I changed it to 3001 because I already had another service on that port. So once it’s up, I can access it at http://192.168.0.200:3001.

As I explained in the Cloudflared article, I route this address through Cloudflare to get a nice domain like https://grafana.fabioakita.dev to access it both locally and remotely, if I want. Go read those other articles afterward to understand how.

And that’s it, now just spin up the containers:

docker compose -f monitor-docker-compose.yml up -d

Done, super simple. I should have done this sooner.

As you can see in the Docker Compose file, when you open the site for the first time, the login is admin with password change-me. Obviously, the first thing to do is change the password:

Dashboards

Now we’re practically done. We can check whether Prometheus is working and integrated. Just go to the “Connections” menu and “Data Sources”:

Grafana can consolidate data from dozens of different sources, and Prometheus is only one of them. In a larger, more complex infrastructure, this list would be huge.

To build dashboards, you can start from scratch and assemble them widget by widget, but it’s easier to import dashboards that already exist. Just click the magnifying glass at the top, in the search box, and choose “Import dashboards”:

On that page you can either import a JSON file, copy and paste the JSON, or search by an ID already registered on grafana.com:

Right off the bat, I learned that there are 3 IDs that are useful for me:

• 1860 - Node Exporter Full - the screenshot at the beginning of this article, with several system metrics like CPU, memory, disk, network, etc. The most useful one for me.
• 13484 - Docker/cAdvisor Exporter - CPU and memory metrics, but broken down per container running on the server.
• 3662 - Prometheus 2.0 overview - a general health summary.

If you like these or other dashboards, just favorite them with a “pin” and you’re set.

For example, here’s the cAdvisor dashboard:

I haven’t explored other dashboards yet. If you have ID recommendations to import, drop them in the comments below!

With this, I have a quick and good-looking view of my home server’s resource usage, and I can spot anything weird happening right away. These services don’t seem to consume many resources, so I can leave them running alongside the other containers.

This way I can access https://portainer.fabioakita.dev (fake example domain) from my phone on 5G, and it hits the portainer Docker container sitting on my LAN at http://192.168.0.200:9000, which was previously only reachable from home. All through a secure tunnel.

Note that I keep saying I’m using fake example domains and IPs throughout these posts, because I don’t want to expose the real domain.

Just by sitting behind a Cloudflare secure tunnel, I’m already under several layers of protection like DDoS defense and many other well-known attack mitigations. And although I take care to configure every service with a strong, random, unique password per service, the risk isn’t zero that someone could find my services and try to brute-force them, or exploit some zero-day vulnerability in one of the many services like Prowlarr, Sonarr, and friends.

So I’m still at the mercy of undiscovered bugs that might pop up out of nowhere, particularly on each service’s login screen or API endpoints.

To avoid that risk, there’s another Cloudflare product worth configuring: Zero Trust. The idea is simple: add a new login layer on top of everything, from Cloudflare itself, to protect each service. That way, someone has to break Cloudflare before they even get a chance to try breaking one of my services.

1. Sign up for the service

This is the easy part. For few services and few users (just me), there’s a free plan, just sign up:

2. Create OAuth Credentials in Google Cloud

Since I pay for Google Workspace, I decided to use integrated login with Google OAuth, but it can be any other provider, pick whatever you prefer. For this I need to go to my Google Cloud console and enter the API & Services section. Here we create credentials for an OAuth client ID and configure it like this:

The name doesn’t matter, it’s what shows up on the login screen. What matters are the Javascript Origins fields, which must follow this format:

https://<your-team-name>.cloudflareaccess.com

And the redirect URIs field, which is where Google’s login screen will redirect to, and it has to point back to our Cloudflare:

https://<your-team-name>.cloudflareaccess.com/cdn-cgi/access/callback

Once that’s done, it gives you the option to download a JSON file with the metadata and secret. What’s important is that at the end it gives you an OAuth Client ID, which is a URL in the format xxxxxx.apps.googleusercontent.com, and most importantly, a Client Secret, which you should store in BitWarden or another secure place.

4. Creating a Login Method in Zero Trust

Now back to the Zero Trust console on Cloudflare, click “Settings” at the bottom of the menu, enter “Authentication”, and create a new Login Method:

After clicking “Add new”, just fill in the data we brought from Google Cloud:

Again, use whatever name you want, paste the OAuth Client ID as “App ID” and the Client Secret, and that’s it, now we have Google OAuth integrated as a login method in our Zero Trust.

5. Creating Policies

Now comes the part where I’m not sure if I did it entirely right. If I didn’t, let me know in the comments below!

My understanding is that we need to create at least two Policies, one for Service Auth and another for the Email I want to allow coming in from Google.

To continue, just click “Add a Policy” and edit the first policy:

The name can be anything. What matters is that under “Action” you pick “Service Auth” and under “Add Rules” you pick Any Access Service Token (this is the part I think is wrong, but I couldn’t find a better option - I didn’t quite understand how Service Token works).

Now we need to create the second policy like this:

This time what matters is having “Action” as “Allow” and adding the Rule “Emails”, picking the email you want to authenticate. In my case, just me.

6. Creating a new Application

Now we can create a new application in Zero Trust:

Click “Add an Application” and pick the type as “Self-Hosted”:

My understanding is that each application supports at most 5 hostnames. I have quite a few more than that, so I’ll have to create more than one application. The configuration in all of them is the same. Under “Basic Information” I just registered the subdomains of each container I want to protect.

Going to the “Policies” tab, just click “Select existing policies” and add the two policies we just created. That’s it.

Finally, under “Login methods”, just pick the Google one we configured earlier. And yes, these are the editing screens of an application I had already created. The new application screen has all of this on a single page, but you get the idea.

And as far as I understand, that’s all. Theoretically, everything is already working!

7. Testing the new Login

If nothing went wrong, just hit one of the hostnames we registered in the Zero Trust application, for example https://immich.fabioakita.dev, and we should see this screen first:

If this shows up, success! Zero Trust blocked access to my application until I logged in to it first.

Later you can customize the text however you want, but since it’s just for me, the “welcome” copy doesn’t matter, because actually, nobody is welcome here, just ME!

Clicking “Google” opens the familiar Google login pop-up, and only after that do I get to reach my application’s own login underneath. And this applies to every hostname I added to the Zero Trust applications.

Once you log in with Google using the correct user, the session stays valid for 24 hours (which is what we configured in the policies and application). During that window you don’t need to log in again on Zero Trust for any of the other applications on my home server. The session is shared across all applications.

8. Logging out Sessions

Once you’ve done the first login with your user, you can control that session by going to “My team” and “Users”:

Clicking on the user, you get a bunch of info and options, in particular see “Revoke Sessions”.

This way we can test the login without waiting 24 hours for the auth token to expire on its own. It’s a way to force logout your users. On the Zero Trust free plan, from what I understand, you can have up to 10 users in your organization.

Conclusion

I had never used Zero Trust before, so I might have made some mistake in the procedure. Read the documentation too. In principle, it seems to be working. I already created more applications and registered the rest of my hostnames.

Just by using CloudflareD for encrypted tunnels, all traffic reaching my home server goes through a series of Cloudflare protections that include: DDoS protection, URL normalization, Origin Rules, IP Address Rules, WAF (Web Application Firewall), Bots Mitigation, Rate Limiting, Cache, Compression and more.

These layers protect against the main well-known attack types, and they’re always adding new rules as each new vulnerability becomes known. Even if you, as a developer, don’t know how to protect against everything in your application, there’s a chance Cloudflare protects you first. That’s why I recommend keeping your domains under Cloudflare’s control.

On top of all of that, we also add Zero Trust, which guarantees that only after a login with my Google account will it start allowing access further down. And at the end of that there’s still one last barrier: the individual login screens of each application running on my home server.

Beyond that, the Cloudflare dashboard has traffic monitoring, analytics and much more. I configured just the basics, but if you want to go deeper, there are plenty of extra policies and rules you can add.

For someone to get through all of this and actually break into my home server and my LAN, they’d have to be really dedicated and really smart - I can’t picture anyone like that wasting so much time just to access a video server 🤣 but they’re welcome to try.

I have a nice Synology DS1821+ that also supports running Docker, but I prefer to keep the NAS doing only NAS duties and avoid risking my data by running other stuff on it. Having a single server that does everything isn’t a “problem” per se, but since I can, I prefer to separate things.

This Intel NUC runs a headless Ubuntu Server LTS (no desktop manager), which I access only via SSH (as I explained in the previous post about SSH). On it I run things like Portainer, Plex, Sonarr, Bazarr, qBitTorrent and much more.

The way it’s set up, I just added an entry to my desktop’s /etc/hosts like this:

192.168.0.100 synology.lan
192.168.0.200 plex.lan

From now on all my domains and IP addresses will be fake, obviously.

Anyway, this way I can open my browser and go to http://plex.lan:32400 to access my Plex, for example. But this has three disadvantages:

1. I can only use these hostnames on my desktop; from my laptop I’d need to edit the hosts file too. On top of that, I need to know the ports for each container.
2. I have no SSL for anything, so all authentication goes in plain text. It’s not a huge problem because the traffic is only internal on my local LAN. I haven’t been accessing it from outside the house.
3. If I ever want to access it from outside, none of this works.

There are dozens of ways to solve this problem, solutions involving WireGuard, or commercial solutions like TailScale and so on. But there’s a reasonably simple solution that solves all the problems at once: Cloudflare Tunnels.

The idea is to create secure tunnels via Cloudflare to access my services on my internal LAN WITHOUT having to expose ports outside my router or anything like that. My network stays isolated and unchanged.

1. Registering a Domain

The first recommendation is to register a real domain. Doing this lets you have real SSL certificates that Cloudflare itself manages and renews automatically.

I have a few domains registered and I’m not going to reveal which one I use for my Home Server. I’ll use this empty fabioakita.dev as an example just for this post.

Domains are cheap, around USD 12 per year, depending on the name you choose. Pick one that’s short and easy to type.

2. Installing Cloudflared

With the domain created, now we need to install the cloudflared service on the Home Server. It’s the one that connects to Cloudflare’s servers and opens the tunnel into my local server. Click the link for installation instructions, but on an Ubuntu Server 22.04 it goes like this:

# Add cloudflare gpg key
sudo mkdir -p --mode=0755 /usr/share/keyrings
curl -fsSL https://pkg.cloudflare.com/cloudflare-main.gpg | sudo tee /usr/share/keyrings/cloudflare-main.gpg >/dev/null

# Add this repo to your apt repositories
echo 'deb [signed-by=/usr/share/keyrings/cloudflare-main.gpg] https://pkg.cloudflare.com/cloudflared jammy main' | sudo tee /etc/apt/sources.list.d/cloudflared.list

# install cloudflared
sudo apt-get update && sudo apt-get install cloudflared

Now we need to log in:

# login
sudo cloudflared tunnel login

# add the homelab tunnel
sudo cloudflared tunnel create homelab

# create the config directory
sudo mkdir -p /etc/cloudflared

Use the same credentials you use to log in to the Cloudflare website. If everything goes well, at the end it will create a file at /root/.cloudflared/xxxxx-yyyy-zzzz...json, which holds your credentials, just like an SSH private key, so don’t throw it on a USB stick or some cloud drive. Keep it very safe. Let’s move it:

sudo mv /root/.cloudflared/*.json /etc/cloudflared
sudo chmod 600 /etc/cloudflared/*.json

Finally, we need to create a configuration file /etc/cloudflared/config.yml with all the hostnames we want for each of the containers, pointing to the correct ports. Here’s a snippet of my config, modified with fake IPs and the fake domain:

# /etc/cloudflared/config.yml
tunnel: homelab
credentials-file: /etc/cloudflared/xxxx-yyyy-....json
ingress:
  ...
  - hostname: plex.fabioakita.dev
    service: http://192.168.0.254:32400
    originRequest:
      httpHostHeader: plex.fabioakita.dev
  - hostname: immich.fabioakita.dev
    service: http://192.168.0.200:2283
    originRequest:
      httpHostHeader: immich.fabioakita.dev
  - hostname: overseer.fabioakita.dev
    service: http://192.168.0.200:5055
    originRequest:
      httpHostHeader: overseer.fabioakita.dev
  - hostname: syncthing.fabioakita.dev
    service: http://192.168.0.200:8384
    originRequest:
      httpHostHeader: qbittorrent.fabioakita.dev
  - hostname: radarr.fabioakita.dev
    service: http://192.168.0.200:7878
    originRequest:
      httpHostHeader: radarr.fabioakita.dev
  - hostname: sonarr.fabioakita.dev
    service: http://192.168.0.200:8989
    originRequest:
      httpHostHeader: sonarr.fabioakita.dev
  - hostname: bazarr.fabioakita.dev
    service: http://192.168.0.200:6767
    originRequest:
      httpHostHeader: bazarr.fabioakita.dev
  - hostname: prowlarr.fabioakita.dev
    service: http://192.168.0.200:9696
    originRequest:
      httpHostHeader: prowlarr.fabioakita.dev
  - hostname: portainer.fabioakita.dev
    service: http://192.168.0.200:9000
    originRequest:
      httpHostHeader: portainer.fabioakita.dev
  - hostname: grafana.fabioakita.dev
    service: http://192.168.0.200:3001
    originRequest:
      httpHostHeader: grafana.fabioakita.dev
  - service: http_status:404 # this has to be the LAST ONE

As you can see, it’s an extremely simple file. I think anyone can understand what it does. Modify it for your own needs.

After saving the file and going back to the command line, you have to manually run this command for each hostname:

sudo cloudflared tunnel route dns homelab plex.fabioakita.dev

But that’s a lot of work with this many hostnames, so I found it easier to create a script /etc/cloudflared/add.sh like this:

# run only once
TUN=homelab
CONF=/etc/cloudflared/config.yml
grep -E '^[[:space:]]*-[[:space:]]*hostname:' "$CONF" \
| awk '{print $3}' | tr -d '"' | tr -d "'" \
| while read -r h; do
    echo "adding $h"
    sudo cloudflared tunnel route dns "$TUN" "$h"
  done

Now just run: sudo sh /etc/cloudflared/add.sh, only once! So make sure your config file is correct first!

Finally, we can configure the service to restart by itself on every boot:

sudo cloudflared --config /etc/cloudflared/config.yml service install

sudo systemctl enable --now cloudflared

3. Accessing your services via the tunnel

If nothing went wrong, you’re done! You can go straight to your browser and test it:

How cool is that. Notice in the URL bar I typed https://immich.fabioakita.dev and it just worked. By the way, if you didn’t know it, Immich works identically to Google Photos, except entirely under your control! It has a smartphone app and syncs nicely.

More importantly, notice it’s on HTTPS, which was provided by Cloudflare for my entire domain. When I log in, it will go through an encrypted tunnel. And more than that: even if I’m on 5G, away from home, on my phone, I’ll be able to access my photos using this address!

From my phone it opens a secure tunnel to Cloudflare, and that cloudflared service on my home server also has a secure tunnel open to Cloudflare, so Cloudflare bridges the two tunnels and done! I have remote access without exposing my server directly to the internet! The best of both worlds!

Don’t forget to configure all your services with strong passwords (I generate them via Bitwarden, but any service works, 1Password or whatever):

4. Bridge to vlans

I don’t remember exactly why, but my Plex container alone gave me more trouble. That’s because it uses a different VLAN than the others. Again, I don’t remember why I did this at the time, but it looks like this:

  plex:
    #image: plexinc/pms-docker:latest
    image: linuxserver/plex:latest
    restart: unless-stopped
    environment:
      - TZ=America/Sao_Paulo
      - PUID=1026
      - PGID=1026
      - VA_DRIVER=IHD
      - PLEX_HW_TRANS_MAX=16
      - PLEX_MEDIA_SERVER_OPTS=--network-host
    volumes:
      - /config:/config
      - /data:/data
      - /transcode:/transcode
      - /Videos:/media
    devices:
      - /dev/dri:/dev/dri
      - /dev/bus/usb:/dev/bus/usb
    healthcheck:
      test: curl --connect-timeout 15 --silent --show-error --fail http://localhost:32400/identity
      interval: 1m00s
      timeout: 15s
      retries: 3
      start_period: 1m00s
    mem_limit: 4g
    cpus: 5.0
    networks:
      plex_macvlan:
        ipv4_address: 192.168.0.254
        mac_address: "xx:xx:xx:xx:xx:xx"

networks:
  plex_macvlan:
    external: true
    name: plex_macvlan

Because of this, cloudflared can’t route to that container. The way to fix it is by creating a bridge from that vlan to my ethernet.

sudo ip link add macvlan0 link enp3s0 type macvlan mode bridge
sudo ip addr add 192.168.0.200/24 dev macvlan0   # free IP in your LAN
sudo ip link set macvlan0 up

Obviously, replace enp3s0 with the correct name of your ethernet interface. Check with ip addr or ip link, for example. Just by running the commands above, if I go to the browser and type https://plex.fabioakita.dev it should already work. To make this permanent, we need systemd:

# /etc/systemd/network/80-macvlan0.netdev
[NetDev]
Name=macvlan0
Kind=macvlan

[MACVLAN]
Mode=bridge

# /etc/systemd/network/80-macvlan0.network
[Match]
Name=macvlan0

[Network]
Address=192.168.0.200/24
Gateway=192.168.0.1
DNS=192.168.0.1

And now, just restart the services:

sudo systemctl restart systemd-networkd
ip addr show macvlan0

The “correct” thing would be to have my Plex configured with network_mode: host, but, again, I don’t remember why it wasn’t, and I wasn’t in the mood to unconfigure everything just to find out. In case anyone has similar setups, this is how you bypass it.

Speaking of Plex, I had heard that you have to be careful with streaming via cloudflared, I don’t know if there are limits or extra costs. I found this article that talks more about it but I didn’t stop to look into it deeply. The only extra thing I did, on the Cloudflare dashboard, was add a rule to not do CACHING on the plex subdomain, to avoid additional costs.

Conclusion

As you can see, it’s extremely easy to have a home server, isolate everything with Docker, and then expose your containers via a secure tunnel for remote access. None of this means instant security. Make sure your containers are always up to date using services like Watchtower, make sure each service has strong, random, unique passwords - don’t use the same password for everything. Make sure your firewall is well configured and active, even on a LAN.

Many modern Wi-Fi routers let you create a separate VLAN for guest Wi-Fi. That way guests can use your Wi-Fi but can’t see your computers, which will be logically isolated on another VLAN. Check your router’s documentation. I use a TP-Link BE1900 Tri-Band Wi-Fi 7, but for most people it’s too expensive and overkill. Do your research. And don’t bother asking me about other models: I haven’t tested all of them to know the differences.

If you saw names like Radarr, Sonarr and don’t know what they mean, read the Servarr Wiki, which consolidates all the documentation for these services in one place. It’s the infrastructure that lets you have your own “Personal Netflix”.

Also, don’t ask me why I use Plex and not Jellyfin. It’s a personal choice, pick what’s best for you. I like Plex, I’ve used Jellyfin, and I don’t plan to switch.

I hate the idea of paying subscriptions for everything and owning nothing. I think it’s money thrown away. I prefer to OWN my own things under MY CONTROL. It’s more expensive, it’s more work. Too much convenience means you have control over nothing and always depend on others.

I hate depending on others.

On Linux, this is a massive pain in the ass, because the SMB protocol (built for Windows) doesn’t properly carry concepts like ownership (chown) or permissions (chmod). It doesn’t understand things like execute permission (everything is executable). For simple stuff like a Downloads, Videos, or Photos directory, it kind of doesn’t matter.

But let’s say you want to edit source code in projects with Git. It’s going to be a nightmare, because SMB overrides the permissions regardless of what’s underneath, and every single time it’s going to conflict with Git, which will think files were modified (changing permissions counts as a change it tracks), and it’ll keep nagging you to commit that, and that will pollute your whole repository (never use Git projects on SMB over the network).

Instead, I’m on Linux, my NAS is Linux. The right call is to use a Linux protocol. That protocol is NFS and I showed how I set it up in last week’s post.

Anyway, I’m using NFS and I have way fewer headaches around file permissions. But then I hit another problem: if you saw my recent posts, you saw I’ve been experimenting A LOT with Docker. Spinning up new images like there’s no tomorrow. My local 2 TB NVMe filled up fast and started complaining about lack of space. So I thought, “Ah, I’ll just move the Docker storage directory, /var/lib/docker, to a directory mounted via NFS on my NAS”

Doing that is very easy:

sudo mkdir -p /mnt/nfs/docker
sudo chown root:docker /mnt/nfs/docker
sudo chmod 771       /mnt/nfs/docker

That creates a new directory inside my NAS and applies the correct permissions.

sudo systemctl stop docker
sudo mv /var/lib/docker /mnt/nfs/docker
# sudo rm -Rf /var/lib/docker (this will free everything up for me)
sudo ln -s /mnt/nfs/docker /var/lib/docker

That will free up all the occupied space. I don’t need to worry about the images because I can rebuild everything from my Dockerfiles. And I don’t need to worry about volumes because I had nothing important that wasn’t mounted externally, in real directories. Rule: never store important things in Docker volumes. It’s bad practice!

# edit /etc/docker/daemon.json
{
  "data-root": "/mnt/nfs/docker"
}

sudo systemctl start docker

Done, this changes the configuration from /var/lib/docker to the new /mnt/nfs/docker and after restarting the service, it’ll start writing everything there. So the post is over, right?

This is as far as most people go. But if you work with this setup for 2 minutes, you’ll feel something is VERY wrong.

I tried building a new image. And it was taking ABSURDLY long. It turns out NFS is indeed faster than SMB, but both were built for file servers, not for operating systems. That makes a huge difference.

These protocols are FILE-BASED, every operation is based on files. And that’s extremely inefficient.

If you watched my videos on file systems, you should already know they are BLOCK-LEVEL. At the filesystem level, you work with fixed-size blocks of bytes, organized in some variation of B-TREE structures (depending on the filesystem). Operations happen at the block level. Files are abstractions over collections of blocks.

When you run a chmod -R, which is a recursive permission change, inside a Dockerfile build, for example, even an empty tree will cause recursion on every entry inside, be it hidden files, ACLs, etc. And it needs to look up every single permission of everything in there. That costs network time (because NFS/SMB are network protocols).

That causes latency. Every permission on every file is an individual remote call, an RPC call. Even on the fastest network, like my 10Gbps one, even if each RPC is under 0.1 ms, it quickly becomes a bottleneck. And that’s exactly what started happening.

I used sudo nfsiostat 1 10 to check. It was showing this:

write: ops/s=2417  kB/s≈295 MiB/s  avg exe=106 ms  avg queue=102 ms

An average of 106 ms per RPC. That’s an eternity. Average wait time in the queue of 102 ms. Even if I make just 100 RPC calls, it will cost more than 10 seconds in latency alone. And in an OS build, installing packages with thousands of files, that will literally cost an eternity. Running Docker build on NFS/SMB is simply not viable.

iSCSI

Obviously I’m not the first to run into this problem and make the mistake of using a file-based protocol for services with high-volume file churn like the Docker cache.

The right move is to create an iSCSI virtual drive on my NAS. If you’ve never heard of it, SCSI is an advanced protocol for drives. Cheap PCs in the 90s and 2000s used IDE/ATA, which is a simpler and cheaper protocol. But the first Macs and UNIX workstations from the 80s and 90s always used SCSI, which was vastly superior.

It was asynchronous, with queue control. You could send dozens of commands at once and the protocol was smart enough to reorder those commands for better efficiency (for example, skipping a file read if a delete command came before it). IDE was blocking, one command at a time. A single SCSI BUS could handle up to 16 devices. Parallel ATA was 2 per cable. Ridiculous. SCSI had advanced error recovery and reporting systems. It had “enterprise” features for hot-swap, power management, timeout behaviors, recovery, clustering, etc. And on top of all that it was agnostic to the transport medium, it could be fiber optic, SAS, or even iSCSI, which is SCSI over Ethernet, which is what we’re going to use.

IDE/ATA had none of that. It was the protocol for poor-person PC disks. Real workstations, SUN, IRIX, Silicon Graphics, Macs, used SCSI.

More importantly, iSCSI is a block protocol, not a file protocol. Anyway, this varies by NAS manager, but in the case of Synology DSM, which is what I use, it has an application called SAN Manager, which lets you create an iSCSI drive:

Just follow the wizard, give it a name, choose the size, enable CHAP if you need extra security (I don’t, I’m on a controlled local network). That creates the iSCSI drive (think of it like a remote USB drive) and a LUN (Logical Unit Number). A LUN is simply one of many potential “virtual disks” that a target can present. A LUN is like a numbered slot in a target’s storage array. A target is a disk server. On a SAN you can have several virtual disks and the way to organize that is with LUNs. In our case there’s no real need to go deep on LUN details.

It takes a while, but when it finishes you just go back to my Manjaro Linux and run:

yay -S open-iscsi
sudo systemctl enable --now iscsi 

That brings up the service to start automatically on every boot. Now to discover my drive on the network:

❯ sudo iscsiadm -m discovery -t sendtargets -p 192.168.0.xx
192.168.0.xx:3260,1 iqn.2000-01.com.synology:TERACHAD.docker.7af6e...
[2804:1b3:....:fe18:3f7d]:3260,1 iqn.2000-01.com.synology:TERACHAD.docker.7af6e...
[fe80::9209:....:3f7d]:3260,1 iqn.2000-01.com.synology:TERACHAD.docker.7af6e...

Every virtual drive has an IQN which is like a “URL” for the drive. Now we log in:

❯ sudo iscsiadm -m node --login
Login to [iface: default, target: iqn.2000-01.com.synology:TERACHAD.docker.7af6e9116c1, portal: 192.168.0.21,3260] successful.

Since I didn’t enable CHAP it doesn’t ask for a password or anything (in a company it should be enabled, obviously). And done. When you do this I hear on GNOME the same little sound as when you plug a pen drive into the PC and it automatically shows up as a disk. If you run lsblk, it’ll show up as a normal /dev/sdX drive. In the GNOME Disks app (or in Windows Disk Management) it really does show up as if it were any other hard drive or SSD. It doesn’t behave like a shared folder, it behaves like a real drive. You even have to format it, just like a new pen drive.

The formatting itself takes quite a while, because in the process it has to check and map block by block. Since it’s over the network, it’s slower than on a real drive, obviously. But it’s a one-time thing. Even on my 10Gbps network it’ll format at around 120 MB/s, and on a 2TB drive that comes out to around 2 hours for a fast format. I forgot you could format with “lazy”:

mkfs.ext4 -v -E lazy_itable_init=1,lazy_journal_init=1 /dev/sdX

Leaving that for next time, now I wait 2 hours. To make sure it mounts automatically at boot, you need to check its node.startup:

❯ sudo iscsiadm -m node -o show | grep -E 'Target:|node.startup'

node.startup = manual
node.startup = manual
node.startup = manual

It’s set to “manual”, to switch it to “automatic” you do:

❯ sudo iscsiadm -m node \
  -T iqn.2000-01.com.synology:TERACHAD.docker.7af6... \
  -o update -n node.startup -v automatic

With the iSCSI daemon running and after logging in, besides /dev/sda, if you run ls /dev/disk/by-path you’ll find something like this:

ip-192.168.0.xx:3260-iscsi-iqn.2000-01.com.synology:TERACHAD.docker.7af6exxxxx-lun-1 -> ../../sdX

Since I’m using the AUTOFS service and it “steals” my /mnt directory, I’m going to mount the iSCSI drive via fstab at /media/docker by adding this line at the end of /etc/fstab:

...
/dev/disk/by-path/ip-192.168.0.xx:3260-iscsi-iqn.2000-01.com.synology:TERACHAD.docker.7af6e9xxxx-lun-1  /media/docker  ext4      _netdev,nofail,x-systemd.automount,x-systemd.requires=iscsid.service,noatime,nodiratime,commit=60  0 2

Then just reload and mount:

sudo systemctl daemon-reload
sudo mount -a

And that’s it. From now on I have a “virtual drive” on my remote NAS. Now, let’s switch Docker over to start using it. I can reconfigure my Docker to cache images and volumes and everything else directly on that drive. Just edit the /etc/docker/daemon.json file with the mount point:

{
  "data-root": "/media/docker/docker"
}

Don’t forget to create the directory on the new drive with the right permissions (and it does accept permissions, because it’s a drive formatted with ext4 like any other):

sudo mkdir -p /media/docker/docker
sudo chown root:docker /media/docker/docker
sudo chmod 771 /media/docker/docker

Finally, just restart the Docker service, wipe out /usr/lib/docker to free up space on my main NVMe, and done. It should be WAY faster than NFS, though not as fast as a local disk, of course. Even at 10Gbps, the network introduces latency, there’s no way around it. But the heaviest part is really the build process, which has a lot of writes. For loading an already-built image, it should be blazing fast. But at least by using a block protocol instead of a file protocol, latency should drop by at least an order of 10x or more.

And is it really faster? YES. In this screenshot of the docker build running, on NFS the times on the right were HUNDREDS OF SECONDS for many commands. Now they’re back below 1 second like they should be. For all intents and purposes, I’m not feeling any difference from what a USB 3.2 Gen 2 external HD would give me.

The MASSIVE difference that a BLOCK-LEVEL virtual drive makes, compared to a FILE-LEVEL NFS/SMB protocol, is brutal, the protocol is gigantic overhead. Incidentally, this is more or less how it works when you go to AWS and hire EBS storage (Elastic BLOCK Store), that’s why it’s called “BLOCK”, because it’s a BLOCK protocol like iSCSI and not NFS. I hope you learned how much of a difference it makes to understand how protocols and filesystems work. If you haven’t watched my videos, there’s this playlist that explains BLOCKS in detail.

But yes, iSCSI has downsides: being a drive, it cannot, or should not, be shared (think of two PCs sharing the same pen drive, it would be a huge sh*tshow). Block devices were built to be used by a single device. Precisely for sharing, that’s why protocols like SMB or NFS exist: to manage concurrent access to the same file over the network.

Another downside: the virtual drive is one big BLOB of bits. Without “mounting” it, there’s no way to see the files. On the NAS server, it doesn’t show the files. It’s like a VirtualBox .vhd file. You need to “turn it on” and “mount” it on the “PC”. That’s why I’m still going to use NFS for the usual stuff, like Videos or Downloads. And the virtual drive is exclusively for the Docker cache, in this case. That works well. No other PC accesses it. If I had a second Linux machine that needed a Docker cache, I’d have to create a second LUN just for it. Remember, it’s a drive. Virtual or not, it behaves like a real drive to your system.

Everyone who follows me, particularly on Instagram, knows about my NAS saga. In particular my new Synology DS1821+ with more than 100TB (4x 12TB HDs and 4x 20TB HDs, and I already have 4 more 20TB HDs to upgrade in the future when I need to). I use it for a lot of stuff, particularly my Personal Netflix that I explained in that other post on how I set it up with Docker.

In the highlights and posts about it on Insta I already discussed details of this NAS, but to summarize:

• I prefer Synology because deep down it is a Linux box with the BTRFS filesystem (which I like), which is almost as flexible but easier to use than ZFS. And yes, I know unraid, TrueNAS and others exist. But I think Synology DSM is very competent (there is an open source version if you want), and it does everything I need.
• I prefer Plex over Emby or Jellyfin. I have tested them, but Plex’s support for TV and mobile apps is much better in my opinion, the player is more stable, and I also think its organization features are more robust.
• Why not just use Cloud? First because it is slow. My fiber is only 1Gbps and upload is half of that. Second because my internal network is ultra fast (10Gbps) and much better. Third because nothing in the “Cloud” has any guarantee, they can block you, they can get hacked, they can delete things by accident. Fourth, everything that is mine, I like it being really mine, within my reach, under my total control, with nobody to mess with my property. It costs more, it is a choice. I literally have more and better movies and series than Netflix with content you will not easily find in any streaming service anymore, for example.

In fact, I trust “cloud” so little that I use it but I keep a copy of everything on my local NAS using the Cloud Sync app from the Synology store:

“Yeah, but what if your NAS dies in some unrecoverable way and you lose EVERYTHING?!”

No backup is complete unless it follows the 3-2-1 Mantra, remember??

• 3 copies, 2 media types, 1 off-site

I have some things recorded on Millennial Discs (mdiscs, which are like Blu-Rays but made of inorganic material that never degrades). And I make copies of the most important things from my NAS to Amazon Glacier using the “Glacier Backup” app that is also in the Synology store.

Glacier is like S3 but much cheaper because it does not have easy access or fast transfer. It is built for backups, where you write once and almost never access again. ENCRYPTED, of course.

Notice how little I trust cloud, like dropbox, that I make a copy on my NAS and send it to another cloud. There are things with multiple copies indeed. I do not trust anyone’s service, some human will always commit a catastrophic mistake one day, it is just a matter of time.

Oh yes, and I do download a lot of things from torrent and I consider myself a Power User and I know what is malware/virus and what is not, and I know how they work. But I am also human, so all my Windows machines have at least Windows Defender active, Malwarebytes which I also like, and even so I still run yet another antivirus on the NAS as well, which is also in the Synology store. I do not trust anything without redundancy of everything, under my total control.

More importantly, and the reason I choose BTRFS both on the NAS and on my PC: snapshots. It takes snapshots of everything periodically. So, if at some point I run a botched rm -Rf, I lose nothing, I just go back to the previous snapshot and everything will be there, intact. Snapshots are smart, they take almost no space, because in BTRFS the same binary content of identical files points to the same physical location. It does not work like rsync that actually duplicates everything and doubles the space used.

On your Linux PC/notebook, with BTRFS, install the Timeshift program to do the same thing. The best part: if one day you update your system and it does not boot or something breaks after an update, it is easy: just reboot and choose the previous snapshot in GRUB and that is it, it will go back exactly as it was before the update. On Arch/Manjaro it is smart and already takes snapshots automatically every time before pacman runs:

Anyway, another use is for personal code projects on my Linux. On the mini-PC I use on the TV as a media center, I run Windows 11. Because I use it connected to another eGPU with an RTX 4060 and I use Steam for games. Yes, many Steam games run on Linux, there is Bazzite, etc. But there are things that only run on Windows and I preferred convenience. I only use it for games and media, so screw it, it works plug and play.

My other home server mini-PC is where my Docker stuff for media center, torrent, plex server and everything else runs. It is a headless Ubuntu LTS. Plus my handhelds like the Rog Ally X that runs Windows. All of them have access to my Synology NAS, where I expose shared folders with SMB and that is it. Plug and play.

I used to do the same on my Manjaro Linux: mount the folders with SMB. But this has a problem if I want to code something and want to use Git: in SMB there are no chmod attributes (like the +x flag for executable scripts) and every time I do git push, it messes up all the permissions of everything. It is a huge pain. Anyone who has used Git on Windows knows this.

I cannot keep scripts executable in this shared directory. So it makes my life hard on Linux. So I decided to fix this once and for all and do the right thing: stop using SMB and start using NFS, which is the native Linux way to access shared folders on the network from another Linux server (if the server were Windows, then it would have to be SMB, but nobody in their right mind sets up a NAS with Windows Server).

In SMB, the service ignores filesystem ownership and permissions. I create users and assign them to the service, not to the files. Just log in with that user and it will have access to the files. So there is no way to do chown and change ownership, nor chmod to change permissions individually on the files, it is all normalized.

In NFS, the service exposes exactly the ownership and permissions on the filesystem directly, without an ACL middleman to check permissions on the side. So I have the same level of access to those attributes that I have on the local files of my Linux, which is what I want.

But Synology does an annoying thing for some reason. On any Linux, the first user and first group created are usually UID: 1000 and GID: 1000. Try it, go to your terminal and run id [your user]:

❯ id akitaonrails
uid=1000(akitaonrails) gid=1000(akitaonrails) groups=1000(akitaonrails),998(wheel),996(audio),994(input),...

So it does not matter whether you do chown akitaonrails:akitaonrails foo.txt or chown 1000:1000 foo.txt. In practice, the identifier number is what matters, not the name.

With NFS, if on the Synology side there is a user with the same UID and GID number, I automatically have permission. But when you create users on Synology, for whatever reason, it starts creating them with IDs above 1020 instead of 1000. So I have an “akitaonrails” user on Synology with UID 1026 instead of 1000.

Even being able to mount the shared folder with NFS, if I try to use Git, for example, it detects that my local UID is 1000 and the files have ownership of UID 1026:

git st fatal: detected dubious ownership in repository at '/mnt/gigachad/Projects/dotfiles' To add an exception for this directory, call: 

	git config --global --add safe.directory /mnt/gigachad/Projects/dotfiles

I cannot keep using mixed UIDs, it becomes a mess as bad as SMB. There are mapping tricks and other hacks, but there are only two “cleaner” solutions:

• change my Linux UID/GID to also be 1026
• create a new user on Synology, change its ID to 1000 and chown all files to this new user.

I chose the second option. The first thing is to enable SSH access on the NAS (the ideal is to leave such things off, for security, but I am on a private local network so the risk is minimal):

At a minimum I always change the default SSH port, out of habit, but it does not really matter. You need to know how to use SSH, I will not teach it here, there is ChatGPT for that.

Now, create a new standard user:

Let’s also make sure the NFS service is on, since I am going to start using it (by default it comes off, with only SMB on because the majority of clients are Windows PCs):

Now SSH into the NAS and edit the UID and GID, first by editing /etc/passwd and finding a line like:

akitaonrails-nfs:x:10xx:10xx:...

And changing it to:

akitaonrails-nfs:x:1000:1000:...

Now editing /etc/group and the same thing, changing the newly created akitaonrails-nfs group to 1000 as well. Done, now I have a user with UID 1000 and GID 1000 created. The name is not the same as on my PC but that does not matter, only the numbers.

In the Synology DSM GUI, do not forget to add permissions for each shared folder, for NFS:

Still over SSH, you have to change the ownership and fix the permissions of ALL FILES and in my case, with terabytes, it took QUITE a while, just leave it running there:

sudo chown -R 1000:1000 /volume1/GIGACHAD
sudo chmod -R 0755 /volume1/GIGACHAD

If I want to disconnect from SSH and leave it running there (since it is going to take a long time), just run it detached from the SSH session with nohup:

nohup chown -R 1000:1000 /volume1/GIGACHAD > /tmp/chown.log 2>&1 &

One thing to be careful about is not doing something like chmod -R -x on everything. First: directories need to have the execute bit and if you remove it, you will no longer be able to navigate directories, so you have to separate fixing directories from files. Second: it is good to apply this fix on directories to make sure no files were left with the executable bit (which can create security problems, imagine random files inside Node modules directories). Third: directories like “#snapshot” are for filesystem control and are read-only, trying to mess with them will only eat A LOT of time, so it is better to filter them out and not touch them.

Or make a script with this and run it with nohup so you can leave SSH while waiting for it to run (in my case, with terabytes, it takes hours):

nohup /tmp/fix-terachad-perms.sh > /dev/null 2>&1 &

There are still some hidden little problems. Still on the Synology SSH you have to change the /etc/exports that NFS uses because it still maps by default to user UID 1026 and 100, you have to change both to 1000:

# old
sudo cat /etc/exports
Password:

/volume1/GIGACHAD       192.168.0.0/24(rw,async,no_wdelay,no_root_squash,insecure_locks,sec=sys,anonuid=1025,anongid=100)
/volume1/TERACHAD       192.168.0.0/24(rw,async,no_wdelay,crossmnt,no_root_squash,insecure_locks,sec=sys,anonuid=1025,anongid=100

The problem with messing with system files like this is that you do not know if touching the NFS permissions GUI, or if updates come, will revert it to how it was before. If anyone knows of a more permanent solution, let me know.

And done, with all this, just unconfigure the CIFS/SMB configuration I had and replace it with NFS. I prefer to use AutoFS to do this. I had an auto.cifs before, which I can now delete. And create a new /etc/autofs/auto.nfs like this:

gigachad  -fstype=nfs4,rw,nosuid,noatime 192.168.0.xx:/volume1/GIGACHAD
terachad  -fstype=nfs4,rw,nosuid,noatime 192.168.0.xx:/volume1/TERACHAD

And replace the auto.cifs line in /etc/autofs/auto.master with this:

/mnt  /etc/autofs/auto.nfs --timeout=60 --ghost --browse

Now restart the service:

sudo systemctl daemon-reexec
sudo systemctl restart autofs

And done, everything mounted with NFS and now I have full support for native Linux ownership and permissions, without the overhead and incompatibilities of crappy SMB. It was a bit scary doing chown and chmod en masse, but it is a NAS with BTRFS. It has RAID to handle data corruption, it has snapshots if I need to revert something. That is the advantage of having safety systems: you can do things knowing that if something goes wrong, there is a way to revert. That peace of mind is priceless.

Every video from my channel, including the original source files, is there. That alone is terabytes. My whole Steam library is there, around 4 terabytes. All my old games, retro games too, including Xbox 360 and PS3 ISOs. That’s another 4 terabytes. My entire Ultra HD (4K BluRay) disc collection I ripped (made a backup), a few more terabytes. Right now I’m already using more than 50 terabytes.

Before anyone jumps in with opinions, yes, this isn’t for everyone. We’re talking about 8 drives of 10.9 TB, plus an upgrade of 2 drives of 20 TB, plus another 2 drives of 20 TB for backup in case one of them fails, plus 2 NVMEs of 1 TB each just for cache, plus a 10 Gbps network card. On top of that I keep several online copies of part of this content on Google Drive or Dropbox, and a full backup on Amazon Glacier, for the rare occasion my house catches fire, for example. I’m always prepared for any catastrophe.

Synology is one of the best home NAS brands. Yes, you can go way more sophisticated too, just build a real server, with CPUs like AMD EPYC ThreadRipper, using systems like TrueNAS. But that’s too much for me, I don’t feel comfortable maintaining it. Like I said, everyone has to know their own limits.

1o HD de 20TB terminou de se integrar ao RAID! Levou 2 dias inteiros!! Colocando o 2o HD ... de 4 😅 pic.twitter.com/Rv6dqBQr05

— Akitando.com (@AkitaOnRails) March 27, 2024

“Why not just keep everything online??” For most people that really is the easiest path. Like I said, my case is unusual. You do NOT want to have more than 50 TERAbytes online. My local wired network is 10 Gbps, my fiber internet is a measly 0.5 Gbps (500 Mbps). Streaming high quality video over the internet is horribly slow. I want everything in real time, and for that it has to be a local wired network.

That said, I built a second mini-computer, using an Intel NUC, connected to the same wired network as the NAS. It’s a Core i7, 32GB of RAM, 2.5Gbps ethernet, with the single purpose of being an empty Ubuntu server (it could be any Linux, I just picked the easiest one) whose job is to run Docker containers.

Even this little PC is overkill, way more than necessary. Many will comment that a Raspberry Pi 5 would be enough, and they’re not wrong. The best solution is the one you can afford and feel comfortable maintaining. Don’t try to copy anyone’s setup exactly, study what fits you best. And don’t waste time bikeshedding other people’s setups, build your own.

Anyway, I decided to share with you all the docker-compose files I’m using to bring up the docker containers on this server. Here’s the GitHub repository.

WARNING: don’t forget you need to edit the directories in the scripts. It’s configured to access my NAS network mount point, which is /mnt/terachad. Fix it to point to the right places on your machine.

WARNING 2: there are a lot of details I’m not covering in this article. This Reddit forum has more details and more discussion. Depending on your specific doubts or problems, it may be answered there.

So you don’t get lost in the middle of so many services, here’s the full picture of what we’re going to build, from the request all the way to watching on the TV, including the protection layers that keep malware and wrong downloads out of your library:

The idea is this: you request in one place (Seerr), the system finds it (Prowlarr and the indexers), downloads it with protection (qBittorrent and SABnzbd, plus the security layers), organizes it on the NAS (with subtitles from Bazarr) and you watch it on any screen (Plex and Navidrome). Let me start explaining the main ones one by one.

Portainer and Utilities

Here’s the snippet (see the full thing in the repository):

volumes:
  portainer_data:

services:
  portainer:
    image: portainer/portainer-ce:latest
    restart: unless-stopped
    volumes:
      - portainer_data:/data
      - /var/run/docker.sock:/var/run/docker.sock
    ports:
      - '9000:9000'

Portainer is a visual manager for the resources managed by the Docker service on your machine. Its purpose is more for me to visually “glance” and check if any container has a problem, or to test some quick little config tweak before editing the docker compose file.

In the utils-docker-compose.yml file there are other services like Organizr that I installed but haven’t stopped to configure yet, nor decided if I really want to keep. There’s Librespeed, which works like those internet speed tests, but for the local network. There’s Watchtower, which monitors image updates for all Docker containers, downloads them and automatically restarts with new versions, to keep all containers always up to date.

And to reach your local services from outside home, there are easy VPN options like Tailscale or ZeroTier, with little configuration. Yes, yes, before someone runs to the comments, you can set up OpenVPN from scratch, but trust me, these options are infinitely more “plug and play” and possibly safer, because everyone installs it, but few have the discipline to keep it maintained and updated all the time. These days I actually ended up moving to a Cloudflare tunnel to expose my services, but to get started, a Tailscale does the job really well.

MakeMKV and HandBrake

To build your personal movie library, legitimately, the correct way is to own BluRay or UHD discs and “rip” them. If that’s your case, I recommend buying disc reader models like the Pioneer BDR-XS07S. You have to be VERY careful because not every reader works.

Check the MakeMKV forum for the most recommended models. Some models got firmware updates that stop them from being used to rip BluRay. You need to find models that either weren’t updated, or that aren’t hard to firmware-downgrade. The ideal is to buy models everyone already knows work, and for that you have to dig through forums like that one.

MakeMKV, as the name says, “Makes MKV video files”, which is the good old Matroska video envelope format, one of the most versatile (because it supports multiple streams not just of audio, but of subtitles too).

Crash course: one thing is the codec that does the encoding of the video streams and audio streams. For example, H.265 or AV1 are video codecs. MP3 or AAC are audio codecs. Now we need to package these two streams (or more, if there are multiple dubs, for example) into an “envelope”, a “container”, a “file format”, like “.mp4” or “.mov” files. Matroska is a file envelope format that’s very flexible, allows multiple streams of everything, including subtitles, which is why it’s so widely used for sharing over torrent.

With this container, you can access the app’s graphical interface through the browser (it’s like a remote Windows client, like VNC). Just have the Blu-Ray reader connected via USB to your computer and it should find it. It will use the libredrive library to decrypt the Blu-Ray data streams, which is why the firmware versions being compatible (already cracked) matters.

Many people don’t know that we call everything “Blu-Ray” but in reality Blu-Ray is for movies encoded in Full HD, which is 1080p. The 4K versions (2160p) come as UHD, which is Ultra HD. The Blu-Ray case is usually blue, the UHD ones are black, you want the UHD ones whenever possible, of course. And I have a huge collection:

"onde eu tô gastando tanto espaço??"

Boa parte é fazendo Rip (backup) de todos os meus UHD (BluRay 4K). E isso porque estou re-encodando em bitrate menor depois de ripar kkkk pic.twitter.com/T1yPa6EIcV

— Akitando.com (@AkitaOnRails) March 26, 2024

The advantage of an app like MakeMKV is being able to decrypt the video stream and copy it bit by bit exactly as it is on the UHD, the best quality possible. This is what’s called REMUX (remuxing), which is extracting the stream from the container and putting it in another container without re-encoding. You’ve probably seen it in some torrent files, it’s the best quality. But it means it will use the same space as on the disc, a range of 50 to 80 GIGAbytes per movie. Every dozen movies will eat up almost 1 terabyte. Depending on how many movies you want to have, your free space will run out very fast.

To minimize this, we can recode or “re-encode”. It’s like when we take a RAW photo and re-encode it to a lower quality format, like PNG or even JPEG. If you know where to look, you’ll find the compression artifacts, but overall, it’s worth it. In my case, for example, encoding to H.265 10-bit, QP 15 Constant Quality.

The best option is to have an NVIDIA graphics card, with hardware video encoding support via NVENC, and use the best video encoding software: Handbrake. In my case, I pick the pre-configured Matroska H.265 MKV 4K60fps profile and switch it to use the NVENC encoder on my RTX 4090, which speeds up encoding several times over.

In docker compose, just like MakeMKV, I can bring up this app for browser access, like this:

  handbrake:
    restart: unless-stopped
    image: jlesage/handbrake
    ports:
      - "5801:5800"
    environment:
      - AUTOMATED_CONVERSION=0
      - HANDBRAKE_GUI=1
      - DARK_MODE=1
      - TZ=America/Sao_Paulo
    volumes:
      - "/home/akitaonrails/handbrake:/config:rw"
      - "/home/akitaonrails:/storage:ro"
      - "/mnt/terachad/Videos/BluRay:/watch:rw"
      - "/mnt/terachad/Videos/BluRayOptimized:/output:rw"
    devices:
      - /dev/dri:/dev/dri

Your PC has at least an Intel graphics card with hardware encoding support via QSV, or better, an NVIDIA one with NVENC support. In both cases, we give the container access to that by mapping the /dev/dri device. Without it, encoding goes through the CPU, and it takes orders of magnitude longer.

Keep in mind that software encoding is slower but has better quality than Intel QSV, which, despite being fast, always gives me inferior video quality. NVENC is the best of both worlds: excellent quality and very fast.

WARNING: in my case, I want the best quality possible, that’s why I back up UHDs, that’s why even over torrent I download BR-DISC, which is a UHD backup, and that’s absurdly heavy as I explained. So I manually redo the encoding to the Matroska H.265 profile, as I also explained.

Many people don’t want this hassle, so be sure to read the Reddit forum, which explains how to configure the next services to not download huge files like this. I won’t show it here, because it’s not my case.

QBitTorrent

No matter how much Blu-Ray and UHD you own or can buy, a lot of stuff doesn’t even exist on physical disc. In that case, your only option is BitTorrent. And yes, piracy is illegal, I am not encouraging piracy blah blah blah…

Every country has its own law, go research it. I’ll assume everyone knows what they’re doing and it’s not my problem. No point commenting about this, I won’t go on about it. The web is already stuffed with material about it, go to Google and search yourself.

The first thing we want is to install QBitTorrent, probably the best and most complete app to download torrents, which also has a web interface and works via Docker:

  qbittorrent:
    image: linuxserver/qbittorrent:latest
    restart: unless-stopped
    environment:
      - TZ=America/Sao_Paulo
      - WEBUI_PORT=8080
      - PUID=1000
      - PGID=1000
    volumes:
      - /home/akitaonrails/qbittorrent/config:/config
      - /mnt/terachad/Downloads/torrents:/downloads
    ports:
      - '8080:8080'
      - '62609:62609'

It’s important to remember to generate a random port in the QBitTorrent settings, as shown in the image below. Any port between 1025 and 65535 should work, but your network might block some port range in that interval, so you have to test. Search web forums, and be careful because a lot of online tips are already outdated.

You need to understand the minimum about networking, understand whether you’re behind a NAT, how to map external ports, whether you have an active firewall blocking ports, etc.

If the torrent isn’t downloading, “it might be” this. I put it in quotes because it’s only a possibility. So click “random” to pick another port, edit the docker compose config to map the new port and restart the container.

For some reason the saving management comes all set to Manual, it’s important to change it to automatic:

Another detail is to tell QBitTorrent to re-announce downloads to the trackers when the port changes. For that, enable the “Reannounce to all trackers when IP or port changed” option, otherwise downloads might not restart.

Now the most important setting in this whole section, and one I learned the hard way: block executables at download time. One fine day I found nine “completed” torrents in my library whose only content was a Windows executable of almost 1 GB, no video at all. And worse, the release name was squeaky clean, stuff like Reacher S04E07 1080p WEB H264-CAKES or Rick and Morty S09E10 ... EDITH.scr. You look at the name and swear it’s the episode. The .exe or .scr was hidden in the file inside.

A name filter doesn’t catch this, because the name is clean. The defense that actually works is blocking the extension at write time. In QBitTorrent, go to Options, Downloads, and enable “Excluded file names” with this list:

*.exe
*.scr
*.bat
*.cmd
*.msi
*.lnk
*.com
*.vbs
*.pif

With that, the malicious file simply never gets written to disk. The torrent “completes” with nothing useful, Sonarr or Radarr notice the import failed, dump that release into the blocklist and go looking for another one on their own. The malware never touches the disk, and you don’t even pass the garbage forward by seeding it. This is the first layer of defense, and the most effective.

Finally, how do I search for things to download? I can manually go to Google or DuckDuckGo or straight to sites like the PirateBays of the world and look for “.torrent” files or magnet links. To understand what that is, watch my video about Cryptography in Practice, where I explain everything about Torrent too.

QBitTorrent supports search plugins. It comes with a few enabled. In the bottom right corner, there’s the “Search plugins” button. We want to add the plugin that covers everything: the Jackett service.

Notice it wants a host named “jackett” enabled on port 9117, and for that we have this snippet in the docker compose:

  jackett:
    image: ghcr.io/linuxserver/jackett:latest
    restart: unless-stopped
    environment:
      - TZ=America/Sao_Paulo
      - PUID=1000
      - PGID=1000
    volumes:
      - /home/akitaonrails/jackett/downloads:/downloads
      - /home/akitaonrails/jackett/config:/config
    ports:
      - '9117:9117'

It brings up a service where we can access the web interface like this:

It’s a service that consolidates dozens of torrent search sites in one place, can test whether they’re online and, more importantly, provides an API key in the top right corner, which we should copy and edit into the QBitTorrent plugin config. With that you can search directly, getting results like this:

Notice in the right column that lists where these results were pulled from: it’s coming from the Jackett service.

An important heads-up, because this changed over time: Jackett here is for THAT, the manual searches you do yourself in QBitTorrent. Don’t fall for the temptation of wiring Jackett straight into Sonarr and Radarr. The automated part is going to be managed by Prowlarr, which I explain later, and it’s Prowlarr that feeds the services’ indexers. I keep Jackett running only for manual searches and, at most, as a bridge for some tracker Prowlarr doesn’t support natively, like RuTracker, which Prowlarr can consume through Jackett.

Radarr (Movies)

Searching manually works, but there are better ways: let a service do the searching and manage the download for you, and to download movies, we can use the Radarr service:

For that, we have this snippet in the docker compose:

  radarr:
    image: ghcr.io/linuxserver/radarr:latest
    restart: unless-stopped
    depends_on:
      - qbittorrent
    environment:
      - TZ=America/Sao_Paulo
      - PUID=1000
      - PGID=1000
    volumes:
      - /home/akitaonrails/radarr/appdata/config:/config
      - /mnt/terachad/Downloads/torrents:/downloads
      - /mnt/terachad/Videos/radarr/movies:/movies
      - /mnt/terachad/Videos/radarr/anime:/anime
    ports:
      - '7878:7878'

Notice I mapped different directories for western movies and anime movies, that’s a preference of mine, you can organize it however you want. As I warned before, don’t copy and paste exactly what I’m showing here: adjust the directories for your machine.

The main thing is that once it’s started, we can navigate the web interface to the “Settings” option. There you’ll find “Download Clients” and we can register the QBitTorrent web service:

I didn’t show it, but in the QBitTorrent settings there’s a place to set the admin password to access its API via web. Don’t forget to set it there and use the same password here. That way Radarr can control the movie downloads directly with QBitTorrent, without you having to get involved.

There’s a setting in Radarr I recommend doing right away, because it ships with a silly hole: by default, EVERY quality definition comes with a minimum size of zero. That lets a tiny fake file passing itself off as 4K slip through. I’ve been a victim: a Superman (2025) WEBDL-2160p.iso of 598 MB came in as if it were the movie and stayed showing in Plex for months. Inside the ISO was a .exe of 597 MB in disguise.

The fix is to go to Settings, Quality, and set a size floor on each quality definition. I use roughly this, in MB per minute of video:

• 720p (HDTV/WEBDL/WEBRip/Bluray): 3
• 1080p (HDTV/WEBDL/WEBRip): 5, Bluray 1080p: 8, Remux 1080p: 25
• 2160p (HDTV/WEBDL/WEBRip): 10, Bluray 2160p: 15, Remux 2160p: 50

It’s low enough to accept a good lean x265 encode, and high enough to refuse a “4K” under 1 GB, which can only be a trap. One annoying detail: Radarr has a bug where saving everything at once doesn’t persist the minimum size, so you have to save quality by quality, one at a time.

Sonarr (TV Shows)

Same thing as Radarr, but for TV shows, we have the Sonarr service:

This service is fantastic. It finds your shows, cross-referencing online databases with details like seasons, each episode’s name and lots of other metadata. So it can not only download old episodes, but also keeps a schedule to download shows that are still airing. Every week a new episode shows up on its own:

And same as Radarr: in Settings and Download Clients, I can register my QBitTorrent, then Sonarr handles downloading the episodes:

And to bring up the service, in the docker compose we have this snippet:

  sonarr:
    image: ghcr.io/linuxserver/sonarr:latest
    restart: unless-stopped
    depends_on:
      - qbittorrent
    environment:
      - TZ=America/Sao_Paulo
      - PUID=1000
      - PGID=1000
    volumes:
      - /home/akitaonrails/sonarr/appdata/config:/config
      - /mnt/terachad/Videos/sonarr:/tv
      - /mnt/terachad/Downloads/torrents:/downloads
    ports:
      - '8989:8989'

Same as Radarr, I mapped different folders for anime and non-anime, which is a matter of your organization preference.

Both in Sonarr and Radarr, there’s a setting that became mandatory for me after I started getting caught by traps: a Release Profile that blocks release names known to be booby-trapped. In Settings, Profiles, I keep a “Must Not Contain” list with terms like these:

BROADCAST
FULL HD
.scr
.exe
SODAPOP
REACHERSON

Each of these has a story. BROADCAST, SODAPOP and REACHERSON are groups that pack split RARs with a Sample/ folder that jams the import (I caught almost 60 GB of junk from a fake Welcome to Derry in one go). FULL HD is the naming signature of the ones carrying .exe. And .scr/.exe in the name is literal. A heads-up for anyone copying this: Radarr, under the hood, uses the same ignored field as Sonarr, even though the interface calls it “Must Not Contain”, so configure the same list in both.

There’s still a class no filter catches on its own: pre-air fakes. A Reacher S04E07 shows up with zero seeds for an episode that only airs next week. Sonarr doesn’t reject a release dated before the air date, so I clean those out by hand. The good news is that, with the QBitTorrent executable block from above, even when one of those downloads, it comes in empty and gets discarded automatically.

Downloads that get stuck

A classic everyday problem: the torrent gets stuck forever in “downloading metadata” or “stalled”, with zero peers, and Sonarr never gives up on it on its own. I’ve had an episode sitting behind a dead magnet for five days while the good release passed by, waiting for a turn that never came.

The solution I set up is a simple script running once a day (a cron or a systemd timer, doesn’t matter). It queries the Sonarr, Radarr and Lidarr queue via the API, finds the items stuck in “downloading metadata” or “stalled with no connections” for more than 72 hours, and removes them with a single command:

DELETE /api/v3/queue/{id}?removeFromClient=true&blocklist=true

That deletes the dead download, dumps the release into the blocklist so it won’t grab it again, and the app goes after another source automatically. The trick is being careful to NOT touch a download that’s merely slow (under 72h) nor an item waiting for a manual import, otherwise you delete good stuff by mistake. The first time I ran it, it cleared out nineteen zombie downloads in one shot.

Prowlarr (indexer)

For both Radarr and Sonarr to know where to pull the show and movie torrents from, they need to search public indexers or trackers (or private ones, if you have good contacts, I don’t, don’t even ask). For that we have Prowlarr. This is the most important service of all, because if this one doesn’t work, nothing downloads properly.

It took me a while to reach a good configuration, but today I have a simple rule that solves it: Prowlarr is the only place that manages indexers. Sonarr, Radarr and Lidarr should NOT have any indexer registered directly, only the ones Prowlarr syncs. Every indexer I had registered by hand inside Radarr was a source of headaches, including a “Lime Torrents” that was exactly the one that served me that Superman ISO malware. I deleted them all and left only Prowlarr feeding the three.

Notice it lists dozens of indexers, or trackers. It’s similar to Jackett. The difference is that Jackett is for manual searches you’ll do yourself, straight from QBitTorrent. Prowlarr’s indexers, on the other hand, get configured straight into Sonarr and Radarr. Here’s where the confusion starts, let’s understand it:

Both in Sonarr and Radarr, besides Download Clients, where we can configure QBitTorrent, there’s also this “Indexers” option, where we can configure which trackers to search for the media.

Think of a tracker like a public mini-Google, it’s where people register their torrents when they want to share. Torrents aren’t magic and don’t appear out of nowhere. Someone has to create a record somewhere. But there isn’t just “one” place: it’s a distributed network and any participant can put up a tracker if they want. It’s as if you could put up your own miniature Google, just to register torrent files.

It’s a pain to keep managing this. Some small trackers decide to go offline, new ones appear, configs get changed, and we have to keep managing this manually, both in Sonarr and Radarr. That’s what Prowlarr is for: to manage the trackers for you. Let’s understand how.

In the Prowlarr settings, we start by configuring where to find our Sonarr and Radarr services:

For example, this is my Sonarr config. Notice the hostname is “sonarr” and that works because in my docker compose I bring all services up on the same VLAN, and by default docker registers the service name configured in the YAML file as the hostname. If you don’t understand Docker, be sure to watch my videos about containers.

We also need to configure access to QBitTorrent:

This other one, SABnzbd, uses another, older download protocol, based on USENET newsreaders. Yes, that old Usenet. Those who know, know. I keep torrent as the primary, but I keep SABnzbd configured as a second source, because a lot of content only exists in those older groups. If you’re going to use it, apply the same extension protection as QBitTorrent: in the sabnzbd.ini file, in the [misc] section, set unwanted_extensions = exe, scr, bat, cmd, msi, lnk, com, vbs, pif and action_on_unwanted_extensions = 2, which tells it to abort the whole download if one of those shows up. That way Sonarr and Radarr see the failure and look for another source.

Finally, many trackers today implement some kind of bot protection using Cloudflare captcha, you know that thing that goes “are you a human?”. To get past that, we need the FlareSolverr service, which we can bring up in the docker compose like this:

  flaresolverr:
    image: ghcr.io/flaresolverr/flaresolverr:latest
    restart: unless-stopped
    volumes:
      - /home/akitaonrails/flaresolverr/config:/config
    ports:
      - '8191:8191'

In Prowlarr it enters as an “indexer proxy”. You register FlareSolverr pointing to http://flaresolverr:8191/, create a tag for it, and mark only the indexers that sit behind Cloudflare with that tag. Those then start solving the captcha under the hood, and the others don’t even need to go through it:

We’re not done yet. On the main screen, you go enabling the indexers that make sense for you. Don’t enable everything: it gets heavy and, worse, some indexers serve more traps than good content. Check each one’s tags. Some indexers are porn-only, some are only Russian or Chinese content, those I skip.

The ones that serve me well today are EZTV and 1337x (both behind Cloudflare, so they depend on FlareSolverr), YTS for movies, and AnimeTosho plus Nyaa for anime. In total I’m left with somewhere between 18 and 21 active indexers per app. And more important than enabling the good ones is disabling the bad ones: I turned off LimeTorrents and TorrentDownload, because, together, they generated more than two hundred downloads and almost no usable import, on top of being a recurring source of malware and fakes. Fewer good indexers is worth more than a pile of bad ones.

At the end, it’s good to click “Test All Indexers” to check which ones are still active or not. And finally, we can click “Sync App Indexers”, which will register all the indexers we know are working into Sonarr and Radarr. Get it? Prowlarr will configure both with the most up-to-date indexers, and remove the ones that are no longer active. We don’t need to configure each of them manually.

It’s important to understand this, because otherwise you’ll see Sonarr and Radarr not managing to download anything and won’t know why: but it’s because the indexers are wrong or obsolete or insufficient. Spend a good amount of time reading the Prowlarr documentation.

Plex (Private Netflix)

To watch everything you download, there are several options. If it’s just for you, you can just download everything to an external drive or something, and watch directly from your PC or laptop. Depending on your Smart TV or game console, you can also connect your drive or flash drive via USB and watch directly.

If you want something a bit more sophisticated, and have a spare PC or Raspberry Pi, you can install Sonarr, Radarr, and share the downloads directory over the network, using protocols like CIFS, which is for Windows shared folders. Again, many Smart TVs and consoles support mounting network shared folders.

But if you want something friendlier and more sophisticated, you can literally have your own “Private Netflix”, with a graphical interface similar to any streaming platform, showing thumbnails, descriptions, movie or show details, a video player and all that, and even suggestions of what to watch, movie suggestions based on what you’re watching, actor info and everything else.

There are several projects that do this. One of them, which the community likes more for being 100% open, is Jellyfin (derived from the old Emby). I recommend testing it, it has everything most people need. Just map the Sonarr and Radarr download folders and it will index everything and organize it for you. Then, from any PC, just access the web interface and it all just works.

But I personally prefer Plex, which is older, more closed (which is why some people don’t like it), but because it offers paid plans with exclusive features, the level of polish is clearly better. The interface is much prettier, and looks more “professional”. Jellyfin, as good as it is, still has that look like a backend programmer made it, with a lot of ill will toward UX and design.

On top of that, to watch on a Smart TV or Android TV, you need an app on the Google Play Store. Jellyfin doesn’t have an app for everything, some TVs, I don’t remember if LG or Samsung, don’t even have the app available. So you have to side-load. Not friendly. Plex, on the other hand, is available in way more places, so it tends to be more “plug and play”, similar to downloading the Netflix or Amazon Prime app.

Plex also has features that “just work”, trivial things you expect in a player, like a “skip intro” button or “resume from where you stopped” or “skip to the next episode” or “search and change subtitle”. That’s what interfaces like these offer you, so the usability really is a lot like watching Netflix. You don’t get that with a generic PC player like a VLC.

Bringing it up is easy, here’s my docker compose snippet:

  plex:
    image: plexinc/pms-docker:latest
    restart: unless-stopped
    environment:
      - TZ=America/Sao_Paulo
      - PUID=1000
      - PGID=1000
      - VA_DRIVER=IHD
      - PLEX_HW_TRANS_MAX=16
    volumes:
      - /home/akitaonrails/plex/config:/config
      - /home/akitaonrails/plex/data:/data
      - /home/akitaonrails/plex/transcode:/transcode
      - /mnt/terachad/Videos:/media
    devices:
      - /dev/dri:/dev/dri
      - /dev/bus/usb:/dev/bus/usb
    network_mode: host