Underneath this whole story there’s a decision almost nobody makes consciously: the filesystem. The Synology used btrfs all along. The QNAP uses ZFS. And your Linux machine, most likely, came from the factory with ext4 without anyone asking you anything. That’s what this is about.

Where the two came from

ext4 is the end of a dynasty of over three decades. ext (extended filesystem) was born with Linux itself, ext2 became the standard in the 90s, ext3 added a journal in 2001 so you wouldn’t need an epic fsck after every power loss, and ext4 arrived in 2008 with extents, delayed allocation and volumes up to 1 exbibyte. The design goal was always the same: be the default filesystem, fast, simple and predictable. It’s excellent at that. And it stopped there.

btrfs was born in 2007, at Oracle, with Chris Mason, with a completely different goal: be Linux’s answer to ZFS. Instead of writing data over data, it’s copy-on-write: every write goes to a new block and the pointers are updated cascading up the tree. That single decision is the mother of everything that matters in it.

What each one delivers, in a list:

ext4:

• journaling (consistent metadata after a crash)
• extents and delayed allocation (good performance)
• mature, stable, everywhere, every tool works

btrfs:

• copy-on-write on everything
• data and metadata checksums on every block
• instant, cheap snapshots
• independent subvolumes inside the same volume
• transparent compression (zstd, lzo, zlib)
• native RAID and scrub (verification sweep)
• snapshot send/receive for incremental backup
• reflinks (copies that don’t duplicate blocks)

The list already answers the title’s question. Let’s go piece by piece.

Even NTFS and APFS have passed ext4

And if the comparison with btrfs seems unfair, look sideways. NTFS was born in 1993, with Windows NT, already with metadata journaling and ACLs, and back in the 2000s it gained a log of every change on the volume plus shadow copies (Windows’ “Previous Versions”). APFS, which Apple launched in 2017 made for flash, is copy-on-write like btrfs: native snapshots (it’s what Time Machine uses for local restores), instant file clones that share blocks, volumes sharing the space of the same container, and per-file encryption.

Read the ext4 list again and compare: no snapshots, no data checksums, no CoW, no compression. Windows’ and Mac’s filesystems, which nobody considers examples of sophistication, protect your data better than Linux’s default, whose last big feature came out in 2008. btrfs is what puts Linux in the same generation as the other two, instead of a decade behind.

Why btrfs is far superior

What copy-on-write is, for those who never stopped to think about it

In classic filesystems, writing is destructive by definition: updating a block means writing the new content over the old one, in the same place on disk. If the machine dies mid-write, the file ends up half old, half new. The ext3/ext4 journal covers that for metadata (the directory structure doesn’t corrupt), but your data itself is left to luck.

Copy-on-write inverts the order: nothing is ever overwritten. Writing to a file allocates a new block, writes the new content there, and only when everything is safely on disk does the filesystem flip the pointers to the new version, atomically. At any instant, you have either the whole old version or the whole new version. Never a mix of the two.

What that buys in practice:

• Nearly free snapshots. The old blocks already persist by CoW’s nature; a snapshot is just the filesystem holding the root pointer of that instant, pure accounting.
• Power loss doesn’t corrupt. An interrupted write leaves the previous version intact, because it was never touched. For anyone who has lost a VM image or a database to a mid-write crash, this is worth the whole filesystem.
• Verification becomes possible. Since each block is written once and never touched again, keeping its checksum forever makes sense. It’s the base of the corruption protection I detail further down.

The cost exists and I pay it knowingly: random-write-heavy files (VM disks, databases) fragment and duplicate blocks, because each rewrite becomes a new block. For those cases you disable CoW per directory with chattr +C, which must be set before the file is created. In my layout, the VM image made with the right +C stayed healthy; the one made without became the textbook fragmentation case.

Snapshots are the feature that changes your life

A btrfs snapshot is an instant photo of a subvolume, and it costs almost nothing, because CoW already keeps the old blocks while someone references them. Creating a snapshot is a seconds operation, not hours. In practice: before every upgrade, the system takes a photo. If the upgrade breaks, you don’t “uninstall packages”: you boot back into the previous photo and you’re done. The boot recovery section, further down, shows this happening for real.

Subvolumes are the right way to organize the disk

A subvolume is an independent namespace inside the same filesystem, with its own snapshot policy. And it has a killer property: snapshots are not recursive. A child subvolume inside another is automatically left out of the parent’s snapshot.

I use this for everything. On my desktop, the cache directories that can be re-downloaded (mise, rustup, npm, opencode, AI caches) are becoming nested subvolumes inside @home: they stay in the same place for any program, but out of every snapshot. That separation alone takes 165 GiB out of snapshot scope, and the big caches (Hugging Face, yay, uv, pip) already live outside the system volume, with symlinks pointing to them: minus 148 GiB of snapshot, part already migrated and part in progress. The snapshot stays small, the restore stays fast, and whatever can be re-downloaded from the internet never enters the photo.

The real cost, measured on the new QNAP (ZFS, but the CoW math is the same): 27 hourly snapshots of a folder with almost 80 TB, and nearly all of them consume megabytes, because between one hour and the next almost nothing changed. Cheap snapshots are what let this routine exist without guilt.

Reflink deduplication

First, the caveat that matters: btrfs has no native online deduplication like ZFS (which keeps a dedup table in RAM and charges dearly for it). What it has are two practical things: reflink and offline dedup. Reflink is a copy that shares blocks: cp --reflink=auto creates a new file that points to the same blocks as the original, instant and free, and each one starts diverging only where written. In the era of dozens of checkouts, worktrees and similar builds, that saves a mountain of blocks. For retroactive dedup there are userspace tools like duperemove and bees.

Real corruption protection

Every btrfs block, data or metadata, carries a checksum. When the filesystem reads and the checksum doesn’t match, it knows the block rotted, and if there’s a second copy (RAID or metadata DUP) it self-heals. ext4 checks no data at all: bit rot passes silently until you open the file years later.

Two pieces of my configuration show what that buys in practice, and what ext4 simply doesn’t offer. The first is metadata in DUP mode. On ext4, each structure on the volume exists only once: a well-placed bad sector corrupts the directory table and the whole drive stops mounting. On btrfs, even with a single drive, every metadata block exists duplicated. The bad sector keeps existing, but the checksum flags the rotten block on read and the second copy answers in its place, without you noticing. It’s the difference between losing the volume and noting a warning in the log.

The second is the monthly scrub. ext4 never re-reads anything to check health: if a block rots, you find out the day someone tries to open that specific file, maybe years later, when there’s no fixing it anymore. The scrub is a scheduled sweep that re-reads the whole volume validating checksums, and finds the rot while it’s still small and recoverable. When the btrfs device stats counter leaves zero, it’s the siren to replace the drive before the disaster, not after.

Transparent compression, measured

Every mount on my desktop uses compress=zstd:1. Level 1 costs a few percentage points of one CPU core, and btrfs detects what doesn’t compress (model blob, video) and writes it raw, at no cost. The result, measured this week on my two new drives:

That’s 630 GB of free space on 2.2 TB: 15% more drive, paying nothing. On the initial migration the gain was even bigger, 45% on the system volume, because there was more text and less binary. Notice the table’s details: programs and libraries compress well, Docker layers compress even better (39%), text logs compress 4.5x, and AI model weights compress nothing, which is expected and costs nothing. Source code, which is what we produce the most in the AI era, compresses beautifully.

Synology uses btrfs, QNAP uses ZFS

Being fair to ZFS: for a big NAS it’s superior, and it’s what I chose for the QNAP. Native checksums and self-healing, mature RAIDZ, excellent send/receive. The price is complexity and resources: ZFS wants RAM for the ARC cache (my QNAP has 7 GB for a 174 TB pool, and I feel the lack), its online deduplication would need about 190 GB of RAM just for the table on my volume, and its provisioning model gave me a free scare when the snapshot was refused with 20 TB free (the thick vs thin story is in the NAS article).

btrfs is the right balance for a consumer machine. Light, native in the kernel, does almost everything ZFS does without demanding a storage administrator. Desktop, laptop, homelab, small or medium NAS: btrfs. Synology proved it in practice, twice: the btrfs volume survived readable with two drives out, and its snapshots saved 11 albums I deleted by mistake in the middle of the cleanup, byte by byte.

Big, serious NAS, with RAM to spare: ZFS. Everything else: btrfs. And for a personal Linux machine, it’s not even a discussion.

Boot recovery via snapshot

The feature that has saved me for real, told on both machines.

On the desktop, Omarchy configures snapper to take a snapshot before every update and limine-snapper-sync publishes each snapshot as a boot entry in Limine. New kernel won’t come up? In the boot menu you pick yesterday’s snapshot and the machine comes back as if nothing happened. On my machine the scheduled routine runs via Timeshift on cron (every boot and hourly), with snapper covering the update points: the way back is the same mechanism. (The gotcha: each snapshot entry carries the kernel image along, so a small ESP fills up fast. My 1 GB one already refused a kernel update for lack of space; the system stayed bootable on the old kernel, which is the right behavior.)

On the server, which runs openSUSE MicroOS, it’s even stronger: the whole system is transactional btrfs, and every update is a new snapshot. Last week, kernel 7.2.9 took down the NVMe controller silently, twice, in the middle of the nightly backup. The fix was a transactional-update rollback and a reboot back to 7.2.6, which ran the whole load (754 GiB of backup reads with concurrent download) without a hiccup. A kernel regression became a scheduled reboot, not a recovery all-nighter. One documented caveat: /etc lives inside the snapshot, so the rollback undoes configuration changes along with it. After any rollback, check your units again.

Maintenance is not optional (and the deleted file that won’t go away)

A modern CoW filesystem delivers a lot, but it charges two routines ext4 never asked for.

The first is the scrub, which already appeared here: re-read everything and check checksums, monthly, on all machines. It’s bit-rot detection in action, and without it the checksum only warns you about the rot when you try to read the file. The second point is subtler: a snapshot pins the blocks of deleted files. When you delete a file inside a snapshotted subvolume, the space doesn’t come back, because the block stays referenced by the photo. That’s exactly what happened on my NAS: the 2.14 TB cleanup I ran on the Synology only freed space for real after the old snapshots were removed; until then, the blocks stayed pinned.

The treatment is simple routine. Retention has to be automatic, with snapper or Timeshift deleting the old snapshots per policy (on the QNAP, the Snapshot Manager’s retention policy plays the same role). Space that doesn’t come back after deleting old snapshots is usually over-allocated chunks: a btrfs balance start -dusage=50 returns the little-used pieces to the free pool (in my house I don’t blindly schedule balance, I monitor the unallocated space and act when it gets tight). And on the restic side, forget alone only removes the snapshot’s reference: the repository space is only reclaimed with prune along, forget --prune.

House cleaning in the AI coding era

Now the part almost nobody connects with filesystems: the agent era produces garbage at industrial speed. I know because I measured.

• Rust is the biggest disk polluter in the dev world. Every build leaves behind 100 to 180 MB test binaries, and they accumulate with each edit. On my desktop, the target/ directories added up to 920 GiB. ai-memory alone accounted for 704 GiB. The hygiene tool I was using is cargo-sweep, weekly. A reader pointed out that the project declares itself unmaintained, and its own README points to cargo-clean-all, which works differently: instead of pruning old artifacts inside one target/, it finds every Rust project under a directory and deletes the whole target/ of the ones you filter, for example cargo clean-all --keep-days 7 ~ to clean only what hasn’t been compiled in a week.
• Docker is right behind. On my machine I found 1.26 TB of build cache plus 938 GB of images. A single cleanup recovered about 600 GB. On the server, the weekly cleanup collects 0.6 to 4.5 GB every Sunday.
• AI and package caches bloat quietly. The Hugging Face cache had 86 GB until I deleted three dead models and it dropped to 21. Pacman had 22 GB of outdated packages. yay, uv and pip, another 125 GB.
• And agents create temporary directories without asking permission. A complex project with several branches means worktrees, and each release worktree has its own CARGO_TARGET_DIR full of test binaries. On ai-memory that has already blown up the disk, and it became a mandatory rule in the project’s AGENTS.md: after every release, cargo clean in each created worktree and removal of finished worktrees, no exceptions.

The solution is to automate the sweeping. My minimal example, a user timer that runs on the 1st and 15th at dawn, with Nice and idle I/O:

• deletes target/ directories of projects idle for more than 14 days
• cleans the yay cache
• prunes only Docker build cache older than 30 days, never images or volumes

And the part that became standard in my house: instruct the agents to clean up what they create. Three real rules from my projects, for you to copy:

In the desktop’s AGENTS.md: “Never run broad cleanups like docker system prune without explicit user authorization after a read-only inventory. Cleanup uses an exact list of IDs; idle, old or unreferenced does not mean disposable.”

In the homelab’s CLAUDE.md: “Before finishing any task, remove the temporary things the task created, and only those. Never delete what you didn’t create or can’t attribute to the task. When in doubt, list it for the owner.”

In ai-memory’s AGENTS.md: “target/ grows without limit: every edit to a shared crate leaves behind a copy of each 100-180 MB test binary (seen at 157 GiB). Weekly cargo sweep --time 7, and mandatory cargo clean after every release.”

And the real backup: restic

A snapshot lives on the same drive, so a snapshot is not a backup: if the drive dies, the snapshot dies with it. The last item in the kit is restic, which for me is the best backup system on Linux today.

restic is a single open source binary, written in Go, that does deduplicated, encrypted and versioned backup to practically any destination: another directory, another NAS via SFTP or REST, S3, Glacier. The central trick is content-defined chunking: it slices each file into variable-size blocks defined by the content, hashes each block and only writes to the repository the blocks that don’t exist there yet. That’s why the delta is cheap: today’s backup carries the blocks that changed since yesterday, not the files that changed. Editing a 100 GB database in the middle costs a few megabytes of backup, not 100 GB again. Each run becomes a repository snapshot, a restore point you browse with restic mount as if it were a filesystem, and restic check audits the whole repository’s integrity whenever you ask.

Configuration matters more than the tool, so here’s ours, mistakes included. On the desktop, the repository lives on the NAS and retention is keep-daily 7, keep-weekly 4, keep-monthly 6, keep-yearly 1. On the server, backup runs daily at dawn to the repository on the QNAP, covering /etc, /usr/local/bin, /home and the container volumes, with the password off the machine and pg_dump before the databases, because a filesystem snapshot doesn’t guarantee consistency for a database that’s writing. The first full load was 729 GiB in under an hour, and restic crossed unharmed, on retries, the window in which the NFS mount went read-only by mistake during the migration.

The most important part of the configuration is what does not go into the backup. Our exclusions, all with a reason:

• AI model caches (Immich, Hugging Face, EasyOCR): if lost, download again;
• Prometheus metrics: history that regenerates itself;
• the Fulcrum database: reindexes from bitcoind;
• the decision classifiers’ weights: re-downloadable by revision hash (and this exclusion’s glob goes inside quotes inside the bash array, because unquoted it becomes a backup source instead of an exclusion);
• node_modules, target/, cargo registry, rustup toolchains: pure build artifacts.

And one retention lesson to close. When we shortened the server policy to “yesterday, 2 dailies, 2 weeklies and 1 monthly”, the correct translation to flags is --keep-daily 3 --keep-weekly 3 --keep-monthly 2, not 2/2/1: restic’s levels overlap, and yesterday’s snapshot is also the newest of this week and this month, so each number needs to be one more than the older points you want to keep. The detail caught us off guard: the script had a literal 2/2/1, and with it the monthly level would never hold anything older than last night. In practice, the intention was to keep 5 restore points, and 2/2/1 delivers only 3: yesterday, plus one daily, plus one weekly. We erred, measured, corrected.

A humility note from the disaster: the old repositories were left on the Synology by my decision (almost 2 TB of abandoned history, because the content was already re-covered on the new QNAP). A backup repository ages too, and revisiting its scope is part of the hygiene.

Conclusion: in the AI era, disk is not a detail

Agents coding all day change the storage math at both ends. On the volume end: builds, caches, worktrees and models fill the disk much faster than any human habit ever did, and without a cleanup routine you find out when it blows up. On the safety end: the more generated code, the more you need a cheap way back, because not every agent change is a good one.

btrfs covers both. Transparent compression gives back 15% of disk for free, and source code is exactly the kind of data that compresses the most. Snapshots and subvolumes give you instant rollback on the system and isolation for what doesn’t need protection. Checksums and scrub watch the health of blocks that ext4 simply ignores. ext4 had its 15 years of well-deserved reign. In 2026, choosing it on a new Linux is giving up protection you’ll need, sooner or later. Choose btrfs and set up the cleanup today.

At that moment I knew I was a hiccup away from losing everything. This post is the post-mortem of what happened, and the first part was written in the heat of the disaster, with the rescue copy still running.

Update of October 11: the copy finished and the ending was a happy one. No file was lost by accident, both NASes are healthy, each with its own RAID-6. The full recovery is in the second half of this post. The first part stays as it was published, untouched.

I’ve been using a NAS for a long time

I’m not new to this. I made an entire video mini-series on storage and filesystems (in Portuguese), and I recommend watching it before asking me “what’s a good beginner NAS?”. There is no such thing.

I’ve used external storage since the second half of the 2000s, when I was a Mac user and Drobo existed. The company is gone, but I had three of their DAS units before migrating to Synology, first on an entry-level model and then on the mid-tier DS1821+, which I’ve been using for more than five years. I’ve written about it here, setting up NFS on Linux and accessing it over iSCSI.

It’s been very valuable to me, and you don’t get to question why I need that much space. It doesn’t matter: everyone decides what they keep on theirs.

When I was producing YouTube videos, I edited 4K footage straight from the NAS, thanks to the NVMe cache and 10 GbE networking. I got used to big storage at 10 gigabits and I can’t go back.

The array was organized in SHR, Synology’s hybrid RAID, which in practice behaves like RAID-5 (more on that later): it survives the loss of one drive without losing any data. I always thought that was enough.

Hard drives die. Always

Make no mistake: hard drives go bad. Never, ever trust a lone drive forgotten in a closet. It will fail, and you will lose everything on it.

The best public data on this comes from Backblaze, which runs hundreds of thousands of drives and publishes failure rates every quarter. In the Q2 2026 report, with 354,415 drives monitored, the annualized rate was 1.73%, and the lifetime rate for the whole fleet stands at 1.41% per year.

That sounds small. Translated to human scale:

• One drive on its own, for 5 years: the chance of it dying in that period is close to 7%. That’s roughly the chance of rolling one specific number on a 14-sided die. Would you bet every family photo on that?
• One drive on its own, for 10 years: about 13%, one in eight.
• Eight drives, like my NAS, for one year: the chance of at least one failing is about 11%. In five years, more than 40%.

And those numbers are from a datacenter, with controlled temperature, clean power and drives spinning all the time. A drawer drive that sits idle for years and gets knocked around in a move doesn’t have statistics that pretty.

Memory cards are worse, and there isn’t even an equivalent statistic to consult. What exists is torture testing: the Great MicroSD Card Survey tested 216 cards over a year of continuous writing, and 52 of them, one in four, had already died.

Genuine cards started throwing errors, on average, after about 2,500 rewrite cycles; counterfeit cards, after 700. A memory card is transport media. Keeping the only copy of anything on a microSD is asking to lose it.

The practical conclusion is the same as in the mini-series: important data needs redundancy. A parity array exists precisely so that the death of one drive, which is a matter of time, doesn’t take anything with it.

What happened

With all that, I should have been at ease with my setup. I wasn’t.

It started the day before yesterday, Sunday. For a while I’d been watching the volume approach the ceiling.

At first it was 8 drives of 10 TB. Over the years I swapped them, one by one, for 20 TB drives (DSM shows binary units; on the label they’re 12 and 22 TB). It’s a slow process: each swap takes days, because the array has to rebuild and redistribute the data.

I decided to swap one more, the seventh. Synology supports hot-swap, so it’s simple: pull the old drive, slot in the new one, add it to the array and let DSM handle the rebuild.

On Sunday night it looked like this: rebuild starting, all eight drives healthy, 108 TB allocated out of 121.8 TB.

On Monday, at 8 a.m., the beep. Drive 1, the new one, with 13 hours of use, marked critical:

And drive 8, one of the old ones, with 22,095 hours powered on (two and a half years), also critical, with a read error at 08:05:

The pool’s message left no room: “Unrecoverable errors have occurred because drive errors occurred after storage pool degradation. Please back up your data immediately.”

Why two drives is the end

In RAID-5, if two drives go down, redundancy is zero, and in theory the whole array goes with them. It’s an unrecoverable situation.

Anyone who watched my playlist knows why. An array like this doesn’t store “files,” one per drive. A file is a collection of blocks, and the blocks are spread across all the drives, along with the parity. Losing a drive is losing a piece of every file. It doesn’t matter that the other seven are healthy, because they only have parts.

Parity is there to recompute the missing piece when one drive disappears. With two missing, the math doesn’t close.

And there’s a cruel detail in the rebuild: to recreate the new drive, the array has to read every other drive end to end. That’s more than 100 TB of continuous reading. If some old drive had a bad sector hiding in a corner nobody had read in a long time, that’s when it shows up. That’s what happened with drive 8.

Where do you copy 90 TB to?

The only way out, at that point, was to thank the heavens that the volumes were still accessible, with the files showing up, and use the window to copy everything out immediately. I couldn’t touch anything else.

The problem: copy about 90 TB to where? Nobody has an empty 100 TB NAS lying around at home.

Normally I’d buy this kind of hardware in the United States, on a tourist trip, and bring it back with me. It’s much cheaper. Buying expensive electronics in Brazil isn’t worth it: the country has charged import taxes worse than Trump’s tariffs for decades, and the final price usually goes past double the American list price.

Except I had to choose. What is it worth to lose 90 TB of data I spent years collecting and that is nearly impossible to recover from anywhere else? It’s one of those situations where the answer is “it’s priceless,” and so, whatever the price, I pay.

I found a storage reseller close to home, Controle Net. Good people, very quick to respond, and they sent me a proposal for equipment with same-day delivery.

Their first suggestion was a QNAP TS-832PX. I researched it before accepting. It has the same 8 bays and ships with 10 GbE, but the processor is a 1.7 GHz ARM Cortex-A57 (Annapurna Labs AL324) with 4 GB of RAM, half the 8 GB minimum that QuTS hero asks for. For RAID-6 on ZFS, with checksums and double parity on every block, that’s weak hardware, as I explain further down. And it would be a downgrade from the DS1821+ I already had, which runs on a Ryzen V1500B.

I asked for another option, and after some back and forth I settled on a QNAP TS-873A, 8 bays, with the same Ryzen V1500B as my Synology, 8 GB of RAM, and 10 Toshiba MG11 24 TB enterprise-class drives.

This is why you have to do your own research and know, objectively, what you want to do with the equipment. The reseller didn’t act in bad faith: they suggested an 8-bay NAS to someone who asked for an 8-bay NAS. The one who knew it would run ZFS in RAID-6 was me. If I had accepted the first proposal in a panic, I’d have paid a lot for a worse machine than the one I was replacing.

R$ 185,320.00. Eye-watering.

What it would cost in the United States

I checked today’s American list prices. About the exchange rate: the dollar is at R$ 4.97 right now, but that’s the effect of Sunday’s first-round election results, which knocked the rate down more than 4% on Monday. Through Friday it closed at R$ 5.22, and that’s the fair rate to compare with, because that’s the rate the reseller’s stock was built at:

At R$ 5.22, about R$ 60 thousand. I paid R$ 185 thousand, more than three times that.

The comparison isn’t perfect: the Brazilian proposal includes installation, training and six years of technical support, and the equipment was on my desk by the end of the same day. But it gives the measure of the Brazil effect.

If the rule of thumb is “double the American price,” normal would be about R$ 120 thousand. The extra R$ 65 thousand is, in my own rough estimate, the sum of services, local stock of enterprise drives and my urgency. Whoever buys on a Monday morning with the array dying is in no position to haggle.

And drives are expensive everywhere

To make it worse, this is the worst possible moment to buy hard drives. The AI bubble is sucking up production.

I thought Seagate had left the consumer market. I checked, and that’s not quite it: it hasn’t left, but in February 87% of its hard drive sales were already nearline drives, for datacenters, and WD gets 89% of its revenue from cloud customers and only 5% from retail. WD’s CEO said the company is “pretty much sold out for calendar year 2026.”

Seagate has said it won’t expand production capacity; growth comes only from higher-density drives. For someone buying drives at home, the practical effect is similar to them having left: you get what the hyperscalers didn’t want, at whatever price the seller asks.

QNAP instead of another Synology

After spending the morning and afternoon between negotiation, paperwork and payment, by the end of the day I had the QNAP powered on next to the Synology.

Choosing QNAP went beyond availability. Synology has been making questionable business decisions.

In 2025 it launched the Plus line requiring its own-brand drives or drives it certified.

The own-brand ones are Seagate, Toshiba and WD drives rebadged with custom firmware, and anyone using something else was left without pool creation, health monitoring, deduplication or firmware updates. After the outcry, it walked that back in DSM 7.3 and allowed HDDs and SATA SSDs from other brands.

Except M.2 pools still require the compatibility list. The uncertainty is left hanging: what stops them from doing it again?

Setting up the QNAP is very easy. QuTS hero, their ZFS-based system, is as simple to use as Synology’s DSM.

Fixing my mistake: RAID-6 and ZFS

This time I knew I had to configure RAID-6, which survives two drives failing at the same time. The downside is less total capacity, because the equivalent of two drives becomes parity instead of one. The upside is that I shouldn’t have to go through an emergency like this again.

One of QNAP’s strengths is supporting ZFS out of the box, well integrated and easy to use.

For those who don’t know it: ZFS was born at Sun Microsystems, for Solaris, in the mid-2000s, and today lives on as OpenZFS on Linux and FreeBSD. It merges into one system what used to be three layers, the RAID, the volume manager and the filesystem. It keeps a checksum of every block, so it notices when a drive silently returns corrupted data, and repairs it on its own from parity. And it never overwrites data in place: every write goes to a new block, which makes snapshots practically free.

Its reputation as a RAM hog comes from two things. The first is the ARC, its cache, which by design takes up almost all the machine’s free memory. That scares anyone looking at the graph, but it’s cache: it gives the memory back when another process asks. The second is deduplication, which keeps a huge table with the hash of every block and needs it in RAM or it becomes unusably slow. The old rule of “1 GB of RAM per TB of disk” comes from there, and only applies to people who turn deduplication on.

Even without it, ZFS is heavier than ext4 or a simple Btrfs. Computing checksums, compression and double parity for every block costs CPU, and with little memory the cache shrinks and everything gets slow. On weak hardware, like an entry-level NAS with an ARM processor and 2 GB of RAM, it suffers.

QNAP asks for at least 8 GB for QuTS hero. Mine came with 8 GB and a quad-core Ryzen, which works for my use, which is storing large files and serving them over the network. Deduplication stays off.

I bought 10 drives for an 8-bay NAS, so two are left in the drawer as spares, for the day one fails.

For my simple setup, three settings:

• 5% over-provisioning. It’s a slice of the pool the system reserves and never hands out for data. ZFS always writes to new space and gets slow when the pool fills up near the limit; this slack keeps performance stable. On SSD pools it also extends drive life, but these are hard drives, so the minimum is enough.
• 5% guaranteed snapshot space. It’s the reserve for snapshots, the “photos” of the state of your files that let you go back after an accident or ransomware. Guaranteeing the space means the snapshots keep existing even if I fill up the rest. My data changes little (it’s mostly cold archive), so snapshots take almost nothing and 5% is plenty.
• Alert at 85%. The system warns me when the pool passes 85% usage, with room for me to act before hitting the ceiling. It was exactly the lack of room that had me swapping drives in a hurry on the Synology.

Adding the two reserves, about 13 TB are set aside and 112.9 TB are left for use.

A detail of my case: I’m copying the 90 TB with snapshots turned off, and I’ll only enable the schedule after the copy finishes. A snapshot protects a file’s previous state, and on an empty NAS receiving its first load there is no previous state worth protecting. The source of truth is still the Synology.

And there’s a practical cost. I stopped and restarted the job several times, deleted folders that had gone in by mistake and copied again. With snapshots on, every deleted thing would keep taking up space inside some snapshot, and I’d be filling the pool with the migration’s own garbage. When everything is on this side and checked, then yes: scheduled snapshots, to protect me from deleting what I shouldn’t and from ransomware.

After that I created one giant “thick” partition. In ZFS you can create a “thin” folder, which only takes up what’s inside it and can promise more space than exists, or a “thick” one, which reserves the full size at creation. Since it’s a pool with one owner and one use, thick is more predictable: the space is mine and nothing competes for it.

Finally, I created the shares and installed the HBS 3 app.

Copying 90 TB

HBS 3 (Hybrid Backup Sync) is QNAP’s app for backup and synchronization. It connects to another NAS, to rsync, FTP, SMB or the cloud.

I connected to the Synology’s SMB share and told it to sync everything.

I had to stop and restart a few times, tweaking the job settings, because it was absurdly slow. And the file count wouldn’t stop growing:

865 TB on a 108 TB NAS, and that’s when I understood what was going on.

I excluded two kinds of path from the job:

• The #snapshot directories. On Synology, when you check “make snapshot visible,” each shared folder gets a #snapshot subfolder with a full view of the folder at each point in time. Inside the Synology that takes no space, because a snapshot only stores the difference. Seen from outside, over SMB, every snapshot looks like a full copy of everything, and HBS was trying to copy them all.
• The restic backup directories. restic stores everything in hundreds of thousands of pack files of a few megabytes each. Network copying is fast with large files and suffers with small ones, because each one costs an open, metadata and a close. And those backups I’ll redo from scratch on the new NAS.

With that, throughput rose to an average of 400 MB/s.

That’s less than half of what a 10 GbE network delivers, but it’s more than three times the gigabit most people have at home. Both sides really are at 10 gigabits:

Why doesn’t it reach 1,000 MB/s? My main hypothesis is drive 8. The source array is degraded, and every read that goes through the affected pieces has to be recomputed from parity, with a drive that on top of that returns errors and retries. It makes sense that the Synology can’t go faster than this.

There’s a second factor, which my own screenshots give away: the Synology has jumbo frames on (MTU 9000), and the QNAP image shows the default MTU of 1500. That screenshot is from earlier; I’ve already changed the QNAP to 9000 as well.

Why it matters: MTU is the maximum size of each packet that travels on the network. At the default 1500 bytes, filling a 10-gigabit link means processing more than 800 thousand packets per second, and each packet costs CPU on both sides, for the header, the checksum and the interrupt. With jumbo frames each packet carries six times more data, so it’s one sixth of the packets for the same volume, and more processor is left for moving files. It only works if everything in the path matches: both cards and the switch. If one side stays at 1500, the connection negotiates down to the smaller value and the other side’s jumbo frames do nothing.

At 400 MB/s, 90 TB takes about two and a half days, and I prefer two and a half guaranteed days.

What Synology explained

In parallel, I opened a ticket with Synology support. After some back and forth, I sent the logs. They responded fast, the same day, and I’ll give them that. The engineer’s explanation, summarized:

• Drive 8 had been logging media errors for over a month. A media error is when the NAS tells the drive to read a sector and the drive answers that it couldn’t, even after several retries. It’s always an internal drive failure, and the drive should have been replaced.
• It was drive 8’s poor health that took it down during the rebuild of drive 1.
• In practice I lost two drives, and the pool should have crashed completely. It didn’t because of how SHR works.

SHR isn’t a single RAID-5. To use drives of different sizes without wasting space, it slices the drives and builds several smaller RAIDs, one per size band, and joins everything into one volume. The engineer sent this example diagram:

In my case, the pool is made of two partitions. In the first, the rebuild of drive 1 was far enough along and the failure of drive 8 was mild enough that both kept working. In the second, both are missing. That’s why the pool didn’t crash entirely and I can still see the files.

His recommendation was the one I was already following: copy everything out before anything else, because there’s no telling whether the situation gets worse or whether it’s fixable. After the backup he can even try forcing drive 8 back into the pool so the rebuild continues, but there’s no guarantee the drive will go back in, or that it will survive the process. In his words, copying and recreating the pool from scratch is probably the quickest way forward.

Classic RAID-5, by the way, requires all drives to be the same size, and to grow capacity I’d have to replace all of them at once. SHR is what let me swap one at a time over the years. Ironically, it’s also what saved me from losing everything at once.

The two mistakes

The way I understood it, there are two mistakes, one mine and one theirs, and that’s where the whole disaster lives.

Mine: I underestimated RAID-6, which on Synology is called SHR-2. With two parity drives, drive 8 failing in the middle of the rebuild would have been just a scare. I traded safety for 20 TB of space and spent five years thinking I’d made a good deal.

Theirs: from the logs, the software knew drive 8 had been failing for at least a month, and I never got a single notification. I have email notifications configured. And they work: when the critical situation blew up, I got a flood of emails. This one, about drive 8’s bad sectors, arrived at one in the afternoon on Monday, five hours after the disaster:

Before that, nothing. No warning about drive 8 until it failed catastrophically.

If I had received one, I would never have tried to swap drive 1 before replacing drive 8. The whole situation was avoidable, and it was the result of a human mistake plus some software bug.

That’s the other reason I didn’t buy another Synology. I have a principle: if the horse throws me, I put the horse down. I can’t trust it again.

Where I am now

So far I’ve copied about 20 TB of the original 90 TB, and HBS 3 keeps working. I still don’t know whether I lost any data permanently. I hope not, but the most likely outcome is that I lost what was in the RAID slice the engineer described as crashed.

I have offsite backup on AWS Glacier of what matters most, but not of everything.

A lot of people ask why I don’t use S3 or some other cloud storage for everything. I return the question: think for ten seconds. How long does it take to transfer 90 TB over your internet connection? Do the math:

• With a solid 1 gigabit of upload, never dropping for a second: more than 8 days.
• With 500 megabits: almost 17 days.
• With 100 megabits, which is good upload for many homes: 83 days.

And that’s to upload. When disaster hits you need to download it all back, with the clock running. Not counting the rent: 90 TB on standard S3 costs about US$ 2 thousand a month at list price. Cloud is great for the critical subset, which is how I use it. For the whole archive, physics and the invoice won’t allow it.

Lessons

I was put in an impossible situation on a Monday morning, I acted right away as best I could, and I’ll probably come out of this mostly unscathed. We’ll see, HBS 3 is still syncing. What I take from it:

1. Single parity isn’t enough for big drives. With 20 TB drives, a rebuild takes days and reads everything. That’s the window in which the second drive tends to show up. RAID-6, SHR-2 or RAIDZ2.
2. Don’t rely only on the vendor’s notifications. Look at SMART and the drive logs before starting any swap. I didn’t, because I trusted I’d be warned.
3. Replace the sick drive first, then the small one. A capacity upgrade on an array with a suspect drive is roulette.
4. Leave slack. I was swapping drives because the volume was at the ceiling. An 85% alert exists for that.
5. Have somewhere to copy to. My way out cost R$ 185 thousand because I had no plan B. A second NAS, even a modest one, with what matters, would have bought me time to shop calmly.
6. RAID is not backup. I know that, I teach that, and I still only had offsite for part of it.

The copy is done and I already know the size of the damage: zero. The full recovery comes next.

Six days later: the end of the story

When I published the first part, I had about 20 TB copied and no certainty. The outcome: I finished evacuating the ~91 TB, verified file by file, and lost nothing by accident. Not even what was on the fallen RAID slice, because it stayed readable until the end.

Who operated the recovery were my two agents. The desktop agent controls my PC and both NASes over SSH, and the homelab agent takes care of the server where my 57 containers run. The two coordinate by message: one leaves a report in the other’s inbox, and I follow the conversation.

)

This part matters because nothing that follows was me typing commands. I made the decisions; they executed, measured and screwed up badly a few times, as you’ll see.

HBS 3 was quietly betraying me

In the first part I told how HBS 3, pulling the Synology shares over CIFS, reached 400 MB/s after the adjustments. What I didn’t know: speed was the smallest of its problems.

Two defects showed up when the agent checked the source inventory against what HBS reported:

• It didn’t go where it had no permission, and told nobody. HBS mounts the Synology as my user. The Bitcoin/data directory, 901 GB with my node’s wallet and chainstate, belongs to uid 101 and is closed to anyone else. CIFS couldn’t get in there, and HBS reported success without recording the omission. A “green” backup that leaves the entire Bitcoin node behind.
• It skipped files with “unsupported” names. 113 files, recorded only on a detail screen nobody opens during a days-long copy.

A backup tool that reports success while omitting what it couldn’t read is worse than one that fails screaming. Verify the backup against an independent inventory of the source, never against the tool’s own log.

We abandoned HBS. The agent opened SSH on both NASes, enabled the rsync server on the QNAP, and the Synology started pushing straight to its rsyncd, with nothing going through the PC. Each job’s wrapper: rsync -a --numeric-ids -W --partial --timeout=600, attempts with retry, and never --delete. The --numeric-ids preserved the original uids with no chown afterwards, --partial turned any interruption into a continuation, and the absence of --delete made the destination always a superset: re-sharding or killing a job never cost data.

The bottleneck wasn’t the broken drive

Some context was missing from the first part. This Synology was born with eight consumer-line Seagate IronWolfs, and over the years I had been swapping them one by one for 20 TB IronWolf Pros. The disaster happened exactly during one of those upgrades: I swapped drive 1 for the new Pro and, during the rebuild, drive 8 failed. One single old drive was left in the array: drive 2.

And it, not drive 8, was the evacuation bottleneck. Measured under load:

• Drive 2 (the old consumer one, with 37 thousand hours of use): 91% utilization, delivering 27 MB/s, each I/O taking 4 to 15 times longer than its siblings.
• The six Pros: between 34% and 45% utilization, sitting idle waiting for it.

In RAID-5 (and in SHR), a sequential read spans the whole stripe, and the stripe only completes when the slowest member answers. One slow drive caps the whole array: because of it, the array couldn’t go past about 200 MB/s while six good drives had room to spare. And its SMART was clean, zero reallocated sectors. Consumer drives usually ship without time-limited error recovery (ERC/TLER) active, so the drive retries a marginal sector for seconds, internally, without accounting for anything. The Synology’s NVMe cache diverted part of the reads from the platters and helped mask the symptom.

The drive that fails and the drive that slows you down don’t have to be the same, and SMART “PASSED” says nothing about latency. A slow array is diagnosed with per-drive utilization and response time, not with SMART.

Parallelizing rsync is science, not guesswork

First instinct: more streams. Wrong, twice.

The first time, a bug of ours: a “clean” job relaunch didn’t kill anything, because the Synology’s busybox ps truncates the command line and the verification grep didn’t find the old processes. Result: 32 rsyncs running into 7 destinations, four to five writers per tree, the QNAP with its receive buffers clogged, and the cable transmitting 516 MB/s while the array read 140. After the cleanup (enumerating /proc/*/cmdline, never trusting ps again), useful throughput rose from 140 to 181 MB/s while total load collapsed.

The second time, measurement inverted common sense: with large-file trees, fewer streams were faster, because the average size of each read request tripled and the array did long sequential runs instead of jumping around.

But small-file trees are another sport. TERACHAD/Emulators has 21 TB in 3.4 million files averaging 6 MB: latency-bound work (stat, creation, metadata), not bandwidth-bound. And there aggressive parallelization worked, with one detail: the right shard is chosen by file count, not by size. A single directory, ROMS_FINAL/xbox-hdd-ready, held 995 thousand of ROMS_FINAL’s 1.59 million files, averaging 1.7 MB per file. Isolating it from the multi-gigabyte ISOs:

Between 2.8 and 3.2 times the single stream’s average throughput, on the same machine, on the same network. And there’s a beautiful reason: with the array missing a member, reads of data that doesn’t live on drive 2 don’t need it, so the extra concurrency let the other drives work while it choked.

“More parallelism” and “less parallelism” are both wrong answers as a general rule. Bandwidth-bound work splits little; latency-bound work splits a lot, and by file count. Measure before choosing.

Copying is half the work; proving is the other half

Every job that finished went through a dry-run of the same rsync, with the same excludes: zero files needing transfer means an identical destination by size and date. And even so it almost went wrong, because the verification inherited the copy’s excludes: a tree excluded from the transfer stayed excluded from its own proof. Re-running the check without excludes turned up 952 files and 191 GB missing in Downloads (exactly the batches I had set aside as “only existing copy”) and the old restic repositories, which a similar directory name on the QNAP pretended to cover: the repositories’ internal ID proved they were different ones, not copies.

There was source drift too: a tree verified as complete received 16 GB of new checkpoints on the Synology after the check, and only the final re-verification caught it. Verification has an expiration date.

And a conscious decision, recorded so nobody accuses me of hiding it: the old backup histories (about 2 TB of restic snapshots and a duplicity set from 2025) were left behind by my choice, because the new repositories on the QNAP already cover the same content from now on. Deliberate loss of history, not of data.

Closing the books: 17 rsync jobs, all exit 0, all verified, about 5.9 million byte-identical files, and the Synology shutdown only happened after every tree was proven.

)

Meanwhile, in the homelab

On the other side of the house, the marvin server, which mounts the NAS over NFS and runs the 57 containers, almost died standing up. Dead NAS plus “hard” NFS mounts plus 57 containers restarting on their own made an infinite queue of uninterruptible I/O: the machine answered ping, but sshd couldn’t complete a handshake. And every insistent SSH attempt only made it worse, because OpenSSH punishes a repetitive source by extending the ban. We knocked on the door and the door added a lock.

The way out was to stop knocking: I paused the torrents through the qBittorrent API (which kept responding), went to the physical console, disabled Docker, rebooted, and the server came back. A watchdog I didn’t even remember existed had, on its own, taken down the 14 NAS-dependent containers before any database corrupted: bitcoind’s LevelDB opened clean on the first try.

The migration itself was elegant: fstab pointed at the QNAP and the old mount points became symlinks into the new one. No docker-compose changed. Then came the QNAP NFS gotcha: permission is per client. The desktop was allowed without squash, but marvin’s writes came in as uid 1001, blocking everything that writes to the NAS. The first fix attempt still flipped the mount to read-only in the middle of the first restic backup, which survived on retries. Final configuration: read and write, no squash for anyone, tested with touch and ls -ln from each client, because “it worked on the desktop” proves nothing about the server.

)

After that it was resurrection tree by tree: each service only came up when its tree was verified on the QNAP. In two days, 57 of 57 containers back, and a brand-new restic backup running daily to the QNAP.

The new kernel took down the NVMe

In the middle of the recovery, a free scare from the server. Seven minutes after the nightly backup started, the NVMe controller vanished silently: every binary took SIGBUS, Docker and sshd died, and nothing remained recorded anywhere, because the log lives on the device that died. Not even a kernel panic: just silence.

The same workload had run fine hours earlier on kernel 7.2.6 and died on 7.2.9. Disabling APST (the NVMe’s power management) didn’t fix it: a second identical crash, a 7.2.9 regression confirmed by elimination. MicroOS is transactional and snapshot-based, so the rollback was a reboot into the 7.2.6 lineage, not a reinstall. Automatic updates were deliberately left broken on that lineage so nobody goes back to 7.2.9 unsupervised, and testing 7.2.8 waits until I can sit next to it.

A transactional operating system turns a kernel regression into a reboot. Without system snapshots, that night would have cost a lot more.

The thick provisioning scare

On the QNAP, the whole copy ran with no snapshots at all: in a bulk migration, snapshots are dead weight. With the copy done, I went to enable the ZFS snapshot schedule and got a dry error: “pool free space is not enough”. With 20 TB free.

)

Both statements were true about different things. The CHAD folder had been created as thick: out of the 100 TB quota, ZFS reserves the full 100 TB, and to snapshot a dataset with a reservation, the pool has to guarantee space to overwrite every referenced block while the snapshot pins the old copies. The math: 80 TB written plus 100 TB reserved, against 126 TB usable in the pool. It doesn’t close. And df lies on both sides, because it measures the folder against the quota, not against the pool.

The fix was converting the folder to thin, which QuTS hero does in-place, hidden on the folder’s Management screen, not in the edit dialog where “Thick” looks like static text. In thin, the number becomes a ceiling instead of a reservation: the 10 TB reservation evaporated, the pool ended up with 39 TB free, and the first snapshot came out consuming 334 KB over 79.9 TB referenced. ZFS needed almost nothing; the reservation was what demanded a 100 TB guarantee to authorize.

)

Now the QNAP has a snapshot schedule with a retention policy, as it should have had since day one:

)

)

Two footnotes for anyone building one of these. If your thin folder has an 80% alert, raise the quota for aesthetic slack, because the alert measures against the quota and not against the pool. And this QNAP’s real bottleneck is RAM: 7 GB for a 174 TB pool. QuTS hero’s official minimum is 8 GB, and sensible practice at this scale is 16 and up. The size of the ARC cache governs metadata speed, and that’s why the whole-tree sweeps during the migration were slow. A RAM upgrade is worth more than any fine-tuning.

The Synology is born again

With everything verified on the QNAP, the Synology could finally be operated on. I swapped drives 2 and 8 for new IronWolf Pros, deleted the wounded pool and recreated it from scratch, this time in SHR-2, the two-parity-drive mode, their RAID-6. The new pool came out with 100 TB, all drives healthy:

)

It goes back to work as a replica target, and no longer as a single point of failure. The QNAP stays as primary, in RAIDZ2, with 87 TB of the 126 TB used and snapshots scheduled:

)

What I take from the second half

1. Distrust silent success. HBS reported green while leaving 901 GB behind. A backup is verified against a source inventory.
2. SMART doesn’t measure slowness. The drive that nearly killed the evacuation was “healthy.” Per-drive utilization and latency tell the truth.
3. Parallelism is calibrated by measuring. Bandwidth-bound splits little, latency-bound splits a lot, and the shard is chosen by file count.
4. Verification doesn’t inherit excludes. If the proof uses the same filters as the copy, it only reaffirms the copy’s assumptions. And verify again immediately before destroying the source.
5. NFS permission is per client. Test writes from every machine that will use it.
6. Don’t hammer a dying sshd. Each attempt extends the ban. One patient attempt, or the physical console.
7. The dangerous repair comes last. The array stayed degraded and slow until the last byte was proven somewhere else; only then did rebuilding stop costing anything.

I came out of this better protected than I went in: two NASes, each with its own RAID-6, scheduled snapshots on the QNAP, daily restic, and Glacier for what’s critical. The disaster cost me a QNAP, two drives and a week. It could have cost 90 TB.

You pay for a book, but you can only read it where Amazon allows. There is no decent native Linux app. Moving to another reader turns into an exercise in archaeology. I bought a new Xteink X4 and, naturally, my library couldn’t simply come along. The file was “in my account,” but it wasn’t under my control.

For years, I worked around this through the official route. I’d sign into Amazon’s website, use Download & Transfer via USB, download the .azw files for books I’d bought, and import everything into calibre. With the DeDRM plugin configured for my Kindle’s serial number, I’d convert them to EPUB and be done with it.

On February 26, 2025, Amazon shut that option down. No decent announcement, no replacement that actually handed over the file, and no concern for anyone who wanted a local backup. My newer books have been trapped in their ecosystem ever since. I tried again last year, updated the plugin, copied files from my Kindle Paperwhite, entered the correct serial number, and got nowhere.

To be fair, on January 20, 2026, Amazon began allowing verified buyers to download EPUB or PDF files when the publisher marks the book as DRM-free. Great. That doesn’t help with protected books, and the decision still belongs to the publisher. Older DRM-free titles must also be confirmed one by one by the publisher. This did not magically add a download button to my library.

I decided to try again now. The new DeDRM version can handle the current DRM, but the process has changed quite a bit. It takes work, requires Windows for one step, and relies on a community tool still in pre-release. It worked. I recovered 106 books I’d bought, tested the EPUB conversion, and can now read them on whatever device I choose.

This is the state of play on July 30, 2026 (updated August 19). Amazon could change the encryption or the app tomorrow. Ignore old tutorials telling you to install Kindle for PC 1.17, disable updates, and enter the device serial number in Calibre. That world is gone.

And let’s get the obvious out of the way: I’m talking about books I bought. I’m not talking about downloading Kindle Unlimited titles, library loans, or somebody else’s books. Laws covering DRM circumvention vary by country. Check the law where you live and take responsibility for what you do.

First things first: don’t throw out an old Kindle

Amazon doesn’t make it easy to repurpose an old Kindle either. The hardware remains perfectly good, the battery is usually replaceable, and an e-ink screen lasts forever, yet the software becomes increasingly closed and limited.

If you want to revive one of these devices, I recommend the Dammit Jeff channel. He covers jailbreaks, KOReader, alternative stores, and ways to keep Kindles useful after Amazon loses interest. His video about AdBreak and recent jailbreaks is a good place to start. Read the current Kindle Modding guide before doing anything, because supported firmware and methods change constantly.

Jailbreaking the device and removing DRM from an ebook are separate problems. You do not need to unlock your Kindle to follow the rest of this article. I mention both because they start from the same premise: hardware and media we’ve already paid for should remain useful without asking the manufacturer for eternal permission.

Crash course: AZW, EPUB, and KFX

DRM and file format are separate things. The format defines how text, images, fonts, the table of contents, and metadata are packaged. DRM adds a cryptographic lock on top and decides which account, app, or device can open it.

These are the three names that matter here:

The extension alone can be misleading. Pieces of a KFX book may show up as .azw, .azw.res, .voucher, and other variations. The KFX Input plugin exists specifically to understand that bundle.

EPUB is much less mysterious. The EPUB 3.3 specification defines it as a single-file container for structured Web content. It’s basically a small packaged website. Any halfway decent reader can support it without depending on a secret Amazon key.

Calibre and the two plugins

calibre is the Swiss Army knife of ebooks. It’s open source, runs on Linux, macOS, and Windows, organizes libraries, fetches metadata, edits covers, transfers books, and converts dozens of formats. It has been around since 2006 and was born precisely because the first Sony Reader didn’t work properly on Linux.

On Omarchy or any other Arch system, install the package:

sudo pacman -S calibre

For other distributions, see the official download page. Don’t install some random package from a download site.

One gotcha for environments using mise: if Calibre dies with ModuleNotFoundError: msgpack or BrokenPipeError, a Python shim probably got ahead of /usr/bin. The Arch package needs the system Python. Start it like this:

env PATH="/usr/bin:$PATH" /usr/bin/python3 /usr/bin/calibre

This has nothing to do with DRM or the plugins. It’s just Calibre’s helper calling the wrong Python. I left a permanent launcher with this PATH on my desktop.

Calibre alone does not remove DRM. We need two plugins:

1. DeDRM recognizes and removes locks from several ecosystems during import. It only runs when the book enters the library. Clicking Convert afterward removes nothing.
2. KFX Input understands Amazon’s modern containers, gathers the pieces of a book, and makes EPUB conversion possible.

At the time of writing, I’m using DeDRM 10.0.28, published on July 14 in the fork maintained by Satsuoni. It is a pre-release. Download the asset named DeDRM_tools.zip, never a standalone executable from some obscure mirror.

Update (August 2026): the Kindle app on the Microsoft Store keeps moving, and each new app version requires a new version of the tool. Since this article was published, the app moved past 1.0.18632: 10.0.29 added support for app 1.22326, and 10.0.30 supports app 1.0.22920. The practical rule is a single one: the executable’s name carries the app version it understands (MSIXKFXArchiverMobi1_22920.exe for Store app 1.0.22920, and so on). Before you start, always check the releases page of Satsuoni’s fork and grab the newest pre-release. Everything else in the process described here stays the same.

I extracted everything into a directory that would later be shared with the Windows VM:

mkdir -p ~/Windows/Kindle-DeDRM/v10.0.28
unzip ~/Downloads/DeDRM_tools.zip -d ~/Windows/Kindle-DeDRM/v10.0.28

The SHA-256 for the DeDRM_tools.zip I used was:

520cce704edf9ae26e43196efe2871daf9b25d6cb489aa56051626801c362947

Check it with sha256sum. This doesn’t magically turn a community binary into safe software, but it at least confirms that you’re using the same file I tested.

Inside that directory is DeDRM_plugin.zip. Do not extract this second ZIP. In Calibre, open Preferences → Plugins → Load plugin from file, choose DeDRM_plugin.zip, accept the warning, and restart the program.

Then go back to Preferences → Plugins → Get new plugins, search for KFX Input, and install it. My test used version 2.33.0. Restart again.

Why copying from the Kindle over USB didn’t work

The simplest path would still be plugging in my new Kindle Paperwhite, copying the book, and configuring the device serial number in DeDRM. That was the first thing I tried.

The file came in as KFX-ZIP, the plugin ran, but it remained encrypted. I opened Calibre in debug mode and the log showed that the book used the ACCOUNT_SECRET strategy. The serial number was correct. The required key no longer came from the device alone.

That’s why so many people follow an old tutorial, try five different serial numbers, and conclude that DeDRM is broken. New books with this DRM need the secret stored by the current Kindle app on Windows. That secret is protected by TPM APIs.

A disposable Windows inside Omarchy

I’m not keeping a dual-boot setup just to download ebooks. I’m also not going to pretend Wine runs everything. Fortunately, Omarchy already ships with everything needed to install and launch a Windows VM: it appears in the Omarchy menu, opens over RDP, and automatically shares ~/Windows with the guest. I didn’t have to build this integration from scratch.

Underneath, it uses dockur/windows. Dockur packages QEMU/KVM inside a container and automates the Windows installation. It remains a real virtual machine with its own disk and kernel. The container handles distribution, configuration, and lifecycle.

To install it from the terminal, run:

omarchy-windows-vm install

Choose the RAM, CPU count, and disk size. The installer creates ~/.config/windows/docker-compose.yml, stores the virtual disk in ~/.windows, and shares ~/Windows with the guest. Inside Windows, that directory appears as Shared, usually on Z:, and is also available at \\host.lan\Data.

The detail that mattered in my case was enabling TPM 2.0. Edit the generated Compose file and add TPM: "Y" to the environment block:

services:
  windows:
    image: dockurr/windows
    environment:
      VERSION: "11"
      TPM: "Y"

Then recreate the container. The Windows disk remains in the persistent volume:

omarchy-windows-vm stop
omarchy-windows-vm launch -k

The -k keeps the VM running after you close RDP. You can also watch it boot in a browser at http://127.0.0.1:8006.

Downloading the books with the right app

Inside Windows, open the Microsoft Store and install Amazon Kindle: Reading App, product 9P8JQ0JJSTLL.

Pay attention here. The old Kindle for PC installer found on blogs is not the same app. The tool I used looks for the Microsoft Store package AMZNKindle.AmazonKindleReadingApp. With the legacy program, it simply replies No AmazonKindleReadingApp installation found.

Sign in to your account and download every book you want to preserve. Opening the cover or seeing the title in your library isn’t enough. The content must be available offline inside the app.

Unfortunately, the most annoying part is still manual. Amazon found room for a one-click purchase button, but apparently not for a “download everything I’ve paid for” button.

I did this for 106 books. Yes, it was a pain in the ass.

Generating KFX-ZIP files and the K4I key

The DeDRM_tools.zip we extracted on Linux is already visible in Windows through the shared directory. Open PowerShell and switch to that directory:

cd \\host.lan\Data\Kindle-DeDRM\v10.0.28
.\MSIXKFXArchiverMobi1_18632.exe

The banner may still mention an older 1.0.15230 build. The versioned executable above looks for Store app 1.0.18632, and that’s the one that worked for me. The number in the file name must match your app version: if the Store already delivered a newer one, download the matching release from the project’s page (see the update in the plugins section).

This executable finds the current Microsoft Store app’s cache, gathers each book’s components, and produces two things:

• archived_kfx/, containing one .kfx-zip for every downloaded ebook;
• oldbooks.k4i, the keyfile DeDRM needs to open those files.

On my machine, the process found and archived 106 books. The SHA-256 of the executable I ran was:

0c78b45ccea2c36a5fbb01b9f66bdd9e5d5960a68a340ba2ee96e138f5cddf4a

It is a 32-bit binary. If it complains about a missing MSVCP140.dll, install the official Microsoft Visual C++ Redistributable x86. Do not hunt for a loose DLL on Google. That’s a great way to turn a book backup into a malware backup.

The oldbooks.k4i file is sensitive. It is derived from your account secret in that Kindle installation. Do not publish it, attach it to a GitHub issue, screenshot its contents, or put it in a public repository.

Configuring the key in DeDRM

Back on Linux, open Calibre and follow this path:

Preferences → Plugins → DeDRM → Customize plugin → Kindle for Mac/PC ebooks → Import Existing Keyfiles

Select:

~/Windows/Kindle-DeDRM/v10.0.28/oldbooks.k4i

Use Import Existing Keyfiles. Do not use the generic Set Keyfile button on the main screen. It records a different kind of key candidate and does not register the K4I key in the right place.

Close the key list, click OK in the main configuration window, and restart Calibre. If you close it with Cancel, it quietly throws the change away.

Importing and converting to EPUB

Before dumping a hundred files into the library, test one.

Choose a .kfx-zip from:

~/Windows/Kindle-DeDRM/v10.0.28/archived_kfx/

Use Add books in Calibre. Internally, the order is:

1. KFX Input recognizes the package;
2. DeDRM tries the K4I keys during import;
3. KFX Input assembles the decrypted book as KFX;
4. Calibre can now open and convert the result.

If the format still appears as KFX-ZIP, it failed. Clicking Convert ten times won’t help. Remove that entry from the library, fix the plugin or key, and import it again. DeDRM only acts on input.

When the book appears as KFX, open it in the viewer and check a few pages. Then click Convert books, choose EPUB as the output, and test the resulting file on the reader you plan to use. Only then should you run the batch import.

In my case, the KFX opened, converted, and the EPUB worked outside Kindle. That told me the entire path was working: Microsoft Store, TPM, archiver, K4I, DeDRM 10.0.28, and KFX Input 2.33.0.

Back up what actually matters

After the conversion, I preserve three things:

• the final EPUBs, which I can read on any device;
• the original KFX-ZIP files, until I’ve verified that every EPUB is intact;
• oldbooks.k4i, encrypted and stored separately from the library.

My plaintext keyfile has 0600 permissions. I also keep an encrypted copy using SOPS and age, with the private key in a separate backup. Do not put a raw oldbooks.k4i in Git just because the repository is private. Private repositories leak too.

The EPUBs follow the same strategy I use for other important files: a local copy, NAS, and off-site backup. I explained this paranoia in Protecting and Recovering Lost Data and showed the same philosophy applied to movies in My Personal Netflix.

Do not delete files from the Kindle or the VM the minute conversion finishes. Open the EPUBs and check the cover, table of contents, images, notes, and a few pages. A backup you’ve never tested is just wishful thinking.

Conclusion

Amazon has remotely deleted books, publishers update covers and content, old apps stop working, and formats change. The end of Download & Transfer via USB proved that a feature available for eighteen years can disappear with one line in a notice.

The classic example is still unbelievable: in 2009, Amazon deleted 1984 and Animal Farm by George Orwell from customers’ Kindles. Later, revisions to Roald Dahl, R.L. Stine, and Agatha Christie were pushed into digital copies people had already bought. The Vice article summarizes those cases. Dammit Jeff makes the same point often: even the original cover you chose can turn into an ugly poster for the streaming adaptation because the publisher and store decided to update your “purchase.”

It doesn’t matter whether the next change is censorship, a legitimate correction, another horrible Netflix adaptation cover, or a plain bug. I bought an edition. I want to preserve the edition I bought.

DRM does not stop piracy. A popular book lands on torrents the day it comes out. DRM only gets in the paying customer’s way, makes accessibility harder, locks up perfectly good hardware, and turns a purchase into an indefinite rental.

Calibre, DeDRM, KFX Input, and a disposable VM gave me my library back. I can now put those EPUBs on the Xteink, a Kobo, a tablet, my phone, or a reader that doesn’t even exist yet. I can switch operating systems and turn off the internet. The files stay with me.

If it isn’t on your machine, it isn’t yours.

The next day, goodalexander reproduced the same behavior on a disposable devnet: an rsync --delete-delay against / swept away /var, /etc and his validators’ state, corrupting the filesystem to the point of needing an OS reinstall. His conclusion: “you can’t use this thing in prod”.

Cases like these pop up every so often since coding agents went mainstream. And every time, the discussion degenerates into the same two extremes: on one side “see? AI is dangerous, never let it run anything on its own”, on the other “whoever lets an agent loose on their system deserves what they get”. I find both sides lazy. Let me tell you my actual experience and what I do about it.

Five months of marathoning, zero accidents

Since February I’ve been marathoning Claude Code, Codex and OpenCode. Almost every day, almost all the time, in very long sessions that sometimes span multiple days. Blowing past the limits of my Anthropic Max plan and my OpenAI Plus plan with room to spare. The concrete result so far: almost 40 new repositories on my GitHub and more than 500 thousand lines of code produced.

And for the past two months I’ve been running everything exclusively in “YOLO mode”. For those unfamiliar with the jargon: by default, these agents ask for permission before each command they’re about to execute in your terminal. YOLO mode turns that safety off. In Claude Code it’s the --dangerously-skip-permissions flag, in Codex it’s --yolo, in OpenCode you configure the permissions to approve everything. The agent runs whatever it wants, whenever it wants, without asking me anything.

In five months under this regime, not once has an agent deleted a file I didn’t ask for. Not once has it run a destructive command out of scope. The frustrations I have with agents are of a different nature (stubbornness, lazy solutions, context getting lost), annoying things but miles away from an rm -rf in my home.

So when I see these reports, my first hypothesis always goes to the same place: the prompt. For an agent to “go berserk” like that, in general the user wrote very bad instructions. Vague, ambiguous, asking for cleanup without defining what counts as trash, telling the agent to “fix it” without giving context. Lack of communication skills combined with lack of technical knowledge to notice the request was poorly formulated. The agent executes the instruction you wrote, with all the ambiguities you left in it.

That said, I do NOT trust LLMs completely. Nobody should.

The difference is that my distrust turns into engineering instead of turning into a tweet.

Layer by layer, this is how I sleep well at night.

Layer 1: sandboxing with ai-jail

When I genuinely suspect a session might do something stupid (I’m about to touch system scripts, I’m going to let the agent run for hours unsupervised on code that touches the filesystem), I open the harness inside ai-jail, a tool of mine, open source, written in Rust.

ai-jail is a sandbox wrapper for coding agents. On Linux it uses bwrap (bubblewrap, the same foundation as Flatpak), on macOS it uses sandbox-exec. The idea: the process sees a filesystem that looks normal, but only the current project directory is real and persistent. The rest of $HOME, the sibling directories, /tmp, everything lives in tmpfs and evaporates when the session ends. The agent’s state (~/.claude, ~/.codex etc) is mounted for real so it stays logged in and configured, but ~/.ssh, ~/.gnupg, ~/.aws and browser profiles never enter the jail.

Installing is one command:

# Arch Linux
yay -S ai-jail-bin

# macOS / Linux via Homebrew
brew tap akitaonrails/tap && brew install ai-jail

# or straight from crates.io
cargo install ai-jail

And using it is even simpler:

cd ~/Projects/my-app
ai-jail claude       # Claude Code in the jail
ai-jail codex        # Codex in the jail
ai-jail --dry-run claude  # shows the policy without running

If the agent decides to run rm -rf ~ in there, it deletes a tmpfs that was going to vanish anyway. My real home was never even mounted. For anyone starting out with YOLO mode without battle scars yet, it’s the cheapest way to make mistakes without consequences.

Layer 2: BTRFS and snapshots (the real safety net)

Now the main point of this post. I’ve been advocating for YEARS that Linux users should use BTRFS instead of EXT4. I made an entire Akitando episode about it in 2023, long before coding agents existed to take the blame. The argument was the same: your data needs to survive human error. An LLM running rm -rf in the wrong place is just a new category of human error, outsourced.

Why is BTRFS superior for desktops? Copy-on-write (CoW): the filesystem never overwrites a block in place, it writes the new version to another block and updates the pointers. Checksums for data and metadata, so silent bitrot gets detected instead of quietly corrupting your file. Transparent compression. Subvolumes, which work like flexible partitions without fixed sizes. And the most valuable consequence of CoW: lightweight snapshots.

A BTRFS snapshot is instantaneous and costs close to zero space the moment it’s created. It freezes the pointers to the current blocks instead of copying data. Only what diverges from that point on (new or modified blocks) starts taking up extra space. In practice you can take dozens of snapshots a day without feeling it. At this very moment my machine keeps 23 system snapshots, and the cost is the difference between them, a few gigs.

Scheduling snapshots

The two classic tools are Timeshift and snapper. Timeshift is the friendlier one: graphical interface, you set “how many snapshots to keep” per hour/day/week/month and it handles the rest. It expects the @ (root) and @home subvolume layout, which is the default for most distros that install on BTRFS today. snapper (born at openSUSE) is more granular: per-subvolume configs, automatic timeline, package manager integration to snapshot before and after every install.

# Timeshift: manual snapshot before a suspicious session
sudo timeshift --create --comments "antes de soltar o agente"

# snapper: create a config for /home and enable the timeline
sudo snapper -c home create-config /home
sudo systemctl enable --now snapper-timeline.timer

My setup runs Timeshift on a schedule, plus grub-btrfsd on standby (I’ll explain why in a moment).

Recovering deleted files

Here’s the magic that everyone who lost files to an agent wishes they had. A BTRFS snapshot is browsable like a regular directory. The agent deleted your project at 3pm? The 2pm snapshot still has everything:

# with Timeshift, snapshots are mounted under /run/timeshift
sudo ls /run/timeshift/backup/timeshift-btrfs/snapshots/

# found the snapshot from before the damage? copy it back and done
sudo cp -a /run/timeshift/backup/timeshift-btrfs/snapshots/2026-07-11_14-00-01/@home/akitaonrails/Projects/meu-app ~/Projects/

With snapper it’s the same thing via .snapshots inside the subvolume, and it also has snapper undochange to revert specific changes between two snapshots. Recovery in seconds, no drama, no forensic tooling.

What if the system won’t even boot?

goodalexander’s case went beyond losing files: the agent swept /var and /etc and the system became unusable, requiring an OS reinstall. With BTRFS not even that would be necessary. This is the scenario grub-btrfs exists for: a daemon that watches your snapshots and regenerates the GRUB menu with a boot entry for each one.

System won’t come up anymore? Reboot, hold the GRUB menu, pick “snapshots” and boot straight into yesterday’s snapshot, read-only. From inside it, run timeshift --restore, pick the good snapshot, reboot again. Entire system back to its pre-disaster state, in less time than it would take you to find the installation USB stick. In the absolute worst case (GRUB itself went down with the ship), any live USB mounts the BTRFS partition, points the default subvolume back to the good snapshot and that’s that.

What about macOS? And Windows?

Mac users have the direct equivalent: APFS, the default filesystem since High Sierra (2017), is also copy-on-write and also has lightweight snapshots. Time Machine is what manages them: with it enabled, macOS takes automatic local snapshots every hour and keeps the last 24 hours on the disk itself, independent of the external backup. And you can create one by hand before letting the agent loose:

# create an APFS local snapshot right now
tmutil localsnapshot

# list the existing snapshots
tmutil listlocalsnapshots /

To recover, open Time Machine itself (“Browse Time Machine Backups” in the menu bar): it browses the local snapshots even without any external disk plugged in, and you restore the file from there. If you prefer the terminal, mount the snapshot read-only with mount_apfs -s and copy it back. In other words, Matt Shumer’s case from the beginning of this post had a native solution on his own Mac: a snapshot from one hour earlier would have brought almost everything back.

On Windows the story is weaker. NTFS has the Volume Shadow Copy service (VSS), which feeds restore points and the “Previous Versions” tab, but it’s aimed at system files and rarely happens to be covering your data at the moment you need it. Nothing with the lightness and predictability of a scheduled CoW snapshot. If you develop on Windows, treat external backup as mandatory, no discussion (and the restic from the next section runs fine on Windows).

Layer 3: external backups with restic

Snapshots live on the same disk as the data. If the NVMe dies, they die together. Which is the other thing I’ve been repeating for years: everybody needs external backups, whether to an external hard drive or a NAS. I use a Synology on the local network, mounted via NFS, and documented the complete setup here.

On top of it runs restic, which in my opinion is the best backup tool for this purpose: deduplication, encryption, incremental snapshots and simple restores. My actual configuration is a script fired by a systemd timer every day at 3am. It checks the NAS is mounted, backs up /home and /mnt/data/Projects skipping caches (node_modules, Rust’s target/, __pycache__ and friends, all in an excludes file), and then applies the retention policy: 7 daily, 4 weekly, 6 monthly, 1 yearly.

Reproducing the essentials looks like this:

# once: create the repository on the NAS
restic init --repo /mnt/nas/desktop-backup

# every day (via cron or a systemd timer):
restic -r /mnt/nas/desktop-backup backup ~ /mnt/data/Projects \
    --exclude-file=~/.config/restic/excludes --exclude-caches

restic -r /mnt/nas/desktop-backup forget \
    --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --keep-yearly 1 --prune

And if the truly bad day arrives (destructive agent, dead disk, ransomware, the cause doesn’t matter):

# list the available snapshots
restic -r /mnt/nas/desktop-backup snapshots

# restore a specific file or directory
restic -r /mnt/nas/desktop-backup restore latest \
    --target /tmp/recuperado --include /home/akitaonrails/Projects/meu-app

# or mount the whole repository as a filesystem and browse at your leisure
restic -r /mnt/nas/desktop-backup mount /mnt/restic

Notice the layered architecture: the BTRFS snapshot solves the accident from minutes ago in seconds. restic solves the disaster from days ago, including dead hardware. An AI agent would have to be very dedicated to punch through both layers, and even then there would still be yesterday’s copy on the NAS, which it can’t even reach.

What about companies? Professionalism

Everything above is about my personal machine. Now let’s raise the bar: corporate environments.

Rule number one is simple and non-negotiable: no AI agent should ever have access to production. Ever. No direct SSH into production servers, no production database credentials, no admin tokens for production services. If your agent can run a DELETE against the database that serves customers, the problem stopped being the AI a long time ago: any intern with the same access is the same time bomb, just with a slower typing speed.

Agents in staging, on the other hand, I advocate for. That’s what staging is for. But even in staging there’s a right way: you resist the temptation of prompting one-shot actions (“run an apt install over there”, “do a systemctl restart”, “connect to postgres and execute this”). A loose command executed by hand is knowledge that evaporates. Your infrastructure needs to be automated and reproducible: Ansible, Kubernetes, Terraform, whatever makes sense for your stack. The agent’s role becomes writing and refining those recipes, which live versioned in git.

A small example, right here from my desk

I have a gaming environment on my main PC, but I refuse to mix emulator packages with my software development environment. So I isolated gaming inside a distrobox. For those who don’t know it: distrobox creates containers (via podman or docker) that integrate with your desktop as if they were native. They share your home (or a dedicated home), GPU access, Wayland, audio. In practice it’s a complete distro inside your distro, with a separate package space.

distrobox-gaming is a repository of Ansible playbooks that builds an Arch box called gaming from scratch: ES-DE as the frontend, standalone emulators (shadPS4, RPCS3, PCSX2, Dolphin, DuckStation, Eden and a dozen more), RetroArch cores, per-game optimized configs, launcher shortcuts on the host. And distrobox-llm does the same for a local LLM work box: CUDA, cuDNN, Ollama, whisper.cpp and LM Studio isolated in a box with --nvidia, while the host only carries the driver. A CUDA major version upgrade no longer rattles my development environment.

Who wrote the overwhelming majority of these playbooks? The agents. Instead of asking “install RPCS3 for me”, my prompt is “write the idempotent Ansible recipe that installs and configures RPCS3, run it, verify the result and fix it until it passes”. Here’s a real excerpt from the role that creates the LLM box:

- name: Check if distrobox exists
  ansible.builtin.shell: "distrobox list 2>/dev/null | awk '{print $3}' | grep -qx {{ dl_box_name }}"
  register: _box_exists
  changed_when: false
  failed_when: false

- name: Create distrobox
  ansible.builtin.command: >-
    distrobox create
    --name {{ dl_box_name }}
    --image {{ dl_image }}
    --home {{ dl_box_home }}
    {{ '--nvidia' if dl_nvidia_enabled else '' }}
  when: _box_exists.rc != 0

Idempotent: it checks whether the box exists before creating it, running it twice gives the same result. If my PC catches fire tomorrow, I recreate both environments in full with one ansible-playbook site.yml in each repo. The knowledge lives in the recipe, versioned, auditable. Zero commands lost in shell history.

Scaling the same principle to production

At a company, swap distrobox for Kubernetes and the principle is identical. The agent writes the manifest, tests it on the staging cluster, you promote it to production manually:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: minha-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: minha-api
  template:
    metadata:
      labels:
        app: minha-api
    spec:
      containers:
        - name: minha-api
          image: registry.exemplo.com/minha-api:v42
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
          resources:
            limits:
              memory: 512Mi

The professional flow: the agent produces and adjusts the manifest (or the Helm chart, or the Terraform module), applies it to staging (kubectl apply -n staging), runs the tests, watches for side effects, bugs, unintended behavior, and iterates until you’re satisfied. Then YOU, a human, promote it to production (via a reviewed PR in a GitOps flow, or a manual and deliberate kubectl apply), monitoring every step. The agent did all the heavy lifting: the artifacts, the procedures, the checklists. You can even ask it to generate the audit documentation at the end. But the finger that presses the production button has a name on a payroll.

Even the “simple” maintenance SQL

A case that looks harmless and bothers me a lot: maintenance SQL commands in production. Plenty of people have always run those by hand, straight into the production client, and are now asking the agent to write AND execute them. That is unacceptable, with or without AI. Your system should have a migrations workflow: the change becomes versioned code, runs in staging first, gets validated, and only then executes in production inside the normal deploy process.

And “production”, by definition, means two things many companies pretend to have. First: real backups, tested, with rehearsed restores.

Never run anything in production that you can’t recover from.

Second: monitoring. If you have no visibility into the state of the system, you don’t have production, you have a server plugged into an outlet and a prayer.

Monitoring: even on my humble PC

And monitoring is so much a part of the discipline that I apply it even to my personal machine. I recently developed a system health widget for my clock-tui (my maintained fork of the original clock-tui, with command widgets in clock mode). It sits open in a corner showing this:

One verdict line (“all systems healthy”) and the state of everything this post described: the restic backup ran 19 hours ago and the next one is at 03:03; Timeshift keeps 23 snapshots; the development cache cleanup ran 32 hours ago; the BTRFS scrub is up to date on each filesystem, with a recent trim, healthy allocation and zero I/O errors; plus zombies, load, memory and each disk’s usage. The widget (tclock-system-health) ships with the repository and the AUR package, and detects most of this on its own.

Behind it are the automated maintenance jobs, all systemd timers: the daily restic at 3am; dev-cache-clean (on the 1st and the 15th) sweeping orphaned Rust target/ dirs and the yay cache, which together give back dozens of gigs; the monthly per-filesystem btrfs-scrub validating checksums; the weekly fstrim. None of it requires me to remember anything. When one of them is late or fails, the widget changes color right in my face.

This is the home-grade level, proportional to a personal PC. At a company you go full-blown: Prometheus collecting metrics, Grafana drawing the dashboards, Alertmanager waking up the on-call person at 3am. The scale changes, the principle is the same: the state of the system visible at all times, and someone (or something) getting notified when it goes off the rails.

Conclusion: discipline

Go back to the two tweets at the beginning. In both cases, a single wrong command caused real loss, and the public discussion became “who do I trust more”. The right question is a different one: why does a single wrong command, from any origin, have the power to cause unrecoverable loss on your machine?

Being a professional is being disciplined, and discipline starts at home. My personal PC operates with the same rigor I would demand from a corporate environment: a filesystem with lightweight scheduled snapshots, recoverable boot via GRUB, daily external backups with a retention policy, a sandbox for suspicious sessions, isolated environments rebuildable from versioned recipes, automated maintenance and a monitor that warns me when any piece falls behind. Monthly maintenance cost of all this: close to zero. It was set up once (in good part by the agents themselves) and runs on its own.

With these layers, the question in the title almost loses its meaning. The agent deleted a directory? Snapshot from an hour ago, two minutes of recovery. Destroyed the system? Boot into a snapshot via GRUB, restore, ten minutes. Fried the disk along with it? restic on the NAS, yesterday’s backup. In five months of YOLO mode I never needed any of these nets for an AI accident. But I sleep well precisely because they’re there.

An LLM running loose on your system is just the most recent stress test of your discipline. If an out-of-control agent can cause permanent loss to your data, the vulnerability was always there, waiting for a mistake of yours, a bad disk or a ransomware. The agent was just the messenger that arrived first.

Email has always been the most annoying piece of this puzzle. My old workflow was to open Thunderbird, let it pull everything from Gmail over IMAP, and back up the ~/.thunderbird directory. Works, but it’s manual, tedious (remembering to open Thunderbird every so often), and locked to Thunderbird itself: if I want to read those emails in another client, they’re sitting in some proprietary internal format.

After a while it clicked: Linux has had native email support forever. The modern protocols (IMAP, SMTP) are open standards. And the local storage format, Maildir, has been around and working well since the 90s. Thunderbird isn’t needed for any of this.

What is Maildir

Maildir was created by Daniel J. Bernstein (djb) as part of qmail in 1995. The idea was to fix the file-locking pain of the mbox format, where the entire mailbox is a single file and two processes writing at the same time could corrupt the whole thing. In Maildir, each message becomes a separate file inside a three-subdirectory structure:

• tmp/ — where the email is written while being delivered
• new/ — newly arrived messages, not yet seen
• cur/ — messages the client has already seen, with flags encoded in the filename (:2,S = seen, :2,F = flagged, etc.)

Delivery is atomic: the MTA writes the message into tmp/, then does a rename() into new/. rename() on a POSIX filesystem is atomic, so there’s no corruption window. No locking, no corruption, and every email is just a plain-text file you can read with cat, index with grep, or version with git if you feel like it.

Thirty years later, it’s still the default local format for most Linux email clients. And it’s the format I’ll use to mirror Gmail.

The tool: mbsync (isync)

The bridge between Gmail’s IMAP and the local Maildir is mbsync, the binary in the isync package. It reads a simple config (~/.config/isyncrc), connects to Gmail’s IMAP, and copies the messages to disk. What makes it different from offlineimap (the better-known alternative) is that mbsync is written in C, it’s significantly faster on large mailboxes, and it has fewer weird edge cases with Gmail.

The config is straightforward. Here’s a sketch of what my ~/.config/isyncrc looks like (no password, obviously):

IMAPAccount my-account
Host imap.gmail.com
User you@example.com
PassCmd "/home/akitaonrails/.config/mbsync/bin/mbsync-password you@example.com"
TLSType IMAPS
CertificateFile /etc/ssl/certs/ca-certificates.crt
AuthMechs LOGIN

IMAPStore my-remote
Account my-account

MaildirStore my-local
SubFolders Verbatim
Path ~/.local/share/mail/mbsync/you.example.com/
Inbox ~/.local/share/mail/mbsync/you.example.com/INBOX

Channel my-backup
Far :my-remote:
Near :my-local:
Patterns * !"[Gmail]/Trash" !"[Gmail]/Spam" !"[Gmail]/Drafts"
Create Near
Sync Pull New Flags
Expunge None
Remove None

Three things worth flagging in that config:

1. Sync Pull: only pulls from the server to local. Nothing I do in the Maildir propagates back to Gmail. That’s accident protection (if I delete a file locally, it doesn’t disappear from Gmail).
2. Expunge None + Remove None: if a message disappears from Gmail (someone deleted it or Google decided to move it nowhere), it stays in my Maildir. The backup accumulates history, it doesn’t mirror the current state.
3. PassCmd instead of a plaintext password: the password goes through a script that does secret-tool lookup against the desktop keyring. No password sitting in a config file on disk.

The password block

The mbsync-password script is a tiny wrapper that isolates the secret:

#!/usr/bin/env bash
set -euo pipefail
account="${1:?usage: mbsync-password ACCOUNT}"
password="$(secret-tool lookup service mbsync account "$account" 2>/dev/null || true)"
[ -z "$password" ] && { echo "missing keyring secret for $account" >&2; exit 1; }
printf '%s' "$password"

The password gets stored once with secret-tool store --label="mbsync gmail" service mbsync account you@example.com. Modern Gmail requires a specific app password (16 characters, generated at myaccount.google.com/apppasswords) because it no longer accepts plain basic auth — you don’t paste your Google password there, you paste a key generated exclusively for the app, revocable any time without touching the rest of your account.

Automation with a systemd user timer

To avoid running mbsync by hand, I set up two systemd user units. First the service, which is just a call to the wrapper script:

# ~/.config/systemd/user/mbsync-backup.service
[Unit]
Description=Mirror Gmail accounts locally with mbsync
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
ExecStart=%h/.config/mbsync/bin/mbsync-run

And the timer that fires the service periodically:

# ~/.config/systemd/user/mbsync-backup.timer
[Unit]
Description=Run mbsync periodically

[Timer]
OnBootSec=10m
OnUnitActiveSec=30m
Persistent=true
Unit=mbsync-backup.service

[Install]
WantedBy=timers.target

OnBootSec=10m waits 10 minutes after boot before the first run (avoids fighting with network-online early in the session). OnUnitActiveSec=30m refires 30 minutes after each run completes. Persistent=true makes the timer catch up: if the machine was off when it would’ve fired, it runs at the next boot instead of skipping. The wrapper itself grabs flock so two mbsync runs never overlap, so if a sync ever takes longer than 30 minutes the next one just waits. Everything logs to ~/.local/state/mbsync/mbsync.log.

To manage the timer, the basic commands:

# Re-read the unit files after creating/editing
systemctl --user daemon-reload

# Start now + enable for the next boot
systemctl --user enable --now mbsync-backup.timer

# Check whether the timer is active and when it fires next
systemctl --user status mbsync-backup.timer
systemctl --user list-timers mbsync-backup.timer

# Trigger an immediate sync without waiting for the interval
systemctl --user start mbsync-backup.service

# Tail the logs in real time
journalctl --user -u mbsync-backup.service -f

# Inspect the status of the last service run
systemctl --user status mbsync-backup.service

# To shut the whole thing down
systemctl --user disable --now mbsync-backup.timer

For the timer to keep running when you’re not logged in (handy on a desktop that stays up 24/7), enable linger once: sudo loginctl enable-linger $USER. Without that, user units only run while you have an active session.

Enable it, check the status once to make sure it’s green, and forget it exists.

Backing up the Maildir with restic

Now that Gmail is mirrored under ~/.local/share/mail/, it rides into the same backup pipeline as the rest of my home directory. I use restic to push encrypted snapshots to the NAS. Restic is incremental via content-addressable storage, so adding 100MB of email to today’s snapshot costs basically the delta, not the full 100MB every time. For the off-site copy, my Synology NAS already has the Glacier Backup package built in, which handles pushing encrypted snapshots to AWS Glacier without me having to configure anything beyond the bucket and the key. The Maildir rides along on the same plan, so the off-site backup is essentially free in terms of extra configuration.

Maildir is friendly to incremental backup by construction: every email is an immutable file (the filename doesn’t even change after download, except for the seen/flagged flag bit). Restic dedups perfectly. A 5GB mailbox snapshot has near-zero marginal cost after the first one.

The open-format upside: many clients

Here’s the part that sealed it for me. Maildir is an open standard, so any client that speaks Maildir reads the same backup. I can:

• Open it in mutt or neomutt when I want something quick in the terminal.
• Open it in aerc when I want a more modern TUI.
• Point Thunderbird or Claws Mail at ~/.local/share/mail/... when I want a GUI.
• Migrate to a different client down the road without an export/import dance.

The emails are mine. They live on my disk. In a plain text format, readable by anything.

Bonus: scripts and bots read directly

A less obvious consequence: if I want to build a filter, a bot, a classifier, or anything that touches the emails, I don’t need to go through Gmail OAuth, create a Google Cloud project, manage refresh tokens, deal with quotas — none of it. I just read the Maildir.

Python has native Maildir support in the standard library’s mailbox module. A trivial example to list subjects from the inbox:

import mailbox
from email.header import decode_header

inbox = mailbox.Maildir("/home/akitaonrails/.local/share/mail/mbsync/you.example.com/INBOX")

for key, msg in inbox.iteritems():
    subject = msg.get("Subject", "")
    sender = msg.get("From", "")
    # decode_header handles Subject in UTF-8 / latin-1 / etc
    decoded = "".join(
        part.decode(enc or "utf-8") if isinstance(part, bytes) else part
        for part, enc in decode_header(subject)
    )
    print(f"{sender[:40]:<40}  {decoded[:60]}")

Runs in milliseconds, consumes zero API quota, has no rate limit, never needs a credential refresh. It’s just file reads. If you want to build something more serious (attachment parsing, LLM-based classification, automated rules), the same principle holds.

Summary

The combination that finally let me relax about this:

• mbsync (isync) pulls from Gmail’s IMAP into a local Maildir, automatically via a systemd timer.
• Maildir keeps the open format, one file per email, pull-only protects against accidental deletion.
• restic snapshots the Maildir to the NAS with ridiculous deduplication, and Synology’s Glacier Backup carries that off-site as an encrypted copy.
• Free choice of mail client — mutt, neomutt, aerc, Thunderbird, Claws, anything reads it.
• Light automation — bot or script reads straight off disk, no OAuth, no API.

Gmail becomes just a convenient web interface for reading and replying. The real ownership of the emails sits on my disk, replicated to the NAS, replicated to an encrypted off-site backup. If Google shuts my account down tomorrow (it’s happened to people), I lose the interface but keep the entire history. For me, this is the definitive way to handle it. Recommended.

In my case, I spent 10 years using Manjaro GNOME as my daily driver. Manjaro is an Arch derivative. The reason was the easier install and the pre-made configs. But these days, with archinstall, I don’t think it’s worth it anymore. I know there are other derivatives, but it’s better to stick with the original.

For Ubuntu folks or those coming from Debian derivatives, Fedora, etc, there’s a small learning curve on the package manager: Pacman and Yay instead of Apt or Dnf. But once you understand the AUR, you’ll never want anything else. It has literally everything you need, without having to deal with third-party repositories.

I took a while to test Omarchy because I was already using Hyprland before Omarchy was created. DHH previously had Omakub, which was a set of scripts to customize Ubuntu into a good development environment. I liked it, but since I preferred Manjaro, I made a fork called Omakub-MJ. After that I decided to uninstall GNOME and GDM and install another set of scripts called ML4W, which makes Hyprland look great as well. So Omarchy wasn’t that novel, for me.

The thing is, I’d been using the same Manjaro for years. It had a ton of old configs and scripts all mixed up. My Manjaro GNOME no longer even had GNOME. My Hyprland was a Frankenstein. The only barrier was my QEMU/Libvirt setup to run Windows with GPU passthrough, which is what I used to edit videos back when I was still posting on my YouTube channel.

But since I stopped editing videos, that Windows isn’t as important anymore. I had another Windows VM with GPU passthrough for games. But currently I have a separate mini-PC with Windows that I use just for games. I also had another VM with another Linux just for my AI experiments, but since then I migrated to running natively while isolating everything with Docker. In other words, my VMs are no longer important. Check my previous blog posts to learn about AI in Docker.

With the release of Omarchy 2.0, it looks like a good time to reinstall Arch from scratch with Omarchy on top and start over clean and beautiful.

1 - My Setup

Every case will be different. This post’s main purpose is to serve as my personal reference for the future, in case I need to reinstall everything again.

No, I’m not going to use NixOS. I really don’t think it’s worth it. It only pays off if you’re a computer lab admin or server infrastructure admin and your day-to-day involves rebuilding identical setups reliably. For a normal user who only reinstalls every 2 or 3 years, the learning curve and maintenance work aren’t even remotely worth it. It’s a waste of time. I’d spend days learning now, finish the setup, go 2 years and I’d already have forgotten everything. Arch Linux by itself is stable enough that I don’t need this.

My setup is still the same from the days I was editing videos for the channel. It’s a Ryzen 9 7850X3D with 16 cores, 32 threads, 5GHz peak. It’s still an excellent CPU with power to spare for my use cases, which is why I don’t see myself wanting to upgrade anytime soon. It’s rare for me to saturate all those cores at those speeds at the same time. Better yet: it has an integrated AMD GPU which makes installing Linux a thousand times easier than if I had NVIDIA.

But I also have a separate discrete GPU, the NVIDIA RTX 4090 (no, I don’t need a 5090, it has the same amount of VRAM). It’ll stay for secondary tasks and not for carrying my graphical environment. Things like Cycles rendering in Blender, or running LLMs with ollama and the like. Before I used GPU passthrough to a VM in QEMU/Libvirt to run Windows in parallel with Linux. Both to edit videos and to game.

Since I no longer edit videos, I don’t need this complicated setup, so I won’t bother installing Libvirt back. If I need Windows, I’ll leave it in dual boot. If I need to edit video, these days we have DaVinci Resolve Studio which runs natively on Linux (a thousand times better than Adobe Premiere). For games I have a separate mini PC with an RTX 4060 hooked up to my TV. The best thing is to have a totally separate PC for games.

So this setup is meant to be pure Linux, with access to my Synology DS 1821+ NAS with 100TB. The only extra thing I’ll need is to configure NFS mounts, as I’ve already detailed in this other post.

One detail is that I have 4 NVMEs in this PC. 2TB for my old Linux. 2 TB for dual boot/Windows VM. 2 TB just as a Steam drive and 1 TB for random stuff. Since I’m wiping the Steam VM, I also no longer need to dedicate an entire NVME just for that. Instead, I’ll make the new Linux’s “/home” directory this entire 2 TB.

This will let me stop worrying about Docker filling my drive and freezing my entire system from lack of space. Last time I had “solved” this by making a remote drive with iSCSI on my NAS (which has infinite space). Since removing my Steam VM will give me 2TB back, I’ll solve the problem locally and with fewer configurations.

RAM I have in spades, 96GB of DDR5. With that, I also don’t need to worry about a swap partition. Swap only matters if you have less than 32GB of RAM, if you’re running lots of Docker and keep hitting the max RAM consumption very fast. Or if you’re on a laptop and want the option of hibernation (being able to record RAM to disk, fully power off the machine, and come back later exactly where you left off — which is different from suspend, which still uses power).

If you want to hibernate, you need a bit more than the size of your RAM. So if you have 32GB, it’s good to leave a margin, around 34GB or more of swap. Since my PC is a desktop at home, I don’t need hibernation, just enable ZRAM for the rare cases where there’s a spike over 96GB (which is super rare). Plus, it’s easier for me to just bump up to 128GB of RAM if I really need it. Swap to compensate for RAM only pays off on laptops that don’t have room for more real RAM.

2 - Why Omarchy?

As I said, I’ve been an Arch user for a decade and I don’t see any package repository bigger and more complete than the AUR. The only other repository that actually competes is NixOS’s. But since Nix is ruled out for my use case, the AUR remains unbeatable.

Arch has Pacman as its package manager. To access the AUR you need a second manager. There are several like the old Pacaur, Paru, and many others. But the best, in my opinion, is Yay. It has exactly the same syntax as Pacman and replaces Pacman. In practice, we just use Yay for everything.

To search packages by name just do yay -Ss spotify.

To remove packages just do yay -Rns spotify.

To update all system packages just do yay -Syu

To remove orphan packages that pile up over time just do sudo pacman -Rns $(pacman -Qdtq)

And those are the commands I use 99% of the time.

I’ve already used every type of desktop manager out there: GNOME (which behaves more like a Mac), KDE (which behaves more like Windows for power users), XFCE (which is GTK-based and lighter), Cinnamon (also GTK-based), i3 (which is a tiling window system), Sway (like i3, also tiling but if I’m not mistaken was designed with Wayland in mind).

In practice, everyone uses apps in two ways: a full-screen window or 2 vertical windows splitting the monitor in the middle (editor on one side, browser on the other), or 3 windows (a vertical editor taking half the monitor, a browser taking 1/4, and a terminal taking another 1/4). Then we organize more open apps into virtual desktops/workspaces. That’s the easiest way to work.

This means being able to drag floating windows is a waste of time. I prefer that windows already open sized into one of these layouts I mentioned above. You can configure GNOME or KDE to do that. But it smells like a kludge. Instead it really is better to use i3 or Sway.

Setting controversies aside (yes, I’m a bit wary of Wayland and I like that XLibre exists). But it’s a fact that X11 is behind the times and is a very old and complicated architecture. Wayland really does have a better “appeal,” despite having taken a while to round off basic things like screen sharing (OBS, Zoom). But I’ve been using Wayland for some years now without headaches. These days I’d say it’s pretty stable. I’ve been using GNOME on Wayland for a long time. I don’t stop using things because I’m wary of them: with technology I’ll always be absolutely promiscuous and use everything.

Knowing I’ll use Wayland and knowing I want automatic window layouts, and don’t need floating windows, the best is a desktop manager with a tiling windows system (tiles are “azulejos” and we’re talking about tiles being side by side and never one on top of another), which organizes windows side by side in a smart and automatic way.

In this case we only have 2 options: Sway or Hyprland (either one is good). Hyprland, installed from scratch, is completely bare — it has absolutely nothing. We need to configure everything through files in .config/hypr. It’s simple enough, but it’s work.

The only good option for having a pre-configured and well-done Hyprland was ML4W, which I was already using. But now we have Omarchy, which installs and pre-configures everything to work smoothly on Hyprland. This is the best option we have today.

Besides configuring Hyprland to work with Waybar, Rofi, and other components, it has its own system for switching themes and styles to leave the system with the “look and feel” you want (and all the ones that come with it are super pretty), but it also installs things I’ve been using for a long time like Neovim with the LazyVim package (in my opinion, the easiest way to make Neovim not fall short compared to VS Code), Mise (the best way to organize development environments to have multiple versions of your favorite programming language).

If you’re installing a distro from scratch, the only two things you need to have are Mise and LazyVim. Plus Docker and LazyDocker. That’s enough to start a productive development environment. And Omarchy already brings all of that pre-configured. No one “needs” Omarchy, but it already brings everything integrated and beautiful. So why bother doing everything from scratch to end up almost the same?

Also, this is Linux. Don’t like one of the components? Just swap it out.

3 - Back Up

The vast majority of everything I have already lives on the remote server, my 100TB Synology NAS. That already helps a lot. Even so, there’s still a good amount of stuff locally on my PC, so the first thing to do is copy everything to the NAS. If you don’t have a NAS, then an external HD. Any server will do; the important thing is to guarantee you won’t lose anything before installing a new OS from scratch.

Don’t forget directories like .config or .local. It’s better to do an rsync -avz --progress .config /mnt/nas/backup/, for example, and copy everything. Even what you think you won’t need. Better to copy more than to be missing stuff. As a rule, the ideal is to copy everything in your $HOME to an external drive or NAS.

Just to make sure I won’t forget anything important, I’m also going to record a list of all the packages I have installed on my old system:

yay -Q > pacotes.txt
flatpak list > flatpak.txt

Also making sure I won’t lose old configs of services I made in systemd, so also copy everything from /etc/systemd/system. It’s also worth recording things like /etc/fstab. Also don’t forget to back up your SSH keys with tar cvfz ssh.tgz ~/.ssh!!

If, like me, you were using QEMU/Libvirt for VMs, it’s good to save the configs of the virtual machines you used. Usually the tool to manage VMs is virsh, so I just did:

sudo virsh list --all
sudo virsh dumpxml <vmname> > <vmname>.xml

If I want to recreate the machines in the future, it will all be in the backup. In my case I keep all virtual drives on my NAS, but if yours are local, you also need to save those files that live in /var/lib/libvirt/images/

rsync -avh --progress /var/lib/libvirt/images/ /backup/libvirt-images/

4 - Is Installing Arch Hard?

Arch, less than Gentoo, has always had a reputation for being the most brutal Linux distro for a beginner to install. That’s not a lie, but the truth is Arch is simultaneously the hardest distro and also the easiest.

Download the official ISO and burn it to a USB drive (on Windows, using Rufus or Balena Etcher). Now press F12 (depends on your BIOS) or configure the BIOS directly to boot from the USB drive. Doing that loads the Arch live environment, which is literally dropping into a command line. With no further help.

From this point you have to follow the Installation Wiki from the excellent ArchWiki and learn how to install a Linux from absolute scratch using only the command line, step by step, through trial and error (more erring than hitting the mark), and in the end you’ll have a much better feel for how a Linux distro actually works.

I didn’t make an Arch video but I did make videos of Slackware to explain how a package manager works. I also made a Gentoo video to show how a distro that compiles everything from scratch works. And I made a video about how the Linux Boot Works to understand where the kernel lives and how it’s structured. But I didn’t make an Arch video because I always left it as an exercise for beginners.

The manual way is viable, but if you’re used to a graphical installer, like Calamaris, which most more popular distros like Ubuntu or Fedora have, you’ll be totally lost. But here comes the bonus: you don’t need to go through all of that to have Arch Linux on your machine.

For that there’s archinstall.

When you boot the Arch USB and drop into the command line, just type archinstall and you’re done — just follow the step by step it provides (which I consider even simpler than Calamaris) and it will install and configure everything automatically for you, the same way installing Fedora or any other distro is. Seriously, it shouldn’t take more than 15 minutes.

archinstall looks like this:

Just go into each of the items on the left and fill in your choices. Configure things like keyboard layout (I normally choose ‘us-acentos’). Edit the hostname of your machine. Choose “NetworkManager” in network configuration (it will configure both ethernet and wifi for you), add Bluetooth and Audio support (choose “Pipewire”). Choose the “desktop” type instead of “minimal” to already get the main packages needed for a desktop. Choose the timezone (“America/Sao_Paulo” in my case).

I also recommend choosing the Hyprland option, I think under Applications or Additional Packages. It will install SDDM to give you a graphical login.

Only two options need more attention: for bootloader, many will be tempted to use Limine, which is more modern and lightweight. But I still recommend using GRUB, I’ll explain why in a moment.

The second important option is “Disk Configuration”. Don’t worry, unlike fdisk, it’s super intuitive. Just choose the correct drive where you want to install (if you have dual boot, careful not to wipe your Windows). Then let it do the automatic partition layout (only edit manually if you have some very special need). Most importantly, choose BTRFS instead of the traditional ext4.

Every Linux always formats its Linux partitions with ext4, which is the oldest, most stable, and best-documented file system. These days I recommend BTRFS; it’s much more modern, safer (has better corruption protection systems), uses less space (learn about CoW - Copy on Write), and most importantly: it has cheap snapshot support (think of Apple Time Machine, only better). It has most of the features of a file system dedicated to a file server but is much simpler and more intuitive to use than the monster that is ZFS (I don’t recommend it for normal users).

It has some maintenance needs that are worth reading about in the Wiki to learn. But overall it’s plug and play, install and forget. In the next sections I’ll show some things you need to know, but trust me: choose BTRFS.

I know Omarchy 2.0 has an option for a pre-configured ISO which I imagine is normal Arch with archinstall and Omarchy on top all at once. But I don’t know if they allow this choice, which is why I preferred going straight to pure Arch first. In the end the effect is the same. Comment below later if the Omarchy ISO also has the same options. Just make sure in the end you have BTRFS. Future you will thank you.

5 - Separate HOME

As I explained before, I used one of my 4 NVMEs as the Linux drive. But now I’ve freed up another NVME and want to move my $HOME there and have 2 TB dedicated just to user space.

I won’t explain how to partition and format that other drive. It’s simple: just use fdisk or later install Gparted which is the graphical version of the parted tool that you can use from the command line. There are several options. The important thing is just to wipe my old Windows partition and also format it with BTRFS.

Each person will have different device and partition names than mine. To find out which are yours, use the lsblk command. See my example:

In my example nvme1n1 is the 1TB drive for random stuff. nvme2n1 is my new Arch Linux. nvme0n1 is the old Windows dual boot drive. nvme3n1 is the Steam drive I formatted with BTRFS to become my new HOME drive.

If you didn’t get it, HOME is the /home/akitaonrails directory. It will live on a separate drive. After archinstall finishes and I boot into the new Linux, I just do the following:

sudo mount -t btrfs /dev/nvme3n1p1 /mnt
sudo btrfs subvolume create /mnt/@home
rsync -aAXHv /home/ /mnt/@home/
sudo umount /mnt

This mounts the new partition, then we use the “btrfs” command to create a new sub-volume called “@home” (again, read the BTRFS Wiki to understand concepts like sub-volumes).

The commands above create the sub-volume and copy everything from the old HOME to the new. Now we need /home to be mounted on this new partition at boot. For this we need to know the UUID of that partition:

~ ❯ sudo blkid /dev/nvme3n1p1
/dev/nvme3n1p1: LABEL="HOME" UUID="84e9a1b7-a30e-...

The blkid command returns the unique UUID of each partition and, with that, we can edit the /etc/fstab file using sudo nvim or sudo nano or the editor of your choice:

# /etc/fstab
...
# /dev/nvme2n1p2
# UUID=da3aea43-...-aeaf8c7224d7 /home      btrfs      rw,relatime,ssd,discard=async,space_cache=v2,subvol=/@home 0 0

# /dev/nvme3n1p1
UUID=84e9a1b7-...-7a6a17fd0395 /home      btrfs      rw,relatime,ssd,discard=async,space_cache=v2,subvol=/@home 0 0
...

See how I commented out the old line that mounted /home and added a new line with the new UUID. The rest is all the same. And that’s it, just boot and from now on my /home will mount on the new drive. Now I have 2 TB dedicated just to my junk downloads lol 🤣

But not only that, I also have space for big things like Docker images or LLM models, without risking filling the operating system drive and running the risk of it no longer booting due to lack of space — as has happened to me several times in the past. Always remember to do maintenance on Docker and LLMs and delete old things.

Speaking of which, BTRFS supports snapshots. And one of the basic mistakes is forgetting that Docker or LLM directories can end up in the snapshots and occupy space without you knowing. To avoid this it’s better to mark those directories to not enter the snapshots. For that we have to tell BTRFS to ignore those directories.

The correct way is to create separate subvolumes for them. For this I’ll mount that same partition on the separate drive where I mounted my $HOME:

~ ❯ sudo mkdir /mnt/temp
~ ❯ sudo mount -o subvolid=5 -t btrfs /dev/nvme3n1p1 /mnt/temp

~ ❯ ls /mnt/temp
 @home   timeshift-btrfs

~ ❯ sudo btrfs subvolume create /mnt/temp/@docker
Create subvolume '/mnt/temp/@docker'

~ ❯ sudo btrfs subvolume create /mnt/temp/@ollama
Create subvolume '/mnt/temp/@ollama'

~ ❯ sudo umount /mnt/temp

Now just edit /etc/fstab to mount correctly just like we did with HOME:

# /etc/fstab
...
# /dev/nvme3n1p1
UUID=84e9a1b7-...-7a6a17fd0395 /home      btrfs      rw,relatime,ssd,discard=async,space_cache=v2,subvol=/@home 0 0
UUID=84e9a1b7-...-7a6a17fd0395 /var/lib/docker      btrfs      rw,relatime,ssd,discard=async,space_cache=v2,subvol=/@docker 0 0
UUID=84e9a1b7-...-7a6a17fd0395 /var/lib/ollama      btrfs      rw,relatime,ssd,discard=async,space_cache=v2,subvol=/@ollama 0 0
...

Done, clean subvolumes. More important in my case: I moved directories that have a tendency to grow without control out of the operating system partition, making it harder to end up with an inaccessible system due to lack of space. Do the same with other similar directories, with things like Kubernetes, for example. Anything that keeps downloading large files, like system images.

Now just mount and bring things back:

# every time you edit fstab, run this:
sudo systemctl daemon-reload

# create the mount points
sudo mkdir /var/lib/docker
sudo mkdir /var/lib/ollama

# mount
sudo mount /var/lib/docker
sudo mount /var/lib/ollama

# copy the contents of the old directory back
sudo rsync -aHAX --inplace /var/lib/docker.old/ /var/lib/docker/
sudo systemctl start docker

Besides space management, what’s the other advantage of making separate subvolumes? Because the next section will be about making the system even more robust by adding snapshot and rollback capability to past snapshots. And the ideal is to keep huge files out of snapshots that will sit in the past. Docker images are easy to recover: just docker pull again, no need for backup.

6 - Installing Omarchy

This is the easiest part. When you’re done installing with archinstall, if you chose the Hyprland option, SDDM will appear for you to log in. And when you enter you’ll be shocked at how raw Hyprland is. You basically can’t do anything.

So this doesn’t become an entire day of manual configuration, we’ll use Omarchy. Hit “Super Enter” to open the terminal and don’t mind how ugly things look right now. It’ll get better.

In the terminal just do this:

curl -fsSL https://omarchy.org/install | bash

Just follow the on-screen instructions and let it do the rest. And that’s it, done, there’s no next step.

I HIGHLY recommend you read the OMARCHY ONLINE MANUAL that DHH made. It’s very clean, simple, and direct. All the keyboard shortcuts, for example, are in the HOTKEYS section. Spend an hour reading the basics to be able to operate it. But by now you already have practically everything you need: Alacritty as terminal, Chromium as browser (you can install Brave if you prefer, as I do).

“Super Alt Space” opens the Omarchy main menu. “Super Space” opens Rofi to type the name of the app you want to open. “Super Enter” opens the terminal. “Super B” opens the browser and so on.

7 - Customizing Omarchy

Omarchy is a customized Hyprland. We can edit whatever we want directly in ~/.config/hypr. And since I was used to ML4W, I decided to tweak a bit to make it good for my setup. Your setup will vary depending on your needs.

Right after I finished installing Omarchy and tried to use Gparted to partition that second NVME of mine for HOME, it had an authentication problem. I don’t remember if this was the solution but it’s good to make sure you have Polkit installed:

# make sure polkit + agent exist
sudo pacman -S --needed polkit polkit-gnome xorg-xhost

# Hyprland: export full env to systemd user bus once
dbus-update-activation-environment --systemd --all

Back to it, the first thing I need to solve is that I have 2 monitors. An Asus 32" 1440p 144fps as the main and another Samsung 24" 4K 60fps vertical on the left side. The way to configure this in Hyprland is by editing a file like ~/.config/hypr/monitors.conf which is normally loaded by ~/.config/hypr/hyprland.conf.

Instead of doing this manually, I prefer to install:

yay -S nwg-displays

Here I configure things like transform 90 degrees to rotate the vertical monitor, resolution scaling, 10-bit color support, etc.

It will save two files: ~/.config/monitors.conf and ~/.config/workspaces.conf. Omarchy doesn’t load this second file, so we have to edit ~/.config/hypr/hyprland.conf to look like this:

...
# Change your own setup in these files (and overwrite any settings from defaults!)
source = ~/.config/hypr/monitors.conf
source = ~/.config/hypr/workspaces.conf`
...

I also took the opportunity to add some extra configurations in this file:

...
# animation
animations {
  enabled = yes
  bezier = easeOutQuint, 0.23, 1, 0.32, 1
  animation = workspaces, 1, 4, easeOutQuint, slide
  # optional extras:
  animation = specialWorkspace, 1, 4, easeOutQuint, slide
  animation = windows, 1, 4, easeOutQuint, popin 80%
  animation = fade, 1, 4, easeOutQuint
}

xwayland {
    force_zero_scaling = false # careful with mixing scaling wayland and x11
}

input {
  numlock_by_default = true
}

# dialogs/splash/utility ONLY
windowrulev2 = float,  xwayland:1, windowtype:^(dialog|splash|utility)$
windowrulev2 = center, xwayland:1, windowtype:^(dialog|splash|utility)$

# never center menus/tooltips
windowrulev2 = tile,   xwayland:1, windowtype:^(popup_menu|dropdown_menu|tooltip)$

Some people don’t like it but I like having a quick animation when I switch from one workspace to another, which is why I added the animations block.

When you use apps made for X11, they’ll run on top of XWayland. One annoying thing is that some of them open “dialog boxes,” which are floating windows. And they can end up in places hard to click, especially on a multi-monitor setup. That’s why I added blocks like xwayland and windowrulev2 rules to force those dialog boxes to always appear centered on screen.

Finally, I have an 8bitdo numpad keyboard connected via Bluetooth, but it doesn’t start up with numlock on. To fix this there’s that input block.

Back to the monitor layout. After nwg-displays generated the files, I still had to modify a bit for my case, like this:

# ~/.config/monitors.conf
# Generated by nwg-displays on 2025-08-28 at 14:30:17. Do not edit manually.

#monitor=HDMI-A-3,2560x1440@143.97,1080x407,1.25,bitdepth,10
#monitor=DP-5,3840x2160@60.0,0x0,1.875,bitdepth,10
#monitor=DP-5,transform,1
#monitor=DP-1,0x0@60.0,-1x-1,1.0
#monitor=DP-1,disable

monitor=HDMI-A-3,2560x1440@143.97,2813x948,1.0,bitdepth,10
monitor=DP-5,3840x2160@60.0,1373x350,1.5,bitdepth,10
monitor=DP-5,transform,1
monitor = DP-1,disable

The monitor file I didn’t have to change anything, really, just remember to disable that DP-1 that appears on my main monitor (I think it’s its DisplayPort, but I use HDMI, so I have to disable DP-1).

# ~/.config/workspaces.conf
# Generated by nwg-displays on 2025-08-28 at 13:07:02. Do not edit manually.

workspace=5,monitor:DP-5,persistent:true
workspace=1,monitor:HDMI-A-3,persistent:true
workspace=2,monitor:HDMI-A-3,persistent:true
workspace=3,monitor:HDMI-A-3,persistent:true
workspace=4,monitor:HDMI-A-3,persistent:true

This one I also didn’t have to change much, in nwg-displays itself I said which workspaces I want on each monitor. In my personal preference I prefer to leave the vertical monitor only with workspace 5 and the others on the main monitor. The difference here was that I added those persistent:true options on all of them, to guarantee that everything will work correctly with the workspaces “pinned” in the right places. Workspaces in Hyprland are quite dynamic; it’s worth reading the documentation later if you want more complicated cases.

This solves my two-monitor setup the way I prefer. Experiment with things like scaling until everything is okay for your setup.

Next I needed to solve the hotkeys (key bindings). I’m used to some combinations that came from ML4W so I decided to copy them directly into ~/.config/hypr/bindings.conf. First, switch chromium for Brave:

# Application bindings
$terminal = uwsm app -- alacritty
$browser = brave # omarchy-launch-browser
...

Then I added a bunch of binds at the end of the same file:

unbind = SUPER, V
bind = SUPER, V, swapsplit
bind = SUPER, M, fullscreen, 1                                                           # Maximize Window
bind = SUPER, T, togglefloating                                                          # Toggle active windows into floating mode
bind = SUPER SHIFT, right, resizeactive, 100 0                                           # Increase window width with keyboard
bind = SUPER SHIFT, left, resizeactive, -100 0                                           # Reduce window width with keyboard
bind = SUPER SHIFT, down, resizeactive, 0 100                                            # Increase window height with keyboard
bind = SUPER SHIFT, up, resizeactive, 0 -100                                             # Reduce window height with keyboard

# Display
bind = SUPER SHIFT, mouse_down, exec, hyprctl keyword cursor:zoom_factor $(awk "BEGIN {print $(hyprctl getoption cursor:zoom_factor | grep 'float:' | awk '{print $2}') + 0.5}") # Increase display zoom
bind = SUPER SHIFT, mouse_up, exec, hyprctl keyword cursor:zoom_factor $(awk "BEGIN {print $(hyprctl getoption cursor:zoom_factor | grep 'float:' | awk '{print $2}') - 0.5}") # Decrease display zoom
bind = SUPER SHIFT, Z, exec, hyprctl keyword cursor:zoom_factor 1 # Reset display zoom

# Workspaces
bind = SUPER, 1, workspace, 1 on HDMI-A-3 # Open workspace 1
bind = SUPER, 2, workspace, 2 on HDMI-A-3  # Open workspace 2
bind = SUPER, 3, workspace, 3 on HDMI-A-3 # Open workspace 3
bind = SUPER, 4, workspace, 4 on HDMI-A-3 # Open workspace 4
bind = SUPER, 5, workspace, 5 on DP-5 # Open workspace 5

bind = SUPER SHIFT, 1, movetoworkspace, 1  # Move active window to workspace 1
bind = SUPER SHIFT, 2, movetoworkspace, 2  # Move active window to workspace 2
bind = SUPER SHIFT, 3, movetoworkspace, 3  # Move active window to workspace 3
bind = SUPER SHIFT, 4, movetoworkspace, 4  # Move active window to workspace 4
bind = SUPER SHIFT, 5, movetoworkspace, 5  # Move active window to workspace 5

bind = SUPER, Tab, workspace, m+1       # Open next workspace
bind = SUPER SHIFT, Tab, workspace, m-1 # Open previous workspace

bind = SUPER CTRL, 1, exec, $HOME/.config/hypr/moveTo.sh 1  # Move all windows to workspace 1
bind = SUPER CTRL, 2, exec, $HOME/.config/hypr/moveTo.sh 2  # Move all windows to workspace 2
bind = SUPER CTRL, 3, exec, $HOME/.config/hypr/moveTo.sh 3  # Move all windows to workspace 3
bind = SUPER CTRL, 4, exec, $HOME/.config/hypr/moveTo.sh 4  # Move all windows to workspace 4
bind = SUPER CTRL, 5, exec, $HOME/.config/hypr/moveTo.sh 5  # Move all windows to workspace 5

Note that the last block of scripts needs a script to move all windows at once to another workspace. I copied this moveTo.sh from ML4W’s GitHub and placed it directly in ~/.config/hypr. Don’t forget to give execute permission with chmod +x moveTo.sh afterwards. The various workspace ones below are how I prefer to move workspaces and move apps to different workspaces.

Omarchy already comes with “SUPER J” to change the orientation of windows, for example, if there are two horizontal windows one on top of the other, with SUPER J they become vertical, but “SUPER V” was missing which I added for “swapsplit” which is swapping windows’ places.

I also like “SUPER M” to temporarily make one of the windows full screen, and “SUPER T” to temporarily make a window floating (like a calculator, for example). I had to unbind the previous SUPER M that opened Spotify, but I barely use Spotify on desktop so I don’t need it. When I want to open that app, I go to SUPER SPACE to open Rofi and type it there.

If you have questions about the default combinations, Omarchy has “SUPER K” to open a window with help for all of them.

I still haven’t modified combinations for specific applications. The ones that come with Omarchy are already very useful, like SUPER A to open ChatGPT as if it were a separate app, or SUPER CTRL A to open Grok in a separate window. SUPER O to open Obsidian is also very useful for me. I recommend playing with these binds to make specific bindings and, again, reading the manual to learn everything else it already brings. But up to here it’s enough to leave Hyprland the way I like to use it.

8 - Timeshift, Snapshots and GRUB

Remember I said to choose GRUB in archinstall? It’s time now. The goal: let the system make periodic snapshots of itself, and in case you install some corrupted package or do things on your system that prevent it from booting, instead of having to resort to a USB drive to try to boot and diagnose the error, it’s easier to just boot, choose a snapshot right before the destructive operation, and come back intact.

Before proceeding, an adjustment I think is important: my 4K monitor makes GRUB letters too small for me to read. I prefer bigger letters. Fortunately GRUB uses bitmap fonts so I can use any. I did the following:

sudo grub-mkfont -s 48 -o /boot/grub/fonts/caskaydia48.pf2 /usr/share/fonts/TTF/CaskaydiaMonoNerdFont-Regular.ttf

This will generate a font in the format that GRUB can load. Choose any one you like; Omarchy already pre-installs several very good ones like JetBrains’s. If size 48 is too big, lower it to 32, experiment until it’s the way you like.

Then just edit /etc/default/grub to, at the end, have this line:

GRUB_FONT=/boot/grub/fonts/caskaydia48.pf2

And now just regenerate the bootloader:

sudo grub-mkconfig -o /boot/grub/grub.cfg

Every time yay -Syu brings a kernel or driver update, GRUB will be regenerated with the same font. Look at how it turned out for me, I think it looks much better (this is a 4K monitor):

Now the entire reason for having chosen GRUB: allowing you to choose BTRFS snapshots to boot into the past, before some destructive operation, as shown in the photo above. For this we need the following:

pacman -S timeshift grub-btrfs
yay -S timeshift-autosnap
systemctl enable --now grub-btrfs.service

Done. From now on, every time GRUB is rebuilt, it’ll have a new option to choose snapshots. We just need to create those snapshots automatically, and for that we have the Timeshift app. Open it and follow its step by step. The important thing is to choose the BTRFS option (yes, you can use RSYNC if you’re on ext4, but that occupies a lot of space, because it will literally keep making copies of your entire system’s files. The beauty of BTRFS is that its snapshots don’t duplicate files).

I leave it making snapshots and accumulating versions of the last hours, last days, last weeks, and months. If later on I remember “damn, there was that file I deleted last month but now I need it” just open Timeshift and browse a snapshot from last month. Like Mac folks do with Time Machine, but occupying almost no extra space.

Besides that, every package upgrade will generate a snapshot beforehand. That way, if by chance a corrupted package comes in or breaks some important config of yours, you can go back exactly to how it was before the update. And if it breaks catastrophically, where your Linux no longer boots, just go to the GRUB extra menu and boot from the previous snapshot, which worked. And everything comes back just like it was before.

Nothing is more important for your system than having automatic snapshots that occupy almost no extra space. This alone makes BTRFS superior to any alternative.

This does not replace external Backup!!

This rollback and the ability to recover things easily is good for day-to-day accidents. But let’s say your NVME breaks somehow, or your PC breaks somehow. That’s what external backups are for (be it external HD, NAS, or Cloud). You want to be sure that, in a catastrophic situation, it’s still possible to recover your files.

I have all the protection layers: local snapshots via BTRFS and Timeshift, backups done remotely to my NAS, and a backup of the most important parts of my NAS on Amazon Glacier which, unlike Amazon S3, is for keeping backups, where storage cost is ultra low but transfer cost is higher (S3 storage is more expensive but transfer is cheap, which is why it serves to serve website assets).

I like being prepared so that, in case my house catches fire, I don’t lose anything important.

9 - NFS on the NAS

Speaking of NAS, one thing I configured differently — since I primarily use Linux instead of Windows — was enabling the NFS server.

But beware that some NAS have a characteristic that can be annoying: your user on the NAS will have a UID and GID different from 1000!

When you create a user on any Linux, the default is to assign UID 1000 to the first user, 1001 to the second, and so on. But since on the Synology NAS the first user receives UID 1026, if I mount with NFS (which passes through UNIX permissions), I’ll have authorization problems.

There are two solutions: try to change the UID on the NAS from 1026 to 1000. I tried that and I’ll tell you up front it’s a waste of time, you can end up breaking your NAS’s configurations catastrophically.

It’s much easier to change the user of your new local Linux to be 1026 and that’s it. It’s pretty easy.

First, we need to log out and leave Hyprland and go back to the SDDM login screen. Then we need to switch to another TTY with CTRL + ALT + F3 (to go to tty3, for example). From there we have to log in as root.

Now we run: systemctl isolate multi-user.target to run in isolated recovery mode, where no other user has processes running. Since we’re going to mess with the user, they can’t be active.

usermod -u 1026 akitaonrails
groupmod -g 1026 akitaonrails
chown -R 1026:1026 /home/akitaonrails

With the commands above we change from 1000 to 1026 and fix the permissions on the local files to reflect this change. Now yes, we can edit /etc/fstab to mount the shared directories on the NAS with NFS:

# NFS
192.168.0.x:/volume1/GIGACHAD /mnt/gigachad nfs4 _netdev,noauto,nofail,bg,x-systemd.automount,x-systemd.requires=network-online.target,x-systemd.after=network-online.target,x-systemd.idle-timeout=60s,noatime,nodiratime,async,rsize=65536,wsize=65536,actimeo=1  0 0
192.168.0.x:/volume1/TERACHAD /mnt/terachad nfs4 _netdev,noauto,nofail,bg,x-systemd.automount,x-systemd.requires=network-online.target,x-systemd.after=network-online.target,x-systemd.idle-timeout=60s,noatime,nodiratime,async,rsize=65536,wsize=65536,actimeo=1  0 0

Don’t forget to create the mount points:

mkdir /mnt/gigachad
mkdir /mnt/terachad

Done, just reboot and everything will work correctly. The file permissions on the NAS — which are UID 1026 and GID 1026 — will be accessible to the local user without problems.

10 - Utilities

Here are the last small tips to keep everything in one place. The first is to replace the ls command with eza which offers a nicer face, with icons and everything, like this:

Just do yay -S eza and add this to your .zshrc or .bashrc:

alias ls='eza --icons --color=auto --group-directories-first'
alias ll='eza -l --icons --color=auto --group-directories-first'
alias la='eza -la --icons --color=auto --group-directories-first'

Second tip is to install Atuin. For anyone who deals with infra, this is extremely important: backup of all the command history you’ve used in your terminal to this day.

Install with yay -S atuin and do:

atuin login
atuin sync

Read the official site, but first you need to create an account and a secret encryption key — yes, you can run it locally on a server at your house, but their cloud option is pretty convenient and secure because it’s all encrypted, so there’s no problem if you upload passwords from your history.

Just put this in your .zshrc:

export ATUIN_NOBIND=true
eval "$(atuin init zsh)"
bindkey "^R" atuin-search

# incremental history search with arrow keys
bindkey "^[[A" history-beginning-search-backward
bindkey "^[[B" history-beginning-search-forward]]

If you’re using ZSH, its autocomplete will work normally but, if you type CTRL+R, it will open the Atuin history. It will save all your commands in a SQLite and sync. That way you can sync several machines and servers you use day-to-day or, as in my case, recover the whole history on a new install.

And to make it better, we can configure it to auto-sync:

mkdir -p ~/.config/atuin
sed -i 's/^#\?auto_sync.*/auto_sync = true/; s/^#\?sync_frequency.*/sync_frequency = "30m"/' ~/.config/atuin/config.toml 2>/dev/null || \
printf 'auto_sync = true\nsync_frequency = "30m"\n' >> ~/.config/atuin/config.toml

Third tip: since we have Docker already installed and configured, I like to use SGPT on the command line for when I want to remember the syntax of some command I rarely use, instead of opening an app or browser. Just add this to my .zshrc:

export OPENROUTER_API_KEY="sk-or-v1-8b8...17"
export OPENAI_API_KEY="sk-or-v1-db...f3"
export OPENAI_API_BASE="https://openrouter.ai/api/v1"

alias sgpt='docker run --rm -e OPENAI_API_BASE=${OPENAI_API_BASE} -e OPENAI_API_KEY=${OPENROUTER_API_KEY} ghcr.io/tbckr/sgpt:latest txt '

I use OpenRouter to have access to different commercial LLMs in a single API, but you can just use your OpenAI key. With this done, I’d use it in the terminal similar to this:

~ ❯ sgpt "what is the pacman command in Arch to find out if a particular file came from an installed package?"
To find out if a particular file came from an installed package in Arch Linux, you can use the following pacman command:

``bash
pacman -Qo /path/to/file
``

Replace "/path/to/file" with the actual path to the file you want to check. This command will show the package which owns the file, or report that the file does not belong to any package if that is the case.

If you want, it supports sessions, so it’ll remember the conversation on each new call. Read the project’s Github.

Since I backed up my old ~/.config, things like open tabs from my Brave browser opened normally in this new Omarchy install and I can keep doing my things exactly from the point I left off. I didn’t lose anything. The same goes for all the other apps like emails that I back up locally via Thunderbird. I don’t need to have it all downloaded again.

If, like me, you have several audio output devices, it’s possible that right after finishing the install you’ll notice that it seems there’s no sound. But that’s not a problem because it may just be that the output is pointing to the wrong device or the wrong profile. Use SUPER ALT SPACE to open the Omarchy menu, choose SETUP and AUDIO to open that program which is text mode (I like text mode programs).

In my case, I use a Topping D70 DAC to process my audio and the “Default” profile doesn’t work but switching to “Pro Audio” worked. Also be careful that there’s a global volume setting and individual volume control per program. Overall, thanks to Pipewire, everything should work normally.

I tested apps like Discord, Blender, Bambu Studio, VLC and everything worked perfectly. Only VLC needed codecs from a separate package:

yay -S extra/vlc-plugin-ffmpeg

Blender too, the best is to install from this package:

yay -S blender-bin

If you use the normal “blender” package, some of its add-ons don’t install right. I had this problem with MACHIN3Tools and MESHMachine, for example.

Speaking of Blender, it’s an X11 app. Wayland runs X11 applications using an adapter layer called XWayland. I had several problems with menu and dialog box positioning, even adding windowrule. One thing that worked, I’m not sure why, was installing Blender and Bambu Studio via Flatpak:

yay -S flatpak
flatpak install org.blender.Blender
flatpak install com.bambulab.BambuStudio

Finally, if you’re new to LazyVim, don’t forget to run the :LazyExtras command to install extra plugins for your favorite languages, like LSP, Linter, syntax highlight, snippets, and more. Pure LazyVim is already complete but without support for many languages so as not to be unnecessarily heavy; you need to install them separately afterwards.

Also learn to use Mise-en-place. In the Omarchy menu there’s an option to install support for various languages and it will use Mise, which is the correct way to manage your languages.

Mise is really easy, in general, from inside your project directory, just do something like:

mise use ruby@3.4.3

With this, only inside your project will Ruby 3.4.3 be available, but outside there can be another global version. Don’t forget to activate it in your .zshrc:

eval "$(mise activate zsh)"

Impressions and Conclusion

Compared to my old Manjaro, there’s a lot I no longer need to configure. For example, because of the amount of space Docker and LLMs needed, I had configured an iSCSI drive on my NAS just for that. But since I freed up 2 TB of local NVME, I no longer need to deal with that, I just use the new BTRFS sub-volumes I created above.

I also no longer need to configure QEMU/Libvirt because I no longer need to use Windows alongside Linux side by side. I kept the NVME with the Windows partitions and I can dual boot to it if I really need something. Overall I never need it, so it’s one less complicated configuration.

I spent months using apps via Flatpak, with the aim of being more secure and isolated from the system. It’s like running apps via Docker. But it’s too annoying. Being isolated means it doesn’t obey system themes, and in the end I’m forced to keep opening permissions for the apps. And if it’s going to keep opening them, it’s easier to install natively via yay and everything works perfectly.

I think that completes my notes of everything I needed to do to leave Omarchy exactly the way I like. As I said repeatedly, what works for me doesn’t mean it’ll work for you. For that there’s documentation like ArchWiki which is “THE” best place to learn everything about Linux. Use and abuse that documentation.

As you can see, there wasn’t all that much I customized in Omarchy, which is why I like it: it already did 95% of the path for me, I just had to touch a few adjustments to leave it the way I need. And now without all the junk that piled up in my old Manjaro. I recommend everyone install Omarchy as a first Linux and learn to actually use it.

On Linux, this is a massive pain in the ass, because the SMB protocol (built for Windows) doesn’t properly carry concepts like ownership (chown) or permissions (chmod). It doesn’t understand things like execute permission (everything is executable). For simple stuff like a Downloads, Videos, or Photos directory, it kind of doesn’t matter.

But let’s say you want to edit source code in projects with Git. It’s going to be a nightmare, because SMB overrides the permissions regardless of what’s underneath, and every single time it’s going to conflict with Git, which will think files were modified (changing permissions counts as a change it tracks), and it’ll keep nagging you to commit that, and that will pollute your whole repository (never use Git projects on SMB over the network).

Instead, I’m on Linux, my NAS is Linux. The right call is to use a Linux protocol. That protocol is NFS and I showed how I set it up in last week’s post.

Anyway, I’m using NFS and I have way fewer headaches around file permissions. But then I hit another problem: if you saw my recent posts, you saw I’ve been experimenting A LOT with Docker. Spinning up new images like there’s no tomorrow. My local 2 TB NVMe filled up fast and started complaining about lack of space. So I thought, “Ah, I’ll just move the Docker storage directory, /var/lib/docker, to a directory mounted via NFS on my NAS”

Doing that is very easy:

sudo mkdir -p /mnt/nfs/docker
sudo chown root:docker /mnt/nfs/docker
sudo chmod 771       /mnt/nfs/docker

That creates a new directory inside my NAS and applies the correct permissions.

sudo systemctl stop docker
sudo mv /var/lib/docker /mnt/nfs/docker
# sudo rm -Rf /var/lib/docker (this will free everything up for me)
sudo ln -s /mnt/nfs/docker /var/lib/docker

That will free up all the occupied space. I don’t need to worry about the images because I can rebuild everything from my Dockerfiles. And I don’t need to worry about volumes because I had nothing important that wasn’t mounted externally, in real directories. Rule: never store important things in Docker volumes. It’s bad practice!

# edit /etc/docker/daemon.json
{
  "data-root": "/mnt/nfs/docker"
}

sudo systemctl start docker

Done, this changes the configuration from /var/lib/docker to the new /mnt/nfs/docker and after restarting the service, it’ll start writing everything there. So the post is over, right?

This is as far as most people go. But if you work with this setup for 2 minutes, you’ll feel something is VERY wrong.

I tried building a new image. And it was taking ABSURDLY long. It turns out NFS is indeed faster than SMB, but both were built for file servers, not for operating systems. That makes a huge difference.

These protocols are FILE-BASED, every operation is based on files. And that’s extremely inefficient.

If you watched my videos on file systems, you should already know they are BLOCK-LEVEL. At the filesystem level, you work with fixed-size blocks of bytes, organized in some variation of B-TREE structures (depending on the filesystem). Operations happen at the block level. Files are abstractions over collections of blocks.

When you run a chmod -R, which is a recursive permission change, inside a Dockerfile build, for example, even an empty tree will cause recursion on every entry inside, be it hidden files, ACLs, etc. And it needs to look up every single permission of everything in there. That costs network time (because NFS/SMB are network protocols).

That causes latency. Every permission on every file is an individual remote call, an RPC call. Even on the fastest network, like my 10Gbps one, even if each RPC is under 0.1 ms, it quickly becomes a bottleneck. And that’s exactly what started happening.

I used sudo nfsiostat 1 10 to check. It was showing this:

write: ops/s=2417  kB/s≈295 MiB/s  avg exe=106 ms  avg queue=102 ms

An average of 106 ms per RPC. That’s an eternity. Average wait time in the queue of 102 ms. Even if I make just 100 RPC calls, it will cost more than 10 seconds in latency alone. And in an OS build, installing packages with thousands of files, that will literally cost an eternity. Running Docker build on NFS/SMB is simply not viable.

iSCSI

Obviously I’m not the first to run into this problem and make the mistake of using a file-based protocol for services with high-volume file churn like the Docker cache.

The right move is to create an iSCSI virtual drive on my NAS. If you’ve never heard of it, SCSI is an advanced protocol for drives. Cheap PCs in the 90s and 2000s used IDE/ATA, which is a simpler and cheaper protocol. But the first Macs and UNIX workstations from the 80s and 90s always used SCSI, which was vastly superior.

It was asynchronous, with queue control. You could send dozens of commands at once and the protocol was smart enough to reorder those commands for better efficiency (for example, skipping a file read if a delete command came before it). IDE was blocking, one command at a time. A single SCSI BUS could handle up to 16 devices. Parallel ATA was 2 per cable. Ridiculous. SCSI had advanced error recovery and reporting systems. It had “enterprise” features for hot-swap, power management, timeout behaviors, recovery, clustering, etc. And on top of all that it was agnostic to the transport medium, it could be fiber optic, SAS, or even iSCSI, which is SCSI over Ethernet, which is what we’re going to use.

IDE/ATA had none of that. It was the protocol for poor-person PC disks. Real workstations, SUN, IRIX, Silicon Graphics, Macs, used SCSI.

More importantly, iSCSI is a block protocol, not a file protocol. Anyway, this varies by NAS manager, but in the case of Synology DSM, which is what I use, it has an application called SAN Manager, which lets you create an iSCSI drive:

Just follow the wizard, give it a name, choose the size, enable CHAP if you need extra security (I don’t, I’m on a controlled local network). That creates the iSCSI drive (think of it like a remote USB drive) and a LUN (Logical Unit Number). A LUN is simply one of many potential “virtual disks” that a target can present. A LUN is like a numbered slot in a target’s storage array. A target is a disk server. On a SAN you can have several virtual disks and the way to organize that is with LUNs. In our case there’s no real need to go deep on LUN details.

It takes a while, but when it finishes you just go back to my Manjaro Linux and run:

yay -S open-iscsi
sudo systemctl enable --now iscsi 

That brings up the service to start automatically on every boot. Now to discover my drive on the network:

❯ sudo iscsiadm -m discovery -t sendtargets -p 192.168.0.xx
192.168.0.xx:3260,1 iqn.2000-01.com.synology:TERACHAD.docker.7af6e...
[2804:1b3:....:fe18:3f7d]:3260,1 iqn.2000-01.com.synology:TERACHAD.docker.7af6e...
[fe80::9209:....:3f7d]:3260,1 iqn.2000-01.com.synology:TERACHAD.docker.7af6e...

Every virtual drive has an IQN which is like a “URL” for the drive. Now we log in:

❯ sudo iscsiadm -m node --login
Login to [iface: default, target: iqn.2000-01.com.synology:TERACHAD.docker.7af6e9116c1, portal: 192.168.0.21,3260] successful.

Since I didn’t enable CHAP it doesn’t ask for a password or anything (in a company it should be enabled, obviously). And done. When you do this I hear on GNOME the same little sound as when you plug a pen drive into the PC and it automatically shows up as a disk. If you run lsblk, it’ll show up as a normal /dev/sdX drive. In the GNOME Disks app (or in Windows Disk Management) it really does show up as if it were any other hard drive or SSD. It doesn’t behave like a shared folder, it behaves like a real drive. You even have to format it, just like a new pen drive.

The formatting itself takes quite a while, because in the process it has to check and map block by block. Since it’s over the network, it’s slower than on a real drive, obviously. But it’s a one-time thing. Even on my 10Gbps network it’ll format at around 120 MB/s, and on a 2TB drive that comes out to around 2 hours for a fast format. I forgot you could format with “lazy”:

mkfs.ext4 -v -E lazy_itable_init=1,lazy_journal_init=1 /dev/sdX

Leaving that for next time, now I wait 2 hours. To make sure it mounts automatically at boot, you need to check its node.startup:

❯ sudo iscsiadm -m node -o show | grep -E 'Target:|node.startup'

node.startup = manual
node.startup = manual
node.startup = manual

It’s set to “manual”, to switch it to “automatic” you do:

❯ sudo iscsiadm -m node \
  -T iqn.2000-01.com.synology:TERACHAD.docker.7af6... \
  -o update -n node.startup -v automatic

With the iSCSI daemon running and after logging in, besides /dev/sda, if you run ls /dev/disk/by-path you’ll find something like this:

ip-192.168.0.xx:3260-iscsi-iqn.2000-01.com.synology:TERACHAD.docker.7af6exxxxx-lun-1 -> ../../sdX

Since I’m using the AUTOFS service and it “steals” my /mnt directory, I’m going to mount the iSCSI drive via fstab at /media/docker by adding this line at the end of /etc/fstab:

...
/dev/disk/by-path/ip-192.168.0.xx:3260-iscsi-iqn.2000-01.com.synology:TERACHAD.docker.7af6e9xxxx-lun-1  /media/docker  ext4      _netdev,nofail,x-systemd.automount,x-systemd.requires=iscsid.service,noatime,nodiratime,commit=60  0 2

Then just reload and mount:

sudo systemctl daemon-reload
sudo mount -a

And that’s it. From now on I have a “virtual drive” on my remote NAS. Now, let’s switch Docker over to start using it. I can reconfigure my Docker to cache images and volumes and everything else directly on that drive. Just edit the /etc/docker/daemon.json file with the mount point:

{
  "data-root": "/media/docker/docker"
}

Don’t forget to create the directory on the new drive with the right permissions (and it does accept permissions, because it’s a drive formatted with ext4 like any other):

sudo mkdir -p /media/docker/docker
sudo chown root:docker /media/docker/docker
sudo chmod 771 /media/docker/docker

Finally, just restart the Docker service, wipe out /usr/lib/docker to free up space on my main NVMe, and done. It should be WAY faster than NFS, though not as fast as a local disk, of course. Even at 10Gbps, the network introduces latency, there’s no way around it. But the heaviest part is really the build process, which has a lot of writes. For loading an already-built image, it should be blazing fast. But at least by using a block protocol instead of a file protocol, latency should drop by at least an order of 10x or more.

And is it really faster? YES. In this screenshot of the docker build running, on NFS the times on the right were HUNDREDS OF SECONDS for many commands. Now they’re back below 1 second like they should be. For all intents and purposes, I’m not feeling any difference from what a USB 3.2 Gen 2 external HD would give me.

The MASSIVE difference that a BLOCK-LEVEL virtual drive makes, compared to a FILE-LEVEL NFS/SMB protocol, is brutal, the protocol is gigantic overhead. Incidentally, this is more or less how it works when you go to AWS and hire EBS storage (Elastic BLOCK Store), that’s why it’s called “BLOCK”, because it’s a BLOCK protocol like iSCSI and not NFS. I hope you learned how much of a difference it makes to understand how protocols and filesystems work. If you haven’t watched my videos, there’s this playlist that explains BLOCKS in detail.

But yes, iSCSI has downsides: being a drive, it cannot, or should not, be shared (think of two PCs sharing the same pen drive, it would be a huge sh*tshow). Block devices were built to be used by a single device. Precisely for sharing, that’s why protocols like SMB or NFS exist: to manage concurrent access to the same file over the network.

Another downside: the virtual drive is one big BLOB of bits. Without “mounting” it, there’s no way to see the files. On the NAS server, it doesn’t show the files. It’s like a VirtualBox .vhd file. You need to “turn it on” and “mount” it on the “PC”. That’s why I’m still going to use NFS for the usual stuff, like Videos or Downloads. And the virtual drive is exclusively for the Docker cache, in this case. That works well. No other PC accesses it. If I had a second Linux machine that needed a Docker cache, I’d have to create a second LUN just for it. Remember, it’s a drive. Virtual or not, it behaves like a real drive to your system.