My thesis: that was true generations ago, when decoding x86 cost a relevant slice of the die. Today x86 is, in practice, a translation layer into micro-instructions — and what separates Apple, Qualcomm, and Intel was never the instruction set. Intel’s crisis was managerial and manufacturing, not architectural. Which is exactly why its comeback, now, makes sense.

What the reviews show

The Panther Lake laptops (Core Ultra series 3, launched at CES on January 5 and on sale since January 27) deliver numbers the Meteor/Arrow Lake era couldn’t dream of. Starting with battery — Dell XPS 14 2026 (Core Ultra X7 358H) versus MacBook Air 15 (M5):

The honest reading: in light use, the Dell ties or wins; under sustained load, Apple remains unbeatable. And DHH posted that his XPS 14 running Omarchy Linux gets over 16 hours of real use, with a 1.4W idle draw — Dell made a point of day-one Linux support.

On CPU, the flagship Core Ultra X9 388H against the M5 (Notebookcheck data):

Single-core remains Apple’s territory — it does the same work on a third of the energy. But in moderate multi-core, Intel caught up. Against AMD (Ryzen AI 9 465) the 388H wins across the board; against the first-gen Snapdragon X Elite too; the X2 Elite Extreme, which arrived in 2026, already handed the Windows CPU crown back to Qualcomm (+24% in single-core). Honesty above all.

On GPU, the verdict is more mixed than Intel’s marketing suggests:

Ars Technica’s summary is the fairest: “Intel’s best laptop CPU in a very long time” — with the caveat that Intel must prove this is the new normal, not an aberration.

Remember this: MacBook battery life in an x86 laptop happened — in light use. In single-core and under heavy load, Apple still sets the pace.

The belief: “to be efficient you have to be ARM”

Every programmer has heard the explanation: ARM has a simple, fixed-length, elegant instruction set. x86 is a chimera accumulated since 1978, with variable-length instructions, real mode, segmentation, decades of mess. “Therefore” ARM is inherently more efficient, and the way for anyone to reach Apple- or Qualcomm-class efficiency is to move to ARM64.

There’s truth in there: x86 does carry historical baggage, and decoding variable-length instructions is objectively more annoying than decoding fixed 32-bit ones. The error is in the conclusion. That difference mattered when decode logic occupied a significant fraction of the chip. That was a long time ago.

Remember this: the ISA difference mattered when decoding occupied a relevant fraction of the die. That ended generations ago.

x86 became a translation layer in 1995

The Pentium Pro, from November 1995, already didn’t execute x86 directly: it translated each CISC instruction into internal RISC-like micro-operations and executed those. AMD did the same with the K5 in 1996 (its “ROPs”). In other words: for thirty years, “executing x86” has meant “translate it into something else and execute the something else.” x86 is a compatibility interface, a translation layer over a RISC engine. And the numbers show how cheap that layer became:

When the micro-op cache hits, the fetch and decode hardware is literally switched off. The Hirki study’s conclusion is dry: “the x86-64 instruction set is not a major hindrance in producing an energy-efficient processor.” And the definitive academic study — Blem, Menon, and Sankaralingam, HPCA 2013, actually measuring ARM against x86 — concluded: instruction count and mix are ISA-independent to first order, performance differences come from microarchitecture, and “the energy consumption is again ISA-independent.”

Jim Keller — the guy who designed AMD’s Zen and Apple’s A4/A5 chips, someone who has lived on both sides — put it even more bluntly in an AnandTech interview: variable-length decode “isn’t dominating the die, so it doesn’t matter that much.” What limits performance today, in his words, is branch predictability and data locality.

And here’s the detail that kills the myth for good: modern ARM does the same thing. The Cortex-A77 has a micro-op cache. Samsung added one to the Exynos M5 explicitly to save fetch and decode power. Fujitsu’s A64FX — the ARM inside the Fugaku supercomputer — decodes the SVE FADDA instruction into 63 micro-ops. Sixty-three. The fantasy of “ARM is one instruction per cycle, simple and pure” hasn’t existed in any high-performance ARM for a long time.

One honest nuance before moving on: x86’s variable length really does make very wide decoders harder to build — Apple decodes up to twice as many instructions per cycle as an x86. That’s a real engineering difficulty. But you pay for it in logic area, which is cheap, not in consumption proportional to the work — which is what defines battery life.

Remember this: for thirty years, no x86 has executed x86 — everything becomes RISC-style micro-ops. The ISA became a compatibility interface.

The transistor math

To understand why the decode burden evaporated, look at the evolution:

In 2000, every block of logic was a tight budget, and x86’s mess was expensive. As transistors became infinite for all practical purposes, the decoder’s fixed cost became pocket change. Along the way, two important shifts:

• Dennard scaling died around 2005. Leakage current stopped clocks from climbing, Intel canceled Tejas in 2004, and the world went multicore. Clocks stalled in the 1-4GHz range and never left.
• Moore’s law became economics, not physics. Cost per transistor stopped falling at 28nm: a leading-edge fab today costs US$20 to 30 billion, and an ASML High-NA EUV machine costs US$350 million.

And what occupies die and consumes energy in a modern chip? Giant caches, branch predictors, dozens of execution ports, GPU, NPU, media engines. The x86 decoder doesn’t even show up in the bread line.

And history hands us the perfect empirical proof: from 2015 to 2021, Intel was stuck at 14nm — Skylake and its derivatives, six years of stagnant process from manufacturing failure. Even so, those old 14nm cores traded blows with AMD’s Zen 2, built on TSMC 7nm. If x86 were the problem, that would be impossible. The bottleneck was the fab. It was the fab all along.

Remember this: six years stuck at 14nm, and Intel still competed with TSMC 7nm chips. The bottleneck was the fab, never the ISA.

What actually makes Apple efficient

The M1 isn’t efficient “because it’s ARM.” Apple has held an ARM architectural license since the A6, in 2012: it designs its own cores from scratch, and only the instruction set is ARM’s. When AnandTech dissected the M1’s Firestorm core, the contrast with contemporary x86 was about engineering choices, not instructions:

In practice: twice the decode width, nearly twice the reorder buffer, six times the instruction cache. Add unified memory soldered into the package and a cutting-edge TSMC process. That’s aggressive microarchitecture, enormous caches, vertical integration, and a leading-edge process — all expensive, all deliberate, and none of it comes free with the ISA.

The reciprocal is also true, by the way: you can make bad ARM. The market is full of mediocre ARM. Efficiency is an engineering and process choice, not an ISA birth certificate.

Remember this: the M1 is efficient because of microarchitecture, cache, process, and vertical integration — not “because it’s ARM.”

Your favorite instructions aren’t x86 or ARM

And there’s something else almost nobody mentions in this debate: much of what a modern CPU executes doesn’t belong to either side’s “classic” instruction set. Look at what runs outside it:

• Video. On Intel, decoding lives in Quick Sync, fixed-function hardware that has existed since Sandy Bridge — a separate block with nothing to do with the x86 legacy. It’s because of it that Frandroid measured the Snapdragon X2 58% slower than Panther Lake in video export.
• Cryptography. AES-NI exists since 2010, SHA extensions since 2016 — dedicated instructions added decades after the “old x86.”
• AI and matrices. AVX-512, then AVX10, and AMX, a matrix tile accelerator. The ARM side has the equivalents: NEON, SVE, SME.

Chips and Cheese ran the perfect experiment: in the same 4K HEVC encode, an Ampere ARM took more than twelve times as long as a Zen 2 with stock ffmpeg; using NEON assembly cut the ARM time by more than 60%. The difference was never ARM versus x86 — it was well-used vector extensions versus ignored vector extensions. In the real world, the heavy lifting lives in the extensions and accelerators, and those are orthogonal to the base ISA.

Remember this: modern heavy lifting — video, crypto, AI — runs on dedicated extensions and accelerators, orthogonal to the base ISA.

The fall was managerial, not architectural

Here’s the part that convinces me completely. Look at the timeline and try to find “x86 was a fundamental limitation” anywhere in it:

• 2005-2006: Intel passes on making the iPhone’s chip — Otellini admitted the regret in his exit interview with The Atlantic: “the world would have been a lot different.” And the tragicomic detail: Intel had an ARM division (XScale) — and sold it to Marvell in 2006 for US$600 million. It wasn’t a lack of technology; it was a lack of vision.
• 2013-2021: three CEOs. Krzanich left in 2018 over an internal scandal. In came Bob Swan, a finance CFO, to run an engineering company at the most delicate moment in its history.
• 2018-2020: 10nm became a joke (Cannon Lake only in limited release) and in July 2020 Intel announced a 7nm delay — the stock dropped 16% in a day. In April 2019, it abandoned the 5G smartphone modem business; Apple bought the modem unit for US$1 billion. In November 2020, the M1.
• 2021-2024: Gelsinger returned with the IDM 2.0 plan, “five nodes in four years.” In December 2024 he was pushed into “retirement” — a board ultimatum, per Reuters and Bloomberg. The year closed with a US$18.8 billion loss, 15,000 layoffs, a suspended dividend, and Intel expelled from the Dow Jones after 25 years — replaced by Nvidia, which at that point was worth more than 30 times Intel. By October 2025, the cumulative layoff count had reached 35,500.

Nothing on that list is architecture. It’s missed products, delayed fabs, wrong decision after wrong decision. x86 was there, competent, while the company dismantled itself around it.

Remember this: the iPhone pass, the XScale sale, the broken 10nm, three CEOs, a US$18.8 billion loss — Intel’s fall was decision after decision, not an x86 limitation.

The unlikely rescue

And then 2025 happened. Lip-Bu Tan took over in March. In August, Trump demanded his head on Truth Social over China ties — and just days later, after a White House meeting, became a fan. On August 22, the US government bought 9.9% of Intel: 433.3 million shares at US$20.47, US$8.9 billion total, converting CHIPS Act grants into equity, with no board seat. SoftBank put in US$2 billion. And the surreal cherry on top: Nvidia — the same company that replaced it on the Dow — invested US$5 billion and signed a partnership to put RTX chiplets into x86 CPUs.

The financial results started showing: Q1 2026 revenue of US$13.6 billion (+7% year over year), Q2 at US$16.1 billion (+25%), seventh consecutive quarter above projections. The stock the government paid US$20.47 for reached more than six times that price in May, when Bloomberg reported talks for Intel to manufacture Apple chips — preliminary reporting, production years away, but the market went wild. Even after cooling off, the US government’s position is still worth five times what it cost. The American taxpayer is, today, a profitable Intel shareholder. Go figure.

Remember this: the US government, SoftBank, and Nvidia as shareholders, and the stock five times above the government’s entry price — Intel became a national cause, and the market bought the comeback.

The comeback, feet on the ground

Panther Lake is the first big product on 18A — RibbonFET (gate-all-around transistors) plus PowerVia (backside power delivery), coming out of Fab 52 in Chandler, Arizona. There’s delicious irony here: the flagship’s 12-core GPU tile is still manufactured by TSMC. And 18A yields (the fraction of good chips per wafer), per Tom’s Hardware, should only reach industry-standard levels in 2027 — the ramp is slow, and Ars Technica asks the right question: is this the new normal or an aberration?

But the point of this article isn’t cheerleading. It’s that Panther Lake empirically closes a debate that was theological. An x86-64 on a competitive process delivers MacBook battery life, ties the M5 in multi-core, and beats the Windows competition in several scenarios. If the ISA were the decisive factor, that would never happen, on any process. What brought Intel down was management; what brings it back is fabs and focus; and what separates Apple, Qualcomm, AMD, and Intel is microarchitecture, cache, process, and integration — exactly as Jim Keller and the literature always said.

I’m rooting for the comeback. Not out of nostalgia from someone who built PCs with Pentiums, but because the efficient-laptop market was settling into a comfortable duopoly — and a comfortable duopoly is the enemy of price and innovation. Let the fight go on.

Remember this: Panther Lake closes a theological debate: competitive x86 exists when the fab is competitive. It was never about instructions.

Every time I mention coding agent harnesses, somebody shows up asking: “but why don’t you use Pi?” or “why not Oh-My-Pi?”

I hate that question when it comes like that.

Not because I have anything against Pi. I don’t. The problem is that almost nobody explains the why. What is the concrete benefit? What problem does it solve better than Claude Code or OpenCode? Which use case is clearly better there? Which limitation does it remove?

Usually, nothing comes back. Just a tool name. That smells like one of two things: fanboy wants validation for the choice he already made, or somebody wants to outsource the homework. “Akita, test this for me and tell me if my favorite tool is good.” Not exactly the kind of pitch that makes me drop what I’m doing.

But I tested it anyway. And, as expected, it did not blow my mind.

Before getting into Pi, OpenCode and Oh-My-Pi, we need to align vocabulary. I already explained this in previous posts, especially in “RANT: Did AI Kill Programmers?”, but it is worth repeating because people still mix everything together.

What is a coding agent harness

A raw LLM does not edit your project. It receives text and returns text. That’s it.

When you open Claude Code, Codex, OpenCode, Oh-My-Pi, Crush or anything similar, you are not talking directly to “the model” in a raw form. You are talking to a program in the middle. That program sets up the LLM’s work environment.

That program is what I call a harness. In English, harness is the thing you use to strap and steer a horse. The metaphor works: the raw model is strong, but without a harness it runs wherever.

The harness does several things:

• builds the system prompt, with behavior rules;
• reads context files like AGENTS.md, CLAUDE.md, local rules, skills and project docs;
• defines which tools the LLM can call;
• executes those tools: read files, search text, edit, run shell, run tests, open a browser, query LSP, call MCP;
• collects tool results and sends them back to the model as new context;
• manages permissions, sandboxing, confirmations for dangerous commands;
• controls the execution plan, task list, parallel subagents;
• detects when context is filling up and triggers compaction;
• tries to preserve memory across turns or sessions;
• decides how to interrupt, resume or steer the agent when you change your mind mid-run.

So two CLIs using the same model can behave differently. Forget “GPT 5.5 vs Opus 4.7” for a second. It is model plus harness. The harness decides what information the model sees, which tools it knows about, which patch format it uses, how it confirms actions, when it summarizes, when it calls a subagent, when it stops.

That layer turns a chat with manual copy-paste into something that can actually work on a real project.

The problem with subsidized plans

I have said this many times: if you use agents for many hours a day, Anthropic and OpenAI’s subsidized plans are much more attractive than paying per token.

Direct API usage is expensive. Very expensive. If you do real Agile Vibe Coding, letting the agent run tests, read code, refactor, compile, rewrite and review, it burns tokens like a weekend barbecue.

A monthly plan changes the math. You pay a fixed amount, get a large quota, and can work without treating every tool call like a taxi meter.

The problem is that Anthropic is strict about where those plans can be used. The Claude Code Legal and compliance docs say OAuth for Free, Pro and Max plans is meant for Claude Code and Anthropic’s native applications, and that third-party developers should use API keys through Claude Console or a cloud provider. They also explicitly say Anthropic does not allow third parties to offer Claude.ai login or route requests using users’ Free, Pro or Max credentials.

So yes, are there hacks to make OpenCode, OpenClaw and other tools use Anthropic plans? Yes. The Claude Code leak helped a lot of people understand how authentication and transport worked. I wrote about that in “Claude Code’s Source Code Leaked. Here’s What We Found Inside.”.

But I do not recommend it.

From what I have seen, Anthropic keeps enforcing the policy. You may get it working today and get blocked tomorrow. Worse, you may lose an account you use every day. Not worth it for me. If you want Opus through an Anthropic plan, use Claude Code. Full stop.

And to be fair, Claude Code is one of the best harnesses around. After hundreds of hours using it, I can praise it without hesitation:

• the planning mode is good;
• the task list is clear and the agent follows it step by step;
• you can interrupt and redirect without destroying the session;
• you can inject prompts while it is working;
• memory and compaction are much more sophisticated than the alternatives;
• after the leak, we could see several layers of context management, not a single dumb summary.

I explained the memory part in more detail in “AI Agent Memory: Karpathy LLM Wiki and agentmemory in Practice”. Claude Code spends real effort preserving what matters when context fills up. Codex and OpenCode do enough. Claude Code goes further.

Codex is the problem, not GPT

OpenAI has Codex CLI to use with its models. I will be blunt: Codex is an inferior harness.

I am not saying GPT 5.5 is bad. Quite the opposite. The model is excellent. The problem is the harness around it.

Codex’s planning mode is weak compared to Claude Code. Interruption is awkward: many times it means stopping everything, losing the flow, and continuing manually. Task organization is looser. More than once it did not finish what it promised until I explicitly told it, “finish this properly.” It feels like a tool that has not decided whether it wants to be a chat, an executor, a planner or a terminal assistant.

Is it usable? Sure. Can you work with it? Yes. But the experience is worse than it should be.

The good news is that we are not forced to use Codex to use GPT. OpenCode exists. In practice, using GPT 5.5 through OpenCode feels much less nerfed. The harness lets the model work better.

OpenCode also documents that it can connect providers and use subscriptions in some cases. In the Providers page, they warn that Anthropic prohibits Claude Pro/Max usage in third-party tools, but list subscriptions like ChatGPT Plus, GitHub Copilot and GitLab Duo as supported in OpenCode. The interface changes between versions, sometimes you see /connect, sometimes people say /login, but the idea is the same: plan OAuth is different from an API key. API key is token credit. OAuth uses the subscription.

So my practical recommendation today is simple:

• Want subsidized Opus 4.7? Use Claude Code.
• Want subsidized GPT 5.5? Use OpenCode.
• Want GPT 5.5 through Codex? You can, but in my opinion you are nerfing the model with a worse harness.

If OpenAI were smart, I would honestly drop Codex CLI and adopt OpenCode as the base. Or at least copy what OpenCode does well without shame. Codex has no concrete advantage for me today.

So where does Pi fit?

Now, Pi.

Pi is a minimalist harness. You can use it by itself, but to me it is like opening NeoVim with no plugins. Yes, it works. Yes, some people love building everything from scratch. Yes, you learn a lot doing it.

But I do not recommend it as the starting point for someone who just wants to work.

Oh-My-Pi is a Bun-based fork with “batteries included.” And of course there is already polarization: Pi extremists say Oh-My-Pi is bloated, that the right thing is pure Pi, building your own tools, composing your own distro, configuring everything by hand.

This reminds me of NeoVim. I like NeoVim. But I do not want to spend my life choosing a statusline plugin, configuring LSP by hand, tweaking file tree keymaps and comparing 12 fuzzy finders. I install LazyVim and move on. When I want to study internals, I study. When I want to deliver work, I use the package that works.

For me, Oh-My-Pi occupies that spot. If you are curious about Pi, start with Oh-My-Pi. If you like it, if you feel the itch, then go to minimalist Pi and build your own distribution. Go for it. Just do not pretend it is objectively better for everyone.

My first impression of Oh-My-Pi is that it works. It is not bad. It also did not make me say, “how did I live without this?”

Out of the box I found it too verbose. I am sure you can customize it, but that is the rabbit hole. I do not want to spend the afternoon tuning prompt, theme, panel size, tool policy and whatever else. I left it default.

In normal use, it feels a lot like OpenCode. Both can work with planning, tasks, tools, subagents, LSP, compaction, interruption and steering. Both are better harnesses than Codex. That is enough for me to take it seriously.

But OpenCode feels more cohesive. More product. More polished. Oh-My-Pi feels more like a lab built by people who enjoy powerful tools full of knobs. That can be praise or criticism, depending on what kind of person you are.

What I checked in the code

To avoid relying only on visual impressions, I asked for an analysis comparing OpenCode, Pi and Oh-My-Pi, then checked parts of the source code myself.

The result matches the intuition.

Upstream Pi

The original Pi is small. The built-in toolset is basically:

• read
• bash
• edit
• write
• grep
• find
• ls

That is clean. I like clean code. But as a modern harness for heavy programming, by itself, it does not compete with OpenCode or Oh-My-Pi.

Again: that does not mean it is useless. It means it is a minimalist base. If you want to build your own agent distribution, great. If you want to open a tool and work today, it is thin.

OpenCode

OpenCode has the normal competent kit for programming:

• shell;
• read;
• glob;
• grep;
• edit/write or apply_patch;
• task/subagents;
• todo;
• web fetch/search;
• skills;
• optional LSP;
• plugins and MCP.

Its read covers what I expect in a code project: source file, directory, supported image and PDF. It has LSP. It has task. It has todo. It does not try to be the Swiss Army knife for every possible artifact.

For normal software, that is enough. For a project mixing SQLite databases, spreadsheets, notebooks, PDFs, archives and web pages, the difference starts to show.

Oh-My-Pi

Oh-My-Pi comes with more tools. A lot more. In its registry you find things like:

• read
• bash
• edit
• ast_grep
• ast_edit
• ask
• debug
• eval
• find
• search
• lsp
• inspect_image
• browser
• task
• job
• recipe
• irc
• todo_write
• web_search
• write
• memory/hindsight tools

The clearest advantage is read.

In Oh-My-Pi, a single read tool understands files, directories, archives (.tar, .tar.gz, .tgz, .zip), SQLite, images, documents (PDF, DOCX, PPTX, XLSX, RTF, EPUB), Jupyter notebooks (.ipynb), URLs in reader mode, and internal URIs like skill://, agent://, artifact://, memory://, rule://, local://, mcp://.

That is where it really wins when the work depends on those formats.

Example: a Rails project with a local SQLite database. In OpenCode, the agent will probably try to use sqlite3 through shell, or write a script, or you will have to instruct it. In Oh-My-Pi, the tool prompt itself teaches:

• file.db lists tables and row counts;
• file.db:table shows schema and samples;
• file.db:table:key fetches a row by primary key;
• file.db:table?limit=50&offset=100 paginates;
• file.db:table?where=status='active'&order=created:desc filters;
• file.db?q=SELECT ... runs a read-only SELECT.

That matters. Of course a human can do this with shell. But the model does not need to invent the command, does not need sqlite3 installed, does not need to get escaping wrong, does not need to dump 30 thousand lines into context. The right path is already described inside the tool, visible to the model.

Same for ZIP, PDF, spreadsheet, notebook. If your project is just TypeScript, Ruby, Go and Markdown, whatever. If your project is data science, investigation, audit, document analysis, database migration, artifact parsing, Oh-My-Pi starts making more sense.

The advantage in pure source code is smaller

For ordinary source code, the difference exists, but it is not huge.

Oh-My-Pi has some good conveniences:

• code reads with structural summaries when the file is parseable;
• multiple ranges in a single read call;
• line-hash anchors that make text edits safer;
• more aggressive LSP routing;
• AST tools (ast_grep and ast_edit);
• a system prompt that insists on using the specialized tool instead of falling back to shell.

The AST part is worth explaining.

grep and rg search text. AST searches syntax structure. Instead of searching the string console.log, you can search for a call console.log($$$) and ignore spacing, line breaks, comments and formatting. Instead of doing a textual replace of an import, you can rewrite an import node. Instead of hunting foo && foo() as text, you can structurally transform it into foo?.() while ensuring the same metavariable appears on both sides.

This is useful in codemods. Useful when you want to change function calls, imports, declarations, error patterns, without catching false positives in strings, comments or docs.

But it is not magic. AST patterns are sensitive. The model can get the shape wrong. Sometimes you need to adjust the query. For a big refactor, LSP rename is still more correct than AST if the language supports it. AST is one more tool, not a religion.

OpenCode is not helpless without it. It has grep, edit, apply_patch, optional LSP, task, todo. For 90% of ordinary coding tasks, that is enough.

My read after checking the code is simple: when the work involves SQLite, notebooks, documents, archives, images, URLs or internal artifacts, Oh-My-Pi has a real advantage. In pure source code, the advantage shows up more in large refactors, type navigation, structural search and codemods. For normal code editing, OpenCode remains just as good and often more pleasant.

Prompt is also a tool

One thing people ignore: it is not enough for a tool to exist. The LLM needs to know when to use it.

Oh-My-Pi’s prompt is explicit. It tells the model to use tools when they improve correctness, parallelize calls when appropriate, prefer LSP for symbol operations, prefer AST for structural search and codemods, prefer read/search/find over shell, delegate to subagents when the work can be decomposed.

That helps big models. Opus and GPT 5.5 follow this kind of instruction well when the tool description is clear.

OpenCode does not fail at the basics. It teaches Read, Glob, Grep, Edit, Write, Bash, Task, patch. It handles the normal flow. The difference is that it does not push LSP and advanced tools with the same insistence as Oh-My-Pi.

Upstream Pi is also minimalist in the prompt. It lists tools, gives guidelines, but does not do heavy routing. Again: good if you want a small base. Less good if you want a harness full of ready-made conveniences.

Memory, planning, interruption

Here Oh-My-Pi and OpenCode are more similar.

Both have context compaction. Both have some plan/todo mode. Both can spawn subagents. Both allow steering, meaning you do not have to treat the task list like scripture. If the agent starts doing nonsense, you interrupt, inject a new prompt, change direction.

That is mandatory for Agile Vibe Coding. Without decent steering, the tool becomes a conveyor belt: starts wrong, keeps going wrong until it hits a wall. Claude Code is very good at this. Oh-My-Pi and OpenCode get close enough. Codex, in my experience, does not.

In Oh-My-Pi, looking at configuration, there are explicit knobs for steeringMode, interruptMode and compaction.strategy. That is its personality: expose harness behavior as configuration. Powerful, but also one more rabbit hole.

OpenCode tends to hide more of that and deliver a more closed product. I like that trade-off for daily work.

But doesn’t Pi have more agents?

Another argument that shows up: “but in Pi you can spawn way more agents.”

So what?

Having more agents available does not mean the task became more parallelizable. Programming is hard to parallelize for the same reason managing teams of programmers is hard: there is almost always a critical path. You can split some of it, sure, but quickly you hit the point where one decision depends on the previous one, which depends on a detail that only appeared after running the test, which changes the architecture, which invalidates half the original plan.

LLMs with subagents suffer the same physics. Agent A needs to know what Agent B decided. Agent C implements on top of an API Agent A is still changing. The main agent needs to consolidate everything, detect conflicts, request fixes, reread diffs, run tests, ask for another patch. Past a certain point, you are not parallelizing. You are inventing management.

I already demonstrated this in “LLM Benchmarks: Is It Worth ($$) Mixing 2 Models?”. I ran variations with a strong planner, cheap executor, free delegation, forced delegation, manual orchestration. Boring result: in continuous coding-agent work, coordination overhead eats the gain. When I did not force delegation, Tier A models simply did not delegate. When I forced it, quality dropped or time increased. In the best case, it tied with more coordination.

That is not a bug. That is the model having common sense.

In a cohesive task, a good model realizes it needs to keep reasoning in the same context. Design and implementation mix. You plan, write, test, discover a premise was wrong, adjust, go back, simplify. Artificially splitting that across 10 agents only creates more edges to break.

Truly parallelizable tasks exist. Translating a huge strings file into 10 languages? Fine. One agent per language. Converting 200 images to WebP? Parallelize it. Applying the same mechanical refactor to 50 independent files? Maybe. The result of one item does not change the other.

How many tasks in your day actually look like that?

I did not see Oh-My-Pi parallelize more than OpenCode or Claude Code in normal programming. And that is correct. If it started opening agents for everything, I would be suspicious. It would spend time, context and tokens consolidating duplicated work.

So no: mixing models, mixing ad hoc agents, increasing the magic worker count, none of that solves hard programming. It solves independent batches. Independent batches are the easy part.

But doesn’t Pi save more tokens?

Another myth: because Pi is more customizable, you control the system prompt of every component, so you can minimize everything and save tokens compared to OpenCode, Codex or Claude Code, which start with bigger prompts.

You are saving cents and losing dollars.

The initial prompt is not where most of the spend happens in a serious coding agent. The spend comes from the work: reading code, writing code, running tests, receiving stack traces, reading logs, rereading files, applying patches, running tests again. One or two verbose logs already eat the entire window of a small local model with 30K tokens. A bundle exec, an npm test, a frontend build spitting warnings, done: there goes your heroic system-prompt savings.

My personal rule: I would not try to use a serious coding agent with less than 200K tokens of context. Below that, the harness has to compact memory constantly. Compaction helps, but it always loses detail. It loses the warning that appeared before the error. It loses the decision made half an hour ago. It loses the chunk of stdout that looked like noise but was the clue.

Below 100K, for me, it is basically unusable for a long programming session. You cannot let the agent read a large log, test output, build stdout, a long diff. You are back to the dumb job of manually choosing what fits in context. That is the opposite of productivity. You become the assistant’s assistant.

That is why I recommend frontier models. Opus and GPT have windows around 1 million tokens, and that matters in long sessions. You can let the harness work without treating every log like contraband. Recent details stay around longer before the next compaction. The answer improves because the model still has the facts in front of it.

This is a big reason I do not recommend local open source LLMs for serious programming. Model quality is only half of it. The other half is KV cache. There is no consumer GPU with enough spare VRAM for giant context on a large model. Even on a beefy Mac Studio, in practice, you barely get near something like 300K before you start sacrificing memory and performance. Still far from 1 million.

At that point, the price comparison changes. Even paying per token can make sense. And if you want to save money, use a previous-generation cloud model, like Kimi K2.6. In my benchmark, it is still much better than any local open source model I tested. It is cheap, and you do not have to pretend 30K context is a serious development environment.

Worse: if you try to save tokens manually, you break the right workflow.

The wrong way is to let the harness suggest a command, copy it to your terminal, run it yourself, grab only a slice of the output, paste it back into chat, and hope you picked the relevant slice. That saves tokens and destroys context. The model goes blind exactly where it matters: the full error, event order, previous warning, real path, stdout vs stderr, the exact command that ran.

The right way is to let the harness run. Yes, it will send output into context. Yes, it costs. But that is how the model gets enough information to choose the next step. The more real, verifiable context it has, the better the chance it fixes the right thing on the next iteration.

That is why I insist on subsidized plans. If you pay per token, you are tempted to drive with the handbrake on. If you are on a Pro/Max/Plus plan, use the tool properly. Let it run tests. Let it read logs. Let it open files. Let the agent work.

And please, do not waste time trying to turn local Qwen into your main programmer because “it is free.” I already tested this in the canonical benchmark “LLM Coding Benchmark: DeepSeek, Kimi, MiMo, GPT”. Local open source models came in Tier C or worse. Local Qwen is fine for lab work, demos, curiosity, study. For serious programming with long tool-calling, large context, tests and refactors, it is still not a replacement for Claude or GPT.

If you cannot pay US$100 a month for a tool that can multiply your productivity by 5 or 10, the problem is not tokens. It is ROI. It is career. It is the kind of work you are doing.

Tweaking Pi to save crumbs of token is not your ticket to better pay. Delivering more, with better quality, using good tools, is.

Is Pi better than OpenCode?

I tried a practical comparison. I took the same PR, akitaonrails/ai-memory#10, and asked three combinations to review it:

• Oh-My-Pi with GPT 5.5;
• OpenCode with GPT 5.5;
• Claude Code with Opus 4.7.

Same PR. Same kind of request. Same intention: find real problems before merge.

None of them found everything.

To simplify, imagine the PR had problems A, B, C and D. Oh-My-Pi found A and B. OpenCode found B and C. Claude found C and D. That was not the exact matrix, but it is the shape of the result: each one caught part of it. Each one missed part of it.

I keep repeating this because people keep learning it the hard way: do not trust a single LLM to audit anything important. It will find some problems. Maybe good problems. Maybe problems you missed. But complete coverage? No. It depends on the context it read, how you asked, which files it opened, which tool it chose, where the reasoning path went.

The interesting part: the same GPT 5.5 gave different answers in two different harnesses. Oh-My-Pi and OpenCode did not become clones just because they used the same model. The harness changes the prompt, tools, context format, ordering, pressure to use LSP or AST, everything. That changes the path.

In this particular case, I would call Oh-My-Pi and OpenCode tied. Both found real problems. Both missed real problems. Claude was a little worse because in one pass it came closer to saying the PR was mergeable while issues were still missing. But that also does not prove Claude is worse. A second pass with a different prompt could flip it.

I am not crowning a winner. I am killing the fantasy.

Pi does not have hidden magic that turns review into perfect audit. Oh-My-Pi is a very good harness. So is OpenCode. So is Claude Code. All of them fail. All of them depend on the model, context and request. That is why I do not know if I would prefer Oh-My-Pi over OpenCode or Claude Code as my main tool. It sits on the same shelf: excellent, useful, but not miraculous.

One more anecdote, not proof of anything. While editing this very post in Oh-My-Pi, I asked for a localized change. With Claude Code and OpenCode, when I do that in a bilingual post, they usually edit the PT-BR section and the matching EN section, only in the requested range. Oh-My-Pi got half of it right: it edited PT-BR in the right place. But for EN it decided to retranslate the whole file from scratch. I do not know why.

It is one case. It proves nothing by itself. But it is funny because it punctures the idea that Pi always saves tokens. In that round, it spent far more. I have never seen Claude Code or OpenCode retranslate the whole post when I asked for a small edit; they are usually good at touching only the piece I asked for.

My first impression is that OMP really does have more tools, and it tries hard to use them. Maybe too hard. Sometimes it feels like it is forcing tooling onto tasks where OpenCode and Claude Code would be more direct, and that starts getting close to making the answer worse instead of better. Maybe that is only a first impression, but that is the feeling I have so far: OMP sometimes tries a little too hard for its own good.

So which one would I use?

For normal programming, I would keep alternating between Claude Code and OpenCode.

Claude Code when I want Opus and the Anthropic plan. OpenCode when I want GPT 5.5 and do not want to suffer through Codex. Simple.

I would keep Oh-My-Pi in the toolbox, not as a mandatory replacement. I would pick Oh-My-Pi when the project had a lot of stuff outside source code:

• PDFs that need to be read;
• spreadsheets;
• SQLite;
• Jupyter notebooks;
• ZIPs and tarballs;
• images;
• web pages that need to become text;
• internal artifacts from previous runs;
• frequent need for AST codemods.

Maybe it shines more in data science projects. Maybe in data investigation. Maybe in audits, migrations, document research, internal tools full of SQLite and CSV and XLSX. In those scenarios, the universal read tool is very convenient.

For Rails CRUD, Go backend, TypeScript frontend, Rust CLI, Hugo blog? OpenCode is just as good and more polished.

And pure Pi? I would only use it if my fun were building the harness itself. Nothing wrong with that. I just will not pretend it is the path for someone who wants to finish work today.

My current recommendation

If you want the short answer:

• Use Claude Code with Opus 4.7 if you want the best proprietary harness and can stay inside Anthropic’s ecosystem.
• Use OpenCode with GPT 5.5 if you want to use OpenAI’s plan without being stuck with Codex.
• Use Oh-My-Pi if you want a more flexible harness, with more tools, especially for projects with varied files and artifacts.
• Avoid Codex if you have an alternative. The model is good. The harness gets in the way.
• Use pure Pi if you enjoy building your own distro. As an immediate work tool, I would start with Oh-My-Pi.

The Linux analogy helps. I do not use Linux because it is “free.” I use it because, for me, it is the best tool for the job. The alternatives do not come close in my workflow. That does not mean every beginner should drop Windows or macOS tomorrow and hand-roll Arch.

Same with editors. NeoVim is technically fantastic. I love it. But there is a reason it is not as mainstream as VS Code: it is not productive on day one unless you already have experience and learned it gradually. If you need productivity now, use the easiest tool that works out of the box. VS Code, Cursor, Windsurf, JetBrains, whatever. Deliver first. Tweak later.

And look, I am not the guy ricing NeoVim all day. I prefer NeoVim because its editing is better than VS Code’s, and because I like staying 100% in the terminal. A GUI editor like VS Code or Zed does not give me that flow. I use the same editor locally or over SSH, combine it with Tmux or Zellij, run everything in a modern terminal like Kitty, Ghostty or Alacritty, and I am productive that way.

Here, LLM harnesses fit me well. Oh-My-Pi, OpenCode and Claude Code are already terminal-based. Any of them fits my workflow.

Pi falls into that category. It is NeoVim. Technically great, elegant for people who like building their own workshop, but not the right choice for everyone. It is a tool for people who enjoy tweaking, understand the cost, and accept spending time tuning the harness as part of the hobby or the work. Most people should start with the package that works.

So: Oh-My-Pi and OpenCode are good. Pick one and work. There is no religion here.

What irritates me is fanboys selling a tool name as if it were an argument. “Use Pi” means nothing. Tell me what it solves better. Tell me where it saves time. Tell me which real bug it avoids. Tell me which workflow is impossible in Claude Code or OpenCode and trivial in Pi.

Then we can talk.

So far, my read is boring: for normal software development, Oh-My-Pi and OpenCode perform similarly when using the same model. Oh-My-Pi has more tools and is more flexible. OpenCode is more cohesive and polished. Claude Code remains the bar for Opus. Codex remains the option I would only use if I had no choice.

Tools help. Harnesses matter. But process still wins: small prompt, clear scope, tests running, careful review, frequent commits, actual engineering.

I bought a Lenovo Thinkpad T14 Gen 6 and installed Omarchy on it. It’s not my main machine, and it’s not supposed to be. It’s a companion: a notebook I can open on the 3D printing office desk, SSH into the desktop, fire up Claude Code, access files on the NAS, debug the network through ethernet without hunting for a USB dongle and a long cable. This article covers the hardware choice, the Omarchy setup on top, the customizations specific to a laptop and to this Thinkpad in particular, and the architecture decisions that might not be obvious.

Mini-review of the Thinkpad T14 Gen 6

Let’s get this out of the way: this isn’t the notebook of my dreams. If I were picking by looks, I’d grab an Asus Zenbook S 14 with an OLED screen. If I were picking by portability, the Thinkpad T14s with the aluminum shell, which is lighter and much better looking. The regular T14 has a 14" 1920x1200 IPS panel, 400 nits, 60Hz, no HDR. It gets the job done, but it’s a far cry from the Zenbook’s screen. The shell is plastic with a rubberized finish, which makes it scratch-resistant but not premium. Reviews from NotebookCheck and XDA Developers land on the same verdict: good price, good connectivity, fine performance for office work, but the screen is dated.

What makes up for it:

• Port selection. Full-size HDMI. Gigabit ethernet. USB-A, USB-C Thunderbolt 4, charges over USB-C (not a proprietary charger). For a debug companion, this is exactly what I wanted. The Zenbook S 14, being thinner, cuts ports.
• Fingerprint sensor that actually works on Linux (Goodix MOC, works with libfprint on kernel 6.11+). I use this, details later.
• Thinkpad keyboard. 1.5mm key travel, trackpoint, classic layout. It’s not the best keyboard in the world in 2026, but it’s reliable and durable.
• Rugged shell. It will fall, it will scratch, it will travel in a backpack. If I put a Zenbook OLED or a Macbook Pro on the 3D printing office desk next to the printer with PLA dust in the air, I’d be nervous. The Thinkpad can take a beating.

If I wanted a gaming machine, I’d get an Asus Zephyrus G14, my favorite gaming notebook. If I wanted a creative work machine, I’d get the Asus Zenbook Duo (UX8406) with two vertical OLED screens, great for video editing and 3D modeling. I can afford any Macbook Pro or Mac Studio, and I chose not to go that way. I’ll explain in the next section.

My use case

My main PC is a desktop with Ryzen 9 7950X3D, 96 GB of RAM, RTX 5090 with 32 GB. That’s where I work, experiment with local models, run containers, edit the blog. For gaming, I have a separate mini-PC with an RTX 4090. Those two cover 100% of what I need to do at home.

The notebook exists to cover the remaining 1%: sitting on the couch, taking it to the 3D printing office, taking it to the kitchen, short trips. It’s not meant to replace the desktop. It’s meant to give me remote access to the desktop when I’m away from it.

In practice it looks like this: open the notebook, split the Hyprland layout, left pane SSHs into the main desktop, right pane is the notebook itself. Same Omarchy on both, same keybindings, same bash. The notebook becomes an extension of the desktop, not a parallel environment I have to re-learn in my head every time I switch.

SSH from outside the house: Tailscale

Inside the local network, SSH is trivial, the notebook talks to the desktop through the internal IP. Outside the house is another story. My home IP is dynamic, opening port 22 to the internet is a terrible idea, and even with DDNS and port forwarding you’re putting SSH on the public internet for any scanner to find.

The solution I use is Tailscale. For those who don’t know: Tailscale is a mesh VPN built on WireGuard that creates a private network between your devices (the “tailnet”). Each machine runs the agent, authenticates once, and gets a fixed IP on the private network (something like 100.x.y.z). Traffic between your own devices goes peer-to-peer, encrypted by WireGuard. It doesn’t route through Tailscale’s central server, they only coordinate NAT traversal. Result: from my notebook in a café anywhere in the world, I run ssh hal9000 and land on my home desktop as if I were on the same network.

More sophisticated options exist: Cloudflare Tunnel with Zero Trust to expose services publicly with SSO auth, self-hosted headscale, raw WireGuard with manual config, Nebula, OpenVPN. Each has its use case. If you need to expose services to third parties, control granular per-identity access, or run the whole infrastructure at home without depending on anyone, those options win. In my case, it’s just notebook talking to desktop, for short periods (it’s not a full week of work, it’s a weekend debug session), so free Tailscale covers it. The free tier accepts up to 100 devices and 3 users, way more than I need.

Setup is as simple as it gets:

# On Arch/Omarchy
sudo pacman -S tailscale
sudo systemctl enable --now tailscaled.service
sudo tailscale up --ssh

The --ssh flag enables Tailscale SSH, which authenticates via tailnet identity instead of a local SSH key. Once you log in through the browser to tailscale, each registered machine can SSH into the others based on ACL policy defined in the admin panel. Zero key management.

I repeat the same on the desktop, log in with the same account, done. Both machines show up in the panel (hal9000 and hal9666 in the screenshot above, both with SSH enabled). From the notebook: ssh hal9000. From the desktop to the notebook: ssh hal9666. No port forwarding, no public IP, no port 22 exposed to the internet. If the notebook is stolen, I remove it from the tailnet with one click.

A practical detail: since the tailnet gives stable names, I added entries to ~/.ssh/config to use these short names:

Host hal9000
  HostName hal9000
  User akitaonrails

Now ssh hal9000 works from anywhere that has Tailscale connected. It’s the closest thing to “it just works” I’ve seen for remote SSH.

Why not a Mac

I have no use for macOS. As a dev, I live better in native Linux. Every tool I need has a first-class Linux version, and on macOS I’d get a second-class version via Homebrew. I don’t do iOS, so I don’t need XCode. For mobile I use Flutter or Hotwire Native, which run on any OS. iTerm2 and Ghostty on Mac are fine, but Alacritty, Kitty and Ghostty itself on Linux work just as well. Every good piece of software lands on Linux first and gets ported afterwards. Arch with AUR covers everything in a single yay.

For creative work, I haven’t done it professionally in years. DaVinci Resolve Studio on Linux is better than Final Cut Pro. Krita or Affinity Photo replace Photoshop for most cases. Clip Studio Paint on Android is better than Procreate. I simply don’t have a workflow that depends on Apple, and the App Store annoys me.

For gaming, Mac is terrible. Apple Silicon has a decent GPU for some things, but the native macOS game library is pathetic compared to Windows or Linux. Game Porting Toolkit exists, CrossOver exists, but for anyone who games seriously, it doesn’t cut it. I won’t be gaming on the Thinkpad, I have the main desktop and the mini-PC with the RTX 4090 for that. But if I ever feel like running Hollow Knight or some indie on a train, I just install Steam and let Proton handle it. Linux became a real gaming platform in the last few years, with Proton/DXVK running almost the entire Steam catalog (check ProtonDB). I recently wrote about running my emulation library in distrobox without polluting the host system. With Mac, those options don’t exist.

Another argument that always shows up: “but a Mac Mini M4 or a Mac Studio with 128GB of unified memory runs large models locally, it’s a ChatGPT replacement.” I already tested that thesis and wrote a detailed benchmark. The conclusion: expensive local hardware to run LLMs is a weekend hobby, not a production tool. The open source models that fit don’t deliver the quality Claude Opus delivers. I have a home server with AMD Strix Halo and 96 GB of unified RAM, I ran models for dozens of hours, and in my real coding flow they’re fine for simple tasks. For complex tasks, Claude Opus. Before spending $4000 on a Mac Studio justifying it with local models, actually test it first. You’ll probably end up paying for Opus again.

Why Omarchy

I’ve been using Omarchy on the desktop for months and documented the path in a series of articles. Omarchy 2.0 has its own installer with LUKS, Btrfs, Limine, snapper and SDDM already configured. I wrote about using the official ISO, about ZSH customizations with atuin, starship, secrets properly organized, about Mise for multiple languages, about LazyVim and LazyExtras, about SSH and Yubikeys, about modern TUIs. There’s also Omarchy 3 with dual GPU AMD + NVIDIA and Crush as a coding agent.

For those who don’t know: Omarchy is plain Arch Linux with a cosmetic layer on top of Hyprland/Wayland. Pre-configured with sane defaults. I could build all of it from scratch, I’ve done it several times in my life, but why redo work someone already did well? I install Omarchy, I stack the tweaks that are mine on top, and I have customized Arch in a fraction of the time.

A point I raise every time I recommend Omarchy: the documentation is excellent. There’s an official manual covering everything from installation to theme customization, keybindings, Hyprland, Waybar, the works. If you’re coming from Ubuntu or Fedora and are wary of Arch, this manual handles most of the doubts. If you’ve never touched Hyprland, open it and read around. It’s not a README dumped on GitHub, it’s a real manual with chapters and an index.

The story of this article starts in a previous thread. I recently migrated my home server, swapping an old Ubuntu box for a Minisforum MS-S1 running openSUSE MicroOS configured with Claude Code. That was the first serious experiment of letting Claude Code drive an entire infra migration, with containers, NFS, services, networking. It worked. And it left an idea in the air: why not use the same approach to configure a new notebook?

That’s what I did. I grabbed the Thinkpad, downloaded the latest Omarchy ISO, burned it to a flash drive, installed it over Windows. From zero to working desktop in under an hour. From there on, it’s all tweaking, which I documented as I went.

I’ll detail the two layers of customization: what any notebook needs, and what’s specific to this Thinkpad.

Laptop-specific configs

A notebook has problems a desktop doesn’t: battery, suspend, lid, brightness, trackpad. Those are the parts Omarchy default doesn’t cover the way I want.

Power management with TLP

Omarchy defaults to power-profiles-daemon. I swapped it for TLP, which gives granular control over CPU scaling, battery thresholds and dynamic profiles based on AC vs battery.

sudo pacman -S tlp tlp-rdw
sudo systemctl mask power-profiles-daemon.service
sudo systemctl enable --now tlp.service

The mask is necessary because upower pulls power-profiles-daemon back in if you just disable it.

Charge thresholds: 60% to start, 85% to stop. The notebook spends most of its time plugged in on the office desk, and keeping the battery at 100% 24/7 wrecks capacity over time. With 60/85, the battery spends most of its time in the healthy lithium range and still has decent usable capacity.

Profiles: balanced with balance_power on battery, performance on AC. TLP’s low-power was too aggressive on the Core Ultra 5 235U, window and terminal response became noticeable. Balanced gives the best consumption/responsiveness ratio for normal use.

There’s a gotcha: tlp auto after tlp ac doesn’t always re-apply the battery profile. I wrote a small script bound to Super+Ctrl+P that reads the current state and calls tlp bat or tlp ac explicitly:

#!/bin/bash
set -euo pipefail

profile=$(cat /sys/firmware/acpi/platform_profile 2>/dev/null || echo unknown)
on_ac=$(cat /sys/class/power_supply/AC*/online 2>/dev/null | head -1 || echo 0)

if [[ "$profile" == "performance" ]]; then
  if [[ "$on_ac" == "1" ]]; then
    sudo /usr/bin/tlp auto >/dev/null
    label="Plugged in — back to AC auto"
  else
    sudo /usr/bin/tlp bat >/dev/null
    label="On battery — normal profile"
  fi
  notify-send -t 2000 "Performance mode: off" "$label" 2>/dev/null || true
else
  sudo /usr/bin/tlp ac >/dev/null
  notify-send -t 2000 "Performance mode: on" "Forcing AC profile" 2>/dev/null || true
fi

In waybar, a custom module custom/perf shows the current state (󰓅 PERF, 󰌪 ECO, or empty for balanced) and accepts a click to toggle. The output script is tiny:

#!/bin/bash
p=$(cat /sys/firmware/acpi/platform_profile 2>/dev/null)
case "$p" in
  performance) printf '󰓅 PERF' ;;
  low-power)   printf '󰌪 ECO'  ;;
  balanced|"") printf ''       ;;
  *)           printf '%s' "$p" ;;
esac

Suspend, hibernate, and the lid

This is what changed compared to the notebooks I used in 2015. The T14 Gen 6 closes the lid, suspends, and wakes up when you open it. No bug, no weird delay, no needing to log in again mid graphical session. Hyprlock kicks in after the suspend, accepts fingerprint or password, and brings me back to the desktop in seconds. This is the behavior Apple has had for years and that on Linux used to be an adventure. In 2026, on modern hardware with kernel 6.11+, it just works.

Mem sleep mode is s2idle (no deep sleep). Hibernation is enabled via a ~30 GB Btrfs swapfile and resume=/dev/mapper/root resume_offset=... on the Limine kernel cmdline. I rarely use it, but it’s there.

Hypridle has aggressive timeouts for a laptop:

• 2.5 min → screensaver
• 5 min → lock
• 5.5 min → DPMS off + keyboard backlight off

After lock, another 5 min and the screen turns off completely. On unlock, it restores screen and keyboard brightness to the previous level. These timings are much shorter than on the desktop (20-40 min there). The battery difference over a day is measurable.

Brightness and keyboard backlight

brightnessctl controls both. Fn+Space toggles the keyboard backlight across three levels. Hypridle saves the current level before turning off and restores it on resume. Command I use in the script:

brightnessctl -sd '*::kbd_backlight' set 0    # save and turn off
brightnessctl -rd '*::kbd_backlight'          # restore

Touchpad

In hypr/input.conf:

touchpad {
    natural_scroll = true
    clickfinger_behavior = true
    disable_while_typing = true
    scroll_factor = 0.4
}
gesture = 3, horizontal, workspace

clickfinger_behavior swaps the 2-finger click as right-click (more comfortable than hitting the lower-right zone). disable_while_typing is basic palm rejection. Three-finger horizontal swipes switch workspaces, the most useful gesture in Hyprland.

Thinkpad-specific configs

This is what’s specific to this model. Some things modern Linux handles on its own, others need explicit configuration.

Fingerprint sensor

Goodix MOC 27c6:6594, works with libfprint 1.94.9+ on kernel 6.11+. Package is fprintd.

sudo pacman -S fprintd
fprintd-enroll                        # right index finger by default
fprintd-enroll -f left-index-finger   # other finger

To make sudo accept fingerprint, I added auth sufficient pam_fprintd.so above the pam_unix.so line in /etc/pam.d/sudo. With sufficient, if the fingerprint passes, it authenticates directly. If it fails or I hit ESC, it falls back to the password prompt. This is genuinely worth it: dozens of times a day, sudo pacman -Syu or sudo systemctl restart something, and I just touch the sensor.

On hyprlock, I use hyprlock’s native configuration, not PAM:

auth {
    fingerprint {
        enabled = true
        ready_message = Scan fingerprint or type password
        present_message = Scanning...
        retry_delay = 250
    }
}

The PAM path gives a double prompt. Configured natively, hyprlock accepts fingerprint or password, whichever succeeds first unlocks.

On SDDM (initial login), I kept password only. The reason: the login password unlocks the GNOME keyring, and the fingerprint can’t provide plaintext for that. Once the keyring is unlocked, hyprlock can use fingerprint without issues.

In practice, sudo looks like that. Touch the sensor, authenticate, move on.

Brazilian Thinkpad keyboard

The Brazilian Thinkpad keyboard has an annoying quirk. The /? key sits where the right Ctrl would be (keycode 97), not in the traditional ABNT2 AB11 position (keycode 89). If you use the standard br(abnt2) layout, that key is inaccessible. It literally prints nothing.

The solution is the br(thinkpad) variant that exists in /usr/share/X11/xkb/symbols/br:

xkb_symbols "thinkpad" {
    include "br(abnt2)"
    name[Group1]="Portuguese (Brazil, IBM/Lenovo ThinkPad)";
    key <RCTL> { [ slash, question, degree, questiondown ] };
};

In hypr/input.conf:

kb_layout = br
kb_variant = thinkpad
kb_model = thinkpad60

And system-wide for TTY/X11:

sudo localectl set-keymap br-abnt2
sudo localectl set-x11-keymap br thinkpad thinkpad60

I wrote a small Python that reads raw scancodes from /dev/input/event* to diagnose these quirks. Useful when a key decides not to work and you need to find out if it’s hardware, kernel, or xkb.

Input method: fcitx5

Alongside the layout I install fcitx5. Input method, for those who don’t know, is the layer that turns a key sequence into characters. It handles deadkeys (tilde for nasalization, acute accent, circumflex), composing characters that aren’t on the keyboard (ç, Ç, uppercase accented letters), emoji support. In Qt or GTK apps, the input method also drives context menus for cedilla and accents.

Packages:

sudo pacman -S --needed fcitx5 fcitx5-configtool fcitx5-gtk fcitx5-qt

And the environment variables for toolkits to find fcitx5. I created ~/.config/environment.d/fcitx.conf:

INPUT_METHOD=fcitx
QT_IM_MODULE=fcitx
XMODIFIERS=@im=fcitx
SDL_IM_MODULE=fcitx

systemd’s environment.d is loaded before graphical sessions, so Brave, Alacritty, VS Code and any GTK/Qt app pick it up automatically. I enable autostart in Hyprland:

exec-once = fcitx5 -d

Typing ~a produces ã, 'e produces é, ç works as it should in every app. On a Brazilian Thinkpad keyboard, this is the difference between typing Portuguese naturally or hunting for each character.

SOF audio

Realtek ALC3306/ALC287 codec via Sound Open Firmware. Without sof-firmware, the kernel module loads but the DSP never boots and PipeWire silently falls back to auto_null. Result: you think the speaker is on mute, but actually PipeWire has no device at all.

sudo pacman -S --needed sof-firmware alsa-ucm-conf pipewire pipewire-pulse wireplumber

If you need to force SOF instead of legacy HDA:

echo "options snd-intel-dspcfg dsp_driver=3" | sudo tee /etc/modprobe.d/alsa.conf

Reload without reboot:

sudo modprobe -r snd_sof_pci_intel_mtl
sudo modprobe snd_sof_pci_intel_mtl
systemctl --user restart wireplumber pipewire pipewire-pulse

Firmware updates via fwupd

fwupdmgr update works, but with Limine there’s a gotcha: fwupd tries to write to /boot/EFI/systemd/ or /boot/EFI/arch/, which don’t exist. The workaround:

sudo mkdir -p /boot/EFI/arch
sudo fwupdmgr update -y --no-reboot-check
fwupdmgr get-history  # should show "Success"

HiDPI / fractional scaling

14" panel at 1920x1200, Hyprland’s free resolution. Omarchy’s auto 1.5x felt too chunky. I pinned it at 1.333x:

env = GDK_SCALE,1
monitor=,preferred,auto,1.3333,vrr,2

Effective 1440x900. GTK with GDK_SCALE=1 renders 1:1 with Hyprland (no double magnification). VRR mode 2 only in fullscreen, because LCD panels tend to flicker on static content with VRR active.

Brave and Chromium flags to render well at this scale:

--ozone-platform=wayland
--enable-features=WaylandFractionalScaleV1,UseOzonePlatform,VaapiVideoDecoder,VaapiVideoEncoder
--enable-gpu-rasterization

VAAPI makes a real difference on YouTube battery life.

Tuning Omarchy to be mine

Omarchy comes with good defaults. What I adjust stacks on top of them, without touching ~/.local/share/omarchy/ (which is clobbered by omarchy-update). All customization lives in ~/.config/.

Infra: Btrfs, snapshots, Snapper

Omarchy already ships with Btrfs and separate subvolumes:

@home separated means ~/.cache, ~/.config/BraveSoftware, etc. don’t bloat root snapshots. Snapshot is for system, not profile.

Swap: 4 GB zram at priority 100 (hit first), plus a 30 GB swapfile at priority 0 (enables hibernation, sized to match RAM).

The snapshot stack: Snapper takes a snapshot before and after every pacman -Syu via snap-pac. limine-snapper-sync writes those snapshots into the Limine menu, so you can boot into a previous snapshot to roll back. If something breaks after an update, you hold a key on boot, pick the pre-update snapshot, boot read-only to verify, and if it’s good, run snapper rollback.

Omarchy leaves snapper-cleanup.timer and snapper-boot.timer disabled by default. I enabled both and configured retention to fit on a 1 TB SSD without blowing up:

TIMELINE_LIMIT_HOURLY=10
TIMELINE_LIMIT_DAILY=0
TIMELINE_LIMIT_WEEKLY=1
TIMELINE_LIMIT_MONTHLY=1
NUMBER_LIMIT=50

A detail that costs debug time if you forget: Docker and Ollama write gigabytes into /var/lib/docker and /var/lib/ollama. If that falls inside @, snapshots go catastrophic. Each Docker image or Ollama model downloaded triples in size. I created nested subvolumes for both, with chattr +C to disable CoW:

sudo btrfs subvolume create /var/lib/docker
sudo chattr +C /var/lib/docker
sudo btrfs subvolume create /var/lib/ollama

This has to be done BEFORE Docker or Ollama writes any data. If they already have data there, you need to migrate.

NFS to the Synology NAS

My Synology exposes three volumes. On the desktop, I mount them the usual way. On the notebook, it has to be more defensive. The notebook moves around, forgets networks, connects to public WiFi. Notebook’s fstab:

nfs4  _netdev,noauto,nofail,x-systemd.automount,x-systemd.idle-timeout=10min,x-systemd.mount-timeout=15s,noatime,nodiratime,nconnect=4,actimeo=10,soft,timeo=30,retrans=2

Critical difference vs the desktop: soft with short timeouts (doesn’t hang forever), x-systemd.idle-timeout=10min (auto-unmount when idle), no network-online.target in the require (doesn’t slow boot). Practical result: cd /mnt/gigachad at home mounts lazily, away from home it fails fast without locking the shell.

Another important detail: my user on the notebook has UID 1026, which matches the share permissions on the Synology. Linux defaults to creating users at 1000, the Synology enforces identity via UID on the wire. If the UIDs don’t match, you can’t read the files, or worse, you write as nobody. I ran the usermod/groupmod from a TTY (with the user logged out) to remap the user to 1026/1026 and did chown -R on /home.

Public WiFi hardening

The notebook will leave the house. In an airport or café, I don’t want to announce hostname, don’t want a trackable MAC, don’t want a service listening on an open port.

/etc/NetworkManager/conf.d/00-macrandomize.conf:

[device]
wifi.scan-rand-mac-address=yes

[connection]
wifi.cloned-mac-address=stable
ethernet.cloned-mac-address=stable
connection.stable-id=${CONNECTION}/${BOOT}

ipv6.ip6-privacy=2
ipv6.addr-gen-mode=stable-privacy

Random MAC per scan (passive anti-fingerprinting). Stable cloned MAC per SSID (captive portals don’t re-prompt you every time) but different MACs between networks. IPv6 with temporary addresses and interface ID derived from a stable secret, not from the MAC (no EUI-64 leak).

/etc/systemd/resolved.conf.d/hardening.conf:

[Resolve]
LLMNR=no
MulticastDNS=no

Kills hostname broadcast on LLMNR and mDNS. I also disable avahi:

sudo systemctl disable --now avahi-daemon.service avahi-daemon.socket

UFW firewall:

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw --force enable

SSH server is already off by default on Omarchy. Nothing listening. Even if the firewall leaks, there’s no surface.

All of it rolls into a single initial setup ritual.

SSH agent persistence (keyring + keychain)

If you use SSH seriously, you want to type the passphrase once per boot and have ssh, git push, scp stay silent for the rest of the day. On current Arch there’s a catch: gnome-keyring 50 dropped its SSH component, and the replacement (gcr-ssh-agent) is a plain in-memory agent with no passphrase persistence. The “remember this key” checkbox you saw in old guides simply doesn’t exist anymore.

The combination that works has three pieces:

1. gcr-ssh-agent.socket managed by systemd serves SSH_AUTH_SOCK at $XDG_RUNTIME_DIR/gcr/ssh
2. pam_gnome_keyring on SDDM login unlocks the keyring with the login password (used by GUI apps like Brave, not by SSH)
3. keychain (wrapper) keeps an ssh-agent alive across logouts, overrides SSH_AUTH_SOCK to point at that persistent agent, and caches the PID in ~/.keychain/<host>-sh

First, install the keyring:

sudo pacman -S --needed gnome-keyring seahorse keychain

Then edit /etc/pam.d/sddm so the login password unlocks the keyring automatically. Add at the top:

-auth      optional    pam_gnome_keyring.so

And at the end:

-session   optional    pam_gnome_keyring.so    auto_start

The leading hyphen (-auth, -session) marks it optional, so if the module fails to load, login doesn’t break.

Pin SSH_AUTH_SOCK for graphical and TTY sessions via ~/.config/environment.d/ssh-agent.conf:

SSH_AUTH_SOCK=%t/gcr/ssh

%t resolves to $XDG_RUNTIME_DIR, something like /run/user/1026. Enable the gcr socket and user linger (so user daemons survive logout):

systemctl --user enable --now gcr-ssh-agent.socket
sudo loginctl enable-linger $USER

Finally, in ~/.config/bash/init.sh, keychain starts or reuses the agent:

if command -v keychain &>/dev/null && [[ -r ~/.ssh/id_ed25519 ]]; then
  eval "$(keychain --eval --quiet ~/.ssh/id_ed25519)"
fi

--eval emits shell assignments (SSH_AUTH_SOCK, SSH_AGENT_PID) pointing to keychain’s agent, overriding the gcr path set by environment.d. --quiet silences the banner once the key is loaded.

Flow in practice: boot → first terminal → keychain prompts for the passphrase once → key stays loaded. Logout → login again → new shells reattach to the same agent (via ~/.keychain/<host>-sh) → zero prompts. ssh-add -l confirms the key is there, echo $SSH_AUTH_SOCK confirms it’s at keychain’s path.

When you’ll re-enter the passphrase: reboot, ssh-add -D, or keychain --clear. In normal use, once a day.

Bash instead of ZSH

On the desktop I use ZSH. On the notebook I went with Bash to align with Omarchy’s default, without having to maintain a parallel stack of modular layers. ~/.bashrc is a symlink to ~/.config/bash/bashrc, which sources Omarchy’s defaults first and then stacks my customizations:

source ~/.local/share/omarchy/default/bash/rc
source ~/.config/bash/envs.sh
source ~/.config/bash/aliases.sh
source ~/.config/bash/mounts.sh
source ~/.config/bash/init.sh
source ~/.config/bash/secrets    # gitignored, chmod 600

envs.sh has what’s mine: OpenRouter base URL, Ollama pointing to the LAN GPU box (192.168.0.14), AWS region, Hugo analytics, zoxide and SSH agent configs. aliases.sh has TLP shortcuts, an alias for shell-gpt via Docker, and functions to harden PATH when running makepkg or yay (prevents binary injection via a malicious user config).

init.sh does the integration work. Atuin with a manual bind for Ctrl-R (so the up arrow keeps bash’s default history-search, which I use more). Keychain loading ~/.ssh/id_ed25519 once per boot and reusing it across shells (no need to re-authenticate SSH every time). Blesh if installed (ZSH-style autosuggestions for Bash). And a function that has PROMPT_COMMAND set the window title to the current pwd and the running command:

__title_idle() { printf '\033]2;%s\007' "${PWD/#$HOME/~}"; }
__title_busy() {
  local cmd="${BASH_COMMAND}"
  [[ "$cmd" == "__title_"* || "$cmd" == *"PROMPT_COMMAND"* ]] && return
  printf '\033]2;%s — %s\007' "${PWD/#$HOME/~}" "$cmd"
}
if [[ -n "${PROMPT_COMMAND-}" ]]; then
  PROMPT_COMMAND="__title_idle; ${PROMPT_COMMAND}"
else
  PROMPT_COMMAND="__title_idle"
fi
trap '__title_busy' DEBUG

Idle, the title shows just the pwd. With a command running, trap DEBUG catches the BASH_COMMAND and updates the title. Waybar’s hyprland/window picks that up and displays it.

For the modern Rust toolbelt, the list:

sudo pacman -S --needed \
  eza bat fd ripgrep sd git-delta dust procs bottom duf tokei hyperfine \
  zoxide atuin tldr starship

eza replaces ls. bat replaces cat with syntax highlighting. fd replaces find. ripgrep replaces grep. sd replaces sed. delta plugs into git for colored side-by-side diff. dust for visual du. procs for ps. btm (bottom) for top. duf for df. tokei counts lines of code. hyperfine for command benchmarking. zoxide is cd with memory (very useful). atuin is shell history with encrypted sync (I point it to my home server via sync_address = "http://192.168.0.90:8888"). starship is the prompt. tldr is a man page in 10 lines.

The atuin key has to be backed up offline. If you lose it, you lose the encrypted history. I saved mine in my self-hosted Bitwarden (Vaultwarden), documented in the Bitwarden self-hosted article.

Git pipes diff through delta:

[core]        pager = delta
[interactive] diffFilter = delta --color-only
[delta]       navigate side-by-side line-numbers hyperlinks, light=false
[merge]       conflictstyle = zdiff3

Hyprland and Waybar

Visual and UX customizations on top of the default.

In hypr/looknfeel.conf, smaller gaps (2/5 vs the default 5/10), slide animation between workspaces, VFR on (reduces consumption when the screen is static), allow_session_lock_restore (if hyprlock crashes, it goes back to the lock screen instead of dumping you on the desktop).

In hypr/bindings.conf:

On Waybar, three custom modules I recommend:

Window title: the hyprland/window module shows what’s focused. In bash, PROMPT_COMMAND updates the title to something like “~/Projects/blog — hugo server”. So in waybar the directory and running command show up, without me needing to look at the terminal.

"hyprland/window": {
    "format": "{title}",
    "max-length": 50,
    "rewrite": { "(.{47}).+": "$1…" }
}

Perf mode: custom/perf runs a script that reads TLP state and shows an icon:

"custom/perf": {
    "exec": "~/.config/scripts/perf-waybar.sh",
    "interval": 3,
    "format": "{}",
    "on-click": "~/.config/scripts/perf-toggle.sh"
}

Empty for balanced, 󰓅 PERF for performance, 󰌪 ECO for low-power. Click toggles. Useful to quickly see what mode the machine is in.

Claudebar: I integrated claudebar as a custom module that shows the Claude Code session usage % and extra spend inline in waybar. No need to open another window.

The clock:

"clock": {
    "format": "{:L%A %d %B - %H:%M}",
    "format-alt": "{:L%A W%V %Y - %H:%M}"
}

Shows “Thursday 17 April - 14:32” by default, click toggles to “Thursday W16 2026 - 14:32” with the ISO week number, useful for planning articles and tasks.

Self-hosted Atuin

Atuin runs on my home server at 192.168.0.90:8888. I made a separate account for the notebook (akitaonrails-thinkpad), I don’t mix history with the desktop on purpose. atuin sync runs every 5 minutes and all history is encrypted before going to the server. The server doesn’t see the commands, only encrypted bytes.

Conclusion: picking a notebook for Linux

One thing I learned on this journey: picking a notebook to run Linux comfortably is not trivial. You have to check the ArchWiki for your specific model before buying. The T14s Gen 6 AMD page, for example, catalogs every hardware quirk and how to work around it. Without that reference, you discover the problems during install.

Rule I follow: never buy a freshly-released model. Let 6 to 12 months go by after launch. That gives the community time to iron out bugs, drivers to land mainline, the ArchWiki to have a decent page, libfprint to include the fingerprint sensor, and the kernel to cover WiFi and the IR camera. Buying on day one means signing up as a beta tester.

Brands with a decent Linux track record: Lenovo (especially Thinkpad, they ship Ubuntu and Fedora pre-loaded), Dell (the XPS and Latitude lines), Asus (Zenbook and ROG have reasonable support), Framework (built for Linux from the factory). Any recent Mac is cut off from Linux mainline or runs through Asahi with caveats.

The Thinkpad T14 Gen 6 isn’t the prettiest notebook I could have bought. But it’s rugged, it has the right ports, the fingerprint sensor works, and the plastic shell lets it fall, get scratched, travel in a backpack without making me nervous. To serve as a remote debug companion, it’s what I needed. If that’s your use case too, I recommend it. If you want OLED and metal, go Zenbook or T14s. Every premium comes with a compromise, no Linux notebook in 2026 is perfect.

Everything I described here is versioned in a private repo of mine. If I need to reinstall tomorrow, I run half a dozen steps in order and I’m back at the same place. That’s the whole point of keeping config in Git: notebook is commodity, config is mine.

The pitch is strong. It’s a multimodal video generation model that takes text, up to 9 reference images, 3 reference video clips, and 3 audio tracks all in the same call. On top of that, it produces up to 15 seconds of output with native synced audio (dialogue, sound effects, ambient music), supports video extension, surgical editing of characters and objects, and prompt-driven camera planning. ByteDance is pitching it at commercial advertising, explainer videos, film production, e-commerce, and gaming. Translation: for content creators trying to climb out of the “random meme” tier and start producing things that look professional.

If you want a quick visual tour before going further, here’s a great walkthrough from Stefan, of the Stefan 3D AI Lab channel, which I already plugged in the post on AI 3D modeling:

The main interface is Dreamina’s “Photo Studio,” where you pick the generation mode and stack your references. So far I’ve only tested the standard multi-reference mode.

Test 1: lip sync with podcast audio

My first test was simple. I grabbed a few seconds of the opening bumper from my podcast The M.Akita Chronicles, which is generated by AI through an ElevenLabs v3 pipeline, passed in my new anime-style avatar as a reference image, and asked Seedance to make the character lip-sync and gesture along with that audio.

The result is decent. All of that came from a single still image. The lip sync follows the speech reasonably well, the expression has some life to it, and the hand gesture lands at the right beat. But this is a YouTube short or Instagram clip at best. Not the kind of animation you’d open a serious video with.

The feature that actually matters: video reference

And here’s what I think is the only real use of these models for actual work: feeding in video as a reference. Text alone is never going to be precise enough. You can describe a scene with whatever vocabulary you want, swap adjectives, specify lens, angle, framing, and every generation will still hand you back something subtly different. It’s a slot machine. Great for memes, terrible for production.

Most amateurs see one random clip that comes out nice on the first try and go straight to flooding their X and Instagram feeds with it. That’s not control. That’s luck. For film, for ads, for show openers, for anything where the next shot has to match the previous one, you have to be able to tell the model: “do exactly this motion, with exactly this camera.” That’s where reference video changes the game, because it replaces hours of 3D modeling, rigging, animation, and rendering that would cost a fortune in a studio.

For my test, I recycled the model I’d already generated in the post on AI 3D with Hunyuan and Nano Banana, where I tested how far you can take 3D modeling from a prompt and even hit “print” on the result. But for something more cinematic I wanted a model with a proper animation cycle. So I grabbed the Darth Talon model from Sketchfab, which has a short, well-made motion sequence, just because it looked cool.

I imported the FBX into Blender, set up a basic camera and lighting, and did a quick Cycles render. Nothing pretty about it, just a sketch to feed in as motion and framing reference.

I uploaded that render to Seedance as a video reference along with three t-pose images of the character (front, back, left side) for visual consistency:

First attempt, with a generic prompt:

Not bad. The character keeps its identity, the lighting got a serious upgrade over my raw render, and the clip is coherent. But the motion and camera position drifted noticeably from the reference. The model basically read it as “there’s a character here that looks like that doing something like that,” and improvised the rest.

Then I remembered Stefan’s tutorial: the trick is to literally drop something like Follow exact motion and camera from reference video into the prompt. Tried again:

A lot better. It’s clearly following the reference choreography, the camera angle matches in several beats, and the character looks more polished than anything I could hand-render in a quick Cycles sketch. But it’s still not a frame-for-frame match. In some shots the model is still taking liberties with timing, position, and framing. There are other prompting tricks and parameters to play with, but I stopped here to write this up while the launch is still fresh.

ComfyUI, Runway, and the rest of the ecosystem

Seedance 2.0 also already shipped as a ComfyUI cloud node, which is great for anyone who wants to drop the model into a bigger pipeline without bouncing between web UIs. If you’ve already got an image-gen workflow built in ComfyUI, this is genuinely handy.

And for anyone who wants volume without doing the per-generation credit math, Runway is offering Seedance 2.0 as part of their Unlimited plan, which runs around $76/month annual or $95/month monthly. “Unlimited” means no per-generation charge, but you do trade for lower queue priority at peak hours and an output storage cap. If you iterate a lot, the subscription model makes way more sense than buying credits one at a time.

The price question

Seedance’s own pricing is credit-based. The Standard plan is $24.90/month annual or $49.90/month monthly, and gets you 2,500 credits a month. Every 10 seconds of generated video burns 6 credits.

The math is brutal. With 2,500 credits on Standard, you get about 416 ten-second generations a month. Sounds like a lot, until you remember my own test: it took at least two tries to get close to what I wanted, and even then it wasn’t a one-to-one match. In practice, plan on 3 to 5 regenerations per usable scene. That drops your real number to something like 80 to 140 short clips per month. Maybe enough for a solo creator doing short-form content. Nowhere near enough for production that needs minutes of polished, continuous video.

The higher tiers scale credits roughly linearly, but the same problem holds: at today’s price, Seedance 2.0 is still a tool for short clips and well-made memes, or for pre-vis sketches before you send the real thing to an actual studio. It doesn’t replace continuous professional production. For that use case, Runway with Unlimited is honestly the cheaper option in practice. Full pricing table here.

What nobody wants to talk about: restrictions and deepfakes

Two things I don’t want to leave out. First: Seedance 2.0 took its sweet time getting to the public partly because ByteDance is being a lot more conservative than the competition when it comes to moderation. The model blocks generation that uses real people as reference (celebrities, politicians, public figures) and filters strong-IP characters from Disney, Marvel/DC, Nintendo, and registered trademarks generally. If you want the details of what passes, what doesn’t, and why ByteDance went that way, this MindStudio piece breaks it down. It makes sense: ByteDance operates across dozens of jurisdictions, lives under heavy regulatory scrutiny worldwide, and has the elephant in the room of Hollywood pushing the entire industry with copyright suits over training data and generation. The release got delayed and arrived with heavier filters precisely because of that.

Second point, and this is the one that matters more: for you, the average reader, the message is that deepfakes are no longer hypothetical. Photos on the internet stopped counting as proof a long time ago, because image models like Nano Banana, Hunyuan, and the rest deliver realism that fools anyone. Now video sits in the same bucket. What we just saw above, where a couple of references and two prompts get you a coherent animated scene in minutes, is just the start. The average person, with no training, isn’t going to be able to tell generated video from real video at a glance anymore. And Seedance’s filters only block above-board, legitimate use. Anyone with bad intent will always find a path through open source models, workarounds, or less strict platforms. That’s the world we’re in now, and the sooner we collectively accept it, the better off we are.

No, this doesn’t replace real artists

Let me cut off the usual speech before it shows up in the comments. No, Seedance doesn’t replace VFX artists, directors, editors, none of that. This is a hammer. The tool is the same for everybody. What comes out the other end depends entirely on who’s swinging it.

Anyone who’s actually studied framing, pacing, continuity, art direction, editing, color, composition is going to use Seedance to 10x the work they were already capable of delivering. Anyone who hasn’t studied any of that is going to crank out the same Instagram meme they were already cranking out, just 10x faster. The AI reflects who you are. If you’re a real artist, the tool multiplies you. If you only think you are, the tool multiplies the mediocrity. There’s no miracle here.

This is exactly why the whole “now everyone can be a film director with AI” wave is the same song we’ve heard with the digital camera, with Photoshop, with After Effects, with Premiere, and with every powerful tool that’s shown up in the last 30 years. The technical bar drops. The bar of taste, reference, and study sits exactly where it always has. Good people get more productive. Everyone else still needs to hire real ones to do real work.

Where this goes next

Seedance 2.0 is clearly a step in the right direction, but it’s not the tool that replaces professional production yet. Text alone stays imprecise, video reference helps but still drifts, and the price per minute of polished output is high relative to what you actually get. The good news is that this story always rhymes: Google has Veo, Runway has its own model and is also reselling Seedance, Kling and Hailuo from China keep pushing, and the open source models are gaining ground. OpenAI had Sora, but they shut it down. As competition tightens, quality goes up and price comes down. A year or two from now we’ll be looking back at this post and finding what I just showed pretty quaint. Same as it ever was.

That’s the April 6th episode of the The M.Akita Chronicles podcast, already generated with the new ElevenLabs v3 pipeline. Subscribe to the show on Spotify so you don’t miss new episodes like this one.

When Qwen3 TTS dropped, back around January this year, everyone on Twitter/X and on the AI newsletter circuit was shouting “ElevenLabs killer”. There’s a Medium piece claiming it’s the first real open source threat to ElevenLabs. There’s a post on byteiota saying 3-second voice cloning beats ElevenLabs. There’s an analysis on Analytics Vidhya calling it the most realistic open source TTS released so far. The enthusiast internet consensus was: we finally have open source that can go toe-to-toe with ElevenLabs, the tables have turned, it’s just a matter of time.

I decided to test it on my own production pipeline, as usual. I built out a whole pipeline on top of Qwen3 TTS 1.7B to generate the weekly podcast for The M.Akita Chronicles, and I documented the behind-the-scenes on the serving-AI-in-the-cloud post. Anyone who wants the detail on cold starts, voice cloning, and sampling parameters that change from one mode to another, that’s the place. I’m not repeating it all here.

This post asks a different question. After almost two months running that setup in production, shipping an episode every Monday, last night I finally pulled the plug on Qwen3 and switched the whole thing to ElevenLabs v3. Let me tell you why.

What didn’t work on Qwen3

Between February 15 and March 30, I made dozens of commits fine-tuning the podcast flow: prompts, sampling parameters, generation order, leading silence on the voice clone sample, volume normalization, pronunciation of tech acronyms. Fixing Marvin’s voice, which was clipping the first syllable because the reference audio started without any lead-in silence. Tuning Akita’s voice to sound more confident and assertive. Adding crossfades between section jingles. Sorting out the “podcast” pronunciation so it didn’t come out as “pódcast”. Making the script generator prefer Portuguese over gratuitous anglicisms so the TTS wouldn’t choke. Each of those fixes was a multi-hour session of listening to audio, regenerating, tweaking a parameter.

The result came out acceptable. “Acceptable” in the sense that I managed to ship every episode without re-recording anything by hand. But if you listen closely, the Qwen3 voice has that unmistakable AI-generating-audio quality. Flat intonation, uniform rhythm. Over long stretches you can feel it’s a machine talking. Good enough to ship, but miles away from what you’d hear on a professional podcast made by humans.

The worst problem was English pronunciation. My podcast covers tech news, so terms like “MCP”, “RAG”, “Claude Opus”, “GPT-5”, “open source” show up in every conversation. Qwen3 took those terms and pronounced them with a Brazilian accent, something like “oh-pen-ee-sohrss-eh”, that kind of thing. Unlistenable for the audience. The workaround I had to implement was manually mapping in the script-generation LLM prompt which English words to swap for Portuguese equivalents. The prompt today has a whole section split between “keep in English” (proper nouns, brands, terms already adopted into Brazilian tech lingo) and “translate to Portuguese” (gratuitous anglicisms), looking roughly like this:

**REPLACE with Portuguese** (common English words that have natural
Brazilian Portuguese equivalents):
- "update" → "atualização"
- "release" → "lançamento"
- "feature" → "recurso/funcionalidade"
- "deploy" → "implantação" or just "colocar em produção"
- "trade-off" → "dilema" or "escolha"
- "performance" → "desempenho"
- "default" → "padrão"
- "insight" → "percepção/sacada"
- "skills" → "habilidades"
- "approach" → "abordagem"
- "highlights" → "destaques"

I call this “scrubbing anglicisms so the TTS doesn’t choke”. The funny part is I didn’t want that level of restriction on my script. I wanted the voice to just pronounce “update” when “update” was the natural word. Since the model couldn’t, I had to mutilate the podcast’s vocabulary so the final result was actually listenable. It’s a band-aid, the kind you add hoping to rip it off later when the tech grows up.

The experience with ElevenLabs v3

Yesterday afternoon I opened an ElevenLabs account, bought the Pro plan ($99/month), and started experimenting with the eleven_v3 model, which shipped in February of this year. Thirty minutes later I had a proof of concept running, and about two hours later the whole podcast system was migrated. The effort gap is an abyss.

The technical details of the migration ended up in an internal project doc, so here’s the summary comparison table that actually matters:

Look at the last line. Qwen3 costs less than ten cents of a dollar per episode. ElevenLabs costs almost thirty times more. And it’s still worth it. The other lines on the table solve problems that were eating hours of my week. I no longer need to write code to scale GPUs in the cloud, I don’t need to wait for the machine to spin up every time, I don’t need to babysit a model or rebuild a Docker image when the weights change. The operation became a single line of configuration.

And here’s the interesting part: the inline emotion tags. The v3 model accepts markers like [sighs], [sarcastically], [dryly], [excited], and changes the delivery accordingly. This works in more than 70 languages, including Brazilian Portuguese. It transformed script generation, because now I can ask the LLM that writes the script to drop emotional tags at the right moments, which gives a liveliness Qwen3 couldn’t deliver in a million years. A concrete example of what comes out of the script later:

AKITA: Isso é simples. [dismissive] Quem ainda acredita que Bitcoin
vai morrer não tá prestando atenção.
MARVIN: [sighs] Mais uma semana, mais uma leva de devs confiando
cegamente em pacotes npm. Previsível.

I even have two separate tag palettes, one for Akita (expressive but controlled, uses [excited], [dismissive], [emphatic]) and another for Marvin (stoic, just [sighs], [sarcastically], [tired], [dryly]). That’s all encoded in the script generation prompts so the LLM knows which character is allowed to use what.

About Marvin’s voice

For listeners who are already used to Marvin’s voice, don’t worry: I cloned him on ElevenLabs using the same audio sample I had used to train on Qwen3. It’s the same voice. It just sounds even better now, because the ElevenLabs model captures nuance and prosody that Qwen3 couldn’t deliver.

Listen to it and tell me

To prove the talk is real, here’s Monday’s episode, April 6th, already generated with the new pipeline:

If you’re already a podcast listener on Spotify and you’ve heard previous episodes made with Qwen3, compare them and tell me in the comments whether you notice the difference or whether it’s the same to you. I’m genuinely curious how much of this is my trained ear after hours of listening to TTS audio and how much is an obvious difference for a casual listener.

Starting next week, every episode of the podcast will be generated by ElevenLabs v3. The newsletter is already plugged into the new pipeline, the scheduled jobs to pre-warm the RunPod GPU have been disabled, and the legacy Qwen3 code stays in the repo as plan B in case v3 starts acting up chronically. Two file edits and I can switch back. I probably never will.

The part about dubbing the YouTube videos

Now the hook for the second half of this post. On the anniversary piece I published earlier today, I told the story of how Claude Code translated nearly half my blog to English over a weekend. In the same spirit, I went after dubbing the videos on the Akitando channel.

The channel has 146 episodes, around 96 hours of technical content in Portuguese, and more than 500k subscribers. I already had the subtitles translated (one curated .srt per episode), and YouTube even offers automatic English dubbing. But the result is like Google Translate circa 2015: gets the idea across, nobody wants to listen for very long.

I tested ElevenLabs’ three voice approaches and only one worked. Speech-to-Speech converts a voice but doesn’t translate. The Dubbing API does translate but creates the voice on its own, with no way to force a specific clone. Only Text-to-Speech could solve this: take the English .srt I already had, send each block to the TTS endpoint with my cloned voice, and assemble the audio aligned to the original video.

But having a translated .srt is not enough. Raw subtitle translation doesn’t survive TTS. Passages with source code, URLs, hex hashes, lists of shell commands — the model switches to spelling mode and the audio comes out at twice the expected duration. Translations that run too long blow past the time window and need to be condensed so the voice doesn’t sound like an auctioneer. Truncated SRTs need to be completed. And after each fix, you re-run the pipeline, listen, find the next problem, fix it, re-run. The whole thing was a cycle of interruptions, manual corrections, and re-runs — a far cry from the “press a button and get a dub” fantasy.

The big challenge was the accent. My cloned voice was trained on Brazilian Portuguese audio, so when it tries to speak English, the thick accent comes through. ElevenLabs has an [American accent] tag that works on v3, but on top of a voice trained in another language it’s weak — the Brazilian accent still bleeds through. The fix was to train a second voice of mine, English only. I recorded a few minutes in my best American accent, uploaded it as a separate Instant Voice Clone in my account, named it “Akita English”, and set the pipeline to use that voice by default. The result comes out more natural, no tag needed, and the voice identity is still mine.

The honest ceiling on my English voice quality

Let’s be frank. I’m a Brazilian who speaks English, not a native speaker, and I don’t have the time or patience to drill pronunciation for hours. A clone can only ever be as good as the source sample — if the source is a non-native reading a script, the clone inherits every imperfection. No clone is going to make me sound like “Fabio speaking perfect English” because that sound doesn’t exist for the model to learn from.

To get to the version that’s running today, I recorded five different training samples throughout the day, each with a different rhythm and script, and ran an A/B battery against the original Portuguese. None sounded native, and that was never the goal. The realistic bar was more modest: sound recognizably like me, read technical content without stumbling, and carry a full one-hour episode without wearing the listener out. The winner ended up being the first iteration, the original “Akita English”. Far from perfect, but good enough to ship. Swapping it out later is literally one line in the config file.

What surprised me most along the way was realizing that the rhythm of the training sample matters more than the timbre. The clone learns the cadence of whoever recorded it: iteration 5 came out 25% slower than iteration 2 reading the exact same text, purely because I recorded that particular sample more slowly. For dubbing tech videos, the right target is the rhythm of a “YouTube tech explainer” — brisk, energetic, around 150 words per minute. No audiobook pacing.

How I align audio to the original video

A non-obvious point: spoken English tends to be longer than equivalent Portuguese. If you just generate the audio for each subtitle and paste it at the original timestamp, the dub drifts later and later. With 30 minutes of video you’re already several seconds out of sync.

My approach was to attack the problem in layers. The first version split the SRT into smaller blocks, called “chunks”. Each chunk maxed out at 700 characters (so v3 wouldn’t hallucinate) and could only cut at a sentence boundary, never mid-sentence. A simple algorithm accumulated subtitles into a buffer until it was about to overflow, then walked back looking for the last cue whose text ended with ., !, or ?, and flushed only up to there. The cues still in the buffer got rolled into the next chunk. Each chunk also stored the start and end timestamps of the subtitles it covered, so the assembly step knew exactly where to drop that piece of audio later.

# For every incoming cue, check whether adding it
# would push the buffer past the char cap.
for cue in cues:
    projected = buffer_char_len(buf) + len(cue.text)
    if projected > max_chars and buf:
        # Walk back for the last cue in the buffer that
        # ends in '.', '!', '?', or '…'.
        sentence_end = last_sentence_end_index(buf)
        if sentence_end is not None:
            buf = flush(buf, sentence_end, chunks)
        else:
            # Run-on sentence longer than max_chars with no
            # terminator. Rare, but happens. Warn and cut.
            log.warning("run-on sentence > %d chars — "
                        "splitting mid-sentence as a last resort",
                        max_chars)
            buf = flush(buf, len(buf), chunks)
    buf.append(cue)

    # Soft-break: if the buffer is already at least 60% full
    # AND the current cue ended a sentence, flush now. Keeps
    # chunks balanced around 60-100% of max.
    if cue.ends_sentence and buffer_char_len(buf) >= max_chars * 0.6:
        buf = flush(buf, len(buf), chunks)

Technically, this was the turning point: instead of letting the dub follow the visual structure of the subtitle file, the chunker started following the real paragraph structure of the script. I won’t retell the whole story of how I got there and why it eventually forced a mass rerender, because I come back to that later in the section about the problems that only showed up after the first upload. The technical point here is simple: once the window represented a real spoken paragraph, the audio started sounding natural, and the stretch target could move from 92% to 95%.

Once the chunking is done, there’s still the problem that English runs longer than the Portuguese equivalent when spoken. The trick here is predicting whether a chunk is going to blow past its target window before generating the audio. If the ratio characters / 16 chars-per-second already exceeds the target duration by a margin, the pipeline passes the speed parameter directly to the ElevenLabs API, asking the model to natively generate faster speech. This preserves prosody way better than compressing afterwards. The ceiling is 1.15× (beyond that the voice starts sounding rushed).

# before calling the API, predict if it will overrun
expected_sec = char_count / EXPECTED_CHARS_PER_SEC  # 16 chars/s
predicted_ratio = expected_sec / target_sec

voice_speed = 1.0
if predicted_ratio > PREEMPTIVE_SPEED_THRESHOLD:   # 1.05
    voice_speed = min(predicted_ratio * 0.98,
                      PREEMPTIVE_SPEED_MAX)         # cap at 1.15
    voice_settings["speed"] = voice_speed

Even with preemptive speed turned on, sometimes the generated audio still comes out a tiny bit longer than the target window. That’s what the second safety net is for: measure the actual audio with ffprobe after it’s generated, compare to the target window, and apply the ffmpeg atempo filter to compress in post if it exceeded the 2% tolerance, capped at 1.20×. Combining native speed (1.15×) with atempo (1.20×) gives an effective compression ceiling of 1.38×, enough to fit naturally even in the worst cases without breaking quality.

# after generating the chunk
actual = ffprobe_duration(chunk_path)
ratio = actual / target_sec

if ratio > FIT_TOLERANCE:       # 1.02
    # compress the audio without touching pitch
    ffmpeg_atempo = min(ratio, FIT_MAX_ATEMPO)   # 1.20
    apply_atempo(chunk_path, ffmpeg_atempo)

When a generated chunk comes out shorter than the target window, the pipeline slightly stretches the audio (without affecting pitch) to reduce the ugly silence after it ends. The goal isn’t to fill the whole window — a natural pause between sentences is fine — just to smooth over the worst offenders that would feel like “cut audio”.

On a big episode, 95 minutes, 194 chunks, the total cumulative drift came out to -0.7%. About 43 seconds of drift across the entire episode, imperceptible while you’re watching.

What saved the budget on this architecture is that each chunk lives on disk as its own .mp3. The pipeline keeps a manifest with the normalized text of every chunk, and before calling the ElevenLabs API, it compares the current text against the cache. If the text hasn’t changed, it reuses the existing audio without spending a single credit. If I rewrite a problematic cue, only the chunks affected by that cue get regenerated — the rest of the episode stays untouched.

This is what made the iterations viable. I’d run the batch, listen to sections, spot a problem (translation too long, a code snippet the TTS couldn’t pronounce, a chunk boundary that landed at a bad spot), fix the SRT or tweak the chunker parameters, and re-run. Each re-run consumed a fraction of the credits and time of the original run, because only the changed chunks were regenerated. Without this cache, every iteration would have cost nearly as much as the first pass, and the total batch cost would have been two or three times higher.

That chunking change also had a heavy cache and cost impact, but I’ll leave that part for the later section where I explain the actual retracing and rerender cycle. Here, the important point is just that it was the right technical move.

When atempo can’t save you: rewriting cues before TTS

Not everything is about window alignment. There’s a kind of cue that breaks the whole pipeline, where neither preemptive speed nor post-process atempo can save it, and I only ran into it once I started running the batch over the more technical episodes of the channel.

The problem is this. The v3 reads at about 16 characters per second when you hand it normal conversational text. But when you hand it a cue stuffed with a literal URL, a hex hash, a binary string, a long sorted number list, or a shell code block, the model shifts into “spelling it out letter by letter” mode and drops to about 9 characters per second. A 500-character cue that was supposed to turn into 30 seconds of audio comes out at 55. The sanity check rejects it (because it’s past 1.8×), the automatic retry tries the same 500 characters again, the five retries all fail in a row, and the chunk gets stuck.

And this did not happen by accident. I used to write those passages that way because I knew the same scripts would later become blog posts, so it made sense to leave the raw URL, the full command, the complete hash, the whole technical detail there for the reader. In Portuguese that worked fine because it matched my original workflow exactly: record the video, then publish the matching text on the blog. What I had never prepared for was the next step, turning that same material into automated English dubbing. I only realized that conflict when the pipeline started failing for real.

I hit this first on ep052, the Ubuntu beginner’s guide, where two cues carried github.com URLs, hkp://keyserver.ubuntu.com URLs, and a 40-character GPG hash. Trying to fix that in post-processing is a waste of time. The 1.20× atempo ceiling will never compress 55 seconds into 30, and even if it did, the result would be an unlistenable chipmunk. The fix has to land upstream, in the text itself.

The solution was to rewrite the cue so it describes what the command does instead of showing the literal command:

- Run: apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 \
-   --recv-keys 0A6A3E7F79F93EF8AAB9E92BAEBB74C8B5A1E44D
+ Run the full command in the video description to import the
+ signing key from the Ubuntu keyserver.

The on-screen English caption still shows the full command, so anyone reading the caption sees exactly what they need to type. The dubbed audio describes the intent in spoken language, which is what the TTS can deliver without choking. The instruction to the viewer stays intact, the characters-per-second ratio goes back to the expected 16, and the generated audio fits the original window without needing any stretch or compression in post.

I went hunting for cues like this across the whole batch. I scanned from ep052 through ep146 looking for cues with a high density of “hard” characters (digits, brackets, operators, long URLs, binary, hex) and ended up rewriting around 30 cues across 13 episodes:

• ep052 Ubuntu for devs: GitHub URLs and a 40-character GPG hash
• ep091 Hello World in C: binary examples
• ep095 640kB memory: address wrap, segment-offset
• ep106 CodeMiner: spelled-out email address
• ep113 Compression: pure binary strings, 75% hard characters
• ep115 SQL Server: long sorted number list
• ep120 Internet: dotted IP addresses
• ep121 Sockets: two JavaScript code blocks
• ep122 Proxies: Chrome User-Agent headers
• ep123 Secure networking: shell one-liners, Docker flags
• ep126 Gentoo: C chroot demo
• ep136 Containers: GitHub release URL
• ep144 Cryptography: signing key address

The pattern was always the same: keep the caption showing the code, URL, or hash verbatim, and rewrite the spoken text so the narrator describes what it does. I could have automated the rewrite by letting an LLM read each cue and propose a substitution on the fly, but it felt safer to review them all by hand. It’s exactly the kind of thing where the model decides to “improve” a command to make it cleaner and ships with broken shell. The manual sweep took about two hours. The automated version would have cost the same time in review, with the bonus anxiety of having shipped a mangled command by accident.

There was also one weird structural case on ep146, about Docker Compose. Two cues had glued themselves together because of a missing blank line in the SRT, and pysrt was treating them as one giant cue. The TTS never even got to process it — the chunker choked first. I fixed it by hand, adding the missing blank line. One-character fix, half an hour to track down.

Reconstructing truncated SRTs

Another trap showed up when I went to run the full batch: three episodes had English captions that were too short relative to the actual video length. Ep056, the Rails episode, was the most absurd case. Nineteen minutes of caption for an 80-minute video. Sixty-two minutes of content with no caption at all. Ep057 (WSL 2) was missing 14 minutes, and ep068 (Git Direito) was missing 2.

This is a leftover from my old translation workflow. At some point I started hand-revising the caption, stopped partway through, and the .en.srt file got saved truncated at the spot where I stopped. You can’t dub an 80-minute video with a 19-minute caption. The chunker produces audio up to where it can read and then simply has no idea what to do with the rest of the video.

The fix became a new script. It diffs the truncated .en.srt against the .pt-orig.srt (the raw YouTube auto-caption, which always covers the whole video), picks up the cues that only exist in Portuguese, sends them to Claude Sonnet 4.6 with a strict JSON schema asking for cue-by-cue translation, and pastes the result back at the tail of the English file. The schema is the detail that matters. When I tried asking for the response as plain text in N|text format, Claude was dropping about 20% of the cues along the way. With a strict JSON schema, the drop rate fell to zero across the three repairs.

The numbers:

• ep068 Git Direito: +50 cues / 2 minutes reconstructed
• ep057 WSL 2: +368 cues / 14 minutes
• ep056 Rails: +1445 cues / 62 minutes (plus dropping 5 fake cues the old translator had invented at the tail to plug the gap)

After the repairs, the three episodes joined the batch normally and got dubbed like any other. The kind of tool you hope you never need, but when you need it, it’s worth writing once and closing the problem for good.

Automatic emotion tags

While I was testing, I decided to add one more step to the pipeline: an “emotion tagger” that reads the English SRT before it goes to TTS and inserts tags like [sarcastic], [thoughtful], [emphatic], [deadpan] at spots where a human narrator would naturally shift tone. The idea is to mimic what a professional voice actor would do — hit the emphasis at the right moments — without turning the video into a theater of overblown emotions.

This part is tricky for two reasons. First: letting an LLM loose on your SRT runs a real risk of it “improving” the text (swapping a word here, rewriting a sentence there) and you ending up with a dub that doesn’t match the caption. Second: LLMs love to overdo it. Ask them to tag emotion and they’ll put a tag on every other line. To guard against both, I pinned a small allow-list of tags and run a round-trip validation after Claude’s response comes back:

ALLOWED_TAGS: frozenset[str] = frozenset({
    "[sarcastic]", "[thoughtful]", "[emphatic]",
    "[deadpan]", "[serious]", "[amused]",
    "[sighs]", "[exasperated]", "[confident]",
    "[matter-of-fact]",
})

def validate_tagged(original, tagged):
    """Ensure the tagged SRT preserves the original with no drift."""
    if len(original) != len(tagged):
        raise TagValidationError("cue count mismatch")

    for o, t in zip(original, tagged):
        # Index and timestamp must match byte for byte.
        if o.index != t.index or o.timestamp != t.timestamp:
            raise TagValidationError(f"cue {o.index}: header drift")

        # Any tag outside the allow-list invalidates the whole response.
        bad = find_disallowed_tags(t.text)
        if bad:
            raise TagValidationError(f"cue {o.index}: {bad}")

        # The text with the tags stripped must be byte-identical to
        # the original. If the LLM swapped a word, rewrote anything,
        # validation fails and the response is rejected.
        if strip_tags(o.text) != strip_tags(t.text):
            raise TagValidationError(f"cue {o.index}: text drift")

When validation fails, the pipeline throws the response away and re-requests (or, if it keeps failing, ships the SRT untagged). On tag density, the prompt sent to Claude explicitly says: aim for N/10 tags in a SRT with N cues, hard floor of 1 tag per 12 cues, hard ceiling of 1 tag per 7 cues, distribution balanced across the four quarters of the episode, and no two tagged cues within 4 cues of each other. That rule set was the sixth iteration — the five before it either blanketed everything in tags or loaded them all into the first quarter.

Don’t let v3 hallucinate

A critical detail that only shows up in production: ElevenLabs v3 tends to hallucinate when you send very long blocks of text. I’ve seen blocks where the input text was around 1,500 characters and the model generated nine minutes of audio when the expected duration was a minute and a half. The model just decides to keep talking on its own, inventing content.

The official ElevenLabs docs recommend keeping each call under 800 characters to avoid this. I went more conservative and cut at 700, always at sentence end (never mid-sentence, because a mid-sentence split sounds horrible when concatenated). On top of that, I bumped stability to 0.9 (Robust mode, more stable but less responsive to emotion tags), turned on apply_text_normalization so the model pronounces numbers and acronyms correctly, and added a sanity check that rejects any generated audio longer than 1.8× the expected duration.

DEFAULT_VOICE_SETTINGS = {
    "stability": 0.9,          # Robust mode
    "similarity_boost": 0.95,
    "style": 0.0,
    "use_speaker_boost": True,
}

EXPECTED_CHARS_PER_SEC = 16.0
CHUNK_SANITY_MAX_FACTOR = 1.8   # reject if actual > 1.8× expected

With those mitigations in place, a 95-minute episode (194 blocks) ran start to finish with zero hallucinations. Exactly one block failed mid-run on a transient 502 error from the API, which the automatic retry caught on the next attempt.

Mastering for YouTube

YouTube doesn’t publish an official number, but industry consensus is that its playback loudness normalization targets -14 LUFS. Deliver audio louder than that and YouTube turns it down; deliver it quieter and it leaves it alone. To land on the target exactly, the pipeline runs ffmpeg’s loudnorm filter in two passes. The first pass measures the entire audio and prints the stats (input_i, input_tp, input_lra, input_thresh) as a JSON blob on stderr. The second pass reads those stats back in and applies a linear static gain to land precisely at -14 LUFS, -1.5 dBTP:

# Pass 1: measure
ffmpeg -i final_en.mp3 \
  -af "highpass=f=80,loudnorm=I=-14:TP=-1.5:LRA=9:print_format=json" \
  -f null -

# Pass 2: apply linear gain with the measured values
ffmpeg -i final_en.mp3 \
  -af "highpass=f=80,loudnorm=I=-14:TP=-1.5:LRA=9\
:measured_I=-18.3:measured_TP=-3.1:measured_LRA=5.4\
:measured_thresh=-28.7:offset=1.2:linear=true" \
  -ar 48000 -ac 2 -c:a pcm_s16le final_en_mastered.wav

Two passes instead of one because single-pass mode runs in dynamic compression and ends up “pumping” the gain during silent stretches. With linear=true on pass 2, the gain stays static on top of the pass-1 measurements, so no pumping. The result lands within ±0.1 LU of the target, which is effectively inaudible. The highpass=f=80 filter in front kills HVAC rumble and mains hum below 80 Hz, which the human ear doesn’t catch but will move peak measurements around.

The point where I thought it was done. And it wasn’t.

This is where the most annoying, and most educational, part of the project starts.

I really thought I had nailed it on the first big pass. The pipeline closed all 146 episodes, I uploaded everything, and it looked done. The durations matched, the files were all there, and the whole process had finally crossed the complete batch of nearly 96 hours of video. But it was exactly after that upload that I started noticing problems I simply had not anticipated.

The first scare was the jingle situation. I already knew many Akitando episodes begin with that short instrumental intro before I start talking, but I underestimated how much trouble it would cause. YouTube’s .srt is almost useless here, because a jingle is not speech. Sometimes it marks [Music], sometimes it gives you nothing useful, sometimes the window is just wrong. In the first assembly pass, some episodes lost the jingle entirely, some got it doubled, and in others the source-fill from the original audio already contained the right jingle and my splice came in on top of it, pushing everything else forward. This was the kind of bug you only catch when you sit down and actually listen to the final mastered file, not when you just look at total duration.

The fix was to stop hoping the subtitle would tell me where the jingle was and go straight to the source. I built a detector using the original video audio itself plus reference jingles, with an audit and repair pass on top of the mastered outputs already generated. That eventually became a small subsystem of its own: detector, audit, cached reassembly, exact-window repair. At the end of that detour, the set of changed episodes collapsed into a well-defined reupload bucket and the jingle problem finally came under control. It was one of those classic production moments: the first solution looked good enough until it hit the whole archive.

Then came the second punch, and this one was more expensive: I had forgotten a basic fact about YouTube subtitles. SRT is built to be read on screen, not to become spoken script. So an idea that was a full paragraph in my original text showed up split into several small cues, each with its own micro time window. In the first version of the pipeline I was using those cues as the generation unit for ElevenLabs. The result was TTS audio that was coherent locally, but the final assembly inserted artificial silences in the middle of sentences. Technically aligned, humanly weird.

Fixing that hurt because it meant walking several steps back. Instead of respecting the cue structure of the SRT, I had to go back to my original scripts and rebuild chunking around the paragraphs I had actually read when recording the videos. Almost every Akitando episode has a matching blog post with a ## Script section, and that is what saved the project. Because I had been meticulous back when I made those videos — writing the scripts first, reading exactly that on camera, and archiving both the videos and the scripts — I had a source of truth. That made it possible to map the script paragraphs onto the timings in the .pt-BR.srt, regroup the .en.srt on top of that, and regenerate the chunks in the right structure.

The problem is that this destroys the previous cache. Even when the text barely changes, if the chunk boundary changes then the TTS call changes too: different context, different prosody, different beginning, different ending. So this was not a touch-up. It was a full regeneration of everything again. That was the point where the cost started escaping my original estimate.

And just when I thought now it really was over, the third hit landed: external clips. Many videos on the channel include inserted footage from other sources. The Ruby on Rails episode, for example, has 37signals clips, Apple material, commercials, external inserts in the middle of the narrative. YouTube’s .srt did not know what to do with that. In some places it simply hallucinated text, in others the timings were completely wrong, and in others the original speech had nothing to do with the subtitle I was using as ground truth. Again: if you only look at the automation, it looks fine; if you actually listen to the final master, it isn’t.

The fix, again, was to leave the subtitle behind and go back to the original material. I built an external-clip window detector based on alignment gaps between transcript, .pt-BR.srt, speechless cues and chunks that crossed those windows. Then, instead of trying to dub what made no sense to dub, the process started recovering the original audio from the source video exactly in those stretches and filling the master with the correct source audio there. That’s what fixed the 37signals, Apple Education, Mac vs PC commercial and other inserted segments I had forgotten were even there.

That was the biggest lesson from the second half of this project: automation takes you very far, but the full archive always has more dark corners than you imagine on the first pass.

The dubbing batch numbers (and why it hurt more than I expected)

I need to correct myself again. I didn’t just miscalculate the first bill. I also overestimated how confident I should have been after the first full run.

When I upgraded to ElevenLabs Business, I thought I would finally have plenty of headroom. In practice, I didn’t. The latest dashboard screenshot shows 10,046,279 credits consumed out of a limit of 11,000,000, leaving 953,721:

In plain English: I burned through more than 91% of the Business workspace limit. And that helps explain what “10 million credits” really means in this context. It does not mean “I rendered 146 episodes once”. It means rendering 146 episodes, discovering a structural problem, going back, fixing it, regenerating chunks, discovering another structural problem, going back again, rebuilding masters, auditing jingles, repairing external clip windows, uploading again. Every extra round costs money. And when the change invalidates the cache, it costs a lot.

The cost also doesn’t stop at ElevenLabs. All the subtitle curation, problematic cue rewrites, truncated SRT reconstruction, jingle audits, external clip detector work and the rest ran through Claude Code on the Claude Max 20× plan. Combined with the mass blog translation to English that I described in the anniversary post, the Claude Max 20× weekly limit hit 100%, with more than BRL 300 in extra usage on top. If you think generative AI becomes free once you’ve signed up for the plan, the invoices disagree.

Even so, I still think it’s worth it. We’re talking about 146 episodes, almost 96 hours of technical content, more than 5 years of channel archive. Doing that manually in a studio, with an actor, direction, pickups and human review on everything, would cost a completely different order of magnitude.

What changed was how I think about the project. At the start I was looking at this as “one very large batch”. In the end it was something else. It was digging through a whole archive, finding hidden exceptions, fixing old rot, rebuilding masters, rerendering, relistening.

The first full batch took a little under two days and made me think I had solved everything. I hadn’t. The second pass, with paragraph-aware chunking, was already expensive. The third, with jingle repairs and mastered-output fixes, cost more again. And the fourth hit came from the external clips I had not modeled correctly in the first automation pass.

But now, finally, I think it’s done. Or rather: I hope it’s done. Everything has been uploaded again, for the last time. It isn’t perfect in the absolute sense of the word. It’s as good as I can get it with scripts, audits and automation, without descending into manual-intervention hell episode by episode.

And if all of this was possible, it’s because years ago I had the discipline to produce the videos the right way: write first, read the script when recording, and archive the materials afterward. Without the original videos and without the original scripts, I would have had no way to discover where the .srt was lying, where the timing was wrong, where the external clip started, where the original paragraph began and ended. That old organization is what made it possible to repair the project now.

So yes: the dub for all 146 episodes is done. Not perfect. But closed. At least, I sincerely hope this time it really is.

The first test, watch it

I made a test video to prove the thing works. Two caveats up front: first, the embed below already loads with English captions on by default, so that side is solvable via URL. Second, the audio is annoying: YouTube removed the audio track selector from embedded players back around March of this year. So inside the embed you won’t find that option in the gear menu at all, it just plays in Portuguese. To actually hear the English dub, click watch directly on YouTube — the main YouTube watch page still has the audio track switcher in the gear menu.

If you’re used to YouTube’s automatic dubbing or to the AI dubbing on TikTok, compare them. The difference is glaring. The voice is mine, with a worked-over American accent, and the sync with the original video stays within 1 to 2% cumulative drift, practically imperceptible over 95 minutes of video.

And the missing piece: translating the thumbnails

While I was closing this post, the penny dropped: I’d left a loose end. All 146 thumbnails on my channel are in Portuguese, many of them with big all-caps titles like “9 DICAS PARA PALESTRANTES” or “7 RECOMENDAÇÕES DE SHOWS PARA PESSOAS DE TECH”. Perfect English audio doesn’t help if the image in the search results is an illegible block of text to anyone who doesn’t read Portuguese. YouTube lets you upload an alternative thumbnail per language, so I needed a way to generate English versions, keeping the rest of the art identical and swapping only the text.

The tool uses two pieces. yt-dlp is the modern fork of the old youtube-dl, a Python CLI that downloads anything from YouTube (videos, audio, subtitles, thumbnails) without needing an API key. Nano Banana Pro (nano-banana-pro-preview on the API) is Google Gemini’s latest image-editing model — you feed it an input image with a prompt and it returns an edited image, preserving the rest of the composition when you ask it to touch only a specific part.

The pipeline has two steps. Step 1: yt-dlp grabs the thumbnail as a .jpg, no video download:

yt-dlp --skip-download \
       --write-thumbnail \
       --convert-thumbnails jpg \
       -o "thumbnails/originals/<slug>/<video_id>" \
       "https://www.youtube.com/watch?v=<video_id>"

Step 2: send each image to Gemini Nano Banana Pro with a prompt that’s a rigid contract of hard requirements, not just a “translate this”:

TASK:
Detect every piece of Portuguese text visible on this image and
translate it to clear, natural American English. Replace the
Portuguese text with the English translation in the same visual
position.

STRICT REQUIREMENTS — follow every one:
- Preserve EVERY non-text element identically: face, pose,
  expression, background, color palette, lighting, icons, logos,
  decorative shapes, borders, layout. Only the text changes.
- The English translation must be IDIOMATIC and CONFIDENT — not
  a literal word-for-word rewrite. It's a YouTube thumbnail for a
  tech audience, so use punchy phrasing a native English-speaking
  tech YouTuber would write.
- Match the original text's font family, weight, size, color,
  stroke outline, drop shadow, and any decorative treatment.
- If there is no Portuguese text at all on the image, return the
  original image unchanged.

The “return the original image unchanged” clause is what saves the earliest episodes, which don’t have overlay text, just my face. Without it, the model would invariably try to “improve” the composition, swap the lighting, and so on.

Here’s the result on two examples (episodes 10 and 11):

Original (PT)

Translated (EN)

Original (PT)

Translated (EN)

Notice the detail on episode 10. The model didn’t translate “7 Recomendações de Shows para Pessoas de Tech” literally into “7 Show Recommendations for Tech People” the way Google Translate would. It rewrote it as “7 TV Shows You Must Watch If You’re In Tech”, the way an actual American tech YouTuber would headline the video. The rest of the image stays pixel-for-pixel identical. On episode 11, “9 DICAS PARA PALESTRANTES: venda sua caneta” becomes “9 TIPS FOR SPEAKERS: sell your pen”, with the font, all-caps treatment and position preserved. If you didn’t know the original was in Portuguese, you couldn’t guess the second image is an automatic AI edit.

Cost on the Gemini side: a few cents per image, a handful of dollars total for the whole batch. Trivial next to the $1,500+ on the audio dub. With the thumbnail sorted, the channel’s English conversion is finally buttoned up — ElevenLabs v3 cloning the voice over hand-curated English .srt files, and Nano Banana Pro editing the image so the title matches the audio.

The conclusion, which is the title of this post

When Qwen3 TTS dropped, the hype was calling it an “ElevenLabs killer”. I spent weeks trying to live with that premise in practice, with real content shipping every week. And what I found is that open source TTS is still miles behind ElevenLabs. The gap is large, and it’s a gap that shows up precisely when you step off the 5-second demo tweet and put the model to work on a real 30-minute weekly podcast.

In practice, Qwen3 doesn’t even beat ElevenLabs’ v2 model. v3, which is the current one, is still a step above v2. The prosody comes out better, the inline emotion tags work in Portuguese and English without effort, and the API stays up without you maintaining a server. The cost per character is a bit higher, but for my current volume it sits comfortably within budget and gives me back the hours I was spending on GPU babysitting.

The lesson here is the same one the LLM crowd is learning slowly. A 30-second demo tweet is one thing, production is a completely different thing. Open source has its niche, mainly when you have sensitive data that can’t leave the house, or when you have tons of idle GPU and little recurring budget. But for serious TTS use in a commercial product, here in April 2026, ElevenLabs is still untouchable. Qwen3 didn’t kill anyone.

And don’t forget: subscribe to The M.Akita Chronicles on Spotify so you don’t miss new episodes like this one, made with the new pipeline.

I bought a Minisforum MS-S1 Max with the new AMD Ryzen AI Max+ 395 chip for one specific reason: this chip supports up to 128GB of unified RAM, and I can allocate 96GB of it as VRAM for the iGPU. That gives me more VRAM than any consumer gaming card, including the RTX 5090 (32GB). And that changes what I can run locally.

Why ditch the Intel NUC

The NUC was a fine Docker server for two years. But the limitation was clear: without a GPU with enough VRAM, I couldn’t run LLMs locally in any usable way. Frank Yomik, my automatic manga translation system, needed CPU-based OCR (slow) and would connect remotely to the Ollama running on my desktop (AMD 7950X3D + RTX 5090) for translation. It worked, but it meant my desktop had to be on for the server to do its job.

With the Minisforum, Frank Yomik now runs entirely on the server. The worker uses ROCm for OCR on the iGPU, and Ollama runs locally with 96GB of VRAM. Zero dependency on the desktop.

You can get a sense of the size from the photo. The NUC is the tiny cube on the left. The Minisforum is bigger but it’s still a mini-PC. It fits on the rack shelf under my Synology NAS without any trouble.

The specs

The chip is the AMD Ryzen AI Max+ 395: 16 cores / 32 threads Zen 5, with an integrated Radeon 8060S iGPU and 128GB of unified LPDDR5X. In the BIOS, I set the UMA Frame Buffer Size to 96GB, which leaves ~30GB of RAM for the operating system and containers. Plus the kernel parameters for TTM (without them, ROCm only sees 15.5GB even with the BIOS allocation in place).

The operating system is openSUSE MicroOS (more on that in the next post). The whole machine pulls under 100W, which is absurd if you’re used to dedicated GPUs that draw 450W+ on their own.

Minisforum vs my desktop: benchmarks

I ran a set of benchmarks comparing the Minisforum against my desktop (AMD 7950X3D, 96GB DDR5, RTX 5090 32GB GDDR7). The results are clear.

CPU

Mixed results. The AI Max+ 395 is better at pure parallelism (multi-core sieve), probably thanks to lower latency in the unified memory architecture. The 7950X3D wins at float and crypto because of its higher clocks and the 3D V-Cache.

LLM inference (models that fit on both)

This is where it gets interesting. For models that fit in the 32GB of the RTX 5090, the comparison is purely about memory bandwidth:

The RTX 5090 is ~7x faster. The explanation is simple: GDDR7 has ~1,792 GB/s of bandwidth. LPDDR5X has ~256 GB/s. The ratio (7x) lines up almost exactly with the measured speed difference (6.7x). LLM inference is a problem dominated by memory bandwidth. Whoever reads weights faster, generates tokens faster.

And what about prompt processing?

Prompt processing is even worse: 7-11x slower. That makes sense, because the prompt has to be processed in full before generating the first token, and it’s an even more bandwidth-intensive operation.

Where the Strix Halo wins: large models

Now we get to the reason I bought this PC. Models that don’t fit in the RTX 5090:

qwen3.5:122b with 81GB of weights running at 19 tok/s. On a mini-PC. That’s simply not possible on an RTX 5090. On the NVIDIA card, that model would have to offload layers to system RAM, dropping to 2-3 tok/s. In practice, unusable.

The difference between MoE and dense models is brutal. qwen3.5:35b runs at 43 tok/s because, despite having 35B total parameters, only ~4B are active per token. A dense 72B model like qwen2.5:72b has to read 40GB+ of weights per token, and at 256 GB/s of bandwidth, the theoretical maximum is ~6.7 tok/s. The 4.5 measured represent ~67% efficiency, which is what you’d expect for an iGPU (overhead from shared bus and drivers).

Summary: when to use which machine

A ROCm bug that’s still around

Not everything works. Models like deepseek-r1:70b, llama3.3:70b and llama4:scout crash with a ggml bug (GGML_ASSERT(ggml_nbytes(src0) <= INT_MAX) failed). The embedding tensor of these models exceeds 2GB and the ROCm copy kernel uses a 32-bit integer for the size. On CUDA (NVIDIA) it’s already been fixed, but on ROCm it hasn’t. Waiting for the fix in Ollama 0.20.0+.

LPDDR5X vs GDDR7: why this difference exists

The next question is: why is LPDDR5X so much slower?

GDDR7 is dedicated GPU memory. It’s soldered onto the graphics card, connected by a wide bus (384 or 512 bits on the RTX 5090) at high clocks. Its only job is to feed data to the GPU. LPDDR5X is unified memory that serves everything: the operating system, applications, and the GPU all at once. The bus is narrower and shared.

In practice: GDDR7 delivers ~1,792 GB/s dedicated to the GPU. LPDDR5X delivers ~256 GB/s that still need to be split between CPU and GPU. LLM inference is basically “read all the model weights from memory, multiply by the current token, generate the next token, repeat.” Whoever reads faster, generates faster. There’s no shortcut.

The Strix Halo’s advantage isn’t speed. It’s capacity. 96GB of VRAM in a 100W chip that costs a fraction of a professional GPU. The RTX 5090 is 7x faster, but it’s stuck at 32GB. Models that don’t fit, don’t run.

The alternatives: who else does this?

If 96GB isn’t enough or you want more speed, the options are limited.

The Framework Desktop uses the same AI Max+ 395 chip with up to 128GB of RAM. Same platform, same performance, but with the differential of being modular and repairable (it’s Framework, after all). In practice it’s equivalent to the Minisforum on specs and price.

Above that, the alternative is a Mac Studio with M3 Ultra. The M3 Ultra chip supports up to 512GB of unified memory, with ~819 GB/s of bandwidth (more than 3x the Strix Halo). Apple manufactures the memory chips on the package, so latency and bandwidth are superior. You could potentially allocate ~400GB as VRAM and run models that don’t fit anywhere outside of professional GPU servers.

Apple’s internal NVMe is also another level: ~7.4 GB/s of sequential read on the M3 Ultra, compared with ~14 GB/s on the Crucial T700 (PCIe 5.0). The T700 is faster on raw throughput, but Apple’s NVMe latency tends to be lower on random I/O thanks to the SoC integration.

The Brazilian price is the elephant in the room. The Mac Studio’s max config costs $9,999 in the US. With import taxes (~60% + state ICMS), it goes past R$ 110,000. The Minisforum with 128GB lands at R$ 12,000-15,000. A nearly 8x price gap buys you a lot.

If you need more than 96GB of VRAM for truly enormous models (DeepSeek-V3 with 671B parameters fits in ~400GB Q4, for example), the Mac Studio with 512GB is the only consumer option. The alternative would be professional NVIDIA A6000 GPUs (48GB VRAM, ~$6,000 each, and you’d need several in NVLink). For everything that fits in 96GB, the Minisforum gets the job done at a fraction of the cost.

And projects that promise to run big LLMs on small GPUs?

There’s a concept called “layer offloading” that projects like llama.cpp already support. The idea: if the model doesn’t fit fully in VRAM, keep some layers on the GPU and the rest in system RAM. The GPU processes the layers it has, hands off to the CPU to process the rest, and back.

In practice, it doesn’t work well. The bottleneck is PCIe: transfer speed between system RAM and GPU VRAM is ~32 GB/s (PCIe 5.0 x16). Each generated token needs to transfer data back and forth. The result is that you drop from 150 tok/s (everything in VRAM) to 2-8 tok/s (partial offload). It’s too slow for interactive use.

VRAM is the fundamental limitation because LLM inference is memory-bandwidth-bound, not compute-bound. The GPU has compute to spare. What’s missing is the ability to read the model weights fast enough. When part of the weights live in system RAM through PCIe, the entire pipeline waits on the transfer.

That’s why unified memory (like in the Strix Halo or Apple Silicon) makes a difference. There’s no PCIe in the middle. CPU and GPU access the same physical memory. The Strix Halo’s 256 GB/s is slow compared to GDDR7, but it’s 8x faster than offloading through PCIe.

Advances in LLM optimization (up to 2026)

To understand why some models run so much better than others on the Strix Halo, you have to understand what’s changed in the ecosystem over the last two years.

Mixture of Experts (MoE)

If you run local models, MoE is the advance that matters most. An MoE model has high total parameters (e.g. 122B in qwen3.5:122b), but only activates a fraction of them per token (e.g. ~4B). The inactive weights stay in VRAM but aren’t read on every token, which drastically reduces the bandwidth needed.

In the Strix Halo benchmarks, MoE models run 3-10x faster than dense models of the same size. qwen3.5:35b (MoE, ~4B active) runs at 43 tok/s while qwen2.5:72b (dense, 72B active) runs at 4.5 tok/s.

DeepSeek and training optimization

DeepSeek V3 (December 2024) showed that it was possible to train 671B parameter models at a cost an order of magnitude lower than predicted. They combined MoE with FP8 quantization during training (not just inference), multi-stage training with curriculum learning, and several inter-GPU communication optimizations. The impact: everyone copied. Qwen, GLM, MiniMax, all of them adopted variations of the technique.

Quantization: from FP16 to Q4 without losing much

Quantization compresses model weights from 16 bits (FP16) to smaller formats: 8 bits (Q8), 4 bits (Q4), or even 2 bits. A 70B model that would take ~140GB in FP16 fits in ~40GB at Q4_K_M. Quality loss exists, but in modern formats (GGUF Q4_K_M, AWQ, EXL2) it’s small enough for practical use.

GGUF (the llama.cpp format) became the standard for local inference. AWQ and GPTQ are alternatives with more sophisticated calibration, but the ecosystem converged on GGUF because it works on CPU, CUDA and ROCm without recompilation.

Distillation: smaller models that know more

Distillation is training a small model using the responses of a large model as the teacher. Microsoft’s Phi-4 (14B) was trained with distillation from GPT-4 and competes with 70B models on several benchmarks. Qwen3 did the same: qwen3:14b is surprisingly capable for its size.

Flash Attention and optimized KV Cache

Flash Attention (Tri Dao, 2022) changed how attention is computed: instead of materializing the full attention matrix in memory, it processes in blocks keeping the data in the GPU’s on-chip SRAM, reducing memory consumption from O(n²) to O(n). Without that, contexts of 128K+ tokens would be impractical. It already went through versions 2 and 3, with optimizations for FP8 and async operations on H100. PagedAttention (vLLM, UC Berkeley) did the same for the KV cache during serving: applies virtual memory concepts to the cache, eliminating fragmentation and improving throughput by 2-4x.

In Ollama, I set OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0 on the server. The first activates flash attention, the second uses 8-bit KV cache instead of fp16, cutting the bandwidth needed per token in half. These are zero-hardware-cost optimizations that improve throughput measurably.

What Qwen, Kimi, MiniMax and GLM are doing

Qwen (Alibaba) has consistently been the best price/performance in open source models. Qwen3:14b is dense and strong; Qwen3.5:122b is MoE and runs surprisingly well in 96GB. GLM-4.7 (Zhipu AI) is notable for offering bf16 full precision versions that fit in 96GB. MiniMax experimented with long contexts (up to 4M tokens). Kimi (Moonshot AI) focused on large context windows with linear architectures.

What runs well in 96GB of VRAM

With 96GB on the Strix Halo, the models that work well for daily use:

Dense 70B+ models (deepseek-r1:70b, llama3.3:70b) are still blocked by the ROCm bug I mentioned. When it gets fixed, they should run at ~4-6 tok/s, usable for batch but not for interactive chat.

Conclusion

I bought the Minisforum to run models that don’t fit in any gaming GPU. For that, it works. It’s not fast. 19 tok/s on a 122B model isn’t the experience you get with Claude or ChatGPT. But it’s local, it’s private, and it runs on my shelf consuming less power than an old lightbulb.

For people asking about the Mac Studio: if you have the budget, it’s the best machine for running local LLMs. 512GB of unified memory, 819 GB/s of bandwidth, mature Metal/mlx ecosystem. You can run DeepSeek-V3 in full Q4. But in Brazil, with import duties, it crosses R$ 110k. The Minisforum with 128GB at R$ 12-15k is the realistic option.

And for people who think you can work around the VRAM limitation with layer offloading: you can’t. PCIe is too slow. The model has to fit fully in VRAM for inference to be usable. It’s the reason gaming GPUs with 32GB of ultra-fast GDDR7 are still capped on model size, and why the unified memory of the Strix Halo and Apple Silicon changed the equation.

In the next post I tell the story of how I migrated the entire home server to the Minisforum using Claude Code, the problems I ran into, and how openSUSE MicroOS behaves as a Docker server operating system.

Since I decided to move back from macOS to Linux, I didn’t want to just return to old Ubuntu (yes, I get bored of doing the same things for too long). So I tried out Fedora 25 and I was delighted by how Gnome 3.22 evolved nicely.

Compared to Ubuntu, Fedora’s defaults felt nicer. In practice, you can force any distro to become whatever you want, but I’d rather not fight the defaults. Ubuntu is heavily customized for Unity and I really, really dislike it. It feels more like a toy than a serious environment to do work.

Fedora 25 looks good with Gnome 3, but it still gave me a few headaches. One thing that didn’t work at all was Gnome’s Online Accounts. GOA collects authentication tokens after you sign in to your social services like Google. Then compatible apps like Evolution and the built-in Calendar can pick them up. But the tokens were getting expired all the time, so the integration was useless. And manually configuring Evolution for Google wasn’t pleasant.

Now the surprise: on Arch Linux, I chose Gnome 3.22 as my desktop and manually installed the gnome-online-account package. I signed in to my Google accounts and I am delighted to report that it’s not expiring and it “Just Works”! This is the kind of polish I expect from a major distro, not the fiasco that’s labeled as “stable” in Fedora.

Why Arch Linux?

Most major distros are divided between “stable” (but with very old packages) and “unstable” (but with the cutting-edge goodness). If you install the Long-Term Support (LTS) versions, you’re doomed to have only old packages from a couple of years ago. If you install the unstable repositories, you’re doomed to have stuff exploding in your face without explanation and losing hours browsing through Stackoverflow.

Now, it seems that Arch figured out the exact “right” level of confidence between stable and cutting-edge. It keeps pushing the most recent version of software without breaking everything else all the time. So, in Ubuntu 16.04 and Fedora 25, if I want to install Postgresql, I am locked in to 9.4 or 9.5, but in Arch I can access 9.6 from the main Pacman repositories. (By the way, “Pac"kage “Man"ager is the most obvious name ever).

You can easily pacman -Sy postgresql and you’re in business.

And if you’re on Ubuntu 14.04 and now you want 16.04, good luck dist-upgrade-ing your way. It’s easier to start from scratch.

So, it seems that the philosophy of Arch is to have the real most recent version of all software that won’t break your system. There is no such thing as a big bang upgrade every 6 months that breaks everything. Instead, you have a constantly rolling upgrade system, where you’re always on the latest version, without having to wait another year for the next big LTS.

Every major distro has “unsupported” repositories for proprietary binaries (codecs for example) or 3rd-party software. Then, there is Arch User Repository (AUR): a collection of small Git repositories from users maintaining simple PKGBUILD text files.

AUR is clever. If you’re from macOS and familiar with Homebrew, you will understand it: it feels like Casks and Formulas. A PKGBUILD file can describe a recipe to download an available DEB package or tarball, disassemble it and rebuild it as a Pacman compatible package. It can describe the necessary dependencies and make the installation process super smooth.

For example, Sublime Text only has an option to download a DEB package or a tarball with the binaries. Same goes for Spotify, Franz, etc. Sometimes you can register Personal Package Archives (PPAs) and then apt-get your way in installing them. But you still need someone to build, maintain and distribute those packages properly. It’s a lot of work.

Now, maintaining a simple Git repository with a simple PKGBUILD text file is far easier. makepkg does the hard work of building the package you need, in your machine, and then pacman can handle installing it like any other package. No more wget-ing tarballs and configuring everything manually!

Maybe I can finally just do pacman -Syu and have everything “really” upgraded without having to worry about the next big LTS that will eventually force me to reinstall everything from scratch.

Arch Linux is perfect for “Beginners”

I’ve been hearing about Arch for a long while and their users are very enthusiastic in trying to convince other people to join. Whenever you see such a heavily loyal fanbase there must be something interesting hidden under the hood. Rolling upgrades, Pacman, AUR are really valuable reasons.

After just one day using it, I’ve come to realize that Arch is good for advanced users, but also for beginners. But not because it is easy. On the contrary: it’s because it is hard in the right way.

Most “Linux users” nowadays just get a trivial-to-install distro, such as Ubuntu or Elementary, and they have no idea what goes on underneath. Blindly clicking “next” in the graphical installers.

Most people have no idea what TTYs are. That you can probe USB devices with command line tools such as lsusb or that you must use tools such as fdisk to partition and then mkfs.ext4 to format them. That memory swap files are partitions with a special format. They are not aware of LVM options for flexible partitioning, or even that LVM exists at all. That the “thing” you choose your kernel from in the boot menu is called Grub and that you can tweak it.

There is a lot going on in assembling a fully functional Linux-based distribution. But graphical installers hide most of it. Arch Linux forces you to go step by step and really feel like you “own” your machine, not the other way around.

If you’re a “beginner”, I really urge you to install a distro like Arch a few times, in different configurations of machines, to really understand what an operating system really looks like.

The Arch Wiki is a very comprehensive and detailed repository of information for everything you ever need to know about installing and maintaining every component of a proper Linux system. You will learn a lot in the process.

But if you’re like me, and you’ve been doing that through all the mid 90’s and early 2000’s (heck, I had to learn my way through Slackware 1.0, I still remember having to use boot and root floppy disks and screwing up my hard-drives not understanding cylinders and sectors through fdisk), you can skip it. For you, advanced/experienced users, I’d recommend you go with Arch Linux Anywhere.

It will custom install Arch but will provide you with enough automation to not waste much time in bringing a properly configured Arch installation up and running, without bloatware.

Pacaur - best way to deal with AUR

Arch users are quick to praise Pacman. For the most part, you can basically do:

sudo pacman -S chromium

And that’s it. And then you can sudo pacman -Syu to upgrade installed packages. This is the basics you need to know.

If you’re a developer I also recommend you to install the basic development packages:

sudo pacman -Sy --needed base-devel

Now, you can manually install AUR packages. You can go to their website and search for “terminix” (a very nice Terminal replacement, similar to Mac’s iTerm2) for example. You will end up in this page and you will have to manually do the following:

git clone https://github.com/gnunn1/terminix.git
cd terminix
makepkg -si

Feels simple, but you can do better by installing Pacaur, a wrapper on top of Pacman. If you’re using a graphical terminal such as Terminal or Terminix DO NOT FORGET to edit the profile to “Run command as login shell”, otherwise there will be a PATH problem and Cower will fail to install .

sudo pacman -S expac yajl --noconfirm
gpg --recv-keys --keyserver hkp://pgp.mit.edu 1EB2638FF56C0C53
git clone https://aur.archlinux.org/cower.git
cd cower
makepkg -si
cd ..
git clone https://aur.archlinux.org/pacaur.git
cd pacaur
makepkg -si
cd ..

In summary, Pacaur can be used not only as a complement to install AUR packages, but also if you want to use a single tool to manage both AUR and official Pacman packages. All commands -S will be Pacman commands. So instead of doing sudo pacman -Syu to upgrade all packages, you can replace it for pacaur -Syu. Everything else mostly “just works”.

When you try to install a package with -S it will first look into the official repos, if not found then it tries AUR. There’s even a nice GUI if you want:

pacaur -S pamac-pacaur

Now, to install the same Terminix, you can do just this:

pacaur -Sy terminix

It will ask you simple yes/no questions such as “Do you want to edit the build file?” You can answer “n” to those and confirm “y” when it asks you if you want to install the dependencies or the generated package.

And that’s it! You can search the AUR repositories with:

pacaur -s spotify

It will give you a lot of options, for example:

$ pacaur -s spotify
aur/spotify 1.0.47.13-1 (1037, 36.09) [installed]
    A proprietary music streaming service
aur/playerctl 0.5.0-1 (127, 11.33)
    mpris media player controller and lib for spotify, vlc, audacious, bmp, xmms2, and others.
aur/blockify 3.6.3-3 (106, 5.61)
    Mutes Spotify advertisements.
...

Common sense, my friends. Read, interpret, choose. Arch requires you to be a smart person, and by “smart” I mean: knowing how to read! Most people skip reading things and just click stuff like morons.

Now that you know the exact name of the package you want, just install it normally like this:

pacaur -S spotify

Pacaur is one of many AUR Helpers. I was initially drawn into Yaourt, but after research, you figure out that you should only try out aurutils, bauerbill or pacaur. I prefer the latter because it’s easier to spell out.

pacaur -Syua

This should keep your system up to date, both the official and AUR packages.

Asdf - the last languages version manager you’ll ever need

If you’re a Rubyist, you’re familiar with RVM, rbenv, chruby. If you’re from Node.js you know the RVM-inspired NVM to manage your different versions of Node. Each new language nowadays needs a version manager as they’re evolving quickly and because if you work with client projects you will eventually need to use an old version to deal with legacy software.

So even though you can install the current stable Ruby 2.3.3 by just doing pacman -S ruby or pacaur -S ruby you will eventually need to switch back to Ruby 2.1 or older for a client project, for example.

Should you install RVM? Or rbenv? And how do you deal with different versions of Clojure, Go, Rust, Elixir?

That sounds like yet another maintenance nightmare to deal with. But someone decided to actually solve this problem in an elegant way. Enter asdf - and if you happen to know Akash Manohar give him a big hug.

Let’s install it (from the project’s README):

git clone https://github.com/asdf-vm/asdf.git ~/.asdf --branch v0.2.1

Then edit your shell config files:

# For Ubuntu or other linux distros
echo -e '\n. $HOME/.asdf/asdf.sh' >> ~/.bashrc
echo -e '\n. $HOME/.asdf/completions/asdf.bash' >> ~/.bashrc

# OR for Mac OSX
echo -e '\n. $HOME/.asdf/asdf.sh' >> ~/.bash_profile
echo -e '\n. $HOME/.asdf/completions/asdf.bash' >> ~/.bash_profile

# For the Fish shell
echo 'source ~/.asdf/asdf.fish' >> ~/.config/fish/config.fish
mkdir -p ~/.config/fish/completions; and cp ~/.asdf/completions/asdf.fish ~/.config/fish/completions

# If, like me, you like ZSH with YADR (you have to install YADR before this)
touch ~/.zsh.after/asdf.zsh
echo -e '\n. $HOME/.asdf/asdf.sh' >> ~/.zsh.after/asdf.zsh
echo -e '\n. $HOME/.asdf/completions/asdf.bash' >> ~/.zsh.after/asdf.zsh

This tool is very self explanatory. Let’s start by installing a bunch of plugins (full table of links in the README file):

asdf plugin-add clojure https://github.com/vic/asdf-clojure.git
asdf plugin-add elixir https://github.com/asdf-vm/asdf-elixir.git
asdf plugin-add erlang https://github.com/asdf-vm/asdf-erlang.git
asdf plugin-add golang https://github.com/kennyp/asdf-golang.git
asdf plugin-add ruby https://github.com/asdf-vm/asdf-ruby.git
asdf plugin-add rust https://github.com/code-lever/asdf-rust.git
asdf plugin-add nodejs https://github.com/asdf-vm/asdf-nodejs.git

If you’re like me, you must be super excited because you already know what we will do next:

sudo pacman -Sy jdk8-openjdk # you need Java for Clojure

asdf install clojure 1.8.0
asdf global clojure 1.8.0
mkdir ~/bin
wget https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein -O ~/bin/lein
chmod +x ~/bin/lein
export PATH=$PATH:~/bin
# echo "PATH=$PATH" > ~/.zsh.after/binpath.zsh # if you use YADR+ZSH
lein

asdf install erlang 19.0
asdf global erlang 19.0

asdf install elixir 1.4.0
asdf global elixir 1.4.0
mix local.hex
mix local.rebar

asdf install golang 1.7.4
asdf global golang 1.7.4

asdf install ruby 2.4.0
asdf global ruby 2.4.0
gem install bundler

asdf install rust 1.14.0
asdf global rust 1.14.0

asdf install nodejs 7.4.0
asdf global nodejs 7.4.0
npm -g install brunch phantomjs

That’s it! We now have every language we need installed and ready to use! What if I need Ruby 2.3.1 for a client project?

asdf install ruby 2.3.1
asdf local ruby 2.3.1

And now I have 2.3.1 locally (I can change it to be the system default using global).

Most of the maintenance effort summarizes to this:

asdf plugin-update --all # update the individual plugins
asdf list-all [language] # to list all available versions

And that’s basically it! You have almost everything you need to develop software.

Useful Software to Install

Now, as usual, let me recommend some software:

# make sure you're up to date
sudo pacman -Syu

# install multimedia codecs
sudo pacman -Sy gstreamer0.10-plugins
sudo pacman -Sy exfat-utils fuse-exfat a52dec faac faad2 flac jasper lame libdca libdv gst-libav libmad libmpeg2 libtheora libvorbis libxv wavpack x264 xvidcore gstreamer0.10-plugins flashplugin libdvdcss libdvdread libdvdnav gecko-mediaplayer dvd+rw-tools dvdauthor dvgrab pulseaudio-equalizer-

# if you need japanese fonts like me
sudo pacman -Sy adobe-source-han-sans-otc-fonts otf-ipafont

# some components that you will need
sudo pacman -Sy fuse-exfat 

# I personally like the Numix theme and Breeze Icons, change them with the Tweak Tool
sudo pacman -Sy numix-themes breeze-icons 

# Ifnstall more good looking fonts
sudo pacman -Sy ttf-dejavu 
pacaur -S ttf-ms-fonts ttf-vista-fonts ttf-liberation adobe-source-sans-pro-fonts ttf-ubuntu-font-family

# Firefox and Java plugin
sudo pacman -Sy icedtea-web firefox

# for devs
sudo pacman -Sy zsh the_silver_searcher gvim imagemagick htop
pacaur -Sy ttf-hack

# Native wrapper for Web apps such as Slack, Hangout, etc
pacaur -Sy franz-bin

# Best native Twitter client for Linux
pacaur -Sy corebird

# No need to explain
pacaur -Sy spotify
pacaur -Sy sublime-text-dev # install these plugins http://www.hongkiat.com/blog/sublime-text-plugins/

# If you like to read RSS
pacaur -Sy feedreader-beta

# if you need Office-like support
sudo pacman -Sy libreoffice-fresh

# if you need Photoshop-like support
sudo pacman -Sy gimp
sh -c "$(curl -fsSL https://raw.githubusercontent.com/doctormo/GimpPs/master/tools/install.sh)"

# if you want a really good video editor
sudo pacman -Sy frei0r-plugins dvdauthor vlc
pacaur -Sy kdenlive

# this can make CPU-intensive software to behave better and guarantee better user experience
pacaur -Sy ananicy-git

# dropbox is the most horrible piece of software, but you may need it:
pacaur -Sy dropbox dropbox-cli nautilus-dropbox

As usual, I like to replace Bash for Zsh and configure Vim with YADR:

sh -c "`curl -fsSL https://raw.githubusercontent.com/skwp/dotfiles/master/install.sh`"
touch ~/.vimrc.before
touch ~/.vimrc.after
echo "let g:yadr_using_unsolarized_terminal = 1" >> ~/.vimrc.before
echo "let g:yadr_disable_solarized_enhancements = 1" >> ~/.vimrc.after
echo "colorscheme gruvbox" >> ~/.vimrc.after

To install and configure Postgresql 9.6:

sudo pacman -Sy postgresql
sudo -u postgres -i
initdb --locale $LANG -E UTF8 -D '/var/lib/postgres/data'
exit

# do not do this in Production machines
sudo sed -i.bak 's/ident/trust/' /var/lib/postgres/data/pg_hba.conf
sudo systemctl start postgresql
sudo systemctl enable postgresql

sudo -u postgres -i
createuser --interactive # create with your username and superuser role
createdb youruser
exit
sudo systemctl restart postgresql

If you want to install Docker:

sudo pacman -Sy docker
sudo usermod -aG docker $USER
sudo systemctl start docker
sudo systemctl enable docker
logout

We always need Redis, Memcached, so let’s install them:

sudo pacman -Sy redis memcached
sudo systemctl start redis
sudo systemctl enable redis
sudo systemctl start memcached
sudo systemctl enable memcached

After you install and remove a lot of software, you may end up with unnecessary packages eating up disk space. You can clean it up with:

sudo pacman -Rns $(pacman -Qtdq)

And as I said before, the Arch Wiki is super useful for you to keep tweaking your system, so make sure you read articles such as this “Improving Performance” page.

Desktop Kernels

One thing to keep in mind about most Linux distros is that the kernel is usually compiled to be better optimized for Servers.

Modern hardware, especially with lots of RAM and equipped with an SSD “should” work well enough. But not always, you may experience some “stutters” or even total unresponsiveness.

There are many reasons why, but the 2 main culprits are application memory being paged out to disk swap and the I/O scheduler of the Linux kernel.

In a server scenario, you want processes to have a fair share of resources, which is why a process scheduler such as CFS - Completely Fair Scheduler - and CFQ - Complete Fairness Queueing - for I/O, are fantastic.

But in the Desktop the story is totally different. You are willing to trade-off high throughput for lower latency in order to have responsiveness. No one wants to have their UI and mouse pointer frozen while copying large files to USB drives, or while waiting for that nasty make -j9 to finish compiling your also nasty gcc-gcj. You may end up with a frozen UI for hours! This is just unacceptable!

What you want for Desktop usage, with dozens of random processes doing random operations, is almost “soft real time” configuration. A more aggressive preemption where the Kernel gives some control back to the UI so you can do other stuff - albeit slower. Low latency is the key to have a smooth user experience.

To increase responsiveness, the most important first thing you want to do is configure this:

sudo tee -a /etc/sysctl.d/99-sysctl.conf <<-EOF
vm.swappiness=1
vm.vfs_cache_pressure=50
vm.dirty_background_bytes=16777216
vm.dirty_bytes=50331648
EOF

Reboot. If you want to know what those settings are, read this.

And you may want to install a customized Kernel with a different Scheduler. There are 3 options nowadays: Zen, Liquorix and CK.

I am still not 100% sure which one is the best, they have a few maintenance concerns.

Out of the 3, you will want to stick to Zen (which is basically Liquorix), as it’s maintained in the official repositories in binary format (believe me, you don’t want to wait for a custom kernel to compile, it takes forever).

sudo pacman -Sy linux-zen
sudo grub-mkconfig -o /boot/grub/grub.cfg

Reboot.

The main change is probably the I/O Scheduler, upgrading from the standard CFQ to BFQ. If you’re using a mechanical hard drive you will want to use the better BFQ (which Zen enables by default). If you’re using an SSD you will want ‘deadline’ instead.

DO NOT INSTALL THOSE KERNELS IN PRODUCTION SERVERS! They are intended for desktop usages only!

For the most part, Zen may have the better balance between stability and tweak set. You may want to use it, especially in older hardware. Modern hardware, as I said, may not notice too much difference.

Conclusion

I am not sure if it is the Arch maintainers that are doing a super job or if it’s RedHat and Canonical that are screwing up their distros so badly in comparison.

I mean, Ubuntu, Fedora, OpenSuse, Elementary, are all fair and nice distros that, most of the time, “just work”.

But for a distro that many consider targeted to “advanced users”, Arch is way more polished. I can’t figure out why.

In the same hardware, the Gnome 3 experience under Arch is noticeably better than the same Gnome 3 over Fedora. Compared to Unity on Ubuntu, it’s miles ahead. It’s fast, fluid, good looking, the defaults all work out nicely.

And all of a sudden I realize that I don’t have to worry about major upgrades. Rolling upgrades to the latest continuously brings me another layer of confidence.

Arch makes me feel like I am in control again without requiring me to waste hours tweaking things to my liking. The defaults are rock solid and I can make small improvements over it whenever I need to.

Kudos to the Arch maintainers, this is the finest Linux distro I’ve ever had the pleasure to play with. I hope this feeling goes on as I keep using this as my daily driver. But so far I am convinced that this is the right choice.

My team, my clients, we’ve been happily using Slack for more than 2 years, as I believe was true for many teams around the world. Although no one ever complained, I was always annoyed by the small things. First of all, as everything of value, it costs. Either I pay USD 6.67/month/user or I live with the restrictions of the free plan. And as most other teams, I lived with the restrictions for as long as I could.

For example, to get rid of the warnings to upgrade because we hit the upload limits, I created a small tool called slack_cleanup (first in Elixir, and then in Crystal just for exercise). This helped for a while.

Our team kept growing, steadily, frequently, as well as clients. The more users we add, the more conversations, the faster we hit the restrictions. History gets lost more frequently, we need to clean up uploads more often. It gets old very fast.

One thing I value above everything is knowledge. As a small example, I keep multiple backups of all my emails, all my projects, all my assets that I produced in the last 20 years. Heck, I have a 6 TB, Raid-5, Drobo system right in my home desk and 3 extra 1TB external drives for backup. I’ve lost very little over the years.

It really annoys me when I lose information.

• Gmail Business, DropBox, AWS S3 buckets, being external services, don’t worry me because I keep copies of everything offline. So if those accounts get busted all of a sudden, I have multiple copies.

• GitHub annoyed me a bit because although I have multiple copies of the repositories, I would lose all the Pull Request and Issues history. That’s one reason I moved to my own GitLab and helped tweak the import process to grab that Pull Request history.

• Slack annoys me a lot for the aforementioned reasons, which is why last week we tried MatterMost first, and then we deployed Rocket.chat.

There, and Back Again

We moved from Slack to Rocket.chat and a couple of days later we are moving again, now to Mattermost.

Yes, this was cumbersome. My team was not very happy about leaving Slack. Slack really is slick, full-featured, good-looking, well-rounded, a proper web product for this generation. Any alternative should be at least almost as good-looking, and at least have bug-free features, including webhooks.

Mattermost fits the bill almost perfectly but the open sourced Team Edition lacked one important feature for me: the ability to disallow normal members to rename and/or delete private groups. Yes, we expect grown-ups to behave, but when you have remote teams, remote clients, external users with no commitment to the company, it’s a hassle.

Yes, I could and I probably should use the paid Enterprise Edition. But for just that small feature, it felt too expensive. So that triggered me to let it go and install Rocket.chat instead. I moved my entire team there (almost 50 people, because I didn’t move the clients yet). That would be the end of the story.

But Rocket.chat has a complex infrastructure to deal with (you must have at the very least 3 boxes or pay extra for a proper Mongodb SaaS). Actually, in my previous post I explained why you MEAN guys should not be careless dealing with Mongodb. In a nutshell: Mongodb was not meant to run in a single instance, you must have a replica set. If you have a single instance Mongodb, you’re doing it wrong.

And the most problematic: the client-side is just too heavy. It frequently spikes out the CPU in not so powerful machines. It’s noticeably and measurably slower to navigate its UI, compared to Slack. MatterMost’s UI was much faster and way more responsive.

I was almost willing to overlook the lack of a proper test suite. I was willing to try to live with Meteor and CoffeeScript. I was willing to deal with the complex MongoDB maintenance.

But slow responsiveness across many of the members of my team is a no-go, a big show-stopper. We turned the beta videochat feature off (as it always triggers CPU spikes across all users), but many members still had a bad experience with a UI that was too slow and a resource hog.

The React-based, ES6-written, properly structured - with good enough client-side test suites - MatterMost was a more suitable candidate. So I decided to really think about the original problem and I came up with a simple solution: add a simple PLPGSQL function to be triggered whenever someone tried to delete a channel. Sure enough, it worked. And that prompted me to call my team again and propose this new change: I believe everybody was on-board as MatterMost was way faster on their machines.

I know I am sounding really harsh towards Rocket.chat, and it’s really not my intention. If we didn’t have any other options, we would still move to Rocket.chat. But as Mattermost proved to be the better choice, it was a no-brainer.

MatterMost Install

As always, I will not bore you with instructions that are already available online. If you want to install everything manually, follow this tutorial, but a better option would be the Docker-based deployment. You can even install/upgrade it together with GitLab.

Make sure you have NGINX in front of the server and that you have both a properly registered domain or sub-domain, and that you have SSL - use Let’s Encrypt.

Because I want to keep tweaking and experimenting with the codebase in a live installation, I installed everything manually and I have this directory structure:

-rw-r--r--  1 mattermost mattermost     6504 Aug 13 20:11 config.json
drwxrwxr-x  5 mattermost mattermost     4096 Aug 13 20:30 data
lrwxrwxrwx  1 mattermost mattermost       33 Aug 13 20:13 mattermost -> mattermost-team-3.3.0-linux-amd64
drwxr-xr-x  9 mattermost mattermost     4096 Aug 13 20:28 mattermost-team-3.2.0-linux-amd64
-rw-rw-r--  1 mattermost mattermost 19968308 Jul 14 16:37 mattermost-team-3.2.0-linux-amd64.tar.gz
drwxr-xr-x 11 mattermost mattermost     4096 Aug 13 20:28 mattermost-team-3.3.0-linux-amd64
-rw-rw-r--  1 mattermost mattermost 20241448 Aug 12 20:41 mattermost-team-3.3.0-linux-amd64.tar.gz
drwxrwxr-x  3 mattermost mattermost     4096 Aug 13 20:23 platform-dev-3.3.0
-rw-r--r--  1 mattermost mattermost 18060203 Aug 13 20:22 platform-dev.tar.gz

So I have a restricted, sudo user mattermost and I have a main mattermost directory pointing to the binary packages you will find in the official download page.

Notice that I have a copy of mattermost/config/config.json in the home directory. I leave it there so every time I download a new version and redo the symlink to mattermost, I can just do:

rm -Rf ~/mattermost/config/config.json
cd ~/mattermost/config
ln -s ~/mattermost/config.json

Also make sure you change at least the following in the config:

...
"FileSettings": {
    "MaxFileSize": 83886080,
    "DriverName": "local",
    "Directory": "/home/mattermost/data",
...

If you want, you can set file uploads to go to some S3 bucket you have, and just fill in the AWS details. But if you choose to have them locally, change the directory to somewhere outside of the mattermost folder, as on every upgrade you will change the folder. With both AWS EC2 or Digital Ocean you can always choose to add a secondary volume that can outlive the virtual boxes, so even if you get to a point where you have hundreds of concurrent users and you want to scale horizontally, you can have all your boxes pointing to a shared volume (AWS EFS, for example).

Speaking of which, in this configuration, upgrading would be like this:

sudo service mattermost stop
wget https://releases.mattermost.com/x.y.z/mattermost-team-x.y.z-linux-amd64.tar.gz
tar xvfz mattermost-team-x.y.z-linux-amd64.tar.gz
mv mattermost mattermost-team-x.y.z-linux-amd64
ln -s mattermost-team-x.y.z-linux-amd64 mattermost
rm -Rf mattermost/config/config.json
rm -Rf mattermost/data
cd mattermost/config
ln -s ~/config.json
cd ..
ln -s ~/data
sudo service mattermost start

The reason I wanted to have it this way is because I can tweak the code and manually push the changes.

For your development machine you should follow this instruction. If you’re in OS X and you choose to use Docker Toolbox, remember that you don’t need VirtualBox anymore as it will use OS X’s native hypervisor now. On my machine, I had to add dockerhost manually in my /etc/hosts because boot2docker was failing to get my ip address.

Then you can just clone the code from Github:

mkdir mattermost
cd mattermost
git clone https://github.com/mattermost/platform
git checkout -b v3.3.0 v3.3.0

Remember to always checkout the correct stable version (v3.3.0 as of the time when I originally posted this article) that you have installed in your server. Again, I will not bore you with what’s already documented in the links above, but you must have Go 1.6(.3), Docker, Docker-Composer, Docker-Machine all installed already.

You can execute make run and after it finishes (and as usual, npm will make sure it takes a very long time) you can open http://localhost:8065 to play with it locally.

Most importantly: you can tweak the React JSX components from platform/webapp/components, for example, the channel_header.jsx and add stuff like this:

# line 493 from v3.3.0
if(isAdmin || isSystemAdmin) {
    dropdownContents.push(
        <li
            key='rename_channel'
            role='presentation'
        >
            <a
                role='menuitem'
                href='#'
                onClick={this.showRenameChannelModal}
            >
                <FormattedMessage
                    id='channel_header.rename'
                    defaultMessage='Rename {term}...'
                    values={{
                        term: (channelTerm)
                    }}
                />
            </a>
        </li>
    );

    if (!ChannelStore.isDefault(channel)) {
        dropdownContents.push(deleteOption);
    }
}

And you know what this will do? Remove the “Rename Group” and “Delete Group” options from the channel menu if the user is not a system admin. Now how do you put this in your server?

First of all, you have to edit the Makefile at this section:

@# Make osx package
@# Copy binary
cp $(GOPATH)/bin/darwin_amd64/platform $(DIST_PATH)/bin
@# Package
tar -C dist -czf $(DIST_PATH)-$(BUILD_TYPE_NAME)-osx-amd64.tar.gz mattermost
@# Cleanup
rm -f $(DIST_PATH)/bin/platform

@# Make windows package
@# Copy binary
cp $(GOPATH)/bin/windows_amd64/platform.exe $(DIST_PATH)/bin
@# Package
tar -C dist -czf $(DIST_PATH)-$(BUILD_TYPE_NAME)-windows-amd64.tar.gz mattermost
@# Cleanup
rm -f $(DIST_PATH)/bin/platform.exe

@# Make linux package
@# Copy binary
cp $(GOPATH)/bin/platform $(DIST_PATH)/bin
@# Package
tar -C dist -czf $(DIST_PATH)-$(BUILD_TYPE_NAME)-linux-amd64.tar.gz mattermost
@# Don't cleanup linux package so dev machines will have an unziped linux package avalilable
@#rm -f $(DIST_PATH)/bin/platform

Notice that the developer that made this file is probably using Linux. That’s because this file builds the Linux-ELF binary as bin/platform, while the OS X Mach-O binary will be at bin/darwin_amd64/platform. Now, if like me you’re on OS X, you have to invert this and make the Linux version point to bin/linux_amd64/platform, otherwise the packages will have the wrong binary.

Then you will notice that there is this section right at the bottom:

setup-mac:
    echo $$(boot2docker ip 2> /dev/null) dockerhost | sudo tee -a /etc/hosts

If you’re on the newest versions of Docker-Toolbox, you don’t need boot2docker anymore, it will use docker-machine instead. So this line doesn’t work and the easiest way to make make test run is to change the dockerhost to localhost in the config/config.json’s PostgreSQL configuration.

If the tests are all passing, you can now run make package to make all packages, then you can send the dist/mattermost-team-edition-linux-amd64.tar.gz to your server and uncompress to the correct location.

As a sidenote for Go noobies like myself, I had my GOPATH pointed to /Users/akitaonrails/.go and my project was at /Users/akitaonrails/codeminer42/mattermost/platform. I was told that this is incorrect and the source of my many hours of frustration. The project MUST BE INSIDE THE GOPATH.

So I changed my GOPATH to /Users/akitaonrails/Sites/go and the project to /Users/akitaonrails/Sites/go/src/github.com/mattermost/platform and now it works. I would probably have symlinked the project path to be inside the GOPATH as well. But now everything compiles and runs just fine.

This is how I will keep experimenting for the time being, until I feel that I am comfortable enough to automate the whole process later. This should work just fine for my team for the time being.

Of course, the snippet above is nothing but a dirty hack, don’t try to contribute like this. A proper implementation would require me to at least create a new setting into the config.json, for example TeamSettings.MembersCanManageChannel to false, and change the api/context.go around line 347 to be something like this:

func (c *Context) HasPermissionsToTeam(teamId string, where string) bool {
    if c.IsSystemAdmin() {
        return true
    }

    if !utils.cfg.TeamSettings.MembersCanManageChannel {
        return false
    }
    ...

Then change the webapp/channel_header.jsx component in the React front-end (as well as proper unit tests to webapp/tests/client_channel.test.jsx), make sure the make test passes, and then finally create a feature request to the core team.

But the idea here was just to show that it was not so difficult to solve the show-stopper for my particular scenario, both using the SQL trigger to safeguard the database and the hack to fix the Web UI.

Conclusion

But what about the most important feature of all? RightGIF support? There is nothing as simple as a rightgif slack command just yet. Fortunately you can compensate most missing niceties like this by installing a Hubot server, and linking it to a user so you can chat with the bot and make it do things for you (set an alarm, fetch a gif, translate text, etc). As a caveat, the hubot adapter requires the use of the mattermost-client which must be synced with the server platform version releases to work properly, so be careful when you’re upgrading.

Overall, Mattermost is a great choice. It’s not for amateurs either, the development environment requires you to know your Docker stuff. It requires you to know proper Go-lang configuration. It requires you to follow proper procedures to contribute, as they should. The project itself is a single codebase divided into roughly 2 applications, a Go-based HTTP API and a React-based front-end to consume the APIs. Everything about the project is automated through the proper usage of Docker images (for mysql, postgresql, openldap instances) and Makefiles to run the test suite, create packages, etc.

And it has some conveniences from Slack that Rocket.chat doesn’t have yet such as a simple shortcut to switch channels (Cmd-K), the ability to reply to messages and organize them as threads, proper and more complete Markdown support. Overall, the features are well-rounded, and not half-baked. What is there is solid and works, it’s always bad to have half-finished features.

With these hacks in mind I can strongly recommend that you use Mattermost. And as I said in previous posts, it’s not just a matter of cost. If you’re a small team, without internal developers or someone that can maintain your own installation, you should definitely pay Mattermost for the Enterprise support, it’s affordable and way cheaper than Slack.

For now, it’s the better choice, both in terms of overall fit-and-finish, well-rounded features, simple and responsive UI, good code quality and overall care on the engineering.

I’ve been using Pivotal Tracker since 2010 up to 2015 and this is because I believe it is far superior to Trello and other glorified to-do lists out there.

The Cost Conundrum

As with all the other reports, let me start with the cost argument. Pivotal Tracker is priced based on the number of users.

If you have a small team and not many outside clients, a basic 10-user plan would be enough, and you will pay just USD 420 a year. Just pay it already.

If you have up to 25 users, it’s still not so expensive, and you will pay USD 1,800 a year. Well worth it.

Even if you have 50, the maximum plan, you will pay USD 3,600 a year. More than that and you have to contact them directly.

The cost is not prohibitive and if you can pay, I advise you to do so.

In my case, the strategy is to get full control over our data and knowledge, which is why I’d rather have ownership of the service.

And because Fulcrum was “almost” good enough, I decided to try it out and see if we could evolve it over time. And I think that even though we are still very far from the full-featured Tracker, what we have is enough for most of our projects.

Let me say it again: you most probably won’t benefit if you’re a small team. Pivotal Tracker is, to this day, the best project management tool for software development teams, and you should definitely try it out first, get used to its workflow (as I will explain below), and then decide if you want to take control and install the open source alternative instead.

There is the factor of cost, but it’s not the driving incentive for this move.

The Important Features

Doing some research I found the Fulcrum project. It’s a very simple, very bare-bones Rails project that mimics Tracker’s most important features.

Any competent project management tool must have at the very least 4 pillars:

• Stories (with discussion threads, assets uploading, tags, Epics organization, and searchability)
• Proper numeric estimation (story points, proportional size for stories)
• Time-boxed Sprints (short, fixed periods of time)
• Velocity (the actual productivity of a particular team in a particular project)

Any tool that has all of the above is useful.

Fulcrum itself has almost everything. We made an internal fork of the project and we started adding some of the missing features. We’ve been using it for around 1 year already and it’s been good enough for our routine. There are still a few bugs and missing features that we intend to add.

If you’re interested, I am making this fork publicly available to conform to the Affero GPL license.

Any contribution is welcome.

On Proper Agile Management

Now sidetracking a bit, I’d like to explain why this kind of tool (Tracker-like) is the best compared to what I called “glorified to-do lists”.

The principles are very simple:

• Every project must start with the most complete backlog possible (uncertainty is at its highest in the beginning, and details will be missing, but it’s important to have a vision of what “minimally complete” means and the order of importance of each story, a proper “priority”).
  • The level of detail doesn’t matter. A story must have enough information that any developer in the team can implement the required tasks. Otherwise it must be tagged “blocked” so the Product Owner knows that (s)he has to add more details and no developer will touch it until the tag is removed.
• All stories must be estimated (more on that on the next section). The unit really doesn’t matter. You can think in hours, in fractions of a day or for what it really is: a proportion. A story sized “2” should be roughly half the complexity of a Story sized “4”. That’s it. Even “blocked” stories should receive sizes.
• Projects must be divided into time-boxed, fixed-sized Sprints, so the team has the opportunity to reassess frequently. This is where estimation can be refined, where more levels of detail in the stories should be required.
• Bugs can’t have any numeric estimation. They will serve to slow down the team’s velocity and they’re an indication of poorly made deliveries.
• Velocity is the actual amount of story points delivered by the end of a Sprint.
  • The total amount of story points of all stories in the project backlog divided by the team’s velocity will give a roughly estimated “End Date”. This is important, because this is how you know when certain groups of features will be delivered over time, and this is what the Product Owner needs to know to prioritize stories, simplify stories or even cut down unnecessary features.
  • The Product Owner’s sole responsibility is to be accountable for the Return on Investment of the implemented features. And the only way to measure cost-benefit is to know “how much” a feature will “cost” in terms of story points divided by velocity.

What the tool will expose:

• If a backlog has a lot of “blocked” stories or if it has many stories delivered without acceptance, it’s a symptom of a Product Owner not doing the work.
• If the velocity varies too much between sprints, it’s a symptom of either poorly written stories (and therefore developers having a hard time understanding what to do), or developers not concentrating on their work and delivering inconsistent results.
• Inconsistent velocities with many bugs in each sprint is the symptom of developers not doing code with proper quality (automated tests to avoid regressions for example).
• A “bug” is not the same thing as the Product Owner changing their mind on a story and rejecting. The team must not implement undocumented changes. Acceptance is tied to what the story says. The Product Owner must change his mind before the story is implemented or create a new story to add the change (and therefore add more cost to the project, there is no free lunch).
• Stories in the “Chilly Bin” (in Fulcrum, or “Ice Box” in Tracker) are “wish-lists” and they will never be implemented until the Product Owner moves the story to the backlog.
• Velocity should be roughly the same between sprints. Peaks or valleys in the velocity are symptoms of a problem that needs to be addressed by the next sprint. This is how a project evolves for the better over time.

On Estimation

Many people worry too much about “proper estimation” to the point of creating so much tension that developers start screaming about “no estimates”. This is nuts. This is why “estimation” is a different word than “prediction”, one is uncertain, a ballpark, the other is exact.

There is no such thing as a “correct” estimation.

I would argue that in many projects if the stories are minimally described (meaning that a developer is able to implement what is written), we could estimate all of them to be size “1” and forget about it.

We run 2 or 3 sprints (usually one sprint should be one work-week - 5 days of at most 8 hours, no more, no less) and measure how many stories were delivered. And this is the velocity.

Now we can see how many stories we have until the end of the project and divide by this velocity. And this is how many weeks it will take to finish the project.

There is no need for consensus based estimation.

Consensus on estimation is usually a waste of time. Estimation is merely the realization that it’s difficult to write down every story to be the exact same size, so we add proportions to them (A story size 2 is half the size of a story size 4).

But if a project is 3 months or larger, the less the proportions matter (unless you do the anti-practice of having heavily disproportionate stories, some size 1 and many sized 5, 8 or even 10, for example). But if you stay closer to less than size 5 for every story, then the larger the project, the less the individual estimates matter. Velocity will average out the differences over time.

Then the task of the Project Manager, Product Owner, the team in general, is to keep an eye on the total amount of points divided by this velocity, which derives the estimated end date of the project. And everybody can negotiate, beforehand, which stories can be done, which cannot, and change priorities, remove stories or simplify stories.

Don’t waste time trying to create formulas to derive estimation consensus.

If you manage the Velocity then you can manage the entire project. This is the core of all proper Agile methodologies. Everything else is an accessory to that end.

Trying to artificially increase velocity is one of the many sins. You can’t. It’s better to invest time in adding more details to a story, or to simplify a story or to decide that some stories are just not needed to deliver a proper product in the end.

Monte Carlo simulations, Six Sigma techniques, Kanban concepts, none of them matter. All traditional industry techniques are meant for production lines in factories, where the time of each task is well defined and you manage the variance, the exception. It’s a Gaussian, Normal world.

In software projects, or any craft and research, you manage expectations on results based on milestones and cutting excesses and uncertainties. It’s a Pareto world, one where there is no such thing as an “average”, there is no medium, no variance.

In a Pareto world everything is interconnected, not statistically independent (a requirement for Gaussians). It’s therefore 80/20: where 20% of the project will yield 80% of the results, where 20% of the scope can be cut down.

Anything that is not “statistically independent” (like tossing a coin) can’t be averaged out, can’t fit in a Normal distribution. It’s most possibly a Pareto, Power Law, Zipf, Exponential, or even a Poisson, not a Gaussian. Software parts are all intertwined in a network, they are not independent events, they can’t be randomly rearranged, they are not easily interchangeable.

Conclusion

This article is very short and I can expand each bullet point above into its own, very long, article. Let me know in the comments section if I should, and which points raise more questions.

I’ve been refining those techniques for the past 10 years, and at least 5 within the constraints of a tool such as Tracker. And I believe I was able to come up with the minimal set of principles that lead to proper management, not superstition, fantasy or delusion you see everywhere nowadays.

Projects suffer when the recommendations above are not followed:

• Product Owners don’t perform their responsibilities: they fail to add details to stories, they neglect delivered features and don’t test them properly, they don’t participate.
• Backlogs are incomplete, lacking detail, lacking estimation, lacking prioritization.
• Developers don’t properly start and finish their tasks with proper code quality.
• Time is wasted in bikeshedding instead of delivery.

Our fork of Fulcrum, called “Central”, is not perfect by any means, but it works well enough and gives us enough control to keep adding features that make our routine more comfortable, and we are just getting started.

I hope this technique and tool is valuable to more teams and I’d love to hear feedback from anyone.

In my previous post I forgot to mention minimal firewall and SSL configuration, but you can check it here.

One of the most difficult pieces to move away from is Slack. Don’t get me wrong, it’s a great tool and it gets the job done. The free-tier is really competent but I really want some of the paid features such as history of all conversations with search capabilities and uploads without limit.

The Cost Conundrum

But then you have to commit to pay USD 8/month per user (USD 6.67 if you pay ahead of time). If you’re a small team (around a dozen people) it’s well worth it and you should definitely pay and be done with it.

If you have more than 20 people then it starts to become a conundrum. On my GitLab cost calculations, some people argued that saving USD 15k yearly is not such a big deal, and they’re not wrong.

If you’re from outside of the US coastal cities, especially from countries in South America or Asia where currency can be more than 3 times weaker, then it quickly starts to become a big deal.

Apart from that, we are currently offloading a whole lot of our IPs and knowledge into the care of companies we don’t really know and where we don’t hold any real stakes.

If you’re a medium to big company it really starts to make sense to want to have more control over your own property. Of course, it costs some work and maintenance, so it’s not for everybody.

What I will argue is that if you have 150 people or more, and you’re still growing, it’s not such a big burden to jump into your own infrastructure.

You avoid paying around USD 12,000 a year or more, and instead you will pay USD 960 a year in Digital Ocean boxes and MMS plus a few hours every month in basic maintenance. That’s 12 times cheaper.

Again, you shouldn’t do it just for the cost savings as the maintenance burden can easily outweigh the savings if you or your team are not devops savvy and you don’t have anyone to keep monitoring it. For most companies, the conversation history is not so important and even the free-tier will be more than enough.

Cost is less relevant depending on your context, and there are many different uses for group chats. For example, if you’re an open source community, you would try Gitter instead.

There are also many other alternatives that you may want to try.

For the sake of this article, let’s assume you’re a small to medium company, in need of private rooms, many external users (for example, your clients), with at least 50 people. Then this can start to make sense. Any other scenario and we have to analyze it differently.

Why I chose Rocket.chat

With the cost reasoning out of the way, then it’s a matter of choosing which alternative to use.

I tried MatterMost first and I really wanted to like it. It has a very similar feature set to Slack and most importantly, it’s written in Go. So it’s both lightweight, very fast, and not so difficult to contribute. It uses MySQL or PostgreSQL which I like.

Unfortunately, it has an important show-stopper for my needs. You can create both public channels or private groups, but anyone can delete a private group. If you want permission control for that you must use their “Enterprise” offering and pay USD 20/year per user.

It’s still at least 3 times cheaper than Slack, so it may be a good option for you.

I really want to have full-control, not just make it cheaper. I could pay Slack already if I didn’t care, so I would not just pay an alternative and not have full control (including the ability to tweak and improve the entire code base).

Then, it took me back to Rocket.chat. Its feature set also rivals Slack, it’s good looking enough. But, it’s made in Meteor. Now, I don’t have anything against Meteor and I really think a Slack-clone is exactly the kind of use case where you could use Meteor to its full potential.

Technically I really think it’s a downside to be forced to use MongoDB (Meteor requires Mongo). For small installations I really prefer to have PostgreSQL. MongoDB is competent, but for medium to big installations there are caveats, and I will show you why below.

I strongly dislike the culture of writing software without minimal care such as having a reasonably complete test suite. You browse through the many packages that comprise Rocket.chat and several of them have no tests whatsoever. Some packages do have jasmine tests, but the majority is lacking.

This is one example I picked randomly, out of the few test files I found:

describe 'rocketchat:katex Client', ->

  it 'should exist', ->
    expect(RocketChat.katex).toBeDefined()

This is sad, sorry to say that, but it is. Not surprisingly I’ve seen katex issues while using the interface. And if you think this was just a poor choice, I picked another test file:

describe 'rocketchat:markdown Client', ->

  it 'should exist', ->
    expect(RocketChat.Markdown).toBeDefined()

Really?

So, the code itself is not so pretty, you have to keep this in mind. I’d much rather use Mattermost (which is much better structured and with enough tests), but it’s not entirely free, so it’s not an option for me. For my scenario, I’d rather have sloppy code (that minimally “works”) that I can tweak than code I can’t see, for this particular venture. Many inexperienced developers may argue about the necessity of having automated tests, but not having them also impacts this:

If we see how the project is evolving over time, we can see that interest is diminishing. The lack of good automated tests also makes it more difficult for new people to contribute without adding regressions, and makes contribution reviews more difficult and time consuming.

On the other hand the feature set is good enough as it is. There is only so much you can add to a Slack-clone. The upside is that if the code doesn’t change too much, I can tweak it more without having many things breaking all the time. It’s the complete opposite from the impressive contribution boost over at GitLab, for example.

Doing a few days of testing through the administration settings, user interface, having dozens of users doing real conversation, convinced me that even though the code quality is bad, the feature set is indeed there. I think I can live with that. Your mileage may vary though.

I am using Franz to open Slack, Rocket.chat, Hangouts, Messenger and any number of other communicators in one single app, which is great. And Rocket.chat does provide mobile apps based off PhoneGap. Again, it’s good, not great.

Many people have heavy bots or integrations into their Slack configuration. The alternatives such as Rocket.chat do support incoming and outgoing webhooks and most of the integrations should be possible (with some tweaking and hacking such as this Giphy example), so make a list of all important integrations and do some Google research on that before continuing.

Planning the Infrastructure

All that having been said, I’d recommend you try Rocket.chat’s demo server first to see if it has the minimal features you need. And also that you install a dry-run in a small Digital Ocean box with the intention of blowing it away later.

For a single-instance, stand-alone installation, temporary dry-run, I recommend you start by installing the MongoDB One-Click Application and follow this instruction to install everything in one machine.

This is the worst possible production environment, which is why I am being so repetitive in saying that you should destroy this box after you do your testing.

As I’ve said before, I am assuming a scenario where you have 50 or more people, with real clients, important communication going on. You don’t want to keep a simple installation like this for production usage.

For a minimal production installation you will need at the very least 4 boxes with 1 GB of RAM each. Now, why 4?

This is an important section for most of the beginner Javascript programmers out there playing with the so called “MEAN” stack (MongoDB, Express, Angularjs, Node.js). I’ve seen people deploy single-instance, stand-alone “MEAN” apps for production usage. And this is super bad.

MongoDB is NOT MEANT to function with just one single instance!

More than that:

DO NOT start an even number of MongoDB nodes, or at least install an Arbiter

So, the minimal setup for MongoDB is a Three-member Replica Set, no less!

The article “MongoDB Gotchas & How To Avoid Them” is a little bit old but you should be aware of several of the documented gotchas, including adding replica sets and avoiding waiting too long to start sharding your data.

The most important part of making Rocket.chat reliable is to make your MongoDB minimally reliable, and this is a hidden cost you must consider.

Installing a Three-Member Replica Set

For the sake of this article I will assume that you created 3 (three) MongoDB One-Click Applications on Digital Ocean’s, at the very least with 1 GB RAM boxes (2 GB RAM machines are a good choice if you’re in doubt, but never the 512 Mb) and another bare Ubuntu 14.04 box to be the web server. Create all machines in the same sub-region, all of them with private networking and backup enabled.

Take note of the public IPs and private IPs, for the sake of this article let’s say you have this:

Public IP     Private IP    Hostname

192.160.10.1  10.0.0.1      mongo1.yourdomain.com
192.160.10.2  10.0.0.2      mongo2.yourdomain.com
192.160.10.3  10.0.0.3      mongo3.yourdomain.com
192.160.10.4  10.0.0.4      yourdomain.com

In the commands and code snippets below, make sure to replace the fake IPs with your own private IPs.

As I said in my GitLab post, create a swap file, configure the locale, set it up for automatic security updates, and upgrade the Ubuntu packages.

The next first thing: configure the firewall in all machines (assuming ufw package is already installed):

sudo ufw reset
sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw allow https
sudo ufw allow ssh
sudo ufw allow in on eth1 to any port 27017
sudo ufw enable

We are allowing MongoDB’s 27017 port only for machines in the same private networking (through eth1, the public IP goes through eth0). We’re also allowing port 22 for SSH and although I am not listing it here in this article you should also consider installing and configuring fail2ban. Digital Ocean also has a good post on UFW for more details.

Next thing to do on the MongoDB machines is to Disable Transparent Huge Pages (THP). For that create a file /etc/init.d/disable-transparent-hugepages, with this content:

#!/bin/sh
### BEGIN INIT INFO
# Provides:          disable-transparent-hugepages
# Required-Start:    $local_fs
# Required-Stop:
# X-Start-Before:    mongod mongodb-mms-automation-agent
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Disable Linux transparent huge pages
# Description:       Disable Linux transparent huge pages, to improve
#                    database performance.
### END INIT INFO

case $1 in
  start)
    if [ -d /sys/kernel/mm/transparent_hugepage ]; then
      thp_path=/sys/kernel/mm/transparent_hugepage
    elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then
      thp_path=/sys/kernel/mm/redhat_transparent_hugepage
    else
      return 0
    fi

    echo 'never' > ${thp_path}/enabled
    echo 'never' > ${thp_path}/defrag

    unset thp_path
    ;;
esac

Now you can register it as a service:

sudo chmod 755 /etc/init.d/disable-transparent-hugepages
sudo update-rc.d disable-transparent-hugepages defaults
sudo /etc/init.d/disable-transparent-hugepages start

Stop MongoDB in all 3 machines:

sudo service mongod stop

Now comes a complicated procedure that you should not do wrong. It’s only complicated because you have to do it in all 4 machines minding their correct private IP addresses. If you already have your own DNS server, you can skip this, otherwise follow through carefully.

You will have to edit both /etc/hostname and /etc/hosts. Each machine will have a particular name and they should be able to find the other machines by name. Let’s start by editing the /etc/hostname on each machine. Use the table in the beginning of this section, for example, in the machine with public IP 192.160.10.1 you must edit its hostname to be mongo1.yourdomain.com, and so on for all the machines.

Once you have it done, you have to replace the line 127.0.0.1 localhost in the /etc/hosts in each of them for the following:

127.0.0.1 localhost mongo1.yourdomain.com
10.0.0.1 mongo1.yourdomain.com
10.0.0.2 mongo2.yourdomain.com
10.0.0.3 mongo3.yourdomain.com
10.0.0.4 yourdomain.com

This snippet assumes you’re in the machine with private IP 10.0.0.1, of course. Make sure you change mongo1.yourdomain.com in the first line for the hostname of the machine you’re in. If this is too difficult for you, you definitely should not try to maintain your own infrastructure, so beware.

Finally, let’s edit the /etc/mongod.conf with these changes:

storage:
  dbPath: /mongodb-data
...
net:
  port: 27017
  bindIp: 0.0.0.0
...
replication:
  replSetName: rs0

This should be exactly the same in all machines. And you must create this /mongodb-data directory like this:

sudo mkdir /mongodb-data
sudo chown -R mongodb:mongodb /mongodb-data

Now go ahead on machine mongo1.yourdomain.com, start up with sudo service mongod start and fire up the mongo command, and you should see something like this:

# mongo
MongoDB shell version: 3.2.8
connecting to: test
>

Let’s now configure the replica set by issuing this command in the MongoDB command-line interface:

rs.initiate()
rs.add('mongo2.yourdomain.com')
rs.add('mongo3.yourdomain.com')

And you can now log in to the secondary machines and sudo service mongod start in all of them. They should all start joining the replica set and syncing.

You can check the status of the replication like this:

> rs.conf()

{
  "set" : "rs0",
  "date" : ISODate("2016-08-09T17:33:10.466Z"),
  "myState" : 1,
  "term" : NumberLong(-1),
  "heartbeatIntervalMillis" : NumberLong(2000),
  "members" : [
    {
      "_id" : 1,
      "name" : "mongo1.yourdomain.com:27017",
      "health" : 1,
      "state" : 1,
      "stateStr" : "PRIMARY",
      "uptime" : 11727,
      "optime" : Timestamp(1470763990, 3),
      "optimeDate" : ISODate("2016-08-09T17:33:10Z"),
      "electionTime" : Timestamp(1470752553, 1),
      "electionDate" : ISODate("2016-08-09T14:22:33Z"),
      "configVersion" : 6,
      "self" : true
    },
    {
      "_id" : 2,
      "name" : "mongo2.yourdomain.com:27017",
      "health" : 1,
      "state" : 2,
      "stateStr" : "SECONDARY",
      "uptime" : 11693,
      "optime" : Timestamp(1470763988, 4),
      "optimeDate" : ISODate("2016-08-09T17:33:08Z"),
      "lastHeartbeat" : ISODate("2016-08-09T17:33:10.026Z"),
      "lastHeartbeatRecv" : ISODate("2016-08-09T17:33:08.887Z"),
      "pingMs" : NumberLong(0),
      "syncingTo" : "mongo3.yourdomain.com:27017",
      "configVersion" : 6
    },
    {
      "_id" : 3,
      "name" : "mongo3.yourdomain.com:27017",
      "health" : 1,
      "state" : 2,
      "stateStr" : "SECONDARY",
      "uptime" : 11693,
      "optime" : Timestamp(1470763988, 4),
      "optimeDate" : ISODate("2016-08-09T17:33:08Z"),
      "lastHeartbeat" : ISODate("2016-08-09T17:33:09.994Z"),
      "lastHeartbeatRecv" : ISODate("2016-08-09T17:33:08.655Z"),
      "pingMs" : NumberLong(0),
      "syncingTo" : "mongo1.yourdomain.com:27017",
      "configVersion" : 6
    }
  ],
  "ok" : 1
}

Each box should be syncing data to another box, one of them will be marked “PRIMARY” and the others will be “SECONDARY”. If one of them “dies” for some reason, one of the others will be elected as PRIMARY until you add a replacement box. Read more details on this procedure in Digital Ocean’s post about implementing replication sets.

Adding your MongoDB environment to MMS

Another thing beginners will not realize is that it’s a good idea to monitor your services. At the very least you should install MongoDB Cloud Manager agents into your servers. Just set up an account at MongoDB Atlas.

It’s beyond the point of this article to explain Atlas, but you should have no trouble installing the agents with the Group ID and API Key provided. Then install the monitoring agent and have the Manager figure out your existing deployment and replica set.

The basic subscription for monitoring will cost you an extra USD 39 a month, but believe me when I say it’s worth it to add all your MongoDB deployments under MMS.

Installing a multi-instance Node.js service

Another mistake beginners in the MEAN stack make is to start up a single Node.js instance. Yes, Node.js’s asynchronous I/O nature makes it “concurrent enough” in 1 single thread. But you do want to maximize the rented machine so you should spin up at least one Node.js instance for each CPU core. A 1 GB RAM machine has just 1 core, but you can spin up at least 2 instances.

First of all, we must set up the yourdomain.com (192.160.10.4 or 10.0.0.4 in the example) to support Node.js, let’s do it:

sudo apt-get install -y npm curl graphicsmagick nginx git bc
sudo npm install -g n
sudo npm install -g forever
sudo npm install -g forever-service
sudo n 0.10.40

If you haven’t already, create a more restricted sudo user:

adduser rocket # add a password when asked
usermod -aG sudo rocket
su - rocket # and always ssh in with `ssh rocket@yourdomain.com`

Within rocket’s home directory let’s install the Rocket.chat codebase:

cd /home/rocket
curl -L https://rocket.chat/releases/latest/download -o rocket.chat.tgz
tar zxvf rocket.chat.tgz
mv bundle Rocket.Chat
cd Rocket.Chat/programs/server
npm install

And now let’s add a way for the machine to start the Rocket.chat whenever it reboots:

cd ~/Rocket.Chat
sudo forever-service install -s main.js -e "ROOT_URL=https://rocketchat42.com/ MONGO_URL=mongodb://mongo1.yourdomain.com:27017/rocketchat MONGO_OPLOG_URL=mongodb://mongo1.yourdomain.com:27017/local PORT=3001" rocketchat1
sudo forever-service install -s main.js -e "ROOT_URL=https://rocketchat42.com/ MONGO_URL=mongodb://mongo1.yourdomain.com:27017/rocketchat MONGO_OPLOG_URL=mongodb://mongo1.yourdomain.com:27017/local PORT=3002" rocketchat2

sudo start rocketchat1
sudo start rocketchat2

Let’s also create SSL certificates (and for that you must have a properly registered domain, of course). Again, Digital Ocean’s documentation on Let’s Encrypt is very good. In summary, all you have to do is:

sudo git clone https://github.com/letsencrypt/letsencrypt /opt/letsencrypt

Now edit the /etc/nginx/sites-available/default and add this inside the server block:

server {
  ...
  location ~ /.well-known {
          allow all;
  }
  ...
}

Now you can:

sudo service nginx reload

sudo openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048

cd /opt/letsencrypt
./letsencrypt-auto certonly -a webroot --webroot-path=/usr/share/nginx/html -d yourdomain.com -d www.yourdomain.com

Again, replace yourdomain.com with your own registered domain, of course. The letsencrypt command will prompt you to enter your e-mail address and accept terms of service.

Let’s Encrypt is a fantastic free SSL provider. Their certificates are short lived and are meant to expire in 90 days, so you must set up auto-renewal. Start sudo crontab -e and add the following lines:

30 2 * * 1 /opt/letsencrypt/letsencrypt-auto renew >> /var/log/le-renew.log
35 2 * * 1 /etc/init.d/nginx reload

This should take care of SSL. Now you can edit your /etc/nginx/sites-available/default and replace it with this template:

# Upstreams
upstream backend {
    server 127.0.0.1:3001;
    server 127.0.0.1:3002;
}

server {
  listen 443 ssl;

  server_name yourdomain.com;

  ssl_certificate /etc/letsencrypt/live/yourdomain.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/yourdomain.com/privkey.pem;

  ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
  ssl_prefer_server_ciphers on;
  ssl_dhparam /etc/ssl/certs/dhparam.pem;
  ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
  ssl_session_timeout 1d;
  ssl_session_cache shared:SSL:50m;
  ssl_stapling on;
  ssl_stapling_verify on;
  add_header Strict-Transport-Security max-age=15768000;

  location / {
    proxy_pass http://backend;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
    proxy_set_header Host $http_host;

    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forward-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forward-Proto http;
    proxy_set_header X-Nginx-Proxy true;

    proxy_redirect off;
  }

}

server {
  listen 80;
  server_name yourdomain.com www.yourdomain.com;
  return 301 https://$host$request_uri;
}

Just for the sake of completeness, again: replace yourdomain.com with your domain. Usually the /etc/nginx/sites-enabled/default is a symlink to /etc/nginx/sites-available/default, just check that as well. This configuration will load balance between the 2 node.js instances we configured and started before. And you can add as many as you want following the same forever-install procedure and adding the new instances to the upstream section in the nginx configuration above.

If everything is correct, you can restart nginx again: sudo service nginx restart.

And now you should have https://yourdomain.com already up and running. The first user that signs up will be the site administrator and from there you can customize Rocket.chat internally.

Conclusion

As you can see, this is not a simple procedure to follow and I am assuming you have experience managing your own infrastructure. If you don’t, you definitely should NOT do this by yourself.

If you do it correctly, you should have a functional Slack-clone with minimal reliability (thanks in large part to Digital Ocean) and minimal cost (USD 80/month in the configuration I described that should be enough for more than 100 users). Rocket.chat also offers more documentation for other environments including Docker configurations you might want to try. But the procedure above is sufficient for my needs.

I said it already but it’s better to repeat it: don’t fall for the trap of installing everything in a single box, with single instance MongoDB, and without proper monitoring. It’s asking for trouble.

For now I am in the middle of the rollout. All developers in my company are already in the new deployment and soon half of the clients should also migrate (some will not be able to leave Slack just yet), but with Franz and the ability of full history and searchability this shouldn’t be a concern even if a developer stays offline for some period of time.

I also don’t advocate that everybody should be online and responding in real-time. It’s infeasible, unproductive, creates unnecessary tension. People should participate when they have free time, and they should be able to concentrate without worrying that they are missing something. That’s why they should opt-out of being notified in the more busy channels and just enable notification on the private groups that matter.

Coincidentally Jason Fried just posted about concerns over group chats at Signal v. Noise, but the gist is that group chat should be purposeful and not yet another tool to create tension. People should definitely get offline when they need to fully concentrate on their work and have the opportunity to catch up with interesting conversations later. And really important communication should go through e-mail or other traditional ways, a simple @all doesn’t cut it for company-wide announcements for example.

I hope this exercise gives you more perspective on what you can have and also raises awareness on the need for companies to regain more control over their own data, particularly knowledge. Erase your communication channels and you’re losing years’ worth of knowledge that can be invaluable.

It’s been a while since my last off-topic, but today I’m so indignant I had to write.

After reading good reviews (I need to find better sources), I took a chance and watched I Am Legend, which had just opened here. I really liked Will Smith’s previous film, The Pursuit of Happyness, and word was he goes dramatic again. That got me to the theater.

The beginning of the film is very good. Great cinematography, fast and dynamic editing. And even though Will Smith is practically the only actor on screen for most of it, he does a solid job. The technical crew recreated a deserted New York beautifully, and the tense moments work. In short, every trick in the book got used.

But the first time one of the monsters shows up, the disappointment kicks in: it brought back the horrendous special effects of films like The Mummy Returns. In a post-Matrix, post-Lord of the Rings era, there’s no excuse left for monsters this bad. Still, that part is forgivable.

The other problem is that the story is only superficially based on Richard Matheson’s novel. A shallow adaptation wasn’t hard to predict, but the film had been building good tension and was fun. Which is why the third act was doubly disappointing. The entire sequence involving Alice Braga is disposable.

It starts with a joke I still don’t get. Alice’s character, Anna, says she came from São Paulo, Brazil. Mid-conversation, Robert, Will’s character, asks if she knows Bob Marley. And she makes a face like: “never heard of him”. She says it out loud!

Ok, the story is set in 2012, not that far away. Even I, practically music illiterate, know Bob Marley and the song they played in the film. What are they trying to say? “In Brazil there’s only jungle, so nobody there has ever heard of Bob Marley?” Not important, but I found it strange. I’ll give the benefit of the doubt, because maybe I didn’t get the joke. Who knows.

Now, the “God sent me” part … no, no, so fucking sorry! I couldn’t swallow it. What does that have to do with the story?? Worse, the story took it seriously!! This Anna character “started listening better to signs from heaven”. She showed up to save Robert the instant he was about to die, heard his radio message because she got a divine signal, and she knows, simply knows, that a survivor colony exists, address and all, more efficient than Google Maps, because God told her … oh well …

And in the end the guy actually has a divine vision and decides to die! And the colony really exists! Wow. I hate it when a film starts thinking too highly of itself.

It left a worse taste than another terrible one: Signs, by M. Night Shyamalan. Same deal: the film starts with a serious mood, builds the tension, goes along great until it completely falls apart, also in the third act. The dying woman delivers a message from heaven: “oh, tell him to swing the bat …” And they show the little green aliens!! And the guy swings the bat and saves the world!

God damn! Ok, I Am Legend has nothing to do with the novel of the same name. The book ends in a completely different way, and the theme is something else entirely, far more interesting. If you go see the film, walk out when Anna shows up. Up to that point it’s a decent film; after that, brace for a huge disappointment.

I’m shocked by the sheer lack of quality in the storytelling. Akiva Goldsman worked on the screenplay, which explains a lot: the same person who managed to ruin the Da Vinci Code adaptation.

I’ve seen bad films by the dozen, but they know they’re bad and don’t take themselves too seriously. Take Resident Evil: mediocre, aware of it, and at least fun for a lazy Sunday afternoon. This one promises a lot up front and ends like that guy who turns around and goes “gotcha!”

The marketing team behind this film is very good! They managed to squeeze plenty of good reviews out of everywhere. Even on Rotten Tomatoes it got 68% approval! Far more than it deserves. Anything above single digits, on a zero-to-one-hundred scale, is already too much.

The next film people are talking up, with some hype already building, is Cloverfield. Whether it’s as good as they say remains to be seen.