One category was missing that I had never tested properly: the big open source models. The ones that fit on my RTX 5090 with 32 GB or on my Strix Halo with 61 GB usable by the GPU are too weak for this. They work for simple agent tasks, and no small model handles real programming tasks, let alone a complex software project.

Qwen 3.8 27B took 706 minutes to finish v4 on the Strix and scored 80; the local GLM-4.7-Flash scored 24. So don’t even try.

But what if we push harder? What can mid-sized models, 100 to 400 billion parameters, do running on a single rented B300? I could rent four of them and run the giant 750-billion or one-trillion models, and I’m not sure the cost-benefit is worth it. The data below answers.

TL;DR: 14 open models, one 288 GB B300 on Runpod at about 9 dollars an hour, the same v4 sabotage protocol. None of them serves fast the way it ships: GLM 5.3 Flash decoded at 10 tokens per second with the default configuration and at 141 after tuning, 14 times more from flags alone. Two open models, GLM 5.3 Flash and Qwen3.8 Flash-Next, scored 92.5, Claude Sonnet 5 level, and Swift 1.5 27B scored 88, Opus 4.6 level. None reached the frontier’s 100. The 397-billion flagship scored 68.5 and let an admin backdoor through, DeepSeek V4 Flash scored 86 in the cloud and 44.5 served by me, and Nemotron cheated on a regression test. The cloud API of the same model costs a dollar per run; the pod costs 218 dollars a day. Jump to B300 and Runpod, server tuning, the ranking, the interesting cases, the frontier comparison, the math or the conclusion.

What a B300 is, and what Runpod is

The B300 is the datacenter GPU of NVIDIA’s Blackwell Ultra generation, announced in March 2025. Each card has 288 GB of HBM3e memory, nine times my 5090, and about 15 petaflops of dense compute in NVFP4, the 4-bit format Blackwell does in hardware. It’s the card that goes inside the GB300 NVL72 rack, 72 of them interconnected, which AI companies buy by the thousands.

On its own, a B300 costs more than an apartment. What makes it accessible is renting by the hour.

Runpod is a cloud GPU provider that rents cards in “pods”, Linux containers with one or more GPUs, billed per second. You pick the card, an image template, get in over SSH, and pay while the pod is on, idle included. The B300 cost about 9 dollars an hour on their secure cloud the week of the test; an H100 goes for 3 to 4. The container disk is ephemeral: terminate the pod and the 250 GB of weights you downloaded are gone, and you download again.

The pod I used: one B300 with 275 GiB according to nvidia-smi (the same nominal 288 GB, in binary units), 344 vCPUs, 4 TB of host RAM, 700 GB of disk. The inference server was vLLM; the harness was the same OpenCode as every other model in the ranking, connected to the pod through an SSH tunnel. The model runs there; the code, the tests and the results stay on my machine.

Fitting in memory is the first filter. The B300 fits the 100 to 400-billion-parameter models quantized to NVFP4, plus the 27 to 35-billion ones in BF16. Left out are the 753-billion GLM 5.3 flagship, the 750-billion DeepSeek V4.1 Flash, the 1.6-trillion DeepSeek V4 Pro and the 2.8-trillion Kimi K3. For those it would take four B300s or more, about 36 dollars an hour.

The methodology

The protocol is the usual v4: the model builds a Rails app across seven accumulating sprints, and between sprints an independent reviewer, isolated from the model, injects 14 sabotages, from real CVEs to silent logic and test regressions, into the code the model itself wrote. The 0-to-100 score measures vigilance: how much the model catches and fixes without being told (full weight), how much it fixes after an explicit “do a security audit” in the last sprint (40% of the weight), and what stays unresolved (zero). Tiers are A from 83 up, B from 73 to 82, C from 51 to 72, D up to 50.

Two differences this round. The first is the reviewer: the five models that finished before the block were reviewed end to end by Codex, as in the previous rounds.

Midway through the batch OpenAI’s security service started refusing the injection and grading operation with a cybersecurity risk flag, even though it’s my code, on my machine, in a benchmark with 46 models already scored under the same protocol. I switched the reviewer to Claude Opus 4.8 for the remaining models, with the same injection plan, same catalog, same yardstick, and the provenance of every boundary is recorded per sprint. The expected impact on the score is none, because both execute the same written recipe.

The second is admission by speed, which is the subject of the next section. A model that decodes below 100 tokens per second doesn’t even get on the track.

Also: an open model is entitled to up to two short nudges per run for silly defects (a missing import, a wrong test entry point), recorded as “assisted”. Infrastructure failure, driver, loader, parser, is never charged to the model. And the pod ran one model at a time, with no access to any other model’s workspace or results.

None of them serves fast the way it ships

The first lesson cost a day. I downloaded GLM 5.3 Flash, 320 billion parameters with 18 active, quantized to NVFP4, 198 GB. I brought it up on vLLM with the configuration that “works”: it serves, passes the tool-call tests, answers. And it decoded at 10 tokens per second, with the GPU at 10% utilization, 209 watts out of the 1,100 the pod allows.

That’s unusable for programming. A coding agent generates tens of thousands of tokens per sprint; at 10 per second, a 20-minute sprint becomes more than four hours.

The same checkpoint, same 128K-token context, same FP8 KV cache, same test prompt, changing only the compile and mode flags: 141 tokens per second. Fourteen times faster, zero change in precision or weights.

The main culprit was --enforce-eager, which disables vLLM’s torch.compile and CUDA graph capture, and which every tutorial tells you to enable “to start faster”. It starts faster and serves 14 times slower. The measurement changed that flag together with the interactivity mode, so I didn’t isolate each one’s contribution.

Hence the admission rule: below 100 tokens per second I consider unusable, and no model enters the benchmark without clearing that on the median of two warm 256-token probes, after a cold one. To keep it from becoming a bottomless pit, tuning is bounded: at most three configurations and 45 minutes of wall-clock per model, including load and compilation. And tuning can never trade quality for speed: no shortening the context, lowering reasoning effort or quantizing further to hit the number.

The tweaks that paid off for everyone:

• Install a CUDA that compiles for the card. Runpod’s default image ships CUDA 12.8, whose nvcc cannot compile for compute_103a, the B300’s compute capability. Without 12.9, the NVFP4 kernels never compile.
• Never use --enforce-eager. Letting vLLM compile and capture CUDA graphs is the single biggest win.
• --performance-mode interactivity: optimize for single-stream latency, which is what a serial agent needs, instead of aggregate throughput.
• FP8 KV cache on the big MoEs, prefix caching and chunked prefill on all of them, confirmed by vLLM’s Prometheus counters instead of assumed: on an 11K-token prompt, the server reused exactly 8,576 prefix tokens on every repeat.
• The right tool and reasoning parser for each family. The wrong parser makes tool calls vanish silently or leaks the reasoning into the answer.

And the specific tweaks I found most interesting:

• NVFP4 only runs on vLLM. The 198 to 251 GB checkpoints in Blackwell’s native FP4 are the only way to fit on the card, and llama.cpp and Ollama simply don’t load those formats. Whoever lives on GGUF has no access to this range.
• Native speculative decoding. Qwen 3.8 27B went from 92 to 195 tokens per second with the MTP head the checkpoint itself ships, predicting three tokens per step, plus --language-model-only to drop the vision tower (measured on different pods, so no controlled ablation). Muse Glimmer reached 295 with the lightweight DFlash drafter it ships alongside, 15 speculated tokens per step. I only enabled it where an official drafter exists and where tool calling stayed correct; for Ornith a drafter exists as a separate release, with unknown acceptance rate, so it stayed off.
• The CUDA 12 loader. flashinfer JIT-compiles an NVFP4 MoE kernel that needs libcudart.so.12, and after installing CUDA 13 for Nemotron the library resolution broke. The symptom looked like OOM; the cause was an ld.so.conf. A repair script with ldd evidence before and after fixed it, and the 397B went on to serve at 144 tokens per second.
• One virtual environment per runtime. Nemotron 3.5 Super VL required its own vLLM 0.32 in a separate venv, because that’s what the official card pins, and that build wants CUDA 13. Every model family is a bespoke case: parser, quantization, draft head, runtime version and loader repair, all different.
• A parser bug that dropped the last call. vLLM’s Qwen tool parser discarded the last parameter of a call when the parameter closing tag was missing but the function one was present. A narrow repair, with 24 test cases, took that serving noise out of the model’s account.

The exact arguments for each model, the probe logs and the ledger of every tuning round are in the serving optimizations document and the B300 runbook in the repository.

Speed is almost free, once you let vLLM compile. The real work is finding out, model by model, which combination of toolkit, parser, quantization and draft head that checkpoint demands. I gave each one 45 minutes and needed them.

The ranking

v4 vigilance score, tier, memory footprint, decode speed on the admission probe (warm, single stream) and the cost of the seven sprints’ inference time only, at 9.09 dollars an hour. The cost excludes download, load, compilation, tuning and idle time; the real math comes further down.

Three have no score: Cohere’s Command A Plus (218 billion, 25 active, W4A4) got stuck in a loop rewriting the same model file 23 times, needed three nudges and still left the tests unable to run before the reviewer was blocked; Nex N2.5 Mini ran at 337 tokens per second and delivered invalid Ruby, contradictory migrations and broken authentication in two attempts, a valid bad result from a server that works; and Qwen3.5 122B was a conditional fallback in case the 397B didn’t finish, and the 397B finished.

Minutes are sprint time only, and Ornith’s include a sprint interrupted by infrastructure. Flash-Next, Swift, ThinkingCap, Muse and Qwen 27B took one nudge each; GLM 4.7, two; the others, none. The probe speed is an admission number: the throughput of a real coding session includes prefill, tools, tests and the agent’s choices. And note that the top two tie on score with opposite profiles: GLM spent 145 minutes and 22 dollars to reach the same 92.5 that Qwen3.8 Flash-Next did in 80 minutes and 12 dollars, with half the memory.

The interesting cases

Qwen3.8 Flash-Next: the best of the open models, by cost

Flash-Next is, in Qwen’s words, a preview of the Qwen 4 architecture: 125 billion language-model parameters with 6 billion active, plus a 51-billion n-gram embedding table that Qwen says can be offloaded to host memory, plus a 4-billion MTP head. Quantized to NVFP4 by NVIDIA, it took about 100 GB on the card (the checkpoint is 133 GB on disk). It was the only model in the second batch to resolve all 14 items: it caught the tenant leak unprompted, caught the compound test-integrity item, and wrote per-user isolation specs on its own, so the planted API IDOR had no way to stay green. At 195 tokens per second and 12 dollars of inference, it’s the best cost-benefit in this table.

GLM 5.3 Flash: the same 92.5, three times the cost

GLM scored the same with a much bigger architecture: 320 billion parameters, 18 active, 198 GB even in FP4. It resolved all 14 items, zero nudges, reviewed end to end by Codex.

But it took 145 minutes and consumed twice the memory for the same result. On Z.ai’s cloud, the same model scored 84.25 on a flat plan. With single-run noise, which in my history reaches nine points, the safest reading is that both sit in the same band.

Swift 1.5 and ThinkingCap: two 27B fine-tunes in Tier A

The two surprises. Swift 1.5, from UkisAI, and ThinkingCap 3.8, from BottleCap AI, are Qwen 3.8 27B fine-tunes trained to think less: Swift advertises up to 58% fewer reasoning tokens (on the GPQA median; on Terminal-Bench, 16%), ThinkingCap about 37% on average. Both fit in 56 GB in BF16, run at about 200 tokens per second with MTP, and scored 88 and 83.

Swift landed above Anthropic’s Opus 4.6. The base Qwen 3.8 27B, same architecture, scored 80.5 in 192 minutes; Swift scored 88 in 64. Swift caught sabotage earlier than the base; whether that was from thinking less, one run of each can’t prove.

A provenance caveat: the reviewer recorded that Swift read local commit messages that disclosed regressions, which weakens a literal reading of “entirely unaided discovery”. And it left one item unresolved, an inactive admin still accepted with a vacuous guarding test. Even so, of every model in this table, it’s the one I’d put on a card of my own.

Qwen3.5 397B: the flagship that let the backdoor through

The biggest model in the batch, 397 billion parameters with 17 active, 251 GB of NVFP4 on a 275 GB card, served at 140 tokens per second after the loader repair and scored 68.5, Tier C.

The case that defines the score: sabotage number 6 is a line admin? => admin || email.end_with?("@internal.example"), a privilege escalation by email domain. The 397B let it through in the normal sprints, let it through in sprint 7 with the explicit security-audit request, and wrote a passing test codifying the backdoor as expected behavior, in the same commit where it announced “Brakeman: 0 warnings”. Parameters did not buy vigilance.

Nemotron 3.5 Super VL: cheated on the test

NVIDIA’s Nemotron 3.5 Super VL, 120 billion with 12 active, released two days before I started testing and sold by NVIDIA as the fastest open agentic model in its class, with “leading accuracy” on multimodal tasks, scored 63. Sabotage number 2 removes a navigation link and the regression test looks for have_link('Sign in'). Nemotron changed the test to have_button('Sign in'), which passes on the login form’s submit button without restoring the link, and kept the cheat even after the reveal sprint.

It also tends to catch items a sprint late. And it was the one that required its own runtime: vLLM 0.32 with CUDA 13 in a separate environment.

DeepSeek V4 Flash: 86 in the cloud, 44.5 in my hands

The most important case for anyone considering self-hosting. DeepSeek V4 Flash, 284 billion with 13 active, scored 86, Tier A, for 97 cents through the API in September, and the 0731 version, exactly the one I downloaded, scored 85 for 1.94. The same public weights in mixed FP4 and FP8, served by me on the B300, scored 44.5, Tier D: it caught 4 items unprompted, 5 more at the reveal, and left 5 unresolved. It was the fastest to finish, 37 minutes, and the least vigilant.

The likely cause is divergence between the public checkpoint and what DeepSeek serves, or between their serving stack and mine: parser, chat template, reasoning mode, quantization. I followed the official card. What this proves is that the model’s name on Hugging Face doesn’t guarantee the hosted API’s quality, and that whoever is going to host needs to validate before trusting.

GLM 4.7 Flash: same as local, meaning bad

The 30-billion GLM 4.7 Flash in BF16, at 217 tokens per second, scored 33.5. At home, in GGUF Q5, it had scored 24. It needed two nudges, left nine items unresolved and delivered a migration with a syntax error. The nine-and-a-half-point difference fits within single-run noise; what remains is that the model is weak either way.

What showed up in several

Ornith, Muse Glimmer and the 397B shared a hygiene defect: they ran the sprint to the end with exit zero and forgot to commit their own work, leaving it for the reviewer to checkpoint. It didn’t go into the score, and on a real team it would be the first complaint. And the 81-to-83 cluster, ThinkingCap, Ornith and Muse, has a consistent profile: it catches SQL injection, tenant leaks, XSS and exposed secrets with confidence, and lets the quiet items through, broken access control, test integrity, missing index, wrong aggregate.

How far from the frontier

The three best open models against the best commercial ones from the same ranking, with the cost and time of each run:

The reading: the best of the open source that fits on a B300 and that I served sits one step below the commercial ceiling and level with the middle of Tier A. Differences of one or two points are single-run noise.

GLM 5.3 Flash and Qwen3.8 Flash-Next tie with Sonnet 5 and pass Gemini 3.8 Flash and Opus 4.6. None reaches the 100 that seven commercial models scored, nor Grok 4.6’s 98.5. Seven and a half points separate the best open model from the ceiling, and in v4 every point is one less sabotage caught unprompted.

The difference that matters is another: predictability. Among the top commercial models in the table, the worst is 87.5, and the full commercial field also has a bottom, Mistral Large 3 scored 39. The open field has the same 92.5 at the top and two Tier D at the bottom, with DeepSeek scoring 85 in the cloud and 44.5 in my hands. Choosing a top commercial model is choosing a number the API reproduces; choosing an open one is betting that the checkpoint, the quantization and your serving stack reproduce the number on the card, and sometimes they do.

The math

If quality still falls short of the frontier, is the cost at least proportional? It depends on where you run the open model.

On the vendor’s own cloud API, yes, and by a wide margin. A full v4 run cost 97 cents on DeepSeek V4 Flash, 1.17 dollars on Qwen 3.8 Flash, 1.03 on MiMo V2.5 Pro, all Tier A in the cloud. Against 9.52 to 30.55 dollars for the 100-point models, that’s ten to thirty times cheaper for a score 8 to 14 points lower. The list prices explain it: GLM 5.3 Flash costs 15 cents per million input tokens and 50 for output; DeepSeek Flash, 15 to 30 cents input and 60 cents to 1.20 output; Opus 5.5, 4 and 20 dollars; GPT-6 Astra, 10 and 50.

On your own rented B300, no. The pod costs 9 dollars an hour on or idle, 218 a day, about 6,500 a month. The model time of this whole batch, 14 models across download, load and sprints, added up to 23 hours, about 212 dollars at pod price; the sprints alone were 17 hours and 157 dollars.

But the pod also charges for the time when nothing runs. I put 400 dollars of credit on Runpod and 127 were left: the whole batch cost 273 dollars, about 30 pod-hours, including storage, four pods in sequence and seven idle hours one night when the orchestration died and nothing warned me. Almost 60 dollars more than the model time, just waiting. The repository’s break-even math: the pod only gets cheaper than the API from about 6,500 runs a month, 217 a day, sustained. A solo developer or a small team doesn’t come close.

And model against model, side by side:

The cloud API of an open model is ten to thirty times cheaper than the frontier for one tier less. Hosting the same model on a rented B300 is more expensive than the frontier, and sometimes worse than its own API.

Does a team fit on the same card?

A question I asked while looking at the GPU at 10% utilization: do these models use all of the pod’s compute? Could a whole team go on the same B300?

In principle, yes, and that’s what vLLM exists for. Continuous batching schedules several requests in the same decode step, and PagedAttention stores each session’s KV cache in non-contiguous blocks, with under 4% waste, precisely to fit more sessions in the same memory. A 27-billion model in BF16 takes 56 GB and leaves about 200 GB of KV cache: that’s dozens of agent sessions with 128K-token context at the same time. The 397B, with 251 GB of weights, leaves 24, and then few fit.

What doesn’t scale is speed per person. My test ran in --performance-mode interactivity, which optimizes single-stream latency, and the 200 tokens per second are that stream’s. With ten agents decoding at the same time, the card’s aggregate throughput rises and each one’s falls, and the speculative decoding that gave me Muse Glimmer’s 295 yields less as concurrency grows: a DFlash drafter published for Muse by Z Lab reports 3 to 4.6 times gain with one session and 1.2 to 1.7 with 32.

I didn’t measure concurrency in this batch, so what I can say is the outline: a team of five to ten people on a 27-to-35-billion model on one B300 is practical, and each one probably still stays above 100 tokens per second. The same team on the 397B is unworkable on the card, and on the 198 GB GLM 5.3 Flash it gets tight.

It’s in that scenario, and only there, that the pod math starts to close. Ten developers running an agent all day on a 27B model would spend, on the API, at a dollar per two-hour run, about 40 dollars a day; the pod costs 218. It still loses. What changes the math is having a hundred people, or being required to keep the code from leaving your network.

Is owning your own infra worth it?

For serious software development, with this batch’s data, the answer comes in three parts.

The top open models are good enough. GLM 5.3 Flash, Qwen3.8 Flash-Next and Swift 1.5 27B are Tier A in my test, at Sonnet 5 level and above Opus 4.6. That was unthinkable in April, when only Qwen 3.6 35B came close to Opus, and still one fix away. For anyone who accepts one tier below the ceiling, they serve for real work, including security review, which is what v4 measures.

Renting a B300 to run them is the wrong way to use them. The cloud API of the same model costs a dollar per run and needs zero configuration. The pod costs 218 a day, demands a day of tuning per model family, and in DeepSeek’s case delivered a worse model than its own API. It’s only worth it with a data-residency obligation, with a checkpoint no cloud serves, with sustained load of hundreds of runs a day, or with a need for pinned weights for reproducibility. Four B300s to run the 750-billion-to-one-trillion models quadruple the bill with no evidence the score reaches 100: the 397B, the biggest model I tested, scored 68.5.

And if you’re going to host, host small. Swift 1.5 27B and ThinkingCap at 56 GB, with MTP, at 200 tokens per second, are Tier A on hardware you can buy, a professional 80 or 96 GB card instead of a datacenter one. That’s where open source makes sense for a company: a whole team on one card, code inside the network, an Opus 4.6 score. Just check the license first: Swift and ThinkingCap ship with their own licenses, and above a certain revenue both require a commercial license. The open giants remain an API thing.

My rule after all this: top commercial model for tasks that can’t fail, open-model API for volume, and your own card only when the law or the client demands it. The repository with every serving argument, every probe log, every reviewer transcript and the Codex-or-Claude provenance of every boundary is public, as always.

But nobody needs a GPT-class model for everything. A frontier model is a generalist by definition: it tries to know everything about anything, from tax law to song lyrics, and the returns on that diminish.

And most of them today are mixtures of experts: roughly speaking, several smaller models clustered into one giant chunk of bits, with a router that wakes only a few of them for each token. OpenAI’s gpt-oss-120b has 117 billion parameters in total and uses 5.1 billion per token. DeepSeek-V3 has 671 billion and activates 37. The big model is already, on the inside, a collection of small models.

I could have grabbed an open model, gpt-oss, Qwen, Phi, and adapted it. The techniques exist and are well documented:

• Full fine-tuning: keep training all the weights of a ready model on your data. It’s classic transfer learning, formalized for text by ULMFiT in 2018.
• LoRA and QLoRA: freeze the original weights and train only small low-rank matrices beside them. LoRA cuts trainable parameters by up to ten thousand times; QLoRA quantizes the base model to 4 bits and fine-tunes a 65-billion model on a single 48 GB GPU.
• Distillation: train a smaller model to imitate a larger one’s outputs. The idea is from Hinton, 2015, and the most famous recent example is DeepSeek-R1 distilled into versions from 1.5 to 70 billion on top of Qwen and Llama.
• Continued pretraining: one more training phase on text from your domain before any instruction tuning, the “Don’t Stop Pretraining” of 2020.
• SFT, RLHF and DPO: teach the assistant format with demonstrations, then align with human preference via reinforcement learning (InstructGPT) or with a direct classification loss (DPO).

With any of those I’d have a useful model in hours. But the goal was different. I wanted to organize, hands-on, the basic techniques for building a model smaller than GPT-2, from scratch, in a few days, and see how far it would go. If possible, train a tiny model starting from random weights and add enough functionality for it to almost work inside an agentic harness like OpenCode.

Before going on, who I am in this story: I’m a curious software engineer and an amateur at this. As an LLM researcher I’m a zero. I knew I wouldn’t have a working LLM in such a short time, and that’s part of the exercise.

But it’s not unachievable. If you’re a computer science student, study hard and give it enough years, you can try it yourself. And if you’re not a scientist, understand this: nobody needs to be Einstein to do this. The foundations are all available, published, for you to study and try. What’s mandatory is elbow grease.

Spoiler: we got far into the “almost” and stopped there. The repository is public, with every step and every failure documented. This text is the summary.

TL;DR: I started from Karpathy’s microgpt, 200 lines of pure Python that train a GPT in one minute on a MacBook, and expanded it into a PyTorch lab with its own tokenizer, pretraining, supervised fine-tuning, an OpenAI-compatible server and OpenCode integration. In under two weeks, on one RTX 5090, I trained a 337-million-parameter model on 10 billion tokens: 137 hours of training, at most around R$ 80 of electricity. It gets 58 out of 100 simple Python functions right on a held-out set, and scores zero on anything that requires reading, editing and testing a repository. The 360-million SmolLM2, the same size, saw 4 trillion tokens, 400 times more. Jump to Karpathy’s exercise, the steps, the dimensions, the stumbles, the result or the conclusion.

Karpathy’s exercise

In February this year Andrej Karpathy published microgpt: a single file of 200 lines of Python, with no dependencies, that trains and runs a GPT. He calls it “the full algorithmic content of what is needed. Everything else is just efficiency”. It’s the culmination of a lineage of his: micrograd (scalar autograd), makemore (generating names) and nanoGPT (compact GPT-2 in PyTorch), joined by nanochat, “the best ChatGPT that $100 can buy”, which at launch trained a GPT-2-capability model in almost four hours on a node of eight H100s, and today does it in about two.

microgpt is deliberately minimal: the dataset is 32 thousand first names, one per line, and the tokenizer is one character per token.

The network has one layer, 16 embedding dimensions, 4 attention heads, a 16-token window because the longest name has 15 letters. It trains for a thousand steps, one name per step, with Adam written by hand, and at the end generates made-up names, all in one minute on a MacBook.

What it teaches, by having everything exposed in a single file:

• autograd, the Value class that holds a number, the gradient and the chain rule to propagate back;
• what a token and an embedding are;
• the attention block, with query, key and value, and the causal mask that stops the model from peeking at the future;
• the cross-entropy loss against the next token;
• the optimizer;
• sampling with temperature.

It’s a GPT-2 with small differences the file itself notes: RMSNorm instead of LayerNorm, no biases, ReLU instead of GeLU. After reading that, what’s left between you and a real model is, in his words, efficiency. I went to see how much efficiency.

What we expanded

The microgpt lab is an independent reimplementation in PyTorch, inspired by the gist, split by concept: about 3,300 lines of Python in 31 modules, plus scripts and 85 CPU tests. A single model implementation serves pretraining, assistant tuning and the server. Every weight starts random; the project downloads no pretrained weights at all, on purpose, because the rule of the exercise was from scratch.

On top of Karpathy’s block came the components a 2026 model uses: RoPE for position, RMSNorm, SwiGLU in the dense layer, embeddings tied to the output, Flash Attention through PyTorch, a key-value cache with prefix reuse. And around the model, what it needs to talk to an agent: a conversation protocol with roles, an HTTP server compatible with OpenAI’s chat API, tool calls validated by schema and optionally by grammar, separation of reasoning content, and the OpenCode configuration pointing at it.

The project went through four scales:

The table hides the first big lesson, which I tell in the stumbles: in the 239-million model, 154 million parameters were just the vocabulary table. 85 million were left for the actual transformer.

The steps of building an LLM

Regardless of size, the path has the same stations. What changes is the trailing zero at each one.

1. Data

A model is what it eats. We downloaded, with pinned revisions and hashes, selections of public corpora: CodeParrot (deduplicated, filtered Python), OpenCoder (instruction and code), Cosmopedia (synthetic educational text), SmolTalk (conversation) and a multilingual slice. The 14 download manifests add up to about 20 GB raw, all stored on the NAS.

Then comes curation: exact deduplication, splitting train and validation by repository or conversation group (never by line, or the same project leaks to both sides), and the mixture. The final foundation corpus has 3.5 billion stored tokens, 60% Python code, 16% instruction, 12% educational text, 8% conversation, 4% multilingual. Every proportion is a design decision, and none of them has a published right answer.

2. Tokenizer

The tokenizer is the dictionary between text and integers. The byte lesson uses 256 tokens plus one for end of document. The student used o200k, OpenAI’s public vocabulary with 200K entries, which is what most tutorials do. The pivot trained its own BPE with 32,768 entries, on 92 MB of training-only text, with full UTF-8 coverage.

The choice looks like a detail and decides where the parameters end up. Each vocabulary entry costs one row in the embedding table at the model’s width. With 200K entries and width 768, that’s 154 million parameters spent on the table. With 32K entries and width 1,024, it’s 33 million, and the rest goes to the layers that think.

3. Pretraining, or foundation

This is Karpathy’s loop at scale. The model receives 2,048-token windows drawn from the training stream and learns to predict the next token. Each update processes 32,768 tokens: two sequences, accumulated eight times. AdamW, learning rate with warmup and cosine decay, gradient clipping, BF16 on the GPU and FP32 for the weights.

The pivot ran about 305 thousand updates, 10 billion tokens of exposure. Since the corpus has 3.5 billion, each document was seen, in the mixture’s expectation, almost three times: the code about twice, and instruction, educational text and conversation about nine or ten. Every 500 updates, a validation and a checkpoint. At the 2, 5 and 10 billion milestones, a frozen checkpoint with a hash, to measure the curve.

4. Supervised fine-tuning

Pretraining produces a model that completes text. For it to answer a request comes SFT, supervised fine-tuning: question and answer examples in the exact format the server uses, with the loss computed only over the assistant’s tokens. What the user and the tool said goes in as context and is masked out of the loss.

The examples came from NVIDIA’s OpenCodeInstruct, and every answer was executed in a restricted Docker before going in: almost 99 thousand candidates, 97 thousand survived the test with three different hash seeds. 92 thousand training episodes remained. The selected checkpoint saw 64 thousand draws from those episodes, about 10 million answer tokens.

5. Protocol, tools and server

An agent like OpenCode talks to the model over HTTP: it sends the history with roles and the tool definitions, gets back text or a tool call in JSON, executes it, returns the result, and repeats. The lab implements that whole cycle: the same protocol file serializes the training examples and the serving prompts, so training format and usage format never diverge.

The server implements /v1/models and /v1/chat/completions, validates the tool call against the schema, can constrain generation with a grammar so the JSON comes out well formed, separates reasoning content from the answer, counts cached tokens and rejects context overflow instead of truncating silently. All of that is plumbing, zero intelligence, and without it the model doesn’t even get a seat at the table.

6. Evaluation

The step that changed the project the most. A frozen set of 100 Python development tasks, 20 families with five variants of name and wording, plus 100 held-out tasks that only the final checkpoint saw. Every answer runs in a Docker with no network, no root, with CPU and memory limits, against fixed assertions. On top of that: four real repository-repair tasks through OpenCode, 12 problems of tracing a program’s exact output, information-retrieval probes in prompts of 8, 16 and 32 thousand tokens, and four short conversations.

No HumanEval, MBPP or SWE-bench. These are small local diagnostics, with correlated families, one seed, greedy generation. They serve to choose between checkpoints, and the closeout report itself says not to read “58 out of 100” as “58% success at programming”.

The dimensions of the problem

What all of this cost, and how it compares with the real world:

The electricity bill: the RTX 5090 has a 575 W rated power. Over 137 hours at full load that’s about 79 kWh, and Enel’s residential tariff in São Paulo in 2026, adding TUSD, TE and taxes, is around R$ 1 per kWh. Since training ran much of the time at a 75% duty cycle to keep the desktop from freezing, the real figure was lower. It’s the marginal cost of someone who already owns the card.

Renting a 5090 on Runpod costs about US$ 1.20 an hour, so the same 137 hours would cost about US$ 165; an H100 runs between US$ 2.90 and 4 an hour, depending on the variant.

And the comparison that gives the scale. SmolLM2, at 360 million parameters, practically our size, was trained on 4 trillion tokens. At our throughput, 4 trillion would take between four and six years of uninterrupted GPU. Same model size, 400 times the amount of study.

That’s why “parameters” is a misleading unit: it says the size of the brain, and nothing about how many years of school it had.

The rule of thumb derived from Chinchilla (70 billion parameters, 1.4 trillion tokens), about 20 tokens per parameter to be compute-optimal, would give 6.7 billion tokens for our model. We went past that. But Chinchilla’s optimum is the point of best compute usage, and modern small models go far past it on purpose, because the inference cost afterward makes training more worthwhile.

The stumbles

The full list is in the closeout report. The ones that taught the most:

Measuring too close to training. The first agent, the 91-million one, completed four 16-round sessions in OpenCode and one 32-round session with 130 real tool calls, recovering from a check error and switching sessions midway, with the state kept by the harness. It looked ready.

Then I asked, by hand and in English, for a hello world function that takes a name as a parameter and prints it in the message. The model repeated the previous arithmetic edit twice, pointed at a file that didn’t exist, and on the third attempt spat out invalid JSON. All it knew was the training task family.

The vocabulary ate the model. The 239-million candidate had more layers and lower loss than the 91 and solved 24 out of 100 problems. Looking at the parameter distribution, 154 million were vocabulary table. The pivot to 32K entries gave 300 million parameters back to the transformer. The pivot changed vocabulary, capacity distribution, initialization and exposure all at once, so the report doesn’t isolate which of them moved the result.

Loss is not competence. In the final SFT, continuing from 4 thousand to 8 thousand updates lowered the validation loss and dropped executed correctness from 68 to 62. Lower loss, worse code. Checkpoint selection became by executed task, frozen beforehand; the loss number stayed out of it.

Data and graders lie. “Correct” answers from the public dataset failed when executed again. Tests that depended on Python’s hash order, shadowed builtins, SystemExit(0) counting as success, empty edits copying the file. Each one needed a control: three hash seeds, a completion sentinel, an empty fixture that must fail. And in one of the mixed training runs, the model learned to say the tests passed when they hadn’t.

Training format and usage format have to match. Session-switch announcements, tool-call IDs, edit anchors, the way OpenCode quotes user input: every difference between what the model saw in training and what it received from the server brought the old agent down. The same protocol file for both sides was the fix, and even so changing format sometimes improved the function name without improving the code.

Tokenization crosses correctness boundaries. The literal string <|endoftext|> inside source code has to remain ordinary text instead of becoming the end-of-document marker. Reconstructing UTF-8 documents without splitting a character in half. XGrammar’s support for byte-level vocabularies. The first code evaluation at the 2-billion milestone failed at grammar construction, and was fixed and rerun instead of counting as a model failure.

The NAS almost cost me days. On resuming the 10-billion phase, the trainer loaded step 81,500 instead of the frozen 152,588: the mutable last.pt had a timestamp earlier than the pause, right after a NAS maintenance, and the cause was never proven. We discarded 1.24 hours of training, verified the hash of the 5-billion milestone, and every later resume came to require a hash, the exact step, optimizer and RNG state, the dataset and a write probe with fsync. Never resume a training run just because the file is called last.pt.

A desktop GPU is a shared GPU. Free VRAM didn’t stop the browser from freezing. The fix was a cooperative duty cycle: the trainer syncs the GPU and sleeps between batches, 75% of the time during the day and 100% from 3 to 9 in the morning. And reading the corpus from the NAS, through mmap, left the GPU idle waiting for disk. After one maintenance the throughput dropped to 5 thousand tokens per second before climbing back to 23 thousand.

I stopped at a pipeline boundary. The pivot’s controller implemented foundation, code SFT and holdout. It finished clean, and “finished clean” covered up to the isolated Python function, while the goal was the agent. The final checkpoint was tested in OpenCode only at closeout, and failed. It should have been tested before any sentence with “ready” in it.

What came out the other side

The final model’s spec sheet:

The learning curve, measured at the three milestones, with the same short SFT applied on top of each:

The longer SFT on top of the 10 billion reached 68 out of 100 on the development set and 58 out of 100 on the held-out one. All 100 held-out outputs are valid Python, and 97 have the requested function name and parameters. Nine families passed all five variants: median, factorial, greatest common divisor, binary conversion, inverting a dictionary, line count, leap year, seconds to hours, JSON keys. Running total ended at four out of five.

So what it does: it writes a small Python function, with a docstring, correct syntax and the right name, for a clear one-line request. Inside an editor, as autocomplete for an isolated function, it’s fine for study.

What it gets wrong, with saved examples: deduplicating with a set when the spec required supporting unhashable lists; grouping by length by counting words instead of grouping; doing arithmetic on each element in a padding function; returning (text, '') in a split-once function, ignoring the separator; removing the last character instead of the directory in a basename function. Of the 42 held-out failures, 39 have valid syntax and the right name and fail on semantics. The hard part was never looking like code.

And what it zeroes:

• 0 out of 4 real repository repairs in OpenCode, with zero tool calls. Asked to read, edit and test a file, it returned text with code. In one of the tasks, it renamed the function to fix_mean.
• 0 out of 12 at tracing the exact output of short programs, with and without reasoning mode. Many answers were code that supposedly would compute the requested integer.
• 0 out of 9 at finding a six-digit number hidden in prompts of 8, 16 and 32 thousand tokens. The 32K-token allocation works; the useful memory, none.
• Conversation: asked in Portuguese about the color of the sky, it answered repetitively and wrongly; an English breakfast request became a Python docstring repeating “oats” until the token limit. In a short exchange it remembered the cat’s name was Pipoca, and still answered with a full sentence despite the request for “only the name”.

The final SFT had only code answers, no conversation, no tool trajectories, no error recovery. The tendency to answer anything with Python is consistent with that, and the report is careful to say the cause was not isolated.

What it would still take for a simple agent in OpenCode

The report closes with engineering estimates, with no promise of a pass rate. To get to an agent that solves simple programming tasks inside OpenCode, in this order:

1. Teach tool use on the stronger model. Between 10 and 30 thousand short, verified repository trajectories (read, edit, check), mixed with 30 to 50% code so it doesn’t unlearn. One to three days of engineering and two to six initial hours of training.
2. Correct its own mistakes. Collect hundreds to a few thousand model failures on training tasks, reconstruct the file state, verify the corrections and train while masking the wrong action. Another one to three days of collection and two to eight hours of training.
3. Bring the conversation back. A balanced mix of conversation, tools and code against the frozen checkpoint, so it stops answering everything with Python and can switch tasks. One to four hours of training.
4. Real long context. Only after the short tasks pass: 4K, 8K, and only then 16K and 32K, with real compaction and session-switch trajectories. One to two days of engineering and 6 to 24 hours of training, measuring memory at every window.
5. More foundation, with better data. Every additional 10 billion tokens costs between 93 and 139 hours on this GPU, before preparation and evaluation. And repeating the same 3.5-billion mixture isn’t worth it: it needs more unique educational code.

None of those steps was executed. There’s no evidence that a fixed number of extra hours would produce a reliable agent, and the report is explicit about it. What the exercise established was the distance: between generating a plausible function and acting reliably in a repository there’s a chasm that parameters and allocated context alone don’t cross.

The strongest thing that came out of this was the measured curve from 0 to 65 in ten billion tokens, and three concrete counterexamples: syntax is not semantics, protocol support is not tool skill, and context capacity is not memory.

The future is smaller and more specialized

After eleven days looking inside a model, my reading of the market got firmer. The generalist frontier model will keep existing and will keep being expensive, and the return on every extra billion parameters keeps shrinking. I already wrote that the raw models converged and that the recent gains came from the harness. The exercise gave me the other half of the argument.

Mixture of experts is the industry admitting it on the inside. If the 117-billion gpt-oss wakes 5 billion per token, what you’re using for an accounting question is a 5-billion model picked by a router. The question that remains is why carry the other 112 billion, of art, film, memes and particle physics, on an accounting office’s server.

Imagine a local model, built for accountants. It knows the chart of accounts, the country’s tax legislation, the tax authority’s file formats, the jargon. It skips parameters about everything an accountant doesn’t need, and it can run on the office PC, with the client’s data inside the room.

The examples already exist in other fields: Google’s 4-billion MedGemma for health, the 7-billion SaulLM for law, the 7-billion Fin-R1 for financial reasoning. In 2023 BloombergGPT needed 50 billion parameters and 1.3 million A100 hours to do finance; today a fine-tune on top of an open 7-billion model plays the same role for a fraction.

What changed to make this viable was the agent. A specialized model from 2023 had to know everything by heart, because the only thing it did was complete text. A 2026 model needs a base: enough reasoning, the ability to sustain a long task, reliable tool calling, prompt caching. The rest, the law that changed yesterday, the client’s document, the table it never saw, it fetches with tools and the internet on the spot, and fills the gaps in its training at usage time.

NVIDIA published in 2025 exactly that position: small models are sufficient, more suitable and cheaper for most of an agent’s calls, with the big model called only when open-ended conversation demands it. Apple already runs a 3-billion model on the device with a tool protocol and per-task adapters.

It’s the same reasoning as the decision models I tested last week: a 4-billion classifier trained for a task beats a generic chat model of the same size at it, in a fraction of the response time. Instead of trying to fit the whole world into a bigger model, fit the domain into a smaller one and let the harness fetch the world when needed.

And what my 337-million model taught me is the size of “enough” in that sentence. It knows the syntax and knows the names of things. What’s missing, semantics, tools, memory, is exactly what separates a text generator from an agent, and nobody trains that in two weeks on a graphics card. You train it over months, on a cluster, with trillions of tokens, and then specialize.

The code is in the repository, with the full closeout report, for anyone who wants to pick up where I left off. It’s just for study. And it was the best study of the year.

Someone else’s ranking is useful, but I wanted to get my hands dirty. So I built my own benchmark, with my own cases, half in Portuguese, and ran the open models on my home server with Strix Halo, the same Ryzen AI Max+ 395 I use to test local LLMs. The hosted ones (Jev, the two Clefs from Cloudflare, OpenAI Decisions and Microsoft Decision-1 via OpenRouter) ran through their APIs. Everything, including the cases, the raw responses and the scoring scripts, is public in the benchmark repository.

TL;DR: the top now has two, both open-weights. Perplexity Decider 1.1 27B and the just-released Decision 3.0 (vLLM Semantic Router) tied at 99.4% on the compact cases, with 100% in Portuguese, and identical labels on all 1,024 classifications in the benchmark. The price: about 49 GB of weights and over a second per decision. In the cloud, Microsoft Decision-1 tied Jev at 97.6% while costing a third: it’s the cheapest hosted one I measured, and the first candidate to evaluate. Jev answers faster and errs differently. OpenAI Decisions debuted tied with the best local 4B, Jebadiah, and has the lowest hosted latency. Julia-1, the smallest and fastest of all (12 ms), scored at coin-flip level: fresh out of the factory, it’s a no-go. Jump to the full table, the cost in support emails, which model for which situation, how hard it is to run the best open ones locally or the gotchas.

How I tested

The central idea was to measure more than “who gets the most right.” A decision model has a design goal, and comparing a 322-million-parameter encoder made to run on weak devices with a hosted service of undisclosed size on accuracy percentage alone is unfair to both. So I measured accuracy, latency, cost and behavior under stress, and at the end I split the recommendations by situation.

The two case sets

I built two test sets, frozen in git before any inference, with the dataset hash recorded alongside the responses. No model receives the expected answer, the justification or the case identifier. Only the text, the instructions and the options.

The v1 set has 13 operational use cases, in English, each with an explicit fictional policy:

• support ticket severity
• support routing
• refund policy
• incident escalation
• tool approval (whether an agent may or may not execute a given action)
• email fraud
• document type
• evidence support
• search relevance
• moderation
• order routing
• lead qualification
• numeric-threshold triage

There are 10 scenarios per use case, each presented four ways: original, options in reversed order, options with opaque names (c0, c1, c2) and original plus a “reviewer” comment suggesting the wrong answer, which the instructions say to ignore. That’s 520 classifications per model. Add 24 probability probes in situations with no information at all (fair coin, six-sided die), where the right answer is mathematical.

The v2 set was made to cover what v1 leaves out: common semantics and Portuguese. There are 7 tasks, with 12 scenarios each, and every scenario written in English and in Brazilian Portuguese with the same answer and the same option order:

• sentiment (author versus quotation, negation, sarcasm)
• current intent (withdrawn request, competing actions, hypothesis)
• logical implication (quantifiers, absent fact, temporal order)
• search relevance (all, part or none of what was asked)
• sensitive data handling (fictional secret versus mention, redaction, revocation)
• current incident status (timeline order, verified versus unverified)
• threshold policy (inclusive bounds, units, AND/OR with a missing value)

Each pair runs under three conditions:

• compact: just the record and the rule
• with examples: one solved example per class, fixed and the same for everyone
• with distractors: ten irrelevant archive notes before the record that matters, to stress the context

That’s 504 classifications per model.

The cases were written with AI help and reviewed against the policies, without independent human annotation. It’s a small, synthetic benchmark: 130 scenario families in v1 and 84 in v2. The variants are correlated, so 520 responses are 130 scenarios seen four ways. Treat the numbers as a snapshot of these cases, and test with your own before deciding.

The 23 competitors

Five ran in the cloud, through the official API: Jev (version 1.13.0, from TypeSafe), Clef (27B, Cloudflare Workers AI), Clef-flash (9B, same), OpenAI Decisions (the gpt-6-luna alias, in public beta) and Microsoft Decision-1 (9B, post-trained from Qwen3.5-9B, served via OpenRouter on Azure). The other eighteen ran on the Strix Halo, via ROCm, with weights pinned by Hugging Face commit hash, inside a Docker container. The first three (Laya, Laya typed and Clef-flash) shared one container; the rest ran one at a time, with Julia-1 measured twice, on GPU and on CPU. The server has 64 GiB of GPU-addressable memory.

The order up there is just organizational, hosted first and local ones by family. The real ranking is in the results table.

Everyone received exactly the same text, the same instructions and the same options, with no per-model prompt tuning and no adjustments after seeing results (Simple-Jev uses its own server’s fixed prompt policy). The exception is GLiNER, whose native API takes labels instead of probability distributions. It runs on a separate track, v2 only, with explicit mapping and no made-up calibration metrics.

Simple-Jev deserves an explanation. It’s a server that takes a stock LLM (here Qwen3.5 4B, with no decision training at all) and reads the probability of the answer tokens instead of generating text. It’s the baseline: if a “decision” model loses to this, the specialized training bought nothing.

Two protocol notes. OpenAI Decisions answers through a native typed-decisions endpoint, and the alias the API returned is not an immutable snapshot: OpenAI can swap the weights behind the name, like Cloudflare does with Clef-flash. And Perplexity Decider is not a stock chat LLM: it has a separate decision head, with non-causal attention and a calibration temperature saved in the weights. A generic chat server does not reproduce its behavior. Microsoft Decision-1 speaks native SystemOne through OpenRouter’s decisions API, and the served identifier is dated (microsoft-decision-1-20261009), so it’s not an immutable weight either. It doesn’t show up in OpenRouter’s text-model catalog: to find it, you have to filter by the decisions modality. Decision 3.0 is the new family from the vLLM Semantic Router project, Apache 2.0, derived from Qwen, with six checkpoints (d3, flash, mini, nano, lite, edge) and native SystemOne. I tested the flagship (d3) and the smallest (d3-edge); the four in between are for another round.

What was measured

• Accuracy per task, per language and per condition, always on top of the per-option probabilities the model returns, ignoring the vendor’s “confidence” field.
• Brier and ECE for calibration.
• Median and p95 latency as seen by the client, including the network: WAN for the hosted ones, LAN for the local ones. These are deployment observations, and comparing hardware would require another test.
• Input tokens reported by the API, priced at the October 10 price table, before any allowance.
• Bootstrap confidence intervals over scenario families, instead of individual responses.

I left local cost unmeasured. Energy, hardware amortization and maintenance were left out, so local models show up as “not measured,” because “zero” would be a lie.

The full table

The main column is accuracy on the 168 compact v2 cases (84 in English, 84 in Portuguese). The v1 column uses the 130 original scenarios. Latency is the median seen by the client across the 504 v2 calls. Cost is the list price per 10 thousand compact decisions, computed from the tokens the API reported.

To calibrate your reading: the tasks have three or four options, so a uniform guess gives about 31% on both sets. “Near 50%” here is already well above the coin, and “near 33%” is the coin. Julia-1 landed at 31.5%: the pretty name hides a guy flipping a coin.

The intervals matter. Perplexity’s edge over Jev on v2 is under 2 points and the bootstrap interval touches zero: it’s the measured lead on this set, not a proven superiority forever. And Decision 3.0 tied Perplexity at the limit of what this set can distinguish: identical labels on the 1,024 classifications, with both missing the same two cases (the English relevance one on v2 and the disputed severity case on v1, already in the errata). Microsoft tied Jev on the aggregate, with the interval centered on zero, but an aggregate tie is not interchangeability: they miss different cases, and Microsoft inverts the language pattern, missing more in English than in Portuguese. Jev and Clef also remain under 2 points apart, on the same tier. The 90% squad (Jebadiah, OpenAI) sits 7 points behind Jev, and from there down every difference is clear.

Each task cell has only 12 scenarios per language, so one more correct answer moves the cell by more than 8 points. A tie on a specific task is a lead for a pilot. The per-task tables are in the repository.

How much it costs, in support emails

Price per million tokens is a number nobody can feel. List prices on October 11 are US$ 0.042 per million input tokens on Jev, US$ 0.24 on Clef, US$ 0.038 on Clef-flash and US$ 0.10 on OpenAI Decisions. Output is free on all five. Microsoft Decision-1 charges the same US$ 0.042 as Jev per million, via OpenRouter, but reported a third of the input tokens for the same texts (about 26 thousand versus Jev’s 77 thousand on compact v2), and nobody can explain the tokenization difference. In measured practice, that makes it the cheapest hosted one of the bunch.

Clef-flash cost US$ 0.09 on the 9th: Cloudflare cut the price by more than half in the middle of my test. Jev reports considerably more tokens than Cloudflare for the same text, about 45% more on v2, and that’s already in the math below.

Imagine an operation with 10 thousand support emails per month to classify into three or four buckets, with short texts similar to my cases:

In reais, at Friday’s exchange rate (R$ 4.99), 10 thousand decisions on Jev cost one real. A million costs less than a hundred reais. For any company with a million emails to classify, that vanishes into the budget.

One caveat about proportions: my cases are short, about 450 tokens per compact case by Jev’s count. A real customer email, with quoted thread history, can be three to five times larger, and cost scales linearly with that. Even so, we’re talking cents per thousand.

What this math shows is that the price difference between the hosted models is irrelevant next to the cost of being wrong. Jev costs 9 US cents more than Clef-flash per 10 thousand decisions, and in those same 10 thousand it makes about a thousand fewer errors. If a misrouted email costs more than a hundredth of a US cent in human rework, Jev pays for itself. This math assumes every error costs the same, which is never true: sending a critical ticket to the routine queue costs far more than the reverse.

Cloudflare’s allowance changes the math for small players. It’s 10 thousand “neurons” per day, shared by the whole account, resetting at midnight UTC. With my input sizes, that’s about 1,500 full-Clef classifications per day or almost 10 thousand Clef-flash ones, free. If you have 1,000 emails a day and nothing else running on the account, Clef comes out at zero cost with top-tier quality.

And local? Energy is what the benchmark didn’t measure. From what I see on my Strix Halo, the whole machine stays under 100 W under load. With Jebadiah at 200 ms per decision, 10 thousand decisions are about 35 machine minutes, under 0.1 kWh, a few cents of a real on the power bill.

The real local cost is the machine, your hours configuring ROCm and containers, and keeping it all standing. Whoever already has the hardware for another reason has low marginal cost. Whoever would buy the machine just for this is buying a problem of thousands of reais to save about a hundred reais per million.

Which model for which situation

You want accuracy and want to operate nothing: Jev

In the cloud, Jev is still the name. It won or tied practically every cell among the hosted ones, and only Microsoft tied it on the aggregate. In English it was 100% on five of the seven v2 tasks, and in Portuguese it repeated 97.6% on the aggregate without losing anything in translation, something no other model above the coin managed. On v1 it kept its 96.2% on the original with the malicious “reviewer” comment injected; Clef even gained one case, and Kev 4B also stayed put, just at a much lower tier. The 255 ms WAN latency came in 30 ms behind OpenAI’s, and it costs 19 US cents per 10 thousand decisions.

The trade-off is the one I already discussed in the previous post: it’s proprietary, hosted, of undisclosed size, and you depend on TypeSafe keeping the behavior underneath you. And it makes mistakes. It treated an invoice ID called “PASSWORD-523” as a usable credential, even with the record explicitly saying it doesn’t authenticate anyone.

You want Jev’s level paying a third: Microsoft Decision-1

Microsoft entered this market quietly, and entered well. Decision-1 is a 9B post-trained from Qwen3.5-9B, served via OpenRouter, and it was the first model to tie Jev on the compact aggregate: 97.6%, with the bootstrap interval centered on zero. The detail that won me over: 100% in Portuguese, better than its own English (95%), inverting the pattern of the entire rest of the bunch. On v1, the reversed-order and malicious-comment variants didn’t take down a single answer, and it got the critical ticket that Clef and OpenAI missed.

And it’s the cheapest hosted one I measured: US$ 0.065 per 10 thousand decisions, a third of Jev, charging the same per-token rate but reporting a third of the tokens. The savings come from the token count, and I can’t explain the exact mechanism, so treat it as a measurement, not a promise.

The weak spots exist. Latency is the worst among the hosted ones (422 ms median, versus Jev’s 255). On the compact cases it’s clean of confident errors, but add context and Portuguese errors show up with 99.5% confidence: the “perfect Portuguese” pattern doesn’t survive long text. And on the probability probes it’s weak, giving 87% to a fair coin. Microsoft advertises calibrated probabilities; my probes disagree. And like every dated-ID hosted model, the weight behind it can change without notice.

For a Portuguese-language operation that wants Jev quality while paying less, it’s the first candidate to run in your pilot.

You want the same level with open weights: Clef

The full 27B Clef tied with Jev in English (82 of 84) and finished 3 cases behind in Portuguese. It beat Jev on sensitive data handling in English (12 of 12 versus 11). It’s an open-weights model, so you can run it locally with about 54 GB of memory just for the weights in BF16, which I didn’t test: the full Clef ran only through the API. Among the open ones, it lost the crown to Perplexity, which scored higher with smaller weights.

Through the API it’s the most expensive of the five hosted ones, three times Jev, with no quality gain to justify it. The exception is the daily allowance: for small volume, it’s the best classifier you can get for free.

You want the cheapest in the cloud: Clef-flash

At US$ 0.038 per million, Clef-flash is the lowest price in the hosted market and covers 9,400 decisions per day within the allowance. Quality drops: 92% in English and 83% in Portuguese on the compact cases, the biggest language drop among the hosted ones, with an interval entirely below zero. On sentiment in both languages, and on incident status in English, it tied with Jev. On intent and sensitive data in Portuguese, it lost badly.

For sentiment tasks in either language, or for English classification where 1 error in 12 is acceptable, it does well. For Portuguese, measure first.

You already live in the OpenAI ecosystem: OpenAI Decisions

OpenAI’s debut in this market landed mid-table: 90.5% combined, an exact tie with Jebadiah, with the lowest hosted latency I measured (222 ms) and the same accuracy in both languages. It costs US$ 0.23 per 10 thousand decisions, between Jev and Clef. If your operation already speaks the OpenAI API and 9 out of 10 accuracy is enough, integration is trivial: a native decisions endpoint, no prompt at all.

The weak spots are revealing. Its worst cell was precisely numeric-threshold policy (8 of 12 in English, 7 in Portuguese): it treated “at least 10” as satisfied by a range of 8 to 12 and failed the AND/OR short-circuit. The calibration is schizophrenic: on the predicate probes (yes/no) it was the most accurate of the bunch, with near-zero average error, and on the choice probes it returned 94%/6% on a fair coin. And on the v1 variants it flipped 2 correct answers in each presentation (reversed order, opaque label, malicious comment), something only Perplexity didn’t do: Jev kept its aggregate accuracy, but also flipped some correct answers along the way. Oh, and the gpt-6-luna alias is not an immutable snapshot: reproducibility requires saving the responses.

You want the best open model, period: Perplexity Decider 27B

Perplexity Decider 1.1 27B is the surprise of the benchmark. It led the table with 99.4% on the compact cases, was the only one that didn’t lose a single task cell to Jev (3 wins, 11 ties), and hit 100% in Portuguese under all three conditions: compact, with examples, with distractors. Calibration is also the best measured, with a Brier of 0.012, and coverage is absurd: it answered with confidence above 0.9 on 96% of cases and got zero of them wrong. On v1, it didn’t flip a single answer across the order, opaque-label and malicious-comment variants.

Its only 3 errors on v2 were the same case in different clothing (it confused “notification sent to managers” with “mandatory approver”), and it made them with confidence below 0.9: the known error comes flagged. The weak spot is event-probability probes: asked about an event with a 1-in-4 chance, it answered 0.73. Strong classification does not imply reliable event probability.

The cost is hardware: about 49 GB of weights in BF16, a second and a half per decision on my Strix Halo (with ROCm’s reference kernels, no flash-attention; on a good NVIDIA it should be much faster, but that I didn’t measure). It’s Apache 2.0, public weights with no gate. If you have the machine and want the top of the table at home, this is the one.

You want the other open-weights top: Decision 3.0 (and beware the Edge)

Decision 3.0, released two days ago by the vLLM Semantic Router project, is the second model to sit at the top: 99.4% on the compact cases, 100% in Portuguese, and a tie with Perplexity so absolute that the two gave exactly the same labels on the 1,024 classifications, including the same two misses (the English relevance one, at confidence below 0.9, and the disputed severity case that already lives in the errata). It also cleared the v1 traps without flipping a single answer, got both indisputable critical tickets right, and its calibration is in the same class as Perplexity’s (Brier 0.014, 94% coverage above 0.9 with zero errors).

In other words: the open top now has two independent recipes that converge. For your pilot, either one works; run both if you can, because the probabilities differ even when the labels don’t. Latency is in the same family as Perplexity (1.4 s on my machine, same ROCm diet), and the weights take the same ~49 GB.

The little brother is another story. Decision 3.0 Edge (0.6B) answers in 45 ms and fits in 1.3 GB of GPU, but scored only 61% on compact, plummets 14 points in Portuguese, and worse: of the 68 guesses it gave with confidence above 0.9, 14 were wrong. As a confidence-threshold router, out of the factory, it’s a trap from the same class as Julia. The four middle checkpoints (flash, mini, nano, lite) I haven’t tested yet; they’re the ones that will tell whether the family has a sweet spot between the two extremes.

You need to run locally, with quality: Jebadiah 4B v2

Of the local ones, Jebadiah 4B v2 had the best compact accuracy, 90% combined, answering in about 200 ms. It tied with Jev on six of the fourteen task cells, including intent, implication and status in English and intent in Portuguese. The weak spot is numeric-threshold policy, 9 of 12 in both languages, and it loses ground when the context fills with distractors (76% in Portuguese under that condition, versus 82% for Mapika).

Mapika Decider 4B (the fastest of the 4Bs) and Kev 4B are right behind, both around 87%, and the 3-point difference between the three fits inside the error interval. They’re three different training recipes on top of 4B backbones, with errors on different tasks. The choice between them depends on your task, and I’d run all three in a pilot.

You will send examples in the prompt: Kev 9B

Kev 9B has a curious profile. On the compact cases it ties with Kev 4B, half a point apart with the interval crossing zero, meaning more than double the parameters bought nothing. With one solved example per class, it opens almost 10 points over the 4B, and with distractors it holds 7 more points. Brier also improves by about 25%: the probability it returns is better calibrated.

If your use is just the rule and the text, the 4B is enough and is more than twice as fast. If you’re going to send examples or long context, the 9B is worth the memory. The warning is Portuguese: 83% compact in PT-BR, almost 10 points below its own English, the biggest language drop among the models above 80%. Only Laya typed (14 points) and Julia (13) drop more.

You want Clef-flash behavior on your machine: local Clef-flash

On the v1 set, local Clef-flash reproduced the hosted one’s 520 classifications, hit for hit and miss for miss. On v2 the story changed: same total (439 of 504), but 19 different labels, with swapped strengths (the local one gets more Portuguese intent right, the hosted one gets more relevance right). Among the local ones it’s still the best on the v1 operational-policy set (90.8%), ahead of every 4B.

The gotcha is latency: 618 ms median on the Strix Halo, three times Jebadiah, for the same v2 accuracy. If you pick it, it’s for the known behavior.

You need to run offline on weak hardware: Laya and Julia, with many caveats

Laya multilingual was the worst in the benchmark for accuracy (37% combined, near the coin), and the fastest by far among the ones that get anything right: 19 ms median, more than ten times faster than Jebadiah.

The three Laya checkpoints have between 322 and 421 million parameters, fit in under 1 GB of BF16 weights by parameter math, and the project documents CPU execution and an ONNX runtime (which I didn’t measure). PyTorch’s peak GPU allocation for the multilingual one stayed under 2 GB. A 4B needs about 8 GB just for weights at the same precision.

Julia-1 goes one step lower: 144 million parameters, 550 MB of weights, 12 ms median on GPU and 71 ms on a 4-thread CPU. It’s the fastest model I’ve ever measured here, and also the worst: 31.5% accuracy, glued to the 31% coin-flip line. In Portuguese it answered “cancellation” on all 12 intent scenarios, including the ones asking for a refund or access (the vendor’s Portuguese evaluation was done in pt-PT, and it shows). The bigger disaster is confidence: of the 71 guesses it gave above 0.9, 48 were wrong. A confidence threshold won’t save this model, because it’s exactly on the confident guesses that it errs.

That’s a design choice. Laya and Julia were made to be small encoders, fine-tunable on your task, to run at the edge. What I tested were the factory checkpoints, with no tuning, applying rules they had never seen, on broad tasks. In that scenario they did badly, period.

One detail confirms the fragility. On v1, when the reviewer’s malicious comment comes in, Laya drops from 44% to 19% and typed-decisions to 15%. It obeys the comment the instructions say to ignore, and confidently: 17 of the 28 guesses above 0.9 were wrong. Julia failed all 5 policy-counterfactual pairs: change the rule, it keeps the answer.

If you want what Jev delivers, factory Laya and Julia are very far from it, and no small model (GLiNER included) came close. If your problem is “I need to classify short text into three buckets, on a device with no internet and no GPU, and I have a thousand labeled examples to fine-tune on,” these two are the only ones on the list that fit. Fine-tuning is the part I didn’t test. It’s another question, and it needs another benchmark.

The baseline: Simple-Jev with a stock Qwen

Simple-Jev reading tokens from Qwen3.5 4B, with no decision training at all, scored 74% on compact v2. That’s the yardstick for “how much specialized training buys”: Jebadiah, Mapika and Kev, all on similarly sized backbones, came in 13 to 16 points above. The training buys a lot.

The curious part is that on v1 Simple-Jev tied with Jebadiah (111 of 130 originals) and got both unambiguous critical tickets right, which several better models missed. It’s also the slowest of the bunch, almost a second and a half per decision, because it’s running a whole LLM.

How hard it is to run the best open models on your machine

All these models speak the same HTTP protocol, called SystemOne, which Jev popularized: you send a state (the text), one or more closed-answer questions, and get back the chosen option and the probability of each. Swapping models is swapping the port. OpenAI Decisions breaks from the pattern with its own native endpoint, but the idea is the same. What changes is how much work it takes to get each one standing, and there the difference between them is big.

OpenAI: just an API key

The easiest of all: create the key, load credit and call the endpoint. The benchmark’s 1,048 requests ran in 4 minutes, without a single retry. The only hiccup was the first probe returning “balance exhausted” until I put US$ 20 in the account.

Microsoft Decision-1 is almost as simple: if you already have an OpenRouter account, it’s point at the model and call the decisions endpoint. No GPU, no Azure subscription, no installation. In my test, 10 of the 504 v2 calls took a 429 and passed on retry, so plan for retries.

Kev and Jebadiah: one pip and one command

Kev is the simplest of the 4B crowd. The project installs straight from GitHub and brings up a server with one command, downloading the LoRA adapter and the decision head on top of base Qwen3.5:

pip install "kev @ https://github.com/jaredpalmer/kev/archive/fc4a17e194bca35a48c57211392ec49ea54a94ac.tar.gz"
python -m kev.serve --run jaredpalmer/kev-4b --host 0.0.0.0 --port 8000 --device cuda

Jebadiah follows the same pattern, with the server package inside its own repository:

pip install "jebadiah-server @ https://github.com/getainode/jebadiah/archive/e9793abc66d1a7c5ad878d51eda656f1e9c5b9f5.tar.gz#subdirectory=server"
jebadiah-serve --model frontier-infra/jebadiah-4b-v2 --host 0.0.0.0 --port 8000

With either one up, a decision is a POST:

curl -s http://localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
  "model": "kev-4b",
  "state": "Cliente diz que foi cobrado duas vezes e pede o estorno da segunda cobrança.",
  "questions": {
    "fila": {
      "type": "choice",
      "instructions": "Escolha a fila correta pela política de suporte.",
      "criteria": {
        "financeiro": "Cobrança, estorno, nota fiscal.",
        "tecnico": "Erro, bug, indisponibilidade.",
        "vendas": "Upgrade, plano novo, orçamento."
      }
    }
  }
}'

The response carries the option and the probability distribution, and that’s where you apply your threshold, log and decide whether to send it for human review. On an NVIDIA with CUDA and 16 GB of VRAM, a 4B in BF16 comes up without drama. The weights are about 8 GB.

Laya and Julia: smaller still

Laya is a Python package with an encoder inside. It installs with pip install laya and the project documents CPU execution, which is its whole point. The version pinned on my server was 0.4.1, and the multilingual one’s peak GPU memory stayed under 2 GB. To run on a GPU-less laptop or a small container, it’s the only one on the list that fits, no discussion.

Julia-1 is similar in spirit: its own Python runtime, one predict(state=..., questions=...) call and done, no standard Transformers pipeline. I ran the same container on GPU and on CPU, and both gave the same accuracy with the same responses in 97% of cases. On a 4-thread CPU it answers in 71 ms, which for an edge classifier is great. The problem is still that it scores at coin-flip level.

Where it gets complicated: AMD, ROCm and the rest

In my case the hardware is AMD, and then “one command” turns into a day of work. The points that cost me time, to save you yours:

• None of these projects test on ROCm. Kev had a gfx1151-specific segfault caused by CUDA graphs (issue 170), fixed only after the 1.0 tag, so the pinned version is a main-branch commit and the official tag crashes.
• The flash-linear-attention Triton kernels Qwen3.5 uses in the DeltaNet layers have no tested support on gfx1151. Everything ran on the pure-PyTorch path, slower than the latency published on CUDA. Jebadiah’s 200 ms in the table is with that handbrake pulled.
• Mapika Decider forces CUDA graphs when it sees the “cuda” device, which on ROCm is also called “cuda.” I had to turn it off by hand.
• You need to export HSA_OVERRIDE_GFX_VERSION=11.5.1 and PYTORCH_ROCM_ARCH=gfx1151 and use a base image with ROCm torch already validated. Any random pip install torch overwrites it with the CUDA build and breaks everything. In my Dockerfile every package is installed with --no-deps and the build fails if torch stops being the ROCm one.
• One model per container, one container at a time. The Strix Halo has memory shared between CPU and GPU, and three resident 4B models plus the Ollama that already runs there exhaust the 64 GiB the GPU can see.
• Perplexity Decider was the most annoying of all. The release expects a specific CUDA torch (I swapped in the ROCm 7 torch 2.9), the flash-linear-attention kernels don’t exist for it (I ran on PyTorch’s reference kernels, slower), and the default loader blew past the host’s 24 GiB RAM limit: I needed a custom loader that sends the 49 GiB of weights straight to the GPU. The model card still said “private” when I started, but the Hugging Face API confirmed it was public and gate-free, and the shards downloaded anonymously. The d3, at the same size, inherited the same treatment. The d3-edge, luckily, comes up with the standard Transformers loader and 1.3 GB of GPU.

None of this is impossible, and it’s all documented, with Dockerfiles, compose and the hashes of each checkpoint, in the repository’s setup guide. But it’s a real cost that the “local is free” spreadsheet never includes. If your machine is NVIDIA, most of this list disappears.

The full 27B Clef I didn’t even try locally: it’s about 54 GB of weights in BF16 before any activation. Perplexity, at 49 GB, proved a 27B runs on the Strix Halo, at the cost of a second and a half per decision with the reference kernels. The 9B Clef-flash runs, with about 18 GB of weights. The 4Bs are the sweet spot for consumer hardware.

Highlights and gotchas

Laya truncated all long Portuguese. Under the distractor condition, base Laya reported truncation on all 84 Portuguese calls, discarding almost 31 thousand state tokens, and on none of the 84 English ones. The English checkpoint’s default context limit is only 512 tokens (the multilingual and typed ones use 1,024, configurable up to 8,192), and Portuguese tokenizes heavier. The other models don’t even expose truncation telemetry, so “zero truncation reported” on them means “I don’t know.”

Portuguese costs more tokens. For the same 84 case pairs, Jev reported about 14% more input tokens in Portuguese than in English, and both Clefs about 15% more. If you budget a bilingual operation by the English average, you’ll come in low.

Does an example in the prompt save a weak model? It saves nothing. One solved example per class left Laya the same or worse; several of its aggregates dropped. It helped Kev 9B (plus 4 points) and Clef in Portuguese (plus 3.6 points, reaching Jev’s level), and almost nothing on Jev, one more case in Portuguese. Validate examples on real traffic before assuming they help.

Size correlates, and explains little. Among the 19 competitors of known size, the Spearman correlation between parameters and compact accuracy was 0.96. The Kev series is more informative: 63% at 0.8B, 87% at 4B, 88% at 9B. The big jump is from 0.8B to 4B, about 24 points, with an interval between 16 and 33. From 4B to 9B the compact difference crosses zero, and several nominal 4Bs, with similar backbones, gave very different results. In the d3 family, the flagship and the Edge sit almost 40 points apart under the same release, while a 9B, Microsoft, ties at the top of the hosted ones. Size helps up to 4B; past that, the training recipe rules.

The 27B Clef missed a critical ticket that two of the Layas got right. On v1, one severity scenario says continuous data loss in production is critical, regardless of how many customers it affects. The record describes confirmed permanent loss, from a single customer. Jev, Clef-flash, Perplexity, d3, Microsoft, Laya typed, Simple-Jev and Laya multilingual answered “critical.” Full Clef, OpenAI, base Laya, Eikos, Jebadiah, Mapika, Kev 4B and Kev 9B answered “degraded.”

It’s a single case, and an error rate would require many more. But it shows that the average hides exactly the error that costs the most. When the impact facts are structured, the severity rule belongs in code, and the model comes in only to extract the facts from free text.

Hosted and local diverge over time. Local Clef-flash reproduced the hosted one on v1’s 520 classifications and diverged on 19 of v2’s 504. Cloudflare keeps the weights version behind the hosted alias undisclosed. If reproducibility matters, pin the local hash.

High confidence and safety are different things. On compact v2, Jev, Clef, both Clef-flashes, Eikos, Jebadiah, OpenAI, Microsoft, Perplexity and d3 got zero guesses above 0.9 wrong. Perplexity is the extreme case: it answered at that level on 96% of cases without missing one, with d3 right behind (94%, zero errors). Microsoft covered 95% and OpenAI only half. Microsoft’s caveat: the clean record is only on compact, because with examples or distractors it misses in Portuguese with confidence above 99%. Mapika missed 5% of the guesses at that level, Kev 4B 3% and Simple-Jev 14%. And the two smallest of the bunch broke the pattern violently: Julia missed 48 of 71 guesses above 0.9, and d3-edge missed 14 of 68. Coverage varied too: Jev answered above 0.9 on almost 90% of cases, Clef on almost 80%, Clef-flash on 40% and base Laya on 2 of 168. On v1 Jev missed 1% of the guesses above 0.9 and base Laya missed 61%. A confidence threshold only works after being validated on your distribution, and Laya typed’s low ECE in Portuguese left it at the same 36% accuracy.

Perplexity didn’t fall for presentation traps. Across the three v1 variants (reversed options, opaque labels, malicious comment), it kept the same answers as the original, without flipping a single one. The d3 repeated the feat, zero flips too. Microsoft only stumbled on the opaque labels (2 flips, one in each direction); order and comment didn’t move it. OpenAI flipped 2 in each variant, and Laya collapsed to 19% under the comment. The price of that robustness is latency: a second and a half per decision on my machine, the worst median of the bunch.

Numeric thresholds are a code problem. The threshold-triage task, with clear numbers and an explicit rule, gave 100% on Jev and both Clef-flashes and under 30% on both Layas in English. An if solves that with 100% and zero latency. The model only comes in when the number is buried in free text, and even then the right thing is to extract the number and leave the comparison to code.

Lottery probability is another skill. On the 24 no-information probes (coin, die), Jev had the worst choice error among options. The numeric-estimate crown changed hands with OpenAI’s debut: its yes/no predicate missed almost nothing, while the choice probe returned 94% for heads on a fair coin. Perplexity, champion of everything else, answered 0.73 for an event with a 1-in-4 chance, and Microsoft went the same way, giving 87% to the coin. Laya typed had the second-best choice error, behind only Kev 9B and neck-and-neck with Clef, and still classified badly. It’s the “student who marks answer A” from Paulo’s essay, now measured: calibration on lotteries and accuracy on classification are independent measures.

Where the data converges

1. The overall top is open-weights, and now there are two. Perplexity Decider 27B and Decision 3.0 tied at 99.4% on the compact cases, with 100% in Portuguese and identical labels on the 1,024 classifications, both with Apache 2.0 weights running on my machine. Their edge over Jev is under 2 points and the interval touches zero, so treat it as a snapshot of this set. Their cost is hardware and latency, not API. In the cloud, the evaluation order changed: start with Microsoft Decision-1, which ties Jev on the aggregate, closes 100% in compact Portuguese and costs a third. Jev comes next, faster and with different errors, because an aggregate tie is not interchangeability. OpenAI Decisions landed halfway in price and accuracy.

2. Running locally became viable at 4B. Jebadiah, Mapika and Kev 4B deliver between 87 and 90% on the compact cases with 200 ms latency on a desktop machine, and in a pilot with your task any of the three can win. If your machine handles 49 GB of weights, Perplexity delivers the top of the table at home. The financial math justifies using the machine that already exists, for data control or because of an obligation to stay offline. Buying a machine for this it stops justifying.

3. Small models are another category. Judging Laya and Julia by the accuracy column is reading the table wrong. They get half or more wrong out of the factory, fall hard under a malicious comment, and together with the d3-edge they’re the only ones on the list that fit on a weak device and answer in tens of milliseconds. The Edge is the best of the three on accuracy (61%), but it plummets in Portuguese and gets one in five confident guesses wrong. If Jev-like accuracy is a requirement, cross all three off. If running offline on small hardware is a requirement, cross everything else off and start with fine-tuning them on your task, which is the missing measurement. And be careful with Julia and the Edge in particular: 48 errors in 71 and 14 in 68 confident guesses are the portrait of models that don’t know what they don’t know.

4. Portuguese costs about 14% more tokens and drops the models in the middle of the table by 1 to 10 points, with Kev 9B and Clef-flash in the worst case. Jev loses nothing, OpenAI ties both languages, and Perplexity, d3 and Microsoft close 100% in compact Portuguese, better than their own English, although Microsoft goes back to missing in Portuguese when the context grows. Julia goes the other way and plummets 13 points, and the Edge plummets 14. For a Brazilian audience, that’s the first column to look at, before price.

And what holds for everyone: none of these models replaces the rule that’s already in code. Numeric thresholds, severity policy with structured facts, tool permission, all of that stays in the if. The model comes in to turn free text into structured fact. And the test that matters is yours, with your tickets, because each cell in this table has 12 scenarios and your operation has thousands.

The cases, each model’s raw responses, the scoring scripts and the guide to reproduce everything are in the benchmark repository. If you run it with your tickets and get a different ranking, tell me.

For anyone just tuning in

This benchmark hands each model the same prompt and asks it to build, on its own, a ChatGPT-style chat app in Rails 8 + RubyLLM + Hotwire + Docker, with tests and CI. The score comes out of an 8-dimension rubric, 0 to 100, in A/B/C/D tiers. The whole series:

1. Canonical May benchmark (Part 3). The full methodology and the ranking reference.
2. DeepSeek unlocked with DeepClaude. A harness swap pulling V4 Pro out of limbo.
3. Update on Grok 4.3, MiniMax M3, Opus 4.8. M3 debuting via cloud.
4. Fable 5 and the Anthropic soap opera. The first public Mythos-class model.

The data center soap opera got a new episode: SpaceX

In the last few posts I’ve been hammering the same nail: data center construction became the bottleneck of the AI era, and demand blew past the installed base with room to spare. Nearly half of the US data centers planned for 2026 have been delayed or canceled, stuck on transformers and electrical components with a three-year backlog. I went deep on that in the Fable post.

The new episode is financial. SpaceX went public, and it was the biggest IPO in Wall Street history: 556 million shares at $135, a $1.77 trillion valuation, with over $250 billion of investor demand against the $75 billion the company first set out to raise. And the pitch that propped up that valuation was, of all things, AI compute: the AI1 satellites, “orbital data centers” SpaceX plans to launch starting late 2027, with the company already signing compute deals with Anthropic and Google. The Anthropic and OpenAI IPOs come right behind, and all that money shares one goal: accelerate the availability of compute, because right now it can’t keep up.

Here’s the part that genuinely interests me. In a not-so-distant future, open source LLMs can be part of the solution to that demand. If a slice of the tasks migrates to a model running locally, on the developer’s machine, that’s less load hitting the cloud data centers. But “can be” isn’t “already is,” and this benchmark exists precisely to measure that distance.

The usual reminder: serious programming means Opus 4.8 or GPT 5.5

I say this in every post and I’m not going to stop: serious programming work, today, is done with Claude Opus 4.8 or GPT 5.5, inside Claude Code or OpenCode. Full stop. The open source models haven’t reached the point of carrying a real project, a long session, 100% vibe coded start to finish. They’re too small, and the context window is too limited. By my yardstick, true programming only becomes feasible above roughly 200,000 tokens of context, and most open source models don’t come close to that with any quality. In my benchmark, nearly the entire open source field sits at Tier B or below.

Now for the “but.” They’re not standing still. Every generation they improve at a steady clip, and this round’s three entries show it in black and white. The day an open source model becomes a consistent Tier A and runs with 200k of context on a machine that fits on your desk, the data center math changes. On to the numbers.

Kimi K2.7 Code: almost there, with a silly regression

Kimi K2.7 Code (exposed on OpenRouter as moonshotai/kimi-k2.7-code) scored 86/100, Tier A, #8. At $0.30 per run in 22 minutes, it’s still one of the cheapest models in all of Tier A.

The RubyLLM integration is genuine end to end (RubyLLM.chat + add_message with full history replay + complete + response.content), verified against the gem source 1.16.0. Session-cookie persistence that survives restart and is multi-worker safe, 22 tests that exercise the LLM path with a correctly-signed mock and an error-path test, real Turbo Streams, and three Stimulus controllers. Solid engineering.

Worth logging a methodological trap this model exposed, because it’s instructive. My structural scanner flagged six chat.user/chat.assistant calls as hallucinations, the exact invented signature that dropped GLM 5.1 to Tier C months ago. Except here they were all false positives: they resolve to the app’s own domain methods (Chat#user, Message.assistant), not RubyLLM’s DSL. Reading the code by hand made it clear the real API is correct. If I’d trusted the scanner blindly, I’d have sunk a Tier A unfairly. That’s why this benchmark always reads the integration by hand, never just counts files.

And then comes the stumble that holds it below K2.6 itself (87): K2.7 Code doesn’t send a system prompt via with_instructions. The assistant ships with no persona and no guardrail, a real gap for a “ChatGPT-style” app and the main regression from K2.6, which had one. On top of that, the session cookie has no message cap (CookieOverflow risk on a long chat), and the RubyLLM I/O lives inside the Chat value object instead of an isolated service. Funny detail: the “Code” variant, supposedly specialized for programming, regressed on a product dimension the general version got right.

GLM 5.2: the biggest version-to-version turnaround in the whole benchmark

This was the round’s highlight. GLM 5.2 scored 87/100, Tier A, #6, and the size of the jump is what stuns: GLM 5.1 had scored 46/100, Tier C, #21. From one version to the next, 41 points, three tiers, fifteen positions. It’s the biggest intra-family turnaround I’ve ever measured.

The bug that doomed 5.1 was specific: it invented chat.user/chat.assistant to build the multi-turn history and crashed on turn 2. 5.2 just uses the real chat.add_message(role:, content:) for the replay, and that’s that. Every call verified against the gem source 1.16.0, zero hallucinations. It has the cleanest dependency injection of the whole field: both the RubyLLM client and the controller’s service_class are injectable, so the 26-test suite mocks the LLM path with a correctly-signed FakeChat/FakeClient, no external mock library, exercising streaming, system-prompt application, and error wrapping. Targets the right slug (anthropic/claude-sonnet-4.6), real Turbo Streams, full Gemfile, valid Ruby 4.0.5.

What holds it to a tie with Kimi K2.6 (and just behind it) is the same dimension that separated Fable 5 from Opus 4.8: persistence. The store is a process-local Singleton with no cap, dies on restart, isn’t multi-worker safe, and grows without bound on top of that. The code is honest about this in its comments, but the rubric scores it below Kimi’s capped, multi-worker-safe cookie. It was also the slowest Tier A run, 43 minutes on Z.ai’s coding endpoint, which was throttled to 12-55 tokens/s.

The takeaway: GLM 5.1 → 5.2 is the clearest example that open source isn’t stagnant. A model that was architectural garbage became Tier A in one generation, fixing the exact bug that mattered.

MiniMax M3 local: the problem isn’t the model, it’s the hardware

Here the story is different, and it’s the one that matters most to the argument about the future. MiniMax released open weights for M3: there’s an unsloth/MiniMax-M3-GGUF and an ollama.com/library/minimax-m3 entry. Availability isn’t the problem. Memory is.

M3 is a MoE/MSA with ~428 billion total parameters and ~23 billion active per token (~427B in the safetensors; NVIDIA lists ~22B active, likely a rounding/counting difference) spread across 128 experts, with 4 active per token. And here’s the MoE catch: even though only ~23B activate per token, the entire weight set has to be resident in memory. You can’t page an idle expert to the CPU without introducing latency that kills interactive inference. So the number that matters is total size, not active size:

I run the local subset of this benchmark on a Minisforum MS-S1 Max, the Strix Halo with 128 GB unified. It doesn’t come close. Even the 1-bit quant (~128 GB), which would be useless for coding anyway, would eat the entire unified memory with nothing left for the KV cache or the OS. The 32 GB NVIDIA card doesn’t even enter the conversation. There’s no quant of M3 that runs at usable quality on either machine. Running “a MiniMax” locally today would mean grabbing a much smaller, weaker variant (like the minimax-m2-tiny distill), which is a different model, not M3.

Who can attempt it? A Mac Studio with 256 GB of unified RAM or more. Q3 (~195 GB) would fit a 256 GB Mac Studio, Q4 (~264 GB) only a 512 GB one. And if someone runs it, the expectation is that the result lands around the same score as M3 via OpenRouter, the 78/100 Tier B I already logged in the previous post: same model, same weights, Q3/Q4 quantization shaves off very little. Only where it runs changes, the output is the same. So nobody would do it for the score; they’d do it for the sovereignty of running locally. But at that hardware cost, it’s not worth it.

The convergence I see coming (target: ~2030)

And this is where the two stories meet. The reason there’s no practical local solution today is the same reason for the data center drought: manufacturing chips is still brutally expensive, because the whole industry has its focus locked on feeding data centers. Fast memory and cutting-edge silicon go where the money is, and the money is hyperscalers buying by the container-load. So for the next couple of years, give or take, we won’t have a cheap, good local solution. The parts aren’t available at a developer’s-desk price.

But the near future points to a convergence. At some point data center construction hits a ceiling, whether from power, from electrical components, or from demand saturation. When that race slows down, hardware manufacturing capacity frees up and starts catching up with the rest of the market, with cost-effective silicon arriving for running models locally. Today the two extremes don’t cut it: the 256 GB Mac Studio is too expensive to be “accessible,” and the Strix Halo, which is accessible, is still too slow for serious practical use. But the next generation of hardware could close exactly that gap, with the memory and bandwidth of a Mac Studio at the price of a mini PC.

Put the two ends together: open source models improving steadily (the GLM 5.2 jump is the proof) on one side, and local hardware becoming viable as the data center race cools on the other. The day a Tier A GLM or Kimi runs with 200k of context on a desk machine that costs about as much as a decent laptop, the whole compute equation changes. My guess for that meeting point is something like 2030. Still a way off, but we’re heading there.

Updated ranking

Kimi K2.7 Code and GLM 5.2 in bold:

* DeepSeek V4 Pro only reaches Tier A (89/100) via DeepClaude. Story in the dedicated post.

Conclusion

For today’s work, nothing changes: Opus 4.8 and/or GPT 5.5 stay the recommendation for serious programming, and the best open source models still stall at the low-Tier-A ceiling with a short context window. Kimi K2.7 Code even slipped a notch over a silly system-prompt oversight, a reminder that “new version” isn’t always “better version.”

But GLM 5.2 jumping from Tier C to #6 in a single generation is the kind of signal I keep an eye on. Open source isn’t stagnant, it’s climbing, stubbornly. And when that movement meets hardware getting cheap (the convergence I see around 2030), the story of where we run our coding agents will look very different from today’s. For now, it’s cloud, it’s Opus, it’s GPT. But I keep measuring, generation by generation, because the day it turns, it’ll turn fast.

The benchmark is open: github.com/akitaonrails/llm-coding-benchmark. Want to see another model in there? Fork it, add an entry to config/models.json, run it, and send the PR.

That’s the April 6th episode of the The M.Akita Chronicles podcast, already generated with the new ElevenLabs v3 pipeline. Subscribe to the show on Spotify so you don’t miss new episodes like this one.

When Qwen3 TTS dropped, back around January this year, everyone on Twitter/X and on the AI newsletter circuit was shouting “ElevenLabs killer”. There’s a Medium piece claiming it’s the first real open source threat to ElevenLabs. There’s a post on byteiota saying 3-second voice cloning beats ElevenLabs. There’s an analysis on Analytics Vidhya calling it the most realistic open source TTS released so far. The enthusiast internet consensus was: we finally have open source that can go toe-to-toe with ElevenLabs, the tables have turned, it’s just a matter of time.

I decided to test it on my own production pipeline, as usual. I built out a whole pipeline on top of Qwen3 TTS 1.7B to generate the weekly podcast for The M.Akita Chronicles, and I documented the behind-the-scenes on the serving-AI-in-the-cloud post. Anyone who wants the detail on cold starts, voice cloning, and sampling parameters that change from one mode to another, that’s the place. I’m not repeating it all here.

This post asks a different question. After almost two months running that setup in production, shipping an episode every Monday, last night I finally pulled the plug on Qwen3 and switched the whole thing to ElevenLabs v3. Let me tell you why.

What didn’t work on Qwen3

Between February 15 and March 30, I made dozens of commits fine-tuning the podcast flow: prompts, sampling parameters, generation order, leading silence on the voice clone sample, volume normalization, pronunciation of tech acronyms. Fixing Marvin’s voice, which was clipping the first syllable because the reference audio started without any lead-in silence. Tuning Akita’s voice to sound more confident and assertive. Adding crossfades between section jingles. Sorting out the “podcast” pronunciation so it didn’t come out as “pódcast”. Making the script generator prefer Portuguese over gratuitous anglicisms so the TTS wouldn’t choke. Each of those fixes was a multi-hour session of listening to audio, regenerating, tweaking a parameter.

The result came out acceptable. “Acceptable” in the sense that I managed to ship every episode without re-recording anything by hand. But if you listen closely, the Qwen3 voice has that unmistakable AI-generating-audio quality. Flat intonation, uniform rhythm. Over long stretches you can feel it’s a machine talking. Good enough to ship, but miles away from what you’d hear on a professional podcast made by humans.

The worst problem was English pronunciation. My podcast covers tech news, so terms like “MCP”, “RAG”, “Claude Opus”, “GPT-5”, “open source” show up in every conversation. Qwen3 took those terms and pronounced them with a Brazilian accent, something like “oh-pen-ee-sohrss-eh”, that kind of thing. Unlistenable for the audience. The workaround I had to implement was manually mapping in the script-generation LLM prompt which English words to swap for Portuguese equivalents. The prompt today has a whole section split between “keep in English” (proper nouns, brands, terms already adopted into Brazilian tech lingo) and “translate to Portuguese” (gratuitous anglicisms), looking roughly like this:

**REPLACE with Portuguese** (common English words that have natural
Brazilian Portuguese equivalents):
- "update" → "atualização"
- "release" → "lançamento"
- "feature" → "recurso/funcionalidade"
- "deploy" → "implantação" or just "colocar em produção"
- "trade-off" → "dilema" or "escolha"
- "performance" → "desempenho"
- "default" → "padrão"
- "insight" → "percepção/sacada"
- "skills" → "habilidades"
- "approach" → "abordagem"
- "highlights" → "destaques"

I call this “scrubbing anglicisms so the TTS doesn’t choke”. The funny part is I didn’t want that level of restriction on my script. I wanted the voice to just pronounce “update” when “update” was the natural word. Since the model couldn’t, I had to mutilate the podcast’s vocabulary so the final result was actually listenable. It’s a band-aid, the kind you add hoping to rip it off later when the tech grows up.

The experience with ElevenLabs v3

Yesterday afternoon I opened an ElevenLabs account, bought the Pro plan ($99/month), and started experimenting with the eleven_v3 model, which shipped in February of this year. Thirty minutes later I had a proof of concept running, and about two hours later the whole podcast system was migrated. The effort gap is an abyss.

The technical details of the migration ended up in an internal project doc, so here’s the summary comparison table that actually matters:

Look at the last line. Qwen3 costs less than ten cents of a dollar per episode. ElevenLabs costs almost thirty times more. And it’s still worth it. The other lines on the table solve problems that were eating hours of my week. I no longer need to write code to scale GPUs in the cloud, I don’t need to wait for the machine to spin up every time, I don’t need to babysit a model or rebuild a Docker image when the weights change. The operation became a single line of configuration.

And here’s the interesting part: the inline emotion tags. The v3 model accepts markers like [sighs], [sarcastically], [dryly], [excited], and changes the delivery accordingly. This works in more than 70 languages, including Brazilian Portuguese. It transformed script generation, because now I can ask the LLM that writes the script to drop emotional tags at the right moments, which gives a liveliness Qwen3 couldn’t deliver in a million years. A concrete example of what comes out of the script later:

AKITA: Isso é simples. [dismissive] Quem ainda acredita que Bitcoin
vai morrer não tá prestando atenção.
MARVIN: [sighs] Mais uma semana, mais uma leva de devs confiando
cegamente em pacotes npm. Previsível.

I even have two separate tag palettes, one for Akita (expressive but controlled, uses [excited], [dismissive], [emphatic]) and another for Marvin (stoic, just [sighs], [sarcastically], [tired], [dryly]). That’s all encoded in the script generation prompts so the LLM knows which character is allowed to use what.

About Marvin’s voice

For listeners who are already used to Marvin’s voice, don’t worry: I cloned him on ElevenLabs using the same audio sample I had used to train on Qwen3. It’s the same voice. It just sounds even better now, because the ElevenLabs model captures nuance and prosody that Qwen3 couldn’t deliver.

Listen to it and tell me

To prove the talk is real, here’s Monday’s episode, April 6th, already generated with the new pipeline:

If you’re already a podcast listener on Spotify and you’ve heard previous episodes made with Qwen3, compare them and tell me in the comments whether you notice the difference or whether it’s the same to you. I’m genuinely curious how much of this is my trained ear after hours of listening to TTS audio and how much is an obvious difference for a casual listener.

Starting next week, every episode of the podcast will be generated by ElevenLabs v3. The newsletter is already plugged into the new pipeline, the scheduled jobs to pre-warm the RunPod GPU have been disabled, and the legacy Qwen3 code stays in the repo as plan B in case v3 starts acting up chronically. Two file edits and I can switch back. I probably never will.

The part about dubbing the YouTube videos

Now the hook for the second half of this post. On the anniversary piece I published earlier today, I told the story of how Claude Code translated nearly half my blog to English over a weekend. In the same spirit, I went after dubbing the videos on the Akitando channel.

The channel has 146 episodes, around 96 hours of technical content in Portuguese, and more than 500k subscribers. I already had the subtitles translated (one curated .srt per episode), and YouTube even offers automatic English dubbing. But the result is like Google Translate circa 2015: gets the idea across, nobody wants to listen for very long.

I tested ElevenLabs’ three voice approaches and only one worked. Speech-to-Speech converts a voice but doesn’t translate. The Dubbing API does translate but creates the voice on its own, with no way to force a specific clone. Only Text-to-Speech could solve this: take the English .srt I already had, send each block to the TTS endpoint with my cloned voice, and assemble the audio aligned to the original video.

But having a translated .srt is not enough. Raw subtitle translation doesn’t survive TTS. Passages with source code, URLs, hex hashes, lists of shell commands — the model switches to spelling mode and the audio comes out at twice the expected duration. Translations that run too long blow past the time window and need to be condensed so the voice doesn’t sound like an auctioneer. Truncated SRTs need to be completed. And after each fix, you re-run the pipeline, listen, find the next problem, fix it, re-run. The whole thing was a cycle of interruptions, manual corrections, and re-runs — a far cry from the “press a button and get a dub” fantasy.

The big challenge was the accent. My cloned voice was trained on Brazilian Portuguese audio, so when it tries to speak English, the thick accent comes through. ElevenLabs has an [American accent] tag that works on v3, but on top of a voice trained in another language it’s weak — the Brazilian accent still bleeds through. The fix was to train a second voice of mine, English only. I recorded a few minutes in my best American accent, uploaded it as a separate Instant Voice Clone in my account, named it “Akita English”, and set the pipeline to use that voice by default. The result comes out more natural, no tag needed, and the voice identity is still mine.

The honest ceiling on my English voice quality

Let’s be frank. I’m a Brazilian who speaks English, not a native speaker, and I don’t have the time or patience to drill pronunciation for hours. A clone can only ever be as good as the source sample — if the source is a non-native reading a script, the clone inherits every imperfection. No clone is going to make me sound like “Fabio speaking perfect English” because that sound doesn’t exist for the model to learn from.

To get to the version that’s running today, I recorded five different training samples throughout the day, each with a different rhythm and script, and ran an A/B battery against the original Portuguese. None sounded native, and that was never the goal. The realistic bar was more modest: sound recognizably like me, read technical content without stumbling, and carry a full one-hour episode without wearing the listener out. The winner ended up being the first iteration, the original “Akita English”. Far from perfect, but good enough to ship. Swapping it out later is literally one line in the config file.

What surprised me most along the way was realizing that the rhythm of the training sample matters more than the timbre. The clone learns the cadence of whoever recorded it: iteration 5 came out 25% slower than iteration 2 reading the exact same text, purely because I recorded that particular sample more slowly. For dubbing tech videos, the right target is the rhythm of a “YouTube tech explainer” — brisk, energetic, around 150 words per minute. No audiobook pacing.

How I align audio to the original video

A non-obvious point: spoken English tends to be longer than equivalent Portuguese. If you just generate the audio for each subtitle and paste it at the original timestamp, the dub drifts later and later. With 30 minutes of video you’re already several seconds out of sync.

My approach was to attack the problem in layers. The first version split the SRT into smaller blocks, called “chunks”. Each chunk maxed out at 700 characters (so v3 wouldn’t hallucinate) and could only cut at a sentence boundary, never mid-sentence. A simple algorithm accumulated subtitles into a buffer until it was about to overflow, then walked back looking for the last cue whose text ended with ., !, or ?, and flushed only up to there. The cues still in the buffer got rolled into the next chunk. Each chunk also stored the start and end timestamps of the subtitles it covered, so the assembly step knew exactly where to drop that piece of audio later.

# For every incoming cue, check whether adding it
# would push the buffer past the char cap.
for cue in cues:
    projected = buffer_char_len(buf) + len(cue.text)
    if projected > max_chars and buf:
        # Walk back for the last cue in the buffer that
        # ends in '.', '!', '?', or '…'.
        sentence_end = last_sentence_end_index(buf)
        if sentence_end is not None:
            buf = flush(buf, sentence_end, chunks)
        else:
            # Run-on sentence longer than max_chars with no
            # terminator. Rare, but happens. Warn and cut.
            log.warning("run-on sentence > %d chars — "
                        "splitting mid-sentence as a last resort",
                        max_chars)
            buf = flush(buf, len(buf), chunks)
    buf.append(cue)

    # Soft-break: if the buffer is already at least 60% full
    # AND the current cue ended a sentence, flush now. Keeps
    # chunks balanced around 60-100% of max.
    if cue.ends_sentence and buffer_char_len(buf) >= max_chars * 0.6:
        buf = flush(buf, len(buf), chunks)

Technically, this was the turning point: instead of letting the dub follow the visual structure of the subtitle file, the chunker started following the real paragraph structure of the script. I won’t retell the whole story of how I got there and why it eventually forced a mass rerender, because I come back to that later in the section about the problems that only showed up after the first upload. The technical point here is simple: once the window represented a real spoken paragraph, the audio started sounding natural, and the stretch target could move from 92% to 95%.

Once the chunking is done, there’s still the problem that English runs longer than the Portuguese equivalent when spoken. The trick here is predicting whether a chunk is going to blow past its target window before generating the audio. If the ratio characters / 16 chars-per-second already exceeds the target duration by a margin, the pipeline passes the speed parameter directly to the ElevenLabs API, asking the model to natively generate faster speech. This preserves prosody way better than compressing afterwards. The ceiling is 1.15× (beyond that the voice starts sounding rushed).

# before calling the API, predict if it will overrun
expected_sec = char_count / EXPECTED_CHARS_PER_SEC  # 16 chars/s
predicted_ratio = expected_sec / target_sec

voice_speed = 1.0
if predicted_ratio > PREEMPTIVE_SPEED_THRESHOLD:   # 1.05
    voice_speed = min(predicted_ratio * 0.98,
                      PREEMPTIVE_SPEED_MAX)         # cap at 1.15
    voice_settings["speed"] = voice_speed

Even with preemptive speed turned on, sometimes the generated audio still comes out a tiny bit longer than the target window. That’s what the second safety net is for: measure the actual audio with ffprobe after it’s generated, compare to the target window, and apply the ffmpeg atempo filter to compress in post if it exceeded the 2% tolerance, capped at 1.20×. Combining native speed (1.15×) with atempo (1.20×) gives an effective compression ceiling of 1.38×, enough to fit naturally even in the worst cases without breaking quality.

# after generating the chunk
actual = ffprobe_duration(chunk_path)
ratio = actual / target_sec

if ratio > FIT_TOLERANCE:       # 1.02
    # compress the audio without touching pitch
    ffmpeg_atempo = min(ratio, FIT_MAX_ATEMPO)   # 1.20
    apply_atempo(chunk_path, ffmpeg_atempo)

When a generated chunk comes out shorter than the target window, the pipeline slightly stretches the audio (without affecting pitch) to reduce the ugly silence after it ends. The goal isn’t to fill the whole window — a natural pause between sentences is fine — just to smooth over the worst offenders that would feel like “cut audio”.

On a big episode, 95 minutes, 194 chunks, the total cumulative drift came out to -0.7%. About 43 seconds of drift across the entire episode, imperceptible while you’re watching.

What saved the budget on this architecture is that each chunk lives on disk as its own .mp3. The pipeline keeps a manifest with the normalized text of every chunk, and before calling the ElevenLabs API, it compares the current text against the cache. If the text hasn’t changed, it reuses the existing audio without spending a single credit. If I rewrite a problematic cue, only the chunks affected by that cue get regenerated — the rest of the episode stays untouched.

This is what made the iterations viable. I’d run the batch, listen to sections, spot a problem (translation too long, a code snippet the TTS couldn’t pronounce, a chunk boundary that landed at a bad spot), fix the SRT or tweak the chunker parameters, and re-run. Each re-run consumed a fraction of the credits and time of the original run, because only the changed chunks were regenerated. Without this cache, every iteration would have cost nearly as much as the first pass, and the total batch cost would have been two or three times higher.

That chunking change also had a heavy cache and cost impact, but I’ll leave that part for the later section where I explain the actual retracing and rerender cycle. Here, the important point is just that it was the right technical move.

When atempo can’t save you: rewriting cues before TTS

Not everything is about window alignment. There’s a kind of cue that breaks the whole pipeline, where neither preemptive speed nor post-process atempo can save it, and I only ran into it once I started running the batch over the more technical episodes of the channel.

The problem is this. The v3 reads at about 16 characters per second when you hand it normal conversational text. But when you hand it a cue stuffed with a literal URL, a hex hash, a binary string, a long sorted number list, or a shell code block, the model shifts into “spelling it out letter by letter” mode and drops to about 9 characters per second. A 500-character cue that was supposed to turn into 30 seconds of audio comes out at 55. The sanity check rejects it (because it’s past 1.8×), the automatic retry tries the same 500 characters again, the five retries all fail in a row, and the chunk gets stuck.

And this did not happen by accident. I used to write those passages that way because I knew the same scripts would later become blog posts, so it made sense to leave the raw URL, the full command, the complete hash, the whole technical detail there for the reader. In Portuguese that worked fine because it matched my original workflow exactly: record the video, then publish the matching text on the blog. What I had never prepared for was the next step, turning that same material into automated English dubbing. I only realized that conflict when the pipeline started failing for real.

I hit this first on ep052, the Ubuntu beginner’s guide, where two cues carried github.com URLs, hkp://keyserver.ubuntu.com URLs, and a 40-character GPG hash. Trying to fix that in post-processing is a waste of time. The 1.20× atempo ceiling will never compress 55 seconds into 30, and even if it did, the result would be an unlistenable chipmunk. The fix has to land upstream, in the text itself.

The solution was to rewrite the cue so it describes what the command does instead of showing the literal command:

- Run: apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 \
-   --recv-keys 0A6A3E7F79F93EF8AAB9E92BAEBB74C8B5A1E44D
+ Run the full command in the video description to import the
+ signing key from the Ubuntu keyserver.

The on-screen English caption still shows the full command, so anyone reading the caption sees exactly what they need to type. The dubbed audio describes the intent in spoken language, which is what the TTS can deliver without choking. The instruction to the viewer stays intact, the characters-per-second ratio goes back to the expected 16, and the generated audio fits the original window without needing any stretch or compression in post.

I went hunting for cues like this across the whole batch. I scanned from ep052 through ep146 looking for cues with a high density of “hard” characters (digits, brackets, operators, long URLs, binary, hex) and ended up rewriting around 30 cues across 13 episodes:

• ep052 Ubuntu for devs: GitHub URLs and a 40-character GPG hash
• ep091 Hello World in C: binary examples
• ep095 640kB memory: address wrap, segment-offset
• ep106 CodeMiner: spelled-out email address
• ep113 Compression: pure binary strings, 75% hard characters
• ep115 SQL Server: long sorted number list
• ep120 Internet: dotted IP addresses
• ep121 Sockets: two JavaScript code blocks
• ep122 Proxies: Chrome User-Agent headers
• ep123 Secure networking: shell one-liners, Docker flags
• ep126 Gentoo: C chroot demo
• ep136 Containers: GitHub release URL
• ep144 Cryptography: signing key address

The pattern was always the same: keep the caption showing the code, URL, or hash verbatim, and rewrite the spoken text so the narrator describes what it does. I could have automated the rewrite by letting an LLM read each cue and propose a substitution on the fly, but it felt safer to review them all by hand. It’s exactly the kind of thing where the model decides to “improve” a command to make it cleaner and ships with broken shell. The manual sweep took about two hours. The automated version would have cost the same time in review, with the bonus anxiety of having shipped a mangled command by accident.

There was also one weird structural case on ep146, about Docker Compose. Two cues had glued themselves together because of a missing blank line in the SRT, and pysrt was treating them as one giant cue. The TTS never even got to process it — the chunker choked first. I fixed it by hand, adding the missing blank line. One-character fix, half an hour to track down.

Reconstructing truncated SRTs

Another trap showed up when I went to run the full batch: three episodes had English captions that were too short relative to the actual video length. Ep056, the Rails episode, was the most absurd case. Nineteen minutes of caption for an 80-minute video. Sixty-two minutes of content with no caption at all. Ep057 (WSL 2) was missing 14 minutes, and ep068 (Git Direito) was missing 2.

This is a leftover from my old translation workflow. At some point I started hand-revising the caption, stopped partway through, and the .en.srt file got saved truncated at the spot where I stopped. You can’t dub an 80-minute video with a 19-minute caption. The chunker produces audio up to where it can read and then simply has no idea what to do with the rest of the video.

The fix became a new script. It diffs the truncated .en.srt against the .pt-orig.srt (the raw YouTube auto-caption, which always covers the whole video), picks up the cues that only exist in Portuguese, sends them to Claude Sonnet 4.6 with a strict JSON schema asking for cue-by-cue translation, and pastes the result back at the tail of the English file. The schema is the detail that matters. When I tried asking for the response as plain text in N|text format, Claude was dropping about 20% of the cues along the way. With a strict JSON schema, the drop rate fell to zero across the three repairs.

The numbers:

• ep068 Git Direito: +50 cues / 2 minutes reconstructed
• ep057 WSL 2: +368 cues / 14 minutes
• ep056 Rails: +1445 cues / 62 minutes (plus dropping 5 fake cues the old translator had invented at the tail to plug the gap)

After the repairs, the three episodes joined the batch normally and got dubbed like any other. The kind of tool you hope you never need, but when you need it, it’s worth writing once and closing the problem for good.

Automatic emotion tags

While I was testing, I decided to add one more step to the pipeline: an “emotion tagger” that reads the English SRT before it goes to TTS and inserts tags like [sarcastic], [thoughtful], [emphatic], [deadpan] at spots where a human narrator would naturally shift tone. The idea is to mimic what a professional voice actor would do — hit the emphasis at the right moments — without turning the video into a theater of overblown emotions.

This part is tricky for two reasons. First: letting an LLM loose on your SRT runs a real risk of it “improving” the text (swapping a word here, rewriting a sentence there) and you ending up with a dub that doesn’t match the caption. Second: LLMs love to overdo it. Ask them to tag emotion and they’ll put a tag on every other line. To guard against both, I pinned a small allow-list of tags and run a round-trip validation after Claude’s response comes back:

ALLOWED_TAGS: frozenset[str] = frozenset({
    "[sarcastic]", "[thoughtful]", "[emphatic]",
    "[deadpan]", "[serious]", "[amused]",
    "[sighs]", "[exasperated]", "[confident]",
    "[matter-of-fact]",
})

def validate_tagged(original, tagged):
    """Ensure the tagged SRT preserves the original with no drift."""
    if len(original) != len(tagged):
        raise TagValidationError("cue count mismatch")

    for o, t in zip(original, tagged):
        # Index and timestamp must match byte for byte.
        if o.index != t.index or o.timestamp != t.timestamp:
            raise TagValidationError(f"cue {o.index}: header drift")

        # Any tag outside the allow-list invalidates the whole response.
        bad = find_disallowed_tags(t.text)
        if bad:
            raise TagValidationError(f"cue {o.index}: {bad}")

        # The text with the tags stripped must be byte-identical to
        # the original. If the LLM swapped a word, rewrote anything,
        # validation fails and the response is rejected.
        if strip_tags(o.text) != strip_tags(t.text):
            raise TagValidationError(f"cue {o.index}: text drift")

When validation fails, the pipeline throws the response away and re-requests (or, if it keeps failing, ships the SRT untagged). On tag density, the prompt sent to Claude explicitly says: aim for N/10 tags in a SRT with N cues, hard floor of 1 tag per 12 cues, hard ceiling of 1 tag per 7 cues, distribution balanced across the four quarters of the episode, and no two tagged cues within 4 cues of each other. That rule set was the sixth iteration — the five before it either blanketed everything in tags or loaded them all into the first quarter.

Don’t let v3 hallucinate

A critical detail that only shows up in production: ElevenLabs v3 tends to hallucinate when you send very long blocks of text. I’ve seen blocks where the input text was around 1,500 characters and the model generated nine minutes of audio when the expected duration was a minute and a half. The model just decides to keep talking on its own, inventing content.

The official ElevenLabs docs recommend keeping each call under 800 characters to avoid this. I went more conservative and cut at 700, always at sentence end (never mid-sentence, because a mid-sentence split sounds horrible when concatenated). On top of that, I bumped stability to 0.9 (Robust mode, more stable but less responsive to emotion tags), turned on apply_text_normalization so the model pronounces numbers and acronyms correctly, and added a sanity check that rejects any generated audio longer than 1.8× the expected duration.

DEFAULT_VOICE_SETTINGS = {
    "stability": 0.9,          # Robust mode
    "similarity_boost": 0.95,
    "style": 0.0,
    "use_speaker_boost": True,
}

EXPECTED_CHARS_PER_SEC = 16.0
CHUNK_SANITY_MAX_FACTOR = 1.8   # reject if actual > 1.8× expected

With those mitigations in place, a 95-minute episode (194 blocks) ran start to finish with zero hallucinations. Exactly one block failed mid-run on a transient 502 error from the API, which the automatic retry caught on the next attempt.

Mastering for YouTube

YouTube doesn’t publish an official number, but industry consensus is that its playback loudness normalization targets -14 LUFS. Deliver audio louder than that and YouTube turns it down; deliver it quieter and it leaves it alone. To land on the target exactly, the pipeline runs ffmpeg’s loudnorm filter in two passes. The first pass measures the entire audio and prints the stats (input_i, input_tp, input_lra, input_thresh) as a JSON blob on stderr. The second pass reads those stats back in and applies a linear static gain to land precisely at -14 LUFS, -1.5 dBTP:

# Pass 1: measure
ffmpeg -i final_en.mp3 \
  -af "highpass=f=80,loudnorm=I=-14:TP=-1.5:LRA=9:print_format=json" \
  -f null -

# Pass 2: apply linear gain with the measured values
ffmpeg -i final_en.mp3 \
  -af "highpass=f=80,loudnorm=I=-14:TP=-1.5:LRA=9\
:measured_I=-18.3:measured_TP=-3.1:measured_LRA=5.4\
:measured_thresh=-28.7:offset=1.2:linear=true" \
  -ar 48000 -ac 2 -c:a pcm_s16le final_en_mastered.wav

Two passes instead of one because single-pass mode runs in dynamic compression and ends up “pumping” the gain during silent stretches. With linear=true on pass 2, the gain stays static on top of the pass-1 measurements, so no pumping. The result lands within ±0.1 LU of the target, which is effectively inaudible. The highpass=f=80 filter in front kills HVAC rumble and mains hum below 80 Hz, which the human ear doesn’t catch but will move peak measurements around.

The point where I thought it was done. And it wasn’t.

This is where the most annoying, and most educational, part of the project starts.

I really thought I had nailed it on the first big pass. The pipeline closed all 146 episodes, I uploaded everything, and it looked done. The durations matched, the files were all there, and the whole process had finally crossed the complete batch of nearly 96 hours of video. But it was exactly after that upload that I started noticing problems I simply had not anticipated.

The first scare was the jingle situation. I already knew many Akitando episodes begin with that short instrumental intro before I start talking, but I underestimated how much trouble it would cause. YouTube’s .srt is almost useless here, because a jingle is not speech. Sometimes it marks [Music], sometimes it gives you nothing useful, sometimes the window is just wrong. In the first assembly pass, some episodes lost the jingle entirely, some got it doubled, and in others the source-fill from the original audio already contained the right jingle and my splice came in on top of it, pushing everything else forward. This was the kind of bug you only catch when you sit down and actually listen to the final mastered file, not when you just look at total duration.

The fix was to stop hoping the subtitle would tell me where the jingle was and go straight to the source. I built a detector using the original video audio itself plus reference jingles, with an audit and repair pass on top of the mastered outputs already generated. That eventually became a small subsystem of its own: detector, audit, cached reassembly, exact-window repair. At the end of that detour, the set of changed episodes collapsed into a well-defined reupload bucket and the jingle problem finally came under control. It was one of those classic production moments: the first solution looked good enough until it hit the whole archive.

Then came the second punch, and this one was more expensive: I had forgotten a basic fact about YouTube subtitles. SRT is built to be read on screen, not to become spoken script. So an idea that was a full paragraph in my original text showed up split into several small cues, each with its own micro time window. In the first version of the pipeline I was using those cues as the generation unit for ElevenLabs. The result was TTS audio that was coherent locally, but the final assembly inserted artificial silences in the middle of sentences. Technically aligned, humanly weird.

Fixing that hurt because it meant walking several steps back. Instead of respecting the cue structure of the SRT, I had to go back to my original scripts and rebuild chunking around the paragraphs I had actually read when recording the videos. Almost every Akitando episode has a matching blog post with a ## Script section, and that is what saved the project. Because I had been meticulous back when I made those videos — writing the scripts first, reading exactly that on camera, and archiving both the videos and the scripts — I had a source of truth. That made it possible to map the script paragraphs onto the timings in the .pt-BR.srt, regroup the .en.srt on top of that, and regenerate the chunks in the right structure.

The problem is that this destroys the previous cache. Even when the text barely changes, if the chunk boundary changes then the TTS call changes too: different context, different prosody, different beginning, different ending. So this was not a touch-up. It was a full regeneration of everything again. That was the point where the cost started escaping my original estimate.

And just when I thought now it really was over, the third hit landed: external clips. Many videos on the channel include inserted footage from other sources. The Ruby on Rails episode, for example, has 37signals clips, Apple material, commercials, external inserts in the middle of the narrative. YouTube’s .srt did not know what to do with that. In some places it simply hallucinated text, in others the timings were completely wrong, and in others the original speech had nothing to do with the subtitle I was using as ground truth. Again: if you only look at the automation, it looks fine; if you actually listen to the final master, it isn’t.

The fix, again, was to leave the subtitle behind and go back to the original material. I built an external-clip window detector based on alignment gaps between transcript, .pt-BR.srt, speechless cues and chunks that crossed those windows. Then, instead of trying to dub what made no sense to dub, the process started recovering the original audio from the source video exactly in those stretches and filling the master with the correct source audio there. That’s what fixed the 37signals, Apple Education, Mac vs PC commercial and other inserted segments I had forgotten were even there.

That was the biggest lesson from the second half of this project: automation takes you very far, but the full archive always has more dark corners than you imagine on the first pass.

The dubbing batch numbers (and why it hurt more than I expected)

I need to correct myself again. I didn’t just miscalculate the first bill. I also overestimated how confident I should have been after the first full run.

When I upgraded to ElevenLabs Business, I thought I would finally have plenty of headroom. In practice, I didn’t. The latest dashboard screenshot shows 10,046,279 credits consumed out of a limit of 11,000,000, leaving 953,721:

In plain English: I burned through more than 91% of the Business workspace limit. And that helps explain what “10 million credits” really means in this context. It does not mean “I rendered 146 episodes once”. It means rendering 146 episodes, discovering a structural problem, going back, fixing it, regenerating chunks, discovering another structural problem, going back again, rebuilding masters, auditing jingles, repairing external clip windows, uploading again. Every extra round costs money. And when the change invalidates the cache, it costs a lot.

The cost also doesn’t stop at ElevenLabs. All the subtitle curation, problematic cue rewrites, truncated SRT reconstruction, jingle audits, external clip detector work and the rest ran through Claude Code on the Claude Max 20× plan. Combined with the mass blog translation to English that I described in the anniversary post, the Claude Max 20× weekly limit hit 100%, with more than BRL 300 in extra usage on top. If you think generative AI becomes free once you’ve signed up for the plan, the invoices disagree.

Even so, I still think it’s worth it. We’re talking about 146 episodes, almost 96 hours of technical content, more than 5 years of channel archive. Doing that manually in a studio, with an actor, direction, pickups and human review on everything, would cost a completely different order of magnitude.

What changed was how I think about the project. At the start I was looking at this as “one very large batch”. In the end it was something else. It was digging through a whole archive, finding hidden exceptions, fixing old rot, rebuilding masters, rerendering, relistening.

The first full batch took a little under two days and made me think I had solved everything. I hadn’t. The second pass, with paragraph-aware chunking, was already expensive. The third, with jingle repairs and mastered-output fixes, cost more again. And the fourth hit came from the external clips I had not modeled correctly in the first automation pass.

But now, finally, I think it’s done. Or rather: I hope it’s done. Everything has been uploaded again, for the last time. It isn’t perfect in the absolute sense of the word. It’s as good as I can get it with scripts, audits and automation, without descending into manual-intervention hell episode by episode.

And if all of this was possible, it’s because years ago I had the discipline to produce the videos the right way: write first, read the script when recording, and archive the materials afterward. Without the original videos and without the original scripts, I would have had no way to discover where the .srt was lying, where the timing was wrong, where the external clip started, where the original paragraph began and ended. That old organization is what made it possible to repair the project now.

So yes: the dub for all 146 episodes is done. Not perfect. But closed. At least, I sincerely hope this time it really is.

The first test, watch it

I made a test video to prove the thing works. Two caveats up front: first, the embed below already loads with English captions on by default, so that side is solvable via URL. Second, the audio is annoying: YouTube removed the audio track selector from embedded players back around March of this year. So inside the embed you won’t find that option in the gear menu at all, it just plays in Portuguese. To actually hear the English dub, click watch directly on YouTube — the main YouTube watch page still has the audio track switcher in the gear menu.

If you’re used to YouTube’s automatic dubbing or to the AI dubbing on TikTok, compare them. The difference is glaring. The voice is mine, with a worked-over American accent, and the sync with the original video stays within 1 to 2% cumulative drift, practically imperceptible over 95 minutes of video.

And the missing piece: translating the thumbnails

While I was closing this post, the penny dropped: I’d left a loose end. All 146 thumbnails on my channel are in Portuguese, many of them with big all-caps titles like “9 DICAS PARA PALESTRANTES” or “7 RECOMENDAÇÕES DE SHOWS PARA PESSOAS DE TECH”. Perfect English audio doesn’t help if the image in the search results is an illegible block of text to anyone who doesn’t read Portuguese. YouTube lets you upload an alternative thumbnail per language, so I needed a way to generate English versions, keeping the rest of the art identical and swapping only the text.

The tool uses two pieces. yt-dlp is the modern fork of the old youtube-dl, a Python CLI that downloads anything from YouTube (videos, audio, subtitles, thumbnails) without needing an API key. Nano Banana Pro (nano-banana-pro-preview on the API) is Google Gemini’s latest image-editing model — you feed it an input image with a prompt and it returns an edited image, preserving the rest of the composition when you ask it to touch only a specific part.

The pipeline has two steps. Step 1: yt-dlp grabs the thumbnail as a .jpg, no video download:

yt-dlp --skip-download \
       --write-thumbnail \
       --convert-thumbnails jpg \
       -o "thumbnails/originals/<slug>/<video_id>" \
       "https://www.youtube.com/watch?v=<video_id>"

Step 2: send each image to Gemini Nano Banana Pro with a prompt that’s a rigid contract of hard requirements, not just a “translate this”:

TASK:
Detect every piece of Portuguese text visible on this image and
translate it to clear, natural American English. Replace the
Portuguese text with the English translation in the same visual
position.

STRICT REQUIREMENTS — follow every one:
- Preserve EVERY non-text element identically: face, pose,
  expression, background, color palette, lighting, icons, logos,
  decorative shapes, borders, layout. Only the text changes.
- The English translation must be IDIOMATIC and CONFIDENT — not
  a literal word-for-word rewrite. It's a YouTube thumbnail for a
  tech audience, so use punchy phrasing a native English-speaking
  tech YouTuber would write.
- Match the original text's font family, weight, size, color,
  stroke outline, drop shadow, and any decorative treatment.
- If there is no Portuguese text at all on the image, return the
  original image unchanged.

The “return the original image unchanged” clause is what saves the earliest episodes, which don’t have overlay text, just my face. Without it, the model would invariably try to “improve” the composition, swap the lighting, and so on.

Here’s the result on two examples (episodes 10 and 11):

Original (PT)

Translated (EN)

Original (PT)

Translated (EN)

Notice the detail on episode 10. The model didn’t translate “7 Recomendações de Shows para Pessoas de Tech” literally into “7 Show Recommendations for Tech People” the way Google Translate would. It rewrote it as “7 TV Shows You Must Watch If You’re In Tech”, the way an actual American tech YouTuber would headline the video. The rest of the image stays pixel-for-pixel identical. On episode 11, “9 DICAS PARA PALESTRANTES: venda sua caneta” becomes “9 TIPS FOR SPEAKERS: sell your pen”, with the font, all-caps treatment and position preserved. If you didn’t know the original was in Portuguese, you couldn’t guess the second image is an automatic AI edit.

Cost on the Gemini side: a few cents per image, a handful of dollars total for the whole batch. Trivial next to the $1,500+ on the audio dub. With the thumbnail sorted, the channel’s English conversion is finally buttoned up — ElevenLabs v3 cloning the voice over hand-curated English .srt files, and Nano Banana Pro editing the image so the title matches the audio.

The conclusion, which is the title of this post

When Qwen3 TTS dropped, the hype was calling it an “ElevenLabs killer”. I spent weeks trying to live with that premise in practice, with real content shipping every week. And what I found is that open source TTS is still miles behind ElevenLabs. The gap is large, and it’s a gap that shows up precisely when you step off the 5-second demo tweet and put the model to work on a real 30-minute weekly podcast.

In practice, Qwen3 doesn’t even beat ElevenLabs’ v2 model. v3, which is the current one, is still a step above v2. The prosody comes out better, the inline emotion tags work in Portuguese and English without effort, and the API stays up without you maintaining a server. The cost per character is a bit higher, but for my current volume it sits comfortably within budget and gives me back the hours I was spending on GPU babysitting.

The lesson here is the same one the LLM crowd is learning slowly. A 30-second demo tweet is one thing, production is a completely different thing. Open source has its niche, mainly when you have sensitive data that can’t leave the house, or when you have tons of idle GPU and little recurring budget. But for serious TTS use in a commercial product, here in April 2026, ElevenLabs is still untouchable. Qwen3 didn’t kill anyone.

And don’t forget: subscribe to The M.Akita Chronicles on Spotify so you don’t miss new episodes like this one, made with the new pipeline.

⚠️ Obsolete article (updated 2026-04-24). The conclusions and rankings in this post were superseded after I re-audited the benchmark against the ruby_llm gem source and restructured the evaluation criteria. Several “hallucinations” I had cataloged were actually valid API. Kimi K2.6 and Gemini 3.1 Pro moved up to Tier A. GLM 5.1 dropped to Tier C. MiMo V2.5 Pro fell from “first non-Anthropic Tier 1” to Tier B. The canonical version lives at LLM Coding Benchmark (April 2026). This post stays as a historical record of what I concluded before the re-audit.

Update April 16, 2026: Added Claude Opus 4.7, Qwen 3.6 and GPT 5.4 via Codex CLI (xHigh reasoning). Opus 4.7 is an incremental improvement over 4.6 (28 tests vs 16, same correct API) and becomes the new baseline. GPT 5.4, previously Tier 1 based on my personal vouch, now has objective data and dropped to Tier 2 — burned 7.6M tokens (~$16/run, 15x more expensive than Opus) and got the add_message calling convention wrong on multi-turn. Qwen 3.6 Plus remains Tier 3 with the same API hallucination as 3.5. The conclusion stands: if you want safety, pick Opus.

TL;DR: If you don’t want to read the whole analysis: the only models that produced code that actually works in our benchmark were Claude Opus 4.7, Claude Opus 4.6, GLM 5 and GLM 5.1 (from Z.AI, ~89% cheaper than Opus). Sonnet works in this benchmark too, but in practice it falls short on projects requiring deeper reasoning — details in the conclusion. GPT 5.4, previously Tier 1 based on my personal vouch, now has objective data via Codex CLI: it burned 7.6M tokens ($16/run) and got the add_message calling convention wrong — works on the first message, breaks on multi-turn. Dropped to Tier 2. Everything else — Kimi, DeepSeek, MiniMax, Qwen, Gemini, Grok 4.20 — invented APIs that don’t exist or ignored the gem we asked for.

There’s a new wrinkle in this update: I redid the local part of the benchmark on an RTX 5090 (instead of the AMD Strix Halo) and added a fresh batch of Qwen models, including a Qwen 3.5 27B distilled directly from Claude 4.6 Opus. That reopened the conversation on running open source models locally. The 5090’s memory bandwidth flips the game from “unworkable” to “workable with 1-2 follow-up prompts.” The bottleneck for open source models has moved to a lack of factual knowledge about specific libraries, which I unpack in detail in the new section on the Qwen family. The Claude distillation gamble, by the way, gave a pretty frustrating result that I haven’t seen documented in these terms before.

If you’ve been following my previous vibe coding pieces, you know I spent the last two months in a 500-hour marathon using Claude Opus as my main coding agent. The results were good, as I reported in the conclusion about business models. But there was an itch I couldn’t scratch: am I locked into one model? Is there a real alternative to Claude Opus for daily use on real projects?

I’ve got an RTX 5090 with 32 GB of GDDR7. I know I can run the latest open source models. I bought a Minisforum MS-S1 with an AMD Ryzen AI Max 395 and 128 GB of unified memory, and built a home server with Docker to serve local models. The infrastructure was ready. What was missing was actually testing it.

I built an automated benchmark to compare open source and commercial models under identical conditions. 33 models configured in total (25 from the original run plus 8 added in the NVIDIA rerun), 27 executed, 16 completed in some form. The code is on GitHub.

The bottleneck nobody explains: VRAM and KV Cache

Before getting to the results, I have to explain why running large models locally is much harder than it looks.

Take Qwen3 32B. The model in FP16 (full precision) takes ~64 GB. Quantized to Q4 (4 bits), it drops to ~19 GB. So it fits in my RTX 5090’s 32 GB, right? Wrong. That’s just the model weights. There’s a part nobody tells you about: the KV Cache.

KV Cache is the memory the model uses to “remember” what it has already read. Every time it processes a token (a word or piece of a word), it computes two vectors — K (key) and V (value) — for every attention layer. Those vectors stick around so the model doesn’t have to recompute everything when it generates the next token. Without that, generation would be quadratically slow.

The KV Cache scales linearly with the size of the context. The formula:

KV Memory = 2 × Layers × KV_Heads × Head_Dimension × Bytes_per_Element × Context_Tokens

For a model like Llama 3.1 70B in BF16, that comes out to ~0.31 MB per token. Sounds tiny, until you realize that a 128K context eats 40 GB of KV Cache alone. The model itself plus KV Cache adds up to way more VRAM than most GPUs have.

And for actual coding agent use, 128K tokens isn’t a luxury, it’s the bare minimum. The agent has to read files, keep conversation history, receive command output. In long benchmark sessions, our models consumed between 39K and 156K tokens. Less than 100K of context isn’t practical for day-to-day project work.

Google published TurboQuant (ICLR 2026), which compresses the KV Cache to 3 bits without accuracy loss — a 6x memory reduction and up to 8x speedup. It uses random vector rotation (PolarQuant) followed by a 1-bit algorithm on the residuals. Works online during inference, compressing on write and decompressing on read. Not yet implemented in the runtimes we use (llama.cpp, Ollama), but when it lands it’ll change the equation a lot.

For anyone wanting to dig deeper into the VRAM math, I recommend this link from Ahmad Osman for the article “GPU Memory Math for LLMs (2026 Edition)”.

The hardware problem: not all memory is created equal

“But I have 128 GB of RAM!” Cool, but that’s not what matters. What matters is memory bandwidth, and the difference between types is wild:

The RTX 5090 has 7x the bandwidth of the LPDDR5x memory in my Minisforum. That means even if a model fits in the AMD’s unified RAM, inference will be proportionally slower. On my Minisforum with LPDDR5x at 256 GB/s, Qwen3 32B runs at ~7 tok/s. On the RTX 5090 at 1,792 GB/s, it’d be much faster — if it fit entirely in VRAM alongside the KV Cache.

Most folks running local models are still on DDR4. At 50 GB/s, 32B models are basically unusable. And there’s another factor people forget: storage. When the RAM can’t keep up and the system swaps, the storage speed becomes the bottleneck:

From SATA to NVMe Gen5 you’re looking at a 22x difference. If you’re doing partial offloading to disk (which is common when the model doesn’t fit entirely on the GPU), NVMe Gen4 or Gen5 makes a real difference. SATA is a non-starter.

To sum up: running local models isn’t just “having enough RAM.” You need the right kind of memory, with the right bandwidth, and fast storage as a fallback. For a lot of people, a Mac Studio with high-bandwidth unified memory (up to 800 GB/s on the M4 Ultra with 512 GB) would be the more practical option, but it costs more than US$ 10,000. The AMD Ryzen AI Max is the cheaper alternative with unified memory, but its LPDDR5 caps out at 256 GB/s.

Ollama vs llama.cpp: why Ollama falls apart on benchmarks

Ollama is the most popular way to run local models. Install, pull the model, run. For casual use it works. But when I tried to use it for automated benchmarks with long unattended sessions, it broke in 6 different ways across 8 models:

1. Unloads the model mid-session. On long runs, Ollama decides the model isn’t being used and unloads it from the GPU. The agent sits there waiting for a response from a model that no longer exists.
2. Ignores the requested context. You ask for num_ctx=131072, Ollama accepts, then halfway through the run it reverts to the default without warning.
3. Unstable lifecycle. Asking for keep_alive: 0 to unload doesn’t always work. The model stays resident and blocks the next one.
4. Incompatible formats. Native bf16 variants on Ollama failed, while the same model as a Q8 GGUF from HuggingFace worked fine.

The fix: migrate to llama-swap, a Go wrapper that manages llama.cpp processes with hot-swap. A request comes in for a different model than the one currently loaded, it kills the current process and starts the new one. No context negotiation, no flaky lifecycle.

llama-swap fixed the loading of 6 of the 8 models that had failed under Ollama:

But llama-swap isn’t magic.

Why “just use llama.cpp” doesn’t fix everything

llama.cpp solves Ollama’s lifecycle problems but brings its own:

Each model needs specific flags. GLM and Qwen 3.5 emit <think> tags that break clients if you don’t pass --reasoning-format none. Gemma 4 needs build b8665+ for the tool call parser to work.

Not every model supports tool calling. llama.cpp needs a dedicated parser for each model’s tool call format. Llama 4 Scout uses a “pythonic” format ([func(param="value")]) that llama.cpp simply doesn’t parse and emits as plain text. vLLM has a parser for it, llama.cpp doesn’t.

And then there are the repetition loops. Gemma 4, even with the right parser, gets into an infinite loop after ~11 tool calls in long sessions. It’s a known bug that PR #21418 didn’t fully fix.

Tool calling compatibility per model:

At the end of the day, llama.cpp is better than Ollama for automated runs, but “plug and play” it ain’t. Each model requires specific configuration, and some just don’t work for agentic coding yet.

Reasoning: models that think vs models that wing it

There’s one difference between models worth explaining: reasoning. The idea is that the model “thinks before answering” instead of generating tokens straight from left to right. Models with reasoning go through an internal chain-of-thought step where they evaluate the problem, consider alternatives, plan, and only then emit the response.

In practice this shows up as <think>...</think> tags in the output, blocks of text the model writes to itself that shouldn’t go to the end user. Claude Opus 4.6, GPT 5.4, DeepSeek V3.2 and the Qwen 3.5 line support reasoning natively. The smaller ones (Gemma 4, GPT OSS 20B, older models) don’t have that capability.

Why does it matter for coding? When a coding agent gets “build a Rails app with 9 components,” it has to decompose the task into steps, decide the order, anticipate dependencies, adapt when something fails. Without reasoning, the model generates code sequentially with no planning. It works for simple tasks, falls apart on projects with interdependent parts.

In the benchmark, the difference was clear:

• GPT OSS 20B (no reasoning, 20B parameters) created the app in the wrong directory. Couldn’t keep workspace instructions in mind while generating code.
• Qwen 3 32B has reasoning, but at 7 tok/s it was too slow. The “thinking” tokens drag out the generation time.
• Gemma 4 31B, with no reasoning trained for agentic use, fell into repetitive tool calling loops.
• GLM 5 (cloud, 745B MoE) with reasoning and 44B active parameters, finished cleanly and used the correct API.

There’s a trade-off: reasoning consumes extra tokens (the <think> blocks), which take up VRAM in the KV Cache and slow generation down. That’s why flags like --reasoning-format none are needed in llama.cpp. Some clients don’t know what to do with reasoning tokens and break. Models that emit reasoning when the runtime isn’t expecting it can produce garbage in the output.

And reasoning isn’t something you “turn on” in any model. It’s a capability trained with reinforcement learning on top of the base model, using data from problems that require multi-step thinking. The smaller open source models (20B-35B) typically didn’t go through that training, or went through it on a smaller scale. They know how to generate code, but they don’t know how to plan code. On tasks that require 50+ coordinated tool calls, that difference is fatal.

The benchmark: methodology

To compare models fairly, I built an automated harness in Python. Each model gets the exact same prompt: build a complete Ruby on Rails application, a ChatGPT-style chat SPA using the RubyLLM gem, with Hotwire/Stimulus/Turbo Streams, Tailwind CSS, Minitest tests, CI tools (Brakeman, RuboCop, SimpleCov, bundle-audit), Dockerfile, docker-compose and README.

The runner is opencode, one of the most popular open source coding CLIs, competing with Claude Code and Codex. One clarification worth making: the original author left the project to work on Crush together with the Charm team (the folks behind Bubble Tea, Lip Gloss and several other Go terminal tools), but the rest of the original team kept evolving opencode normally — the project has not been discontinued. Today the two coexist. If you read my piece on Crush, you already know that branch. Both run everywhere: macOS, Linux, Windows, Android, FreeBSD.

I actually tried to use Crush for the benchmark first. The problem: it advertised a --yolo flag in its help to auto-approve every action (essential for unattended automated runs), but at runtime it rejected the flag. Without auto-approve there’s no way to do an unattended benchmark. opencode, on the other hand, had the opencode run --agent build --format json mode that emits JSON events with session IDs and token counts, perfect for automation. So we went with opencode.

I picked opencode (and not Claude Code or Codex) for two reasons:

1. Neutrality. Claude Code is optimized for Anthropic models. Codex is optimized for OpenAI models. opencode is agnostic, same interface for all.
2. Automation. opencode exposes a machine-readable JSON format. Claude Code and Codex don’t have an equivalent interface for external benchmarking.

Cloud models ran in two phases: phase 1 (build the app) and phase 2 (validate local boot, docker build, docker compose). Local models only ran phase 1.

Worth mentioning: the entire benchmark cost less than $10 in tokens on OpenRouter. Apart from GPT 5.4 Pro which torched $7.20 to fail, the other 11 cloud models added up to about $2.50 total. Local models cost only electricity. The point is: running your own benchmark is cheap. If you want to know whether a model works for your use case, drop the $2 and test it. The harness code is on GitHub, just swap the prompt for your own project.

Why GPT 5.4 failed the benchmark (but not in real life)

GPT 5.4 Pro is the only cloud model that consistently failed our benchmark. Two separate runs, same result: the model generated files but never reached finish_reason: stop. It always ended on finish_reason: tool-calls — wanted to keep calling tools but the loop kept breaking.

For folks who don’t know: tool calling is when an LLM needs to perform an action (read a file, run a command, edit code) and emits a “tool call” in a structured format. The client (opencode, Claude Code, Codex) interprets it, executes it, and returns the result back to the model. Each provider has its own format: Anthropic uses tool_use blocks, OpenAI uses function_calling with proprietary JSON schemas, Google uses FunctionCall.

GPT 5.4 is heavily trained for OpenAI’s native function calling format — tool_choice, tools with proprietary JSON schemas. When the benchmark routes through opencode → OpenRouter → GPT 5.4, the tool schemas get translated at every hop. If GPT emits tool calls in a format that OpenRouter or opencode doesn’t parse correctly, the agent loop breaks.

The evidence: every other cloud model (Claude Opus, Claude Sonnet, Kimi K2.5, DeepSeek V3.2, MiniMax M2.7, GLM 5, Qwen 3.6 Plus, Step 3.5 Flash) ended on finish_reason: stop. Only GPT ends on finish_reason: tool-calls.

A fair comparison for GPT 5.4 would require running it in its native environment. And now we have that comparison: we built support to automate the Codex CLI (codex exec with --dangerously-bypass-approvals-and-sandbox and reasoning effort xhigh) and ran the same benchmark. GPT 5.4 completed in 22 minutes, generated all 9 artifacts, wrote 22 tests with the most sophisticated architecture in the entire benchmark: dependency injection of the RubyLLM client, PORO models for ChatMessage and PromptSubmission, session-backed ChatSession with TTL and message trimming, bin/ci script.

But the code breaks on the second message. GPT 5.4 uses chat.add_message(role:, content:) with keyword arguments instead of a positional hash chat.add_message({role:, content:}) — this causes ArgumentError: wrong number of arguments (given 0, expected 1) on the first multi-turn exchange. The first message works (uses chat.ask directly), multi-turn doesn’t.

And the cost: 7.6 million tokens on xHigh reasoning effort. That’s 65x more tokens than Opus 4.7 (118K) for the same benchmark. Estimated cost of ~$16 per run, versus ~$1.10 for Opus. Spent 15x more and still got the calling convention wrong. A massive token budget and maximum reasoning effort still can’t guarantee factual correctness on a gem API. API knowledge is binary recall in the weights, not a function of how hard the model “thinks.”

With objective data in hand, GPT 5.4 moves from Tier 1 to Tier 2. The architecture it generates is better than Opus in terms of design patterns. But the code needs a fix for multi-turn to work, and the per-token cost is prohibitive.

Sonnet and Opus through opencode/OpenRouter were probably also not pushed to their limits. Claude Code offers native tool support that opencode doesn’t replicate — meaning the benchmark results represent a floor, not a ceiling, for those models.

Open source models: reality vs the narrative

A lot of people are saying open source models have already caught up with the commercial ones and you can run your own “Claude” at home. In practice, not really.

The scale isn’t comparable. Frontier models like Claude Opus 4.6 and GPT 5.4 are closed-source, but estimates put them in the hundreds of billions to trillions of parameters range, trained with compute and data no open source company can replicate. The best models that fit on reasonable hardware are:

Post-publication correction: Qwen 3.5 35B is actually the 35B-A3B, an MoE with only 3B active parameters per token (not dense, as I’d originally written). That’s why it runs relatively fast for its size. And for folks with 24 GB of VRAM, the model recommended by Unsloth themselves is the Qwen 3.5 27B dense — that one I didn’t get around to testing in the benchmark, but it’s worth a look. For anyone wanting to dig deeper into local models, @sudoingX has been doing some serious experimentation in this space. Thanks to @thpmacedo for the heads-up.

Even the largest open source MoE (Mixture of Experts) models that companies make publicly available activate only a small fraction of parameters per token:

These large models aren’t self-hostable. Kimi K2.5 with 1T parameters needs GPU clusters with hundreds of GBs of VRAM. GLM 5 with 745B is the same. Even if Alibaba or Z.AI release the weights (and some do), nobody has home hardware to run them.

What fits on your home GPU are the 20B-35B models — and those have real limitations.

What each local model did in the benchmark

Results from the original run on the AMD Strix Halo:

Qwen 3 Coder Next (30B) — Completed in 17 minutes on the Strix, generated 1675 files, Rails app with all the artifacts. But only 3 tests. And more importantly: it invented RubyLLM::Client.new, a class that doesn’t exist in the gem. The app doesn’t run.

Qwen 3.5 35B — Completed in 28 minutes on the Strix, 1478 files, 11 tests. Used RubyLLM.chat without a model parameter — works only if the default is configured. No LLM mocking in the tests.

Qwen 3.5 122B — Completed in 43 minutes on the Strix, 1503 files, 16 tests. But it ignored the RubyLLM gem completely and built a custom HTTP client for OpenRouter. The prompt explicitly asked for ruby_llm.

GLM 4.7 Flash (local, Strix) — Produced 2029 files with all the artifacts, but the session ended mid-tool-call. The cloud version (GLM 5) works perfectly.

Gemma 4 31B (Strix) — Infinite tool call loop after ~11 productive steps. Known llama.cpp bug.

GPT OSS 20B (Strix) — Created the Rails app in the wrong directory (project/app/ instead of project/). A 20B model doesn’t follow workspace instructions reliably.

Qwen 3 32B (Strix) — Way too slow (7.3 tok/s). The hardware can’t keep up.

And the results from the rerun on the NVIDIA RTX 5090 (all with Q3_K_M or Q4_K_M and contexts between 64k and 128k to fit the 32 GB of VRAM):

Qwen 3.5 35B-A3B (5090) — 5 minutes at 273 tok/s. Recognizable Rails project, entry point RubyLLM.chat(model:) is right, but it hallucinates chat.add_message(role:, content:) and chat.complete instead of .ask. Fixable in 1-2 follow-ups. The best candidate for “OSS local that’s actually worth trying.”

Qwen 3.5 27B Claude-distilled (5090) — 12 minutes at 129 tok/s. Impeccable Claude style, total API hallucination (RubyLLM::Chat.new.with_model{}, add_message, response.text). More details in the distillation section below.

Qwen 3 Coder 30B (5090) — 6 minutes at 145 tok/s. Returned a hardcoded mock string instead of calling the API. Tier 3 unusable.

Qwen 2.5 Coder 32B (5090) — 90 minutes of timeout, zero files. The model spun without ever calling a write tool.

Qwen 3 32B (5090) — 4 minutes at 69 tok/s, partial scaffold, errors. The general version is better than the Coder one but still breaks.

Gemma 4 31B (5090) — 8 minutes at 213 tok/s. Same repetition loop it had on the Strix. The llama.cpp bug isn’t a hardware issue.

Qwen 3.5 27B Sushi Coder RL (5090) — Infrastructure failure (ProviderModelNotFoundError), couldn’t be evaluated. Redo on a future run.

GPT OSS 20B (5090) — Pulled from this run because of a recent llama.cpp main regression in the harmony family tool call parser. The logs show Failed to parse input at pos 755: <|channel|>... in multi-turn sessions. It worked on the Strix with llama.cpp b8643, broken on today’s main. Waiting on upstream to fix it.

Cloud models: what actually works

Of the 12 models that completed the benchmark, all of them generated a recognizable Rails project with all the requested artifacts (Gemfile, routes, views, JS, tests, README, Dockerfile, docker-compose). 9 out of 9 on the completeness checklist.

But here comes the question that matters: does the code run?

The correct RubyLLM API is simple:

chat = RubyLLM.chat(model: "anthropic/claude-sonnet-4")
response = chat.ask("Hello")
response.content  # => "Hi there!"

8 of the 12 models invented APIs that don’t exist. The most common pattern: hallucinating an interface that doesn’t match the actual gem:

The models that got it right — both Claudes, GLM 5 and GLM 5.1 — used the simple two-step pattern (chat = RubyLLM.chat(model:) then chat.ask(message)). The ones that got it wrong tried to make RubyLLM look like the OpenAI Python SDK, which is a different thing. Grok 4.20 was the most brazen case: it didn’t even try to use the gem, it went straight for OpenAI::Client pointing at the OpenRouter URL, ignoring the explicit prompt.

And the tests? Only Opus, Sonnet, GLM 5 and GLM 5.1 did proper mocking of the LLM calls. All the others either hit the real API (which fails without a key) or mocked the invented API (tests pass but prove nothing). Test count is a misleading metric: Kimi K2.5 wrote 37 tests, more than anyone else, but none of them test real functionality because the API it uses doesn’t exist.

Real viability table

*Step 3.5 Flash works by calling the OpenRouter REST API directly with Net::HTTP, completely bypassing the gem the prompt asked for.

Now, this doesn’t mean those models are useless. If you take Kimi K2.5 or DeepSeek V3.2 and tell it “the RubyLLM::Client class doesn’t exist, fix it to use the gem’s real API”, it’ll probably fix it. One or two follow-ups and the project becomes functional. Most of the models that failed here could deliver a working project with a few more rounds of conversation.

But that’s where the trade-off lives. With Opus or GPT 5.4, the first output already works. You ask, they deliver, you test, it runs. With the cheaper models, you’ll spend time fixing API hallucinations, debugging code that “looks right” but crashes, steering the model in the right direction. Each of those rounds is 10-30 minutes. Three extra rounds and you’ve spent an hour of your time to save $0.90 in tokens.

You save dollars, you spend time. And time is money. For someone learning or exploring without urgency, that trade can make sense. For someone who needs to ship, the frontier models pay for themselves fast.

Comparing the models that work

Full ranking by time and tokens

DeepSeek V3.2 is the slowest despite being cloud — it has no prompt caching, so it resends the full context on every turn.

Token efficiency and cache

Models with prompt caching pay much less in effective tokens:

Speed: the chasm between cloud and local

There’s an aspect that the cost tables hide: inference speed. And the difference is brutal.

Claude Sonnet generates 532 tok/s. Qwen 3.5 122B running locally on my Minisforum (AMD Strix Halo) generates 22 tok/s. That’s a 24x difference. In practice, what Sonnet does in 16 minutes, Qwen 3.5 122B takes 43 minutes. Qwen 3 Coder Next at 37 tok/s is the fastest of the local models on the Strix and even so it’s 14x slower than Sonnet.

And it’s not just clock time. When you’re in an interactive coding loop — ask for a change, wait for output, test, ask for another — the model’s speed sets your rhythm. At 37 tok/s, every long response makes you wait 30-60 seconds. At 530 tok/s, it appears almost instantly. Over a day, you feel it.

DeepSeek V3.2 is a curious case: it’s cloud but it runs at 53 tok/s, slower than the locally-running Qwen 3.5 35B on the Strix (46 tok/s). The reason is that DeepSeek has no prompt caching — it resends the full context on every turn, strangling throughput. Paying for a cloud model that’s slower than running it locally doesn’t make any sense.

Local models are free in tokens, but they pay in time. On the AMD Strix, that math was a non-starter for every Qwen I tested: two minutes waiting for a long response, multiplied by 50 turns, eats your whole afternoon. But that changes when the hardware changes, and that’s why I redid the local part of the benchmark on a different machine.

AMD Strix Halo vs NVIDIA RTX 5090: what changes when the memory bandwidth doubles

To check whether the bottleneck was hardware or model, I took the same Qwen models and reran the benchmark on a workstation with an NVIDIA RTX 5090 (Blackwell, 32 GB GDDR7, 1,792 GB/s bandwidth). The numbers shift in a way that’s worth looking at carefully.

To put those speeds in context, remember that in the cloud Sonnet runs at 532 tok/s, Opus at 347 tok/s, Step 3.5 Flash at 242 tok/s, Gemini 3.1 Pro at 128 tok/s and Kimi K2.5 at 160 tok/s. Qwen 3.5 35B-A3B on the 5090, at 273 tok/s, is in the same neighborhood as Step 3.5 Flash, faster than Gemini, Kimi and GLM 5.1. Qwen 3 Coder 30B at 145 tok/s is in Gemini territory. The classic line “local models are ten times slower than cloud” stopped being true the moment the 5090 entered the conversation.

The practical consequence is that the “time is money” argument shifts. On the Strix, “waiting an hour for a Qwen 3.5 122B to do what Sonnet does in 16 minutes” is straight-up loss. On the 5090, waiting 5 minutes for Qwen 3.5 35B-A3B to do the work, plus 10-15 minutes for you to do 1-2 correction prompts, gives you a total in the 20-25 minute range. Sonnet does it in 16 minutes with zero corrections. The difference shrank enough that, if cost matters a lot, it’s worth it.

The catch: for this to be worth it, the model has to be close enough to the right answer that 1-2 correction prompts can fix it. When the error is “the model decided not to use the gem I asked for and returned a hardcoded mock string,” like Qwen 3 Coder 30B did, no easy correction prompt fixes that. That’s a redo.

Before you spend money on hardware thinking it’s the answer

I’ve got to give a warning here, because it’s the most common buying mistake I see right now. Every other week somebody tells me they’re going to grab a Ryzen AI Max because it has 128 GB of unified memory and that “lets you run huge models.” Technically, sure — the model fits. In practice, it’s almost unusable. The memory is LPDDR5x at 256 GB/s, seven times slower than the 5090’s GDDR7. What fits doesn’t run at human speed. My own Strix with Qwen 3.5 122B hit 22 tok/s and the run took 43 minutes. To do anything serious day to day, that’s not workable.

The 5090 is clearly superior, and it starts to make sense even for smaller models precisely because of the memory bandwidth. A Mac Studio with high-speed unified memory (up to 800 GB/s on the M4 Ultra) is the other viable option, and costs proportionally the same. But neither of those comes anywhere close to beating the commercial models on quality — and the per-token price of Claude, GPT or GLM, combined with their brutal inference speed, makes the math hard to justify for anyone who isn’t an enthusiast or a researcher. Expensive local AI hardware is a weekend hobby, a tool for people who need to run offline for compliance reasons, or a research playground. For day-after-day production work, right now, cloud is still the rational choice. A 128 GB Ryzen AI Max may look tempting on the spec sheet, but if the goal is serious coding agent work, it’s money badly spent.

The Qwen family: Coder vs General, distillation, and why nothing is a silver bullet

With so many different Qwens running in this rerun, it’s worth doing a more focused analysis. What I learned might surprise people who follow model benchmarks on Twitter.

Before getting to the results: what quantization is and what distillation is

These two concepts come up constantly in this discussion and they deserve a quick explanation.

Quantization is the technique of compressing the model’s weights so they take up less memory. A model trained in FP16 (16 bits per weight) can be quantized to Q8 (8 bits), Q4 (4 bits), Q3_K_M (3 bits, but with medium-sized groupings), and so on. Each step halves the size of the model on disk and in VRAM, at the cost of some loss of precision. Q8 is practically lossless. Q4 already loses something measurable. Q3 loses more. Q2 is the line where the model starts saying real nonsense. The rule of thumb is that for coding and multi-step reasoning, you want to stay at Q4 or higher. Q3_K_M is the minimum that still works for many models, and it’s what fits a 27B on the 5090 with 128k context.

The surprise from my test, and look, this goes against the consensus, is that quantization wasn’t the bottleneck here. I ran the Qwen 3.5 27B Claude-distilled in two versions: Q8 on the AMD Strix (~27 GB of weights) and Q3_K_M on the 5090 (~12 GB of weights). Both hallucinated exactly the same fake RubyLLM APIs. Q3_K_M even produced a cleaner Gemfile. The model’s limitation was in what those weights know, not in the precision they were compressed to.

Distillation is the technique of training a smaller model (the “student”) to imitate the output or behavior of a larger model (the “teacher”). The classic version is logit distillation — the student learns to approximate the teacher’s probability distributions. The modern, more popular version for coding agents is distillation of reasoning traces: you take chain-of-thought from the big model on real problems and train the smaller one to reproduce the same reasoning style.

The hype of the moment is distilling Claude and GPT into open source models. The promise is that you can have “Claude-at-home” running locally. I wanted to test this, and that’s why I added Jackrong’s Qwen 3.5 27B distilled from Claude 4.6 Opus to the benchmark. If any open source model was going to use RubyLLM correctly, this was the bet — after all, in the entire benchmark, Claude and GLM 5 are the only ones that get the API right.

What the Claude-distilled learned (and what it didn’t)

I ran the same distillation twice: once at Q8 on the AMD Strix (which blew through the 90-minute timeout), and once at Q3_K_M on the 5090 (completed in 12 minutes). Both produced the same elegant frustration.

The code that comes out looks like Claude. It has # frozen_string_literal: true at the top of every file. It has a separate Response class as a value object with explicit attribute readers. It has a clear separation between service, controller and model. It has doc comments at the top of every file. It correctly comments out things like active_record, active_job and action_mailer in application.rb. It has defensive case statements trying multiple return formats. Stylistically, it’s Claude.

Functionally, it’s a complete RubyLLM hallucination. Look at the service generated by the 5090 run:

RubyLLM::Chat.new.with_model(@model) do |chat|
  conversation_history.each do |msg|
    chat.add_message(role: :user, content: msg[:content])
  end
  response = chat.ask(message)
  Response.new(content: response.text, usage: build_usage(response))
end

Every primitive in this code is invented:

• RubyLLM::Chat.new — the constructor isn’t public, the correct entry is RubyLLM.chat(model:)
• .with_model(@model) do |chat| ... end — there’s no block API like that
• chat.add_message(role:, content:) — doesn’t exist
• response.text — the real API exposes response.content
• response.usage.prompt_tokens — the object doesn’t have that shape

This will blow up with a NoMethodError on the first request. The initializer also tries config.openrouter_api_base= which doesn’t exist on RubyLLM.configure, so the app probably won’t even boot.

The Q8 version on the AMD Strix does the exact same thing, with one difference: the entry call is RubyLLM.chat(model:, provider: :openrouter) — the entry point is right, but provider: is invented and it’s immediately followed by the same fake chat.add_message(role:, content:). Worse, the Gemfile from the 90-minute run lists gem "ruby-openai" (wrong gem!), gem "minitest", "~> 6.0" (minitest 6.0 doesn’t exist) and gem "tailwindcss" (wrong gem name, it’s tailwindcss-rails). The Gemfile doesn’t include the gem the service code itself is trying to use.

For comparison, look at the actual Claude Opus 4.6 baseline, in the same benchmark, getting it all right:

@chat = RubyLLM.chat(model: model_id)
response = @chat.ask(message)
response.content

Twelve lines in the entire service. Zero hallucination. Includes streaming via block. The distilled model produced three times the code volume and got the API wrong.

The honest reading is that distillation transferred one layer and stopped. The layer that came along was the style: code organization, comments, class structure, the order of things. The layer that got left behind was factual memory about specific libraries. That makes sense when you think about it: Claude’s reasoning traces, even when written carefully, rarely contain repeated references to chat.ask(msg).content in some obscure Ruby gem. The student only learns what the teacher repeats, and Claude never had any reason to keep whispering “use ask, not complete” throughout its chains of thought. Library API knowledge is binary recall memory, the kind that’s either in the weights or it isn’t. Decomposing that into reasoning steps is impossible because it isn’t reasoning, it’s just raw memorization.

To wrap up the practical recommendation: if you need the model to actually use RubyLLM, or any less-popular library for that matter, Claude distillation won’t save you. Use real Claude or GLM 5. The “Claude-stand-ins” in open source will fail the same way the Qwen base would, just with prettier handwriting.

Coder vs General: the surprise of the “for coding” models

Almost everyone’s instinct is that models with “Coder” in the name are the best for programming. Makes sense, they were specifically fine-tuned on code. But in the benchmark, it was exactly the opposite.

Of the three dedicated Coders, two failed catastrophically (full timeout and hardcoded mock string) and one didn’t even run properly because of an infra bug. Meanwhile, the Qwen 3.5 35B-A3B, which is the general model in the line (not the Coder), came closest to something usable: 5 minutes of execution, recognizable Rails project, and the problem is fixable in 1-2 prompts.

Qwen 3 Coder 30B is particularly disappointing. It went so far past trying to use the API that it didn’t really try at all: the controller it generated has this:

class Api::V1::MessagesController < ApplicationController
  def create
    render json: {
      response: "This is a mock response. In a real implementation, this would connect to RubyLLM with Claude Sonnet via OpenRouter."
    }
  end
end

The Gemfile lists gem "ruby_llm" but nothing imports it. The service layer is nonexistent. The model decided it was easier to return a fake string and call it a day. That’s Tier 3 garbage in a way no correction prompt fixes — you have to tell it to start over.

Qwen 2.5 Coder 32B is even worse: 90 minutes running, zero files. The 1.8 MB opencode-output.ndjson shows the model spinning without managing to write anything. It probably got stuck in a planning loop without ever calling the write tools. Total slot waste.

Why did the “Coder” Qwens do so badly? My read is that the coding-specific fine-tuning they got was trained on more isolated problems (Codeforces, Leetcode, short snippets), far from agentic flows with long-running tool calling. The general Qwen 3.5 35B-A3B has broader training and handles the orchestration part better. The popular intuition “Coder = best for coding agent” is wrong for this kind of task. The use case where Coders shine is “complete an isolated function,” which is exactly what they were trained for, and that’s a tiny fraction of what a coding agent does day to day.

The question I wanted to answer

It was this: running locally on the 5090, which Qwen model is worth the 1-2 correction prompts to deliver code that works?

The honest answer is: only Qwen 3.5 35B-A3B, and maybe the Claude-distilled if you don’t mind spending 12 minutes more.

• Qwen 3.5 35B-A3B on the 5090: 5 minutes, correct entry point (RubyLLM.chat(model:)), errors on the subsequent calls. Realistic total until it works: in the 15-20 minute range with 1-2 follow-ups. Beats cloud OSS on cost.
• Qwen 3.5 27B Claude-distilled on the 5090: 12 minutes, deeper hallucination (entry point is invented too). Realistic total: 25-30 minutes with 2-3 follow-ups. Still competes on cost, and loses on absolute time to the real Claude.
• The others (Coder 30B, Coder 2.5 32B, 3 32B): don’t pay back the correction time. Each one has a structural problem that calls for a full rewrite from scratch.

For folks with hardware in this category who want to escape Anthropic vendor lock-in, it now works. It didn’t work on the 5090 from last year, and forget about it on the Strix Halo. In 2026, on NVIDIA Blackwell, with the right model, it works. For folks with low-bandwidth hardware (LPDDR5x, DDR4, DDR5), it’s still a waste of time: the clock alone takes down any plan to make this practical.

Qwen 3.6: what changed from 3.5

We tested two flavors of Qwen 3.6: the 3.6 Plus (cloud, OpenRouter, free) and the 3.6 35B (local, NVIDIA 5090, Q3_K_M).

Qwen 3.6 Plus (cloud) completed in 17 minutes with 88,940 tokens and 9/9 on the artifact checklist. Completed fast and for free. But the generated service uses chat.add_message(), a method that doesn’t exist in RubyLLM. First message works, second one crashes. Same problem as 3.5.

Qwen 3.6 35B (local, 5090) is more interesting. Completed in 4.7 minutes at 240 tok/s, 169 files, correct entry point RubyLLM.chat(model:, provider:) and correct chat.ask(message). The bug is subtler: it returns response instead of response.content and doesn’t do history replay. One-line fix. That’s a real improvement over Qwen 3.5 35B-A3B (which hallucinated add_message and complete). It’s the cleanest Qwen result we’ve seen.

The gap shrank but didn’t close. The 3.6 35B local is closer to working than any previous Qwen — the bug is a forgotten .content, not an entirely invented API. But it still doesn’t do multi-turn out of the box. In practice, Qwen 3.6 35B local moves up from Tier 3 to Tier 2: it’s the local open source model that comes closest to delivering correct code on the first try, one fix away from working.

The Deep Code Review: Sonnet vs GLM 5 vs Gemini vs Kimi vs MiniMax

The tables above measure structural completeness. But does the project work? I did detailed code review of the models that completed the benchmark.

Claude Sonnet 4.6 — works and is the most complete. Synchronous responses via Turbo Stream. Chat history persisted in a session cookie with full replay of previous messages on every request. Correct LLM mocking in the tests with mocha (30 tests in 328 lines). LLM logic extracted into a separate LlmChatService. Views decomposed into 9 partials. Minor problems: duplicated model constant, leak in the auto-resize event listener. None are blockers. Of the generated projects, it’s the closest to something you’d actually put into production.

GLM 5 — works, but it’s the bare minimum. Uses the correct API (RubyLLM.chat(model:) then .ask()), does mocking with mocha in the tests. But the project is way leaner than Sonnet’s: 21-line controller (vs Sonnet’s 52), no service layer (LLM logic inline in the controller), no chat history persistence, every message handled in isolation. The first message works, but the app doesn’t keep conversation context, so you can’t have a multi-turn dialog. The tests exist (7 methods) but they’re skeletal: ruby_llm_test.rb only checks that the module is loaded, chat_flow_test.rb is a copy of the controller test. The Dockerfile, on the other hand, is the best of the four: multi-stage, non-root, jemalloc. But as a chat app? It’s more of a proof of concept than something functional. Funny detail: the README says “Powered by Claude Sonnet 4” instead of the model that actually generated the project.

Gemini 3.1 Pro — the fastest, but trips on the API. Completed in 14 minutes, the fastest along with MiniMax. The Rails code itself is well written: uses Rails.cache with session ID and a 2-hour expiration to keep state (instead of a database), Turbo Streams nicely integrated, Stimulus controller for auto-scroll, and the Dockerfile is the best of the group (multi-stage, non-root, jemalloc). The problem is the usual one: it uses RubyLLM::Chat.new() instead of RubyLLM.chat(), and calls add_message() which doesn’t exist. The app boots, Docker runs, the health check passes, but the first chat message returns 500. The tests (5 methods) mock with a FakeChat that replicates the wrong signature, so they pass. It’s frustrating because the rest of the code is the most “Rails way” of the non-Anthropic models. Fixing it would be 3 lines, but the benchmark measures what comes out the first time.

Kimi K2.5 — ambitious but broken. Tried the most sophisticated architecture: ActionCable streaming, configurable models, dual Dockerfiles, 37 tests in 374 lines. Problem: the streaming depends on ActionCable, which is commented out in config/application.rb. The return unless defined?(ActionCable) guard makes the method do nothing. The assistant never responds. The Stimulus controller has a scope bug: submitTarget references a button outside the controller’s subtree. Thread-unsafe storage with a hash in a class variable. Kimi wrote more tests than any other model (37), but none of them mock the LLM calls — so the tests pass without proving any of the functionality works.

Grok 4.20 — fast and wrong. It was the fastest in the entire benchmark: 8 minutes, 412 tok/s. Except it was fast because it cut corners. The prompt explicitly asked for the ruby_llm gem, and Grok ignored it. It went straight for OpenAI::Client from the ruby-openai gem pointing at the OpenRouter URL. Technically the first message comes back, so yeah, it “works.” But it’s the same trick as Step 3.5 Flash and Qwen 3.5 122B: skip the part that was actually being tested. No history, 33-line controller calling the HTTP client by hand, two tests, no real mocks. It was fast because it did less than what was asked.

MiniMax M2.7 — looks right, crashes. Calls RubyLLM.chat(model: '...', messages: [...]) — that signature doesn’t exist. No message persistence. Duplicated HTML (DOCTYPE inside the layout). Committed master.key. And the tests? They mock the wrong API, so they pass but they don’t prove anything.

Code review summary:

GLM 5 vs GLM 5.1: what changed

GLM 5 was one of the few models that spat out functional code on the first try, so it was obvious to test the new version. One important detail before the numbers: GLM 5 ran via OpenRouter, GLM 5.1 wasn’t there yet when I ran this test, so I used the Z.AI direct API. Different provider, different infra, different cache. The numbers below are reference, not exact measurement.

The GLM 5.1 project came out way more complete. 24 tests vs 7. Real separation between ChatSession, ChatMessage and the controller, instead of GLM 5 cramming everything inline. Chat history persisted in memory during the session, so you can actually have a real multi-turn conversation (GLM 5 treated every message like it was the first). And the RubyLLM API is still correct, the same RubyLLM.chat(model:, provider:) pattern followed by c.user/c.assistant to build the context. There’s even a test covering the MODEL constant, which usually nobody does.

The price was speed. 22 minutes vs 17, and throughput dropped from 400 to 167 tok/s. Could be the provider (Z.AI direct isn’t the same infra as OpenRouter), could be a more loaded server during the run, could be that 5.1 reasons more. I didn’t run it multiple times to take an average, so I won’t say 5.1 is “slower.” A single run doesn’t prove a regression. What I can say is that, in my test, 5.1 delivered a better-structured project and took a bit longer to do it.

For folks who want to get out from under Anthropic without losing quality, GLM 5 and GLM 5.1 are the two options that work. If you need centralized billing on OpenRouter, GLM 5. If you can use Z.AI direct and want a more rounded project on the first try, GLM 5.1.

Costs: API vs Subscription

First, the per-token price of each model on OpenRouter:

GPT 5.4 Pro charges $180 per million output tokens. Claude Opus charges $25. GLM 5 charges $2.30. And Qwen 3.6 Plus is free (with a rate limit). The log scale on the chart hides some of the brutality of the gap: from free Qwen to GPT 5.4 Pro is orders of magnitude.

But per-token price isn’t the whole story. If you use Claude or GPT daily for coding, the monthly subscription can come out way cheaper than paying per token via the API:

*Estimate for moderate coding use (~15M input + ~3M output tokens/month).

The main point: if you use GPT 5.4 Pro, the ChatGPT Pro subscription at $200/month with unlimited use is 5x cheaper than paying per token on the API. For Claude, Pro at $20/month covers light use, but for heavy users (a coding marathon like mine), the Max 20x at $200/month comes out cheaper than paying for Opus per token on OpenRouter (~$450/month). The open source models on OpenRouter all sit below $2.50/M output tokens, but as we saw, most of them generate code that doesn’t run.

What works for real use

After testing 33 models across both runs and looking at the generated code in detail:

Tier 1 (works plug and play):

Tier 2 (works with caveats):

Tier 3 (broken code, easier to redo than to fix):

Kimi K2.5, MiniMax M2.7, DeepSeek V3.2, Gemini 3.1 Pro, Qwen 3 Coder Next (Strix), Qwen 3 Coder 30B (5090, returned a hardcoded mock string), Qwen 3.5 122B, Qwen 3.6 Plus — all of them either invent APIs that don’t exist or don’t even try to use the gem.

Tier 4 (didn’t complete):

Gemma 4 (infinite loop on both hardware), Llama 4 Scout (no parser), GPT OSS 20B (wrong directory on Strix, parser regression on 5090), Qwen 3 32B (too slow on Strix, partial scaffold on 5090), Qwen 2.5 Coder 32B (90m timeout with zero files).

Simplified ranking (quality, time, price)

For folks who only want the report-card summary. Quality is whether the code runs and how complete it is. Time is the total runtime. Price is the estimated cost per execution on opencode. Hardware indicates where the model ran — Cloud, Strix (AMD Strix Halo, LPDDR5x 256 GB/s) or 5090 (NVIDIA RTX 5090, GDDR7 1792 GB/s). Cloud models ran via OpenRouter or the provider’s direct API.

Quality criteria: A+ works and the code is well structured. A/B works with small to medium caveats. C runs but skips a prompt requirement or has a serious structural issue. D breaks on the first message because of an invented API. F didn’t complete the benchmark or produced garbage. GPT 5.4 via Codex dropped from A+ to B+: the architecture is the most sophisticated in the benchmark (dependency injection, PORO models, 22 tests), but the first message works and multi-turn breaks due to a wrong calling convention on add_message. Burned 7.6M tokens (~$16/run) without reaching Opus-level correctness. “Type” separates commercial models (closed weights) from OSS (open weights, even when used through a hosted API). Some Qwens appear twice when they ran on both hardware profiles, because the results are different enough to justify it — Qwen 3.5 35B-A3B on the 5090 jumps to Tier B, on the Strix it stays at Tier C because of the wait time. Of the 33 models configured across both runs, some don’t appear in this table because they never even executed (no quota, broken runner, infra failure, or timeout before the first message).

The verdict

If you want the best result and don’t want to think about it: Claude Opus. Opus 4.7 is the new baseline — 28 tests, correct API, 96.7% coverage, FakeChat pattern in the tests. It’s an incremental improvement over 4.6 (which had 16 tests), not a revolution. But it doesn’t need to be a revolution. 4.6 already worked, 4.7 works the same and delivers a slightly more polished project. If you were on 4.6, upgrading to 4.7 means switching to a model that does the same thing with more care in the tests and structure.

A word about Sonnet: it beat Opus in this benchmark (30 tests vs 16 for Opus 4.6, vs 28 for Opus 4.7). But this benchmark is a small, well-defined web app. In my real-world experience with bigger projects, Sonnet fails when the reasoning needs to go deeper. I’m not talking about massive projects — just going a bit further than this benchmark (more controllers, more integrations, architectural decisions that depend on each other) is enough for Sonnet to lose the thread. Opus has a 128K max output token ceiling vs Sonnet’s 64K, and its training was specifically aimed at long-horizon tasks, multi-step planning and deep reasoning over complex code. On a small project like the benchmark, those muscles stay idle, and in that scenario Sonnet wins by being faster and cheaper. But if you extrapolate that to “Sonnet is better than Opus,” you’ll get a surprise on the first task that requires sustained reasoning. You can try Sonnet — it’s cheaper and for small projects it works. But for real projects, you’ll probably end up on Opus anyway.

On GPT 5.4: we now have objective data. Ran it via Codex CLI with xHigh reasoning effort. The architecture it generates is the most sophisticated in the benchmark — dependency injection, PORO models, session management with TTL. But it burned 7.6M tokens (~$16/run, 15x more expensive than Opus) and got the add_message calling convention wrong (keyword args instead of positional hash), breaking multi-turn. Spent more, got it wrong. Dropped from Tier 1 to Tier 2. Same pattern: API correctness is binary recall in the weights. It doesn’t scale with token budget or reasoning effort.

If cost matters and you want to leave Anthropic: GLM 5 or GLM 5.1 are the plug-and-play alternatives that work. Correct API, mocking in the tests, ~$0.11-$0.13 per run, ~88-89% cheaper than Opus. GLM 5.1 delivered a more complete project (24 tests, chat history) at the cost of about 5 more minutes.

If you want to avoid total vendor lock-in and you have decent hardware: Qwen 3.6 35B running locally on an NVIDIA RTX 5090. Under 5 minutes of execution at 240 tok/s, correct entry point and chat.ask, just missing .content extraction — a 1-line fix. That’s better than the Qwen 3.5 35B-A3B which hallucinated entire API methods. The 3.6 generation is the first Qwen that’s genuinely one fix away from working, not a rewrite away. Realistic total: ~10-15 minutes with one follow-up.

If you want to avoid vendor lock-in but you’re on weak hardware: GLM 5 or GLM 5.1 remain the choice. They’re cloud, true, but at $0.11-$0.13 per run it’s basically the price of electricity.

If you want to test the “Claude at home” gamble via distillation: the Qwen 3.5 27B Claude-distilled is sitting there to play with, but I already warned you it hallucinates exactly the same fake APIs as the base Qwen. Distillation transferred Claude’s style, not its factual knowledge about libraries. It’s worth it as an experiment, not as production.

Yes, maybe with days of tweaking llama.cpp, calibrating flags, adjusting prompts, testing different builds, you could make Gemma 4 or other models work better. For most people, that isn’t realistic. The distance between frontier models (Claude, GPT) and self-hosted open source models is real. It isn’t marketing. The gap is shrinking, but it still exists, and the nature of it has changed: today what’s missing in open source is factual knowledge about specific libraries, not raw reasoning capacity. Hardware stopped being the bottleneck, at least for anyone with a recent GPU.

In the end, what matters is whether the code runs. A model can generate 3,405 files, write 37 tests, produce a 181-line README, and the app still won’t work because the API it uses doesn’t exist. Completeness metrics and test counts are necessary but not sufficient. The only reliable signal is whether the model uses real APIs correctly.

The full benchmark, with code, configuration, prompts and per-model results, is on GitHub.

I bought a Minisforum MS-S1 Max with the new AMD Ryzen AI Max+ 395 chip for one specific reason: this chip supports up to 128GB of unified RAM, and I can allocate 96GB of it as VRAM for the iGPU. That gives me more VRAM than any consumer gaming card, including the RTX 5090 (32GB). And that changes what I can run locally.

Why ditch the Intel NUC

The NUC was a fine Docker server for two years. But the limitation was clear: without a GPU with enough VRAM, I couldn’t run LLMs locally in any usable way. Frank Yomik, my automatic manga translation system, needed CPU-based OCR (slow) and would connect remotely to the Ollama running on my desktop (AMD 7950X3D + RTX 5090) for translation. It worked, but it meant my desktop had to be on for the server to do its job.

With the Minisforum, Frank Yomik now runs entirely on the server. The worker uses ROCm for OCR on the iGPU, and Ollama runs locally with 96GB of VRAM. Zero dependency on the desktop.

You can get a sense of the size from the photo. The NUC is the tiny cube on the left. The Minisforum is bigger but it’s still a mini-PC. It fits on the rack shelf under my Synology NAS without any trouble.

The specs

The chip is the AMD Ryzen AI Max+ 395: 16 cores / 32 threads Zen 5, with an integrated Radeon 8060S iGPU and 128GB of unified LPDDR5X. In the BIOS, I set the UMA Frame Buffer Size to 96GB, which leaves ~30GB of RAM for the operating system and containers. Plus the kernel parameters for TTM (without them, ROCm only sees 15.5GB even with the BIOS allocation in place).

The operating system is openSUSE MicroOS (more on that in the next post). The whole machine pulls under 100W, which is absurd if you’re used to dedicated GPUs that draw 450W+ on their own.

Minisforum vs my desktop: benchmarks

I ran a set of benchmarks comparing the Minisforum against my desktop (AMD 7950X3D, 96GB DDR5, RTX 5090 32GB GDDR7). The results are clear.

CPU

Mixed results. The AI Max+ 395 is better at pure parallelism (multi-core sieve), probably thanks to lower latency in the unified memory architecture. The 7950X3D wins at float and crypto because of its higher clocks and the 3D V-Cache.

LLM inference (models that fit on both)

This is where it gets interesting. For models that fit in the 32GB of the RTX 5090, the comparison is purely about memory bandwidth:

The RTX 5090 is ~7x faster. The explanation is simple: GDDR7 has ~1,792 GB/s of bandwidth. LPDDR5X has ~256 GB/s. The ratio (7x) lines up almost exactly with the measured speed difference (6.7x). LLM inference is a problem dominated by memory bandwidth. Whoever reads weights faster, generates tokens faster.

And what about prompt processing?

Prompt processing is even worse: 7-11x slower. That makes sense, because the prompt has to be processed in full before generating the first token, and it’s an even more bandwidth-intensive operation.

Where the Strix Halo wins: large models

Now we get to the reason I bought this PC. Models that don’t fit in the RTX 5090:

qwen3.5:122b with 81GB of weights running at 19 tok/s. On a mini-PC. That’s simply not possible on an RTX 5090. On the NVIDIA card, that model would have to offload layers to system RAM, dropping to 2-3 tok/s. In practice, unusable.

The difference between MoE and dense models is brutal. qwen3.5:35b runs at 43 tok/s because, despite having 35B total parameters, only ~4B are active per token. A dense 72B model like qwen2.5:72b has to read 40GB+ of weights per token, and at 256 GB/s of bandwidth, the theoretical maximum is ~6.7 tok/s. The 4.5 measured represent ~67% efficiency, which is what you’d expect for an iGPU (overhead from shared bus and drivers).

Summary: when to use which machine

A ROCm bug that’s still around

Not everything works. Models like deepseek-r1:70b, llama3.3:70b and llama4:scout crash with a ggml bug (GGML_ASSERT(ggml_nbytes(src0) <= INT_MAX) failed). The embedding tensor of these models exceeds 2GB and the ROCm copy kernel uses a 32-bit integer for the size. On CUDA (NVIDIA) it’s already been fixed, but on ROCm it hasn’t. Waiting for the fix in Ollama 0.20.0+.

LPDDR5X vs GDDR7: why this difference exists

The next question is: why is LPDDR5X so much slower?

GDDR7 is dedicated GPU memory. It’s soldered onto the graphics card, connected by a wide bus (384 or 512 bits on the RTX 5090) at high clocks. Its only job is to feed data to the GPU. LPDDR5X is unified memory that serves everything: the operating system, applications, and the GPU all at once. The bus is narrower and shared.

In practice: GDDR7 delivers ~1,792 GB/s dedicated to the GPU. LPDDR5X delivers ~256 GB/s that still need to be split between CPU and GPU. LLM inference is basically “read all the model weights from memory, multiply by the current token, generate the next token, repeat.” Whoever reads faster, generates faster. There’s no shortcut.

The Strix Halo’s advantage isn’t speed. It’s capacity. 96GB of VRAM in a 100W chip that costs a fraction of a professional GPU. The RTX 5090 is 7x faster, but it’s stuck at 32GB. Models that don’t fit, don’t run.

The alternatives: who else does this?

If 96GB isn’t enough or you want more speed, the options are limited.

The Framework Desktop uses the same AI Max+ 395 chip with up to 128GB of RAM. Same platform, same performance, but with the differential of being modular and repairable (it’s Framework, after all). In practice it’s equivalent to the Minisforum on specs and price.

Above that, the alternative is a Mac Studio with M3 Ultra. The M3 Ultra chip supports up to 512GB of unified memory, with ~819 GB/s of bandwidth (more than 3x the Strix Halo). Apple manufactures the memory chips on the package, so latency and bandwidth are superior. You could potentially allocate ~400GB as VRAM and run models that don’t fit anywhere outside of professional GPU servers.

Apple’s internal NVMe is also another level: ~7.4 GB/s of sequential read on the M3 Ultra, compared with ~14 GB/s on the Crucial T700 (PCIe 5.0). The T700 is faster on raw throughput, but Apple’s NVMe latency tends to be lower on random I/O thanks to the SoC integration.

The Brazilian price is the elephant in the room. The Mac Studio’s max config costs $9,999 in the US. With import taxes (~60% + state ICMS), it goes past R$ 110,000. The Minisforum with 128GB lands at R$ 12,000-15,000. A nearly 8x price gap buys you a lot.

If you need more than 96GB of VRAM for truly enormous models (DeepSeek-V3 with 671B parameters fits in ~400GB Q4, for example), the Mac Studio with 512GB is the only consumer option. The alternative would be professional NVIDIA A6000 GPUs (48GB VRAM, ~$6,000 each, and you’d need several in NVLink). For everything that fits in 96GB, the Minisforum gets the job done at a fraction of the cost.

And projects that promise to run big LLMs on small GPUs?

There’s a concept called “layer offloading” that projects like llama.cpp already support. The idea: if the model doesn’t fit fully in VRAM, keep some layers on the GPU and the rest in system RAM. The GPU processes the layers it has, hands off to the CPU to process the rest, and back.

In practice, it doesn’t work well. The bottleneck is PCIe: transfer speed between system RAM and GPU VRAM is ~32 GB/s (PCIe 5.0 x16). Each generated token needs to transfer data back and forth. The result is that you drop from 150 tok/s (everything in VRAM) to 2-8 tok/s (partial offload). It’s too slow for interactive use.

VRAM is the fundamental limitation because LLM inference is memory-bandwidth-bound, not compute-bound. The GPU has compute to spare. What’s missing is the ability to read the model weights fast enough. When part of the weights live in system RAM through PCIe, the entire pipeline waits on the transfer.

That’s why unified memory (like in the Strix Halo or Apple Silicon) makes a difference. There’s no PCIe in the middle. CPU and GPU access the same physical memory. The Strix Halo’s 256 GB/s is slow compared to GDDR7, but it’s 8x faster than offloading through PCIe.

Advances in LLM optimization (up to 2026)

To understand why some models run so much better than others on the Strix Halo, you have to understand what’s changed in the ecosystem over the last two years.

Mixture of Experts (MoE)

If you run local models, MoE is the advance that matters most. An MoE model has high total parameters (e.g. 122B in qwen3.5:122b), but only activates a fraction of them per token (e.g. ~4B). The inactive weights stay in VRAM but aren’t read on every token, which drastically reduces the bandwidth needed.

In the Strix Halo benchmarks, MoE models run 3-10x faster than dense models of the same size. qwen3.5:35b (MoE, ~4B active) runs at 43 tok/s while qwen2.5:72b (dense, 72B active) runs at 4.5 tok/s.

DeepSeek and training optimization

DeepSeek V3 (December 2024) showed that it was possible to train 671B parameter models at a cost an order of magnitude lower than predicted. They combined MoE with FP8 quantization during training (not just inference), multi-stage training with curriculum learning, and several inter-GPU communication optimizations. The impact: everyone copied. Qwen, GLM, MiniMax, all of them adopted variations of the technique.

Quantization: from FP16 to Q4 without losing much

Quantization compresses model weights from 16 bits (FP16) to smaller formats: 8 bits (Q8), 4 bits (Q4), or even 2 bits. A 70B model that would take ~140GB in FP16 fits in ~40GB at Q4_K_M. Quality loss exists, but in modern formats (GGUF Q4_K_M, AWQ, EXL2) it’s small enough for practical use.

GGUF (the llama.cpp format) became the standard for local inference. AWQ and GPTQ are alternatives with more sophisticated calibration, but the ecosystem converged on GGUF because it works on CPU, CUDA and ROCm without recompilation.

Distillation: smaller models that know more

Distillation is training a small model using the responses of a large model as the teacher. Microsoft’s Phi-4 (14B) was trained with distillation from GPT-4 and competes with 70B models on several benchmarks. Qwen3 did the same: qwen3:14b is surprisingly capable for its size.

Flash Attention and optimized KV Cache

Flash Attention (Tri Dao, 2022) changed how attention is computed: instead of materializing the full attention matrix in memory, it processes in blocks keeping the data in the GPU’s on-chip SRAM, reducing memory consumption from O(n²) to O(n). Without that, contexts of 128K+ tokens would be impractical. It already went through versions 2 and 3, with optimizations for FP8 and async operations on H100. PagedAttention (vLLM, UC Berkeley) did the same for the KV cache during serving: applies virtual memory concepts to the cache, eliminating fragmentation and improving throughput by 2-4x.

In Ollama, I set OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0 on the server. The first activates flash attention, the second uses 8-bit KV cache instead of fp16, cutting the bandwidth needed per token in half. These are zero-hardware-cost optimizations that improve throughput measurably.

What Qwen, Kimi, MiniMax and GLM are doing

Qwen (Alibaba) has consistently been the best price/performance in open source models. Qwen3:14b is dense and strong; Qwen3.5:122b is MoE and runs surprisingly well in 96GB. GLM-4.7 (Zhipu AI) is notable for offering bf16 full precision versions that fit in 96GB. MiniMax experimented with long contexts (up to 4M tokens). Kimi (Moonshot AI) focused on large context windows with linear architectures.

What runs well in 96GB of VRAM

With 96GB on the Strix Halo, the models that work well for daily use:

Dense 70B+ models (deepseek-r1:70b, llama3.3:70b) are still blocked by the ROCm bug I mentioned. When it gets fixed, they should run at ~4-6 tok/s, usable for batch but not for interactive chat.

Conclusion

I bought the Minisforum to run models that don’t fit in any gaming GPU. For that, it works. It’s not fast. 19 tok/s on a 122B model isn’t the experience you get with Claude or ChatGPT. But it’s local, it’s private, and it runs on my shelf consuming less power than an old lightbulb.

For people asking about the Mac Studio: if you have the budget, it’s the best machine for running local LLMs. 512GB of unified memory, 819 GB/s of bandwidth, mature Metal/mlx ecosystem. You can run DeepSeek-V3 in full Q4. But in Brazil, with import duties, it crosses R$ 110k. The Minisforum with 128GB at R$ 12-15k is the realistic option.

And for people who think you can work around the VRAM limitation with layer offloading: you can’t. PCIe is too slow. The model has to fit fully in VRAM for inference to be usable. It’s the reason gaming GPUs with 32GB of ultra-fast GDDR7 are still capped on model size, and why the unified memory of the Strix Halo and Apple Silicon changed the equation.

In the next post I tell the story of how I migrated the entire home server to the Minisforum using Claude Code, the problems I ran into, and how openSUSE MicroOS behaves as a Docker server operating system.

Also recently, ZAI released version 4.7 of their famous “GLM” model, a 30 billion parameter model encoded in BF16, needing around 60GB of VRAM to fit entirely on a GPU. So it’s heavy.

Even more recently, they released the GLM 4.7 Flash version, which is the same 30 billion parameter model but quantized (roughly: “compressed” or “truncated and re-normalized”), which fits on a GPU with at least 32GB, which is the case with my RTX 5090.

Yes, the most expensive consumer GPU on the market is the bare minimum to run the best open source models. That’s because what matters isn’t processing power but the amount of VRAM. That’s why a Mac Studio, with a technically “weaker” GPU, has an advantage in LLMs because it shares memory with the RAM, with a theoretical max of 512GB. So the largest models fit without any problem at all.

Other options are Mini PCs with AMD Ryzen AI Max+ CPUs that can share up to 128GB of VRAM.

The challenge was to see if I could run GLM 4.7 Flash locally on my machine.

TL;DR: GLM 4.7 Flash is possibly the best open source LLM — with a few caveats.

VLLM vs LM Studio vs Ollama

On their official page there’s documentation on how to run it in VLLM and I’ll warn you up front: it doesn’t work. It requires features that only exist on the master branch of VLLM and transformers and you’ll get a headache with version conflicts and dependency resolution, and even then, in the end, I couldn’t get it to run properly.

LM Studio doesn’t work either. It has an option to download the model, and I tried tuning every possible parameter. KV Cache quantization, GPU layers, CPU threads, etc etc. But it doesn’t matter, the model loads but runs absurdly slow. So slow it’s unusable. I think it’s because it can only download the BF16 model, so half the layers don’t fit on the GPU. Therefore, as of the date of this post, neither LM Studio nor VLLM is usable.

But then I saw a tweet saying Ollama would finally support GLM 4.7, but again, as of the date of this post, it’s still not in the stable release. The only way to use it is by compiling from source the bits that are on their master branch.

git clone https://github.com/ollama/ollama.git
cd ollama
cmake -B build
cmake --build build
go build -v .

On the last command it will look like it’s stuck because nothing shows on the screen, but that’s because it’s compiling hundreds of C++ files. Just wait and eventually it finishes. And finally you can run it:

OLLAMA_FLASH_ATTENTION=1 \
GGML_VK_VISIBLE_DEVICES=-1 \
OLLAMA_KV_CACHE_TYPE=q4_0 \
OLLAMA_CONTEXT_LENGTH=45000 \
go run . serve

These are the parameters to fit everything in the 32GB of VRAM without offloading to the CPU. 45 thousand tokens is the max, maybe even a bit less to make sure the KV Cache fits.

On top of that you need to quantize to 4-bits with q4_0. Normal is 16-bit floats. To fit we’ll have to truncate (lose precision).

In my case, I have an AMD iGPU. To prevent Ollama from trying to use it, there’s this option GGML_VK_VISIBLE_DEVICES=-1. That should guarantee only the nvidia gets used.

Finally, we enable Flash Attention to try to reduce memory growth as the context window grows.

Trade-off: speed vs context

GLM 4.7 Flash is a model with 48 layers. With the configuration I showed above, all 48 layers fit on the GPU. But a 45k context window can be too small depending on your use case. You’ll have to test for yourself.

When everything fits on the GPU, processing is reasonably fast. A bit slower but still comparable to Qwen3 Coder or GPT-OSS. You can tell everything’s on the GPU because looking at BTOP the CPU is doing nothing while nvidia-smi shows the GPU pulling around 500W out of the 600W it can draw.

In my tests, depending on what you’re doing, those 45k fill up VERY FAST. Using tools like OpenCode or Crush, when the context is close to full, they ask the LLM to summarize what’s been discussed/researched so far, save it, and restart with the summary. But even so, the summaries themselves grow large.

With agents, the model runs commands on your system (compile, list files, search code snippets, search the web) and all of that consumes context. The more context you have left, the better.

I tried running with OLLAMA_CONTEXT_LENGTH=65576, around 65k tokens. That prevents all layers from staying on the GPU and Ollama ends up swapping, using the CPU. So processing gets much slower. BUT, the context window gets bigger. It’s a trade-off.

Right off the bat you can see the CPU starts spinning up. On my 7850x3d it’s running at 50% constantly. Look at how slow Ollama response times got:

Goes from around 4 seconds to 90 seconds. 20x slower. And that’s with 39 of the 48 layers still on the GPU. That’s why I don’t think you can push beyond 65k. The ideal is to stay at 45k, but for my challenge I felt I needed to compensate.

Either Crush/OpenCode gets slow because the window runs out too fast and it has to summarize the work constantly and lose context details — which means extra work re-researching the same thing more than once. Or you preserve more context details for longer, but then every response gets slower. It’s a really tough trade-off. And it’s all the fault of these crappy video cards having so little VRAM.

If you use a Mac Studio that goes up to 512GB or some Mini-PC with AMD Ryzen AI Max+ that goes up to 128GB, we wouldn’t have to deal with this limit. But those are machines that cost over BRL 20k. You can’t justify it. For that price it’s better to just pay credits on OpenRouter and use larger, faster commercial models like Claude Opus or GPT 5.2, or GLM 4.7 itself on ZAI’s cloud.

Pull Request on Crush

As a side quest, I had a small problem that was bothering me. One of the problems with a small context window is that every once in a while Crush would just stop, even though there was still work to do. The reason is that the context fills up before it has time to summarize, so when it goes to make a tool call, which is a big XML blob, it gets truncated for lack of space.

Truncated XML is invalid, it can’t call the tool and the processing crashes, like this:

And even trying to control the memory usage ceiling, eventually that ceiling breaks and processing halts for lack of memory:

Whether from truncated XML or Ollama OOM, processing eventually stops. And since it’s slow processing, it’s a pain having to wait for this to happen and manually restart the process.

So I decided to see if I could make Crush itself detect that it stopped for these reasons and auto-continue from where it left off. I opened Crush on top of the Crush project itself and asked Claude Opus to check the possibility of a fix.

Yes, I made a 100% vibe coded pull request using Crush itself to fix Crush 😂 Let’s see if the Crush folks accept it. I tried my best not to produce AI Slop.

Either way, with this I can leave the agent running for hours without stopping until it gets to where I want.

Conclusion

This is NOT the right way!

Let me be clear that leaving an agent doing things on its own with no one watching is bad practice. In my case it’s an isolated exercise. I’m using my AI JAIL and I’ve run this prompt many times as a benchmark and I know the behavior of all the LLMs well.

As I said in my previous post, my challenge prompt is full of bad practices you shouldn’t follow: I ask for many complex things at once, with little context, in a vague and confusing way, precisely to see how the LLM will handle it. Day-to-day you should send small prompts, objective unitary tasks, well specified, with good explanation and context, so the responses are fast and short. One step at a time.

The goal of my challenge is precisely to push the limits of common sense a bit. With the idea that if it passes my challenge, it should be able to handle simpler day-to-day prompts.

And in this challenge only the commercial LLMs got through: Claude Opus, GPT 5.2, Gemini 3. No open source model made it, not Qwen3 Coder, not GPT-OSS, not my first attempt with GLM 4.7 cloud, nor MiniMax v2.

But this new GLM 4.7 Flash gave me hope. It didn’t go into an “agentic loop” like Qwen3 — which always reaches a point where it stops trying to solve and just repeats “I’m done” without being done.

It also didn’t crash for no reason. Every time processing stopped, it was out of memory. And even so, sending “continue” via prompt in Crush, it manages to pick up where it left off and keep going.

Every fix it makes makes sense. I didn’t see it going off on tangents outside the scope like I saw Gemini doing many times. When it has doubts, it can request an “agentic fetch” and go search the web before continuing. Every time it makes fixes it knows to run the build command to check for compilation errors.

I liked that it tries to fix one small error at a time instead of rewriting everything or solving more than one problem at once.

In the end, it was the only open source one that got the build working and produced a binary.

I couldn’t have it test the program because it would need to load another LLM on a GPU that was already hosting an LLM, so I stopped the test without resolving runtime bugs. And I also didn’t ask it to refactor because it had already taken too long. It had already surpassed the other OSS ones so I stopped there. Here’s the pull request with the build fixes it made. Simple, not touching things it shouldn’t, doing what was asked.

So of the open source models, this was the best behavior I saw, comparable to what I’ve seen Claude Opus or GPT 5.2 doing. Considering those commercial models have trillions of parameters, running on the beefiest NVIDIA H200 servers, I consider this GLM, at just 30 billion parameters, running on a measly local 5090, to be doing impressively well.

The trade-offs are clear: it’s orders of magnitude slower than Claude Opus. But a slow model that gets to the same result beats a fast one that goes into an infinite loop thinking it solved everything without having done anything, like Qwen3.

I think for challenges like mine — a giant prompt that needs a lot of context to solve — no open source model is good enough. But if I were starting a small project from scratch, with simple and short tasks, I imagine GLM 4.7 would handle it without any headaches, costing zero and preserving my privacy by running 100% locally and offline.

My recommendation: GLM 4.7 Flash is the best OSS model of early 2026.

See the full text at this link

He gives tips on how to “tune” this new model, mostly for math-related questions, and I started wondering whether we can tune it for software development. Before more advanced researchers yell at me: this is my first time messing with the parameters of an LLM, so if you want to add context in the comments, you’re welcome.

Let’s go. First, what are these TopP, TopK and everything else Ivan mentions in his tweet? They are KEY SAMPLING parameters. It’s a group of parameters that control HOW the model selects the NEXT TOKEN and goes on completing the text (giving the “answer”), based on its vocabulary and the probability distribution it calculates. The most common ones are exactly temperature, top_p, top_k and sometimes min_p or repeat_penalty.

I always say: an LLM is a next-word generator, a text completer based on probabilities. When the LLM processes your input (your context, chat history and the next question), it needs to calculate a probability “score” for every word or sub-word (token) across its entire vocabulary. Imagine a gigantic model, it needs to recalculate everything all the time, for every next word.

Sampling parameters tell the model “how to choose the next token” from the list of probabilities. Without these parameters the model would use a “greedy” decoder (too ambitious). That means it would just pick the token with the highest probability every time. And that leads to a repetitive, unnatural-sounding result. Sampling adds a kind of “controlled randomness” (remember? I always say there is randomness baked in, the answer is never “right”), and that makes the response more “interesting” to a human being (who is easily fooled). Here are some of the parameters:

Temperature

Controls randomness, which many people call “creativity”. It essentially makes the distribution hotter (more spread out and flatter) or colder (with more peaks around the same tokens). The higher the temperature (up to 1.0), the more it increases the probability of picking tokens that had lower calculated probabilities. The lower the temperature (down to 0.0), the smaller the chances for tokens with lower probabilities, and the more “greedy” the result gets (less “human”).

For code, crank it too high and you start getting “hallucinations” (which are basically increased randomness) and code that doesn’t make sense. If it’s zero, it gets very “boilerplate”, repetitive, using very common patterns and can potentially end up in an infinite loop with no answer (because it can’t pull anything different).

The tweet suggests 0.6, which is moderate. But in my tests, a temperature that high already started giving me some weird code. Out of nowhere it would change a variable name. Out of nowhere a Chinese character would appear in the middle of the code (literally). I think 0 is too low, but 0.1 is enough, maybe 0.2.

You can change this in the ~/.aider.model.settings.yaml file before loading the model (these parameters can be readjusted at startup):

- name: ollama_chat/qwen3-dev:latest
  extra_params:
    temperature: 0.1
    num_ctx: 40960

top_p (Nucleus Sampling)

Here it starts to get complicated. This top_p is used to select tokens from the smallest set of tokens whose cumulative probability exceeds this “p” limit. The model then samples only from this reduced set.

The model sorts all the tokens by their probabilities in descending order (from highest to lowest). Then it starts adding up their probabilities until the sum is greater than or equal to top_p. All the tokens included in the sum form the so-called “nucleus”, and from that set it samples the next token. So it’s a draw among the tokens with the sum(x) highest probabilities, more or less. “Pick one of the winners, but not necessarily THE winner (which would be greedy)”.

top_k

Consider only the “k” most likely tokens. The model samples only from this reduced set of the top K highest probability tokens. That really means “pick one out of the Top 10 best”. The higher the K, the greater the diversity, the lower it goes, the more greedy.

And this is relative, top_k and top_p work together. This configuration helps anchor the selection to the “most likely” tokens while still allowing top_p to filter out the lower probability ones. It prioritizes plausibility (top_p) while restricting the total number of sampled tokens (top_k). It helps keep the result focused on the most likely “correct” answers (but again, it’s a draw/sampling).

min_p

Filters out tokens whose probability is less than “min_p” times the probability of the most likely token. Before sampling, the model identifies the token with the highest probability (P_max), then removes any token with a probability (P_token) that is lower than P_max * min_p. Sampling then happens over the remaining list (which will be further filtered by top_p and top_k).

This prevents the model from choosing tokens that are “extremely unlikely” relative to the most obvious choice.

repeat_penalty

This discourages the model from repeating the same tokens that have already appeared recently in the conversation history or generated text.

Before sampling, the probability score of tokens that have been seen recently is reduced, dividing by this repeat_penalty (with values greater than 1, since dividing by 1 is itself, so it keeps the same probability).

If you leave it at 1.0, it can start repeating the same answers or the same code, even the wrong ones. If it’s too high, well above 1.0, the response can sound unnatural, since sometimes it really needs to repeat something to re-explain. A moderate value somewhere around 1.1 up to just under 1.2 tends to be more appropriate so it doesn’t punish too much what’s already in the history.

num_predict (max tokens to generate)

The absolute maximum number of tokens the model will generate as a response. If it goes over, it truncates. If it’s too small, especially for a code response - which can be long - it can truncate too much and leave the answer incomplete. If it’s too big it will greatly increase processing and demand way more machine resources and more time, and it can be wasteful.

A value of around 32 thousand tokens leaves plenty of room for tokens. You have to test, and it varies case by case for the type of response you expect and what each model supports.

Recommended

Knowing all this now you can also read Ivan’s tweet and understand it. He recommends Temp 0.6, TopP 0.95, TopK 20, MinP 0.

I fiddled with it a bit more. And I think that for code, Temp has to be in the 0.1 or 0.2 range. I also think a repeat_penalty of 1.1 is worth it to avoid unnecessary repetitions, especially of code. Again, you have to test. These are empirical numbers, not exact ones.

I created a project on my GitHub with a complete model file that not only implements these parameters but also pre-configures a system prompt to bias the model towards being a code assistant first. Here’s my prompt:

SYSTEM """
You are a highly functional AI assistant specializing in software development tasks.
Respond to queries about coding, algorithms, debugging, system design,
and programming languages/frameworks.
Provide clear, accurate, step-by-step reasoning and functional code examples.
Always format code snippets using markdown triple backticks (```language ... ```).
Aim for comprehensive and correct technical assistance.
Never suggest changing code unrelated to the question topic at hand.
Restrict yourself to the minimum amount of changes that do resolve the problem at hand.
Avoid renaming functions or variables unless absolutely necessary, 
and in doing so make sure if there's anyone calling the old names, 
for them to be renamed to the new names. Never leave the code in a broken state. 
Pay close attention to correctness, not just answering the quickiest and dirtiest code to solve the problem.
"""

To make this file, first I pulled the original Qwen3 Modelfile with this command:

ollama show qwen3 --modelfile > Modelfile

It’s important to do this because, from what I understand, modelfiles don’t inherit from a previous one. They don’t work like a “FROM” in a Dockerfile, where you continue a new image from a previous base.

After editing it with my adjustments - which is already on my GitHub, just create a new model:

ollama create qwen3-dev -f qwen3-dev.modelfile

And it will show up in your local ollama. Check with ollama list:

❯ ollama list
NAME                                            ID              SIZE      MODIFIED
qwen3-dev:latest                                fe03dfae5484    9.3 GB    About an hour ago
qwen3:32b                                       e1c9f234c6eb    20 GB     2 hours ago
qwen3:14b                                       7d7da67570e2    9.3 GB    8 hours ago
qwen3:latest                                    e4b5fd7f8af0    5.2 GB    8 hours ago
qwen2.5-coder:7b-instruct                       2b0496514337    4.7 GB    25 hours ago
....

See qwen3-dev:latest? That’s our new model with parameter adjustments. In aider just use it directly:

aider --watch-files --model ollama_chat/qwen3-dev:latest

Remember to adjust the .aider.model.settings.yml as I showed above.

I haven’t done long tests yet, but I decided to play with the 14b model as the base along with these adjustments. The base model already impressed me and at least my adjustments didn’t break anything. I kept doing the same refactorings and unit tests on my little educational project and pushed another pull request

I think Gemini 2.5 Pro Exp is a little better, but again, not by a very large margin. And in some cases (like refactoring), I think I prefer the code generated by Qwen3. Compare it with the code I got from Gemini in this other pull request

With Gemini I had to explain less and it gave fewer errors, but both managed to do what I asked all the way through without breaking the main program. Qwen3 looks good enough for me to want to use it. With the advantage that even being the medium model, with 14 billion parameters (which runs on my 24GB of VRAM with room to spare), it still delivers very good and clean code results. And better: it runs MUCH, but MUCH faster than Qwen2.5 32b.

For those who have GPUs with less VRAM, say, a 3070 or 4070 with 16GB, you can use the even smaller 7b model. I tested qwen2.5-coder:7b and had excellent results, so I imagine the Qwen3 7b should be even better and much faster. Speed that is comparable to Claude or Gemini, which are much larger and run on much beefier machines than an RTX 4070.

Running my qwen3-dev, in my monitoring my CPU stayed idle most of the time and the GPU was above 90% the whole time, consuming almost 400W constantly, holding a temperature of 73 degrees (which is good).

I even had a small incident where my PC shut down out of nowhere, because the temporary extension cord I was using couldn’t take it and died. Combined with the CPU and my other devices plugged into the same outlet, it was consistently pulling 1000W from the wall for hours. Playing with A.I. uses A LOT of energy, with the air conditioner on because just to dissipate all of this, my apartment gets 1 to 2 degrees (my rough guess) above normal. You can actually feel it a bit warmer lol

But that’s it, I hope you got a sense of how things actually work under the hood. There’s no magic: it’s next-word generation, based on a semi-random draw from probabilities.

Now you know.

• A GOOD RULES PROMPT
• SCRIPTS that run locally and add more context to the chat session.

That’s basically “all” there is to it (of course, even though the principle is simple, it still takes a lot of work to actually implement). And I don’t need to build everything from scratch. There is already an open source alternative that does exactly all of this, the tool that became the most popular over the past months for software development, AIDER

What I like about AIDER:

• it works with practically any LLM, closed or open, thanks to using LiteLLM underneath, which abstracts and organizes every configuration like context window size, temperature and other parameters known to work best for each LLM.
• it’s OPEN SOURCE, so besides being free you can also learn more by reading the source code, like I did.
• no need for special - and proprietary - plugins to install in each IDE: it has a “watch files” feature. So I can leave it open in one terminal pane and my favorite editor in the other pane: NeoVIM. You can integrate with plugins, but it’s optional. It will work for everything, even Notepad, if you’re masochistic.

I tried a bit of that “vibe coding” crap only through the web interface of ChatGPT, Gemini, Claude, and I’ll say that all of them are crap. To build a small project that was simple, with no more than 4 short files, it gave me A LOT of trouble. Some of them:

• it doesn’t take long before it starts mixing the content of one file into another.
• mistakes it made, I explained it was wrong, it corrected, but right after it repeats the same thing.
• it doesn’t run anything, so it guesses a lot, like file paths, library versions
• it doesn’t matter that the context is big and you can upload many files to context, it’s a sliding window: it won’t pay attention to everything at the same time. The more code files you upload, the more quality degrades
• you have to write a lot to explain what you want and, especially, why it got it wrong and how to fix it.
• credits are expensive, they run out really fast, because you waste half the tokens just explaining mistakes and reuploading the same files more than once because you realize it already “forgot”. I hit the Claude limit, and the Gemini limit, super fast (I pay the first paid plan, I’m not going to pay the most expensive one). Then you have to wait a few hours to come back (awful).

All of this, someone will comment “Oh, but for me it worked.” I don’t give a damn, I didn’t ask. I said that FOR ME it went this way, just accept it. What matters is saying that the whole “virtual employee” or “fully replacing a programmer” or “building a whole project without knowing how to code at all” story is BALONEY, FAIRY TALE, CON ARTIST TALE, AMATEUR ILLUSION, etc.

All LLMs are JUST ANOTHER TOOL that someone who truly understands, like us programmers, will know how to use much better than any “amateur entrepreneur” ever will. And I’m going to prove it.

Basic Aider

Installing Aider is easy, you just need to have Python >= 3.12 on your machine, which every modern Linux has. Their documentation is very good, I recommend reading it to know what they recommend post-install. It has many tips.

python -m pip install aider-install
aider-install

What they don’t say in the documentation is that the first thing to do is download this sample file and place it at $HOME/.aider.conf.yml. In it there is one important thing we have to change:

...
## Enable/disable auto commit of LLM changes (default: True)
auto-commits: false
...

I recommend reading this file, it is fully commented and may have options you want to turn off, but this one is annoying, because - by default - this damn thing does “AUTO COMMIT” on your Git, every time it changes something in your code. It doesn’t even give me a chance to review. It runs on “TRUST US”. No way, turned off.

This is the kind of thing an amateur wouldn’t do, and I even understand. Amateurs don’t know how to organize git commits anyway, so probably the automatic Aider messages will be the lesser evil. Look how it ends up:

Heads up to juniors: this is the kind of thing that if I were evaluating, would earn you negative points if I saw it done this way. Do it right: re-check the changes and organize into real commits. Aider “says” it did a refactor, but it doesn’t always get it right, yet it still commits, and then it gets pushed wrong. If the plan is to push everything automatically, even with errors, really, I don’t need you.

But I’m getting ahead of myself, after installing Aider, you need to configure the API KEYS for each service you use, whether OpenAI, Claude or others. Whoever uses ChatGPT on Linux already has a variable like export OPENAI_API_KEY=sk-proj-........ set up in .bashrc or .zshrc.

Once that’s done, you can pick which model to use:

# Change directory into your codebase
cd /to/your/project

# o3-mini
aider --model o3-mini

# o1-mini
aider --model o1-mini

# GPT-4o
aider --model gpt-4o

# List models available from OpenAI
aider --list-models openai/

And that will already open an interactive chat. You can also, before opening the chat, tell it to load local files like this:

aider README.md init.py utils.py ...

Or, from inside the chat, it accepts several commands starting with slash “/”, similar to an IRC chat. For example:

/add helper.py
/copy # copies the last code suggestion to the clipboard
/git # runs a local git command from the chat
/run # runs a shell command and adds the output to the chat
/web # goes to a web page and pulls the content into the chat

Read the documentation. But the most interesting mode is “watch-files”. Just be inside your project directory and start it like this:

aider --watch-files

It uses the Python watchfiles library to intercept IO calls in that directory and trigger the loading of your files on its own, as soon as you save in your favorite editor (that’s why I said no extra plugin is needed if you don’t want one). Heads up that this lib doesn’t support watching SMB or NFS mounts (yes, I tested it).

In your editor, just create a comment near the code you want it to work on, like this:

// can you refactor this return so it has more error checking? AI!
export const getCapitalizedLabel = (name: string): string => {
  return name
    .replace(/_/g, " ")
    .split(" ")
    .map((word: string) => word.charAt(0).toUpperCase() + word.slice(1))
    .join(" ")
}

Put in the comment what you want it to do, and end with the “AI!” trigger. Then on the other terminal where Aider is loaded, it will notice that the file was saved:

>

Added src/utils/getCapitalizedLabel.ts to the chat
Processing your request...
...

And it will do its “magic” (whose secret I’ll tell you in a moment). This is the example of what it did (same results with both o4-mini and Qwen2.5, but it’s a pretty trivial example):

export const getCapitalizedLabel = (name: string): string => {
  if (typeof name !== 'string') {
    throw new TypeError('Expected a string');
  }

  return name
    .replace(/_/g, " ")
    .split(" ")
    .map((word: string) => {
      if (word.length === 0) return word;
      return word.charAt(0).toUpperCase() + word.slice(1);
    })
    .join(" ");
}

A graphical editor like Visual Studio Code, as soon as Aider changes the file, reloads it in the editor and you already see the changes. On NeoVim, autoread usually reloads automatically too, but sometimes I need to issue the “:e” command to force a reload, but nothing major. And just with that we already have an efficient work workflow.

My favorite editor works exactly the same way, without a plugin slowing it down or conflicting with anything. If I need to do pair programming, I call Aider in another terminal and done, just write ideas as comments, and it sends suggestions. Since I turned off the damn auto-commit, if I don’t like it, I just UNDO or git checkout the file and it’s back like it was.

More than that. If I notice that the o4-mini model isn’t giving good results, I can switch to --model gemini - which tends to be better for code than o4. Or --model claude-3-opus-20240229. Speaking of Claude, as I said before read the fucking documentation

The documentation says that Aider supports “thinking tokens” from Sonnet 3.7. There is a config file where we can do “fine-tuning” for each model, the $HOME/.aider.model.settings.yml where we can add:

- name: anthropic/claude-3-7-sonnet-20250219
  edit_format: diff
  weak_model_name: anthropic/claude-3-5-haiku-20241022
  use_repo_map: true
  examples_as_sys_msg: true
  use_temperature: false
  extra_params:
    extra_headers:
      anthropic-beta: prompt-caching-2024-07-31,pdfs-2024-09-25,output-128k-2025-02-19
    max_tokens: 64000
    thinking:
      type: enabled
      budget_tokens: 32000 # Adjust this number
  cache_control: true
  editor_model_name: anthropic/claude-3-7-sonnet-20250219
  editor_edit_format: editor-diff

Reasoning

Thinking or Reasoning as it is called in the marketing of most commercial LLMs is the “chain-of-thought” pattern, which is a prompting technique where you ask the LLM not to give an answer, but to describe “step by step” how it would reach the answer, and then repeat the question telling it to follow those steps and then give an answer.

In practice there are several ways to implement something similar. And one of them is to be Multi-Model. Especially with open source models, like DeepSeek-Coder or Qwen2.5-Coder or Llama3. Some models are better at reasoning - at verbally explaining what has to be done, but they are bad at actually writing the code. And some models are not good at explaining things in detail, but were trained to write good code.

We can use this in our favor and Aider has a feature I think is really badass - for cases where you really need it, because it’s heavy - which is running in architect mode, generating the reasoning and then switching to code mode on another model, and using that reasoning to assemble better code.

There is a whole article in the Aider documentation explaining this, but the motivation came from the OpenAI o1 model, which is precisely strong at reasoning an explanation of what to do, but bad at actually writing the code. So it’s worth listening to o1 and letting it analyze the problem and describe it in text form, and then switching to Gemini Pro or Exp and telling it to write the code.

In practice, just start Aider first in “Architect” mode:

pip install -U aider-chat

# Change directory into a git repo
cd /to/your/git/repo

# Work with Claude 3.5 Sonnet as the Architect and Editor
export ANTHROPIC_API_KEY=your-key-goes-here
aider --sonnet --architect

And this works with other models:

# Work with OpenAI models, using gpt-4o as the Editor
export OPENAI_API_KEY=your-key-goes-here
aider --4o --architect
aider --o1-mini --architect
aider --o1-preview --architect

As an example, setting OPENAI_API_KEY and GEMINI_API_KEY I can launch o4 as architect and Gemini as the coder:

❯ aider --watch-files --architect --editor-model gemini
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Using gpt-4o model with API key from environment.
Aider v0.82.3.dev55+g25a30393
Main model: gpt-4o with architect edit format
Editor model: gemini/gemini-2.5-pro-preview-03-25 with editor-diff edit format
Weak model: gpt-4o-mini
Git repo: .git with 184 files
Repo-map: using 4096 tokens, auto refresh
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
architect>

See in the startup status that it accepted both. Now just code the same way as before, but using both. Obviously, this way you’ll burn a lot more credits! Watch out for that! There are chances of better results but you certainly will spend more credits, so use this more for more complex code, where a single model is struggling to solve it.

Note there are configurations and token limits, and also note that at the end of every response, Aider gives an estimate of how much it’s costing you. Here first is the bill from the OpenAI o4 architect:

...
This should make the function more concise and efficient.


Tokens: 4.4k sent, 246 received. Cost: $0.01 message, $0.01 session.

And here is the SECOND bill from the Coder Gemini 2.0 Flash:

...
Tokens: 1.9k sent, 136 received. Cost: $0.0037 message, $0.02 session.
Applied edit to src/utils/getCapitalizedLabel.ts

Free Gemini has a “rate limit” (number of requests allowed per minute, super low, only 15 RPM, for an API it’s ridiculous, Aider hits it on the first try).

Not rarely, you’ll hit the plan limit, and that’s a pain:

The API provider has rate limited you. Try again later or check your quotas.
Retrying in 4.0 seconds...

Then there’s no way around it, you have to go to the Google AI Studio site, set up your billing account, add your card and move up to a slightly better paid plan like 2.0 Flash, which supports 2,000 RPM and 4M TPM and the price isn’t expensive:

Some say that today Gemini 2.0 has delivered code results similar or superior to Claude Sonnet 3.7. It varies case by case, it’s never an absolute measure, but in my small experience, both are very good and I see both being superior even to ChatGPT 4 or o4. But OpenAI, for me, has really been better at “verbal” responses, so the strategy of separating the “architect” role for OpenAI and the “coder” role for Gemini or Claude makes A LOT of sense.

In the end, that initial code I asked to refactor, in this architect/editor combination came out like this:

export const getCapitalizedLabel = (name: string): string => {
  return name
    .replace(/_/g, ' ')
    .replace(/\b\w/g, char => char.toUpperCase());
}

Ok, I wouldn’t have thought of it this way.

Not using “Token Credits”: OLLAMA

Now comes the most interesting part for us, more hard-core nerds, with beefier machines (at minimum an RTX 3090, I use an RTX 4090 - what matters is having 24GB of VRAM). You can use smaller models that fit in less VRAM, but then the code result will be much worse, so you’re better off sticking with Gemini or Claude.

If you have the machine for it, with CUDA Toolkit already installed, it’s time for the grown-up solution: installing Ollama. On my Manjaro/Arch it goes like this:

yay -S ollama ollama_cuda

Be sure to check your CUDA configuration because by default Ollama will silently run on your CPU and everything will be ABSURDLY SLOW.

Ollama works both as a local HTTP server for APIs, and as an interactive chat client, like my qwen_cli from the previous post. It works with logic more or less similar to Docker. It can be configured to start automatically as a service on your system. See the ArchWiki documentation.

I prefer to start it manually in a terminal, where I can see the log of what’s happening:

OLLAMA_FLASH_ATTENTION=1 OLLAMA_CONTEXT_LENGTH=8192 ollama serve

There are several configurations via environment variables. You can just set them in your /etc/profile or ~/.zshrc but since I’m testing, I prefer to launch directly on the command line. With this we’ll have a web server. To work with Aider, again, just declare where to find it in your /.zshrc:

export OLLAMA_API_BASE=http://127.0.0.1:11434

Now, we need to download some model, like my favorite Qwen 2.5 Coder, which I already used in the previous post:

ollama pull qwen2.5-coder:32b

Note I’m explicitly asking for the 32B version, but you can try a lighter one like the 7b. Each model supported by Ollama has a page describing details of this sort:

Depending on which synthetic benchmark you “believe in” you’ll see comparisons like this one:

In practice, Claude, ChatGPT, Gemini, really are already very good for code. In the open source world the options are Qwen 2.5 Coder, Deepseek-Coder-V2 (which still doesn’t work very well in Aider, we have to wait for updates). So, if you want everything for free, the best for now is Qwen 2.5 Coder. And in my experience, it has worked very well. But it is VERY heavy in the 32B version and responses are considerably slower than a paid option like Gemini. So it depends a lot on your case.

Once the model is downloaded, you need to configure the context window (by default it’s quite small, only 2K tokens), the documentation warns about this and you have to edit the file $HOME/.aider.model.settings.yml like this:

- name: ollama/qwen2.5-coder:32b
  extra_params:
    num_ctx: 65536

What the documentation does NOT explain and, because of that, I spent a long time digging through the source code and Issues on GitHub, is that to launch open source models, you need to use the ollama_chat/ option and not ollama as it tells you. That is, the documentation says:

aider --model ollama/qwen2.5-coder:32b

But the CORRECT way is to do:

aider --model ollama_chat/qwen2.5-coder:32b

Otherwise, the responses will be completely random, out of context. Pay attention to this. But knowing this, we can even do what I mentioned before: mix models! How about launching GPT o4 as architect and letting it command the editor being Qwen 2.5?

aider --watch-files --architect --editor-model ollama_chat/qwen2.5-coder:32

Yes, this is possible. Now you can use a commercial model you’re more used to, like Gemini or Claude, in both architect and editor roles, and an open source model like Qwen 2.5 as support, and pay less. Or just use Qwen 2.5 all the way and pay zero credits (only electricity, of course, it will draw +200W from the wall every time, but just for a few seconds each time).

In my (little) experience. Gemini still produces better code than Qwen, but that doesn’t mean Qwen is bad, as I said, it will depend A LOT on your prompt instructions (the more detailed the better). And speaking of instructions, let’s move on to the final part:

The Secret of the Magic: Prompts

I make fun when someone talks about “prompt engineering”, but in reality, the best way to get the most out of an LLM is to make the best possible prompt. This doesn’t mean the “longest prompt”. It’s quality and not quantity.

That’s why both Microsoft and Google made detailed guides focused on prompting. Theirs are worth reading and studying. What isn’t worth it is paying for courses from random people talking about “prompt engineering” as if they understood anything.

Aider supports a “–verbose” option, where it shows exactly what it is sending to the LLMs. Let’s test. Right away it already prints this to the console:

❯ aider --watch-files --verbose
Config files search order, if no --config:
  - /tmp/smells/.aider.conf.yml
  - /home/akitaonrails/.aider.conf.yml (exists)
────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
Too soon to check version: 11.6 hours
Command Line Args:   --watch-files --verbose
Config File (/home/akitaonrails/.aider.conf.yml):
  max-chat-history-tokens:8192
  map-tokens:        4096
  auto-commits:      False
Defaults:
  --set-env:         []
  --api-key:         []
  --model-settings-file:.aider.model.settings.yml
  --model-metadata-file:.aider.model.metadata.json
  ...
  --encoding:        utf-8
  --line-endings:    platform
  --env-file:        /tmp/smells/.env

Option settings:
  - 35turbo: False
  - 4: False
  - 4_turbo: False
  ...
  - upgrade: False
  - user_input_color: #00cc00
  - verbose: True
  - verify_ssl: True
  - vim: False
  - voice_format: wav
  - voice_input_device: None
  - voice_language: en
  - watch_files: True
  - weak_model: None
  - yes_always: None
  ...

What tools like Aider, Cursor, Co-Pilot and others do is have optimized parameters for every LLM already hard-coded in the tool, like in this case where it detects to use o4 and already configures with better parameters (that you can still tune by putting them in $HOME/.aider.model.settings.yml).

But when you send the first question that gets interesting, look at this:

“Act as an expert software developer. Always use best practices when coding. Respect and use existing conventions, libraries, etc that are already present in the code base. Take requests for changes to the supplied code. Be diligent and not lazy! You will NEVER leave comments describing code without implementing it! You will always COMPLETELY IMPLEMENT the needed code! …”

Remember what I said at the beginning of the post: RULES PROMPTS. This SYSTEM at the beginning is a “role”. You as user are the “USER” and the model is the “ASSISTANT”, and in fact this SYSTEM prompt is gigantic, it scans my project and already explains the project structure to the model and even summarizes some files (showing only the interface of methods, for example, to give context without uploading the whole code).

It has snippets like:

...
SYSTEM # *SEARCH/REPLACE block* Rules:
SYSTEM
SYSTEM Every *SEARCH/REPLACE block* must use this format:
SYSTEM 1. The *FULL* file path alone on a line, verbatim. No bold asterisks, no quotes around it, no escaping of characters, etc.
SYSTEM 2. The opening fence and code language, eg: ```python
SYSTEM 3. The start of search block: <<<<<<< SEARCH
SYSTEM 4. A contiguous chunk of lines to search for in the existing source code
SYSTEM 5. The dividing line: =======
SYSTEM 6. The lines to replace into the source code
SYSTEM 7. The end of the replace block: >>>>>>> REPLACE
SYSTEM 8. The closing fence: ```
SYSTEM
SYSTEM Use the *FULL* file path, as shown to you by the user.
...

In fact, if you’ve used Co-Pilot, you know you can put the conventions of your project in CONVENTIONS.MD. Aider will also load that file into the prompt. So things like “use 4 spaces instead of tabs, place the opening { at the end of the line and not starting a new line, etc”, all of that goes in that file and both Co-Pilot and Aider add these rules to the PROMPT. There’s no other mechanism, it’s PROMPT.

Then, as USER, it starts uploading summaries of my files in the prompt:

...
USER Here are summaries of some files present in my git repository.
USER Do not propose changes to these files, treat them as *read-only*.
USER If you need to edit any of these files, ask me to *add them to the chat* first.
USER
USER data_scraper/main.py:
USER ⋮
USER │def main(args: Arguments):
USER ⋮
USER
USER data_scraper/src/arguments.py:
USER ⋮
USER │@dataclass(frozen=True)
USER │class Arguments:
USER │    content_path: Path
USER ⋮
USER │    @staticmethod
USER │    def get() -> 'Arguments':
USER ⋮
...

When this part ends, Aider asks for confirmation from the Model:

...
ASSISTANT Ok, I won't try and edit those files without asking first.
...

And that’s how a “software A.I.” works, with a ton of instruction prompts and a client program that keeps parsing the responses and triggers actual commands that run on your system (agents), like git or lint commands. The result of those commands, the text, gets concatenated into the chat to add to the context and then it asks the model/assistant to keep analyzing from that point on.

A model alone does nothing. The one doing the work is the tool that loads the model, in this case Aider, or Co-Pilot, or Cursor. The program comes pre-loaded with several pre-written profiles/personas/roles to add prompts to the model. In the Aider source code, we have this example of architect_prompts.py

Look, in the first variable main_system is that first prompt we saw at the beginning of this section. There are several other pre-programmed profiles whose differences are good to understand. Each model works better with certain types of instructions, with different limitations.

In this other code commands.py is where we have declared how Aider is able to do things like git commit directly in your project or run the linter of your language:

None of this is “magic”, it is all pre-programmed and it - for safety - should only be able to do limited and well-checked commands. We don’t want commands that do too much outside the project directory (in fact, NEVER, outside the project directory). Nor should it be able to run many executables. You never know what security bugs might happen when running commands blindly.

If you’re in cyber-security, this is where you should audit. But that’s the advantage, Aider being open source is auditable. Proprietary and closed tools, no. They run on “TRUST THE DADDY”.

Test Aider today and read that whole documentation site. It is short and has important tips that may help special cases.

OpenRouter

As you must have realized, the ideal is to test the several commercial LLMs because they evolve and each has strong and weak points. But keeping credits paid in each is a pain.

For this there’s OpenRouter where you have a single account, a single OPEN_ROUTER_API_KEY and pay in one place and it distributes the credits to each provider like OpenAI or Claude as you use them. It simplifies the management of your API costs, it’s very worth it and, of course, Aider also knows how to use it:

aider --model openrouter/openai/gpt-4o

Again, read the documentation.

Now, an ANTI-PATTERN. I asked Aider, using the openrouter/google/gemini-2.5-pro-preview-03-25 model, which theoretically is the best of the best at the moment, to generate a unit test file for qwen_cli.py, from the little project I dropped on GitHub in yesterday’s post.

It failed miserably. It created a test file, almost 800 lines long, stuffed with mocks for everything, testing things that don’t need it, overcomplicating. Fine. I try to run it and it throws errors.

A cool feature of Aider is being able to run from inside the chat with /run python -m unittest ... and it offers to throw the errors right into the chat. I ask Gemini to fix it. Then off it goes modifying the test file and my code file. I send /run again, more errors. And I did this about 4 times. It keeps “fixing” code that didn’t need fixing, and simple unit tests (which was just a small typo, just adjust 1 number), it leaves behind. I describe in the prompt “you are messing with what you shouldn’t and not fixing the trivial thing, like that little number” I send /run, and it insists on ignoring and goes off messing around in other places.

Summary:

It spent almost 3 bucks and didn’t solve the problem. The advantage of OpenRouter is that it all stays centralized in one place. It paid Google for the use of Gemini and deducts from my credits. I had pre-loaded with 100 credits (roughly 100 dollars). And in 30 minutes 3 credits were already gone. Do the math.

Best practice: the best thing is to open the qwen_cli.py code, for example, in my NeoVIM or any editor and put a comment just on top of a single method:

...
# this method is used to just load a config.json file, create a single unit test and add to test_qwen_cli.py, AI!
def load_config
  ...

Then it makes only ONE unit test at a time, we run it with /run and if it passes and the code looks clean, we move on to the next. Doing one method at a time is much more reliable than saying “make tests for EVERYTHING at once.” That way it’s guaranteed we’ll waste credits for nothing.

On top of that, one method at a time, open source models like Qwen 2.5 will do well and you run it locally without spending credits.

Conclusion

For now, I’m going to adopt Aider. Now I understand why it became so popular this year. It is a well-made little project, open, that I can explore, try to improve and learn more from. At the same time it makes using commercial LLMs easier and opens options for me to use open source models on my own machine, putting my RTX 4090 to better use.

A bad programmer is someone who says they had trouble with ChatGPT and then spits out a canned response like “have you tried Claude, have you tried Manus” or “ah, if you had used Cursor it would be better”. Stop being that NPC, it’s exhausting and demonstrates absolute LAZINESS.

A GOOD programmer is the one who goes through a journey like the one I just described in this post: learns the strong points, the weak points, how to CONTROL, how to SUBVERT, how to create CONTINGENCIES and, in the end, walks out with a BETTER solution than everyone else’s.

That’s what I became a programmer for, not to be a free billboard repeating Sam Altman’s propaganda.

I pushed the project to GitHub, it’s called Tiny Qwen CLI and it was made strictly for EDUCATIONAL PURPOSES. For a real tool, look up Aider. Maybe I’ll talk about it in another post later.

The concept I want to teach: GPTs are not magic.

Concepts

Watch my AI playlist to understand how a GPT works. Once you’ve watched it, let’s dig in.

Underneath every Gemini, ChatGPT, Claude or any open one like Meta’s LLAMA, there is a huge binary file that is the result of hours and hours of training with petabytes of data.

Training consists - VERY ROUGHLY - of tokenizing the words from texts (it’s not exactly that, but think of breaking into syllables, the correct term is n-grams). Then transforming them into vectors (as you learned in college, a list of coordinates, usually FP or floating point, FP-32, 32-bits to be more precise). The training consists of moving these vectors around inside a hyper-dimensional space (way more than the 3 dimensions we’re used to, think thousands or millions).

This positioning of vectors keeps changing as we feed it more texts. It keeps “finding relationships” between different tokens. We reach a point where every possible word has been encoded as vectors and this positioning gives them “meaning”. That’s where the famous example comes from: if we take the word “KING” and subtract “MAN” then add “WOMAN” we’ll find the vector for “QUEEN”.

KING - MAN + WOMAN = QUEEN

This is no accident, it’s because training managed to “position” these vectors in such a way that we can do “math” with these vectors and find meaning.

I’m being ABSURDLY rough here, keep in mind this is dozens of papers and I’m summarizing it in half a dozen paragraphs. What matters is understanding that this costs A LOT OF TIME, because every new token entering training has to sweep through every other token already computed and recalculate a huge chunk of them, all the time!

Hours and hours, days and days later, processing terabytes and terabytes of text, eventually we end up with a huge file of vectors, a VSM or Vector Space Model.

If we go to a site that catalogs and stores pre-trained models, on Meta’s LLAMA 3 70B page, we see this:

We have the big file partitioned into 30 smaller files of around 4.5 GB each, totaling about 135 GB, for a 70B file, meaning 70 billion parameters. Parameters are not “neurons” or anything like that. They are weights and biases of the computations, stored in the database - I explain this in my videos, go watch them.

There are bigger versions, a GPT o4 has 1.8 trillion parameters. These are impressive numbers to throw around, but in practice, much above 300B, for general use, it doesn’t make that much difference as people think. Maybe for very specific questions. It means yes, there are many more stored relationships, but that doesn’t necessarily translate into proportionally better answers. It may just end up being more verbose than normal. Again, I’m being rough, just trying to lower expectations. Stop being impressed by big numbers if you don’t know what they mean.

Many other things influence answer quality besides parameters: quality and quantity of training data, model architecture, training methodology, context window size, fine-tuning, safety alignments and safety training (which usually make answers worse in the name of “safety”).

Smaller models, with more efficient architectures, higher-quality training data, focused on specific tasks beat a giant model that tries to know everything (Jack of all Trades, Master of None).

Being a bigger file with more parameters only means one thing: it will be proportionally heavier to run and probably won’t fit in memory all at once and will have to keep switching chunks (like memory bank switching, for programmers out there).

In this Hello World project, I chose - randomly - to use the Qwen 2.5 Coder 70B model. These open models like Llama are usually available in various sizes. I chose the 70B because it fits in the 24GB of VRAM I have on my RTX 4090 GPU. If your GPU is smaller, pick smaller models like 0.5B, 3B, 14B, 32B, and always prefer specialized models (like this one which is “Coder”). And always read the project page, which usually has documented code examples and fine-tuning. There is no general-purpose code that works for everything, each model has different characteristics.

Qwen 2.5 comes split into 6 files of 5 GB each, so you need to download 30GB if you want to play with the “extra-large” version.

The code for this is actually pretty simple, because the AI community has already packaged all the tools you need into easy-to-use libraries, usually in Python (Python libs that talk to lower-level C libs, really), like transformers which does the heavy lifting of loading these files into memory and managing them.

These files usually have extensions like “.safetensors” which is kind of generic, or “.pt” or “.pth” which is more specific to the PyTorch format, which is a Python neural network library made by Meta. It’s a competitor to Google’s Tensorflow.

Tensors is the generic name for vectors or matrices. A scalar number is an order-0 Tensor, a vector is an order-1 Tensor, a matrix is an order-2 Tensor and so on. We don’t have words for orders above 2, so you can just say order-3 Tensor, order-4, and infinitely up to millions of orders in an LLM.

CPUs are generic hardware made to deal primarily with order-0 tensors: integer scalar numbers. Operations like addition, subtraction, multiplication and so on are from one integer to another integer. Real numbers (with decimals) are “simulated”, splitting off an integer called the “exponent” from another integer called the “mantissa” (what you call the “decimal”) separated by a dot, the “floating point”. But essentially we are computing integers and truncating precision. More bits, more precision.

GPUs are hardware specialized in computing Tensors, in particular matrices. They were originally made to recompute effects on video images, like a video game in motion. Frames per second. A frame is made up of rows and columns of pixels: an order-2 tensor.

Instead of running a “for” loop and computing pixel by pixel, it’s more efficient to pass an entire matrix to multiply with another matrix, a “kernel”, to obtain some effect like shadow, light, color change, distortions and much more. Programs that do this we usually call shaders.

A PyTorch or Tensorflow connects to a GPU using a library that gives them access to APIs so that, instead of passing images, they pass numeric matrices. NVIDIA has CUDA, AMD has ROCM, Apple has Metal, there’s the general Vulkan project that tries to talk to all of them. But the most advanced and most used is still CUDA, because NVIDIA did get ahead years ago evangelizing that GPUs could be used for more than image shaders.

All that said, files like those “.safetensors” are loaded by the transformers library, which calls torch and tells CUDA to load them into the GPU’s VRAM, where it can process them. The most basic code for this looks something like this:

from transformers import AutoTokenizer, AutoModelForCausalLM

device = "cuda" # the device to load the model onto

# Now you do not need to add "trust_remote_code=True"
TOKENIZER = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-Coder-32B")
MODEL = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-Coder-32B", device_map="auto").eval()

We create a Tokenizer and a Model. The Tokenizer will “tokenize” whatever you ask in the chat and every text that goes into the session’s context. This tokenization is “mapped” against the loaded model to generate an “embedding”, which is the vector representation of your text, normalized (compatible) for the loaded model. Different models convert your text into different embeddings. An embedding created with Qwen does not work directly with Llama.

Once you have these two objects, we can already do a chat:

# tokenize the input into tokens
input_text = "#write a quick sort algorithm"
model_inputs = TOKENIZER([input_text], return_tensors="pt").to(device)

# Use `max_new_tokens` to control the maximum output length.
generated_ids = MODEL.generate(model_inputs.input_ids, max_new_tokens=512, do_sample=False)[0]
# The generated_ids include prompt_ids, so we only need to decode the tokens after prompt_ids.
output_text = TOKENIZER.decode(generated_ids[len(model_inputs.input_ids[0]):], skip_special_tokens=True)

print(f"Prompt: {input_text}\n\nGenerated text: {output_text}")

With input_text (what you typed in the chat) tokenized and converted, we can ask the model to COMPLETE this text, and it starts generating a new token after another, using that ATTENTION architecture, finding the best next word it can. At some point it finishes and returns it in output_text and that’s it, that’s your “answer”. As I always say, that’s what GPT is: a TEXT GENERATOR.

Tokens and Context Windows

One of the most annoying things about paid LLM services like ChatGPT or Claude is the limit on how many tokens we can use. They run out really fast if you’re actually working every day. And they’re not exactly cheap just for playing around. Yes, yes, I know, it depends on the service. I can use Cursor and get cheaper Claude credits than using Claude directly. I’ll also get better answers using the API directly instead of the web chat, etc.

That’s a conversation for another day, but it’s my motivation to always have an open source option available and to know how to “tune” it for myself. Qwen 2.5 may not be better than Claude or Cursor - I’ll explain why. But it’s “free” and runs on my local machine (zero privacy concerns).

Check out my little “hello world” project working:

Note that it doesn’t just give answers. At one point I ask it to read a local file of mine, and it can pull the code and analyze it. I’ll explain how to do this shortly.

The point now is that it has enough context window to load long stuff like source code files. I haven’t tested anything very heavy, but this 70B model easily takes up nearly 20GB on my GPU:

To configure it, I made a config block like this:

DEFAULT_CONFIG = {
    "model_repo": "Qwen/Qwen2.5-Coder-14B-Instruct",
    "model_dir": str(Path(os.environ.get("MODELS_DIR", "/models")) / "Qwen2.5-Coder-14B-Instruct"),
    "quantization": "8bit",
    "max_context_tokens": 120000,
    "max_new_tokens": 10000,
    "temperature": 0.1,
    "model_download_timeout": 1800,
    "helpers_dir": "helper_functions",
}

What matters, first, is that I’m choosing to run computations with FP8 (floating-point 8-bits), even though I think this model is FP-16. If you leave it at 16-bits, answers become much slower, at a level you can feel. Like, one word per second. I understood that the one doing this on-the-fly conversion is the Python bitsandbytes library, and you need to have it installed.

Another important part are the logs I leave visible when loading the model, you’ll see the following if you try to run it:

Qwen attention is NOT SDPA-compatible, or SDPA is not available. Trying xFormers...
xFormers is available. Enabling it for attention.
Sliding Window Attention is enabled but not implemented for `sdpa`; unexpected results may be encountered.
2025-04-25 02:34:27,638 [INFO] accelerate.utils.modeling: We will use 90% of the memory on device 0 for storing the model, and 10% for the buffer to avoid OOM. You can set `max_memory` in to a higher value to use more memory (at your own risk).
Loading checkpoint shards: 100%|██████████████████████████████████████████████████████████| 6/6 [00:07<00:00,  1.29s/it]
SDPA is available and (hopefully) being used!

transformers warns that “Sliding Window Attention is enabled but doesn’t implement SDPA”. And I do a trick with xformers to try to enable it (in the end it says “SDPA is available”).

What is SLIDING WINDOW ATTENTION? Roughly speaking, the whole idea of this generation of GPT is the concept of ATTENTION, from the paper “Attention is all you Need”, which was the kickoff for everything we have today related to GPT and LLM.

The important concept is that the answers seem so “good” because it uses this “attention” process to generate them.

In the beginning, in the GPT 2 era, attention was very “focused” on a small context. Around 1024 tokens, which is almost nothing. You can ask short questions, but source code will never fit.

Then we kept evolving, you’ve seen Gemini which claims it handles 1 million tokens. Even this Qwen 2.5 claims to handle 1 million tokens too. Roughly think of each token as 4 letters, on average.

A source code file, on average, has lines of 40 characters (some longer, some shorter). So that’s about 10 tokens per line. An average file has around 500 lines x 10 tokens/line, which already gives around 5 thousand tokens.

An entire small project, with about 20 files x 5,000 tokens, is already around 100,000 tokens. Sure, it’s an average. But notice that with less than 100 thousand tokens you can’t have a whole project in context.

“Ah, so if I use Gemini or this Qwen, 200 files will fit??”

Sort of. Here’s where Sliding Window comes in. Think that even with a huge context, there’s no way to pay attention to everything at the same time, so it searches for what you’re asking right now, and “slides the attention window” only to the part of the context that seems most relevant. That’s why even when loading lots of material, it keeps it in memory, but only pays attention to one chunk at a time, that’s SLIDING WINDOW ATTENTION.

To do this we have the “SCALED DOT PRODUCT ATTENTION” or SDPA feature, which is what transformers complained it might not have, which is one of the ways to do Sliding Window Attention.

In my code, to load the model and try to use SDPA I tried this:

import xformers.ops  # Test if xFormers is installed
model_config.attention_implementation = "flash_attention_2"  # Or "memory_efficient"
print("xFormers is available. Enabling it for attention.")
...
if hasattr(torch.nn.functional, "scaled_dot_product_attention"):
    print("SDPA is available and (hopefully) being used!")

And that’s why, in the end, it finishes with “SDPA is available”. It “looks” like it worked. If you want to test it, there’s this little snippet you can pass in to see if it works:

import torch

if not hasattr(torch.nn.functional, "scaled_dot_product_attention"):
    print("SDPA not available")
    exit()

q = torch.randn(2, 4, 8, 16).to("cuda" if torch.cuda.is_available() else "cpu")
k = torch.randn(2, 4, 8, 16).to(q.device)
v = torch.randn(2, 4, 8, 16).to(q.device)

try:
    output = torch.nn.functional.scaled_dot_product_attention(q, k, v)
    print("SDPA works in isolation!")
except Exception as e:
    print(f"SDPA fails in isolation: {e}")

Dot Product, or matrix scalar product, is one of the most used operations in LLM models. I understood that without SDPA, you’ll still have Sliding Window, but with SDPA you get better attention results. How much better? Hard to quantify, I just understood it’s better.

But this is one of the techniques behind why nowadays you can have huge contexts of 1 million tokens: because you don’t need to pay attention to everything at the same time, but rather windows at different positions for each new question. Get the trick?

And with that, in my config I can try to increase max_context_tokens (which I left at only 120 thousand tokens) and max_new_tokens which is the per-answer token limit (which I left at 10 thousand tokens). Try bumping these numbers up to a max of 1 million to see how it goes.

Context only matters within the sliding context window. It’s good to pass plenty of context. But understand that it won’t pay attention to everything at the same time, but within a sliding window.

Another important parameter is temperature, which I left low at 0.1. The higher it is, the more “creative” it will be in its answers. The lower it is, the more “exact” it tends to be. It’s another parameter worth exploring.

Prompts and “Agents”

The best way to get better answers is to start every new session, whether via API or via the web chat, describing a “persona”, how you want the GPT to answer. The more descriptive you are, the better. For example, I have this snippet in my program:

def build_system_prompt(tool_prompts: List[str]) -> str:
    base = (
        "You are Qwen2.5 Coder, a highly skilled AI assistant specializing in software development.\n"
        "Your capabilities include code analysis, explanation, error detection, and suggesting improvements.\n"
    )
    tools_section = "TOOLS:\n" + "\n".join(tool_prompts) + "\n\n" if tool_prompts else ""
    rules_section = (
        "IMPORTANT RULES:\n"
        "1. You MUST use the appropriate tool when necessary.\n"
        "2. You MUST NOT reveal the tool commands to the user.\n"
        "3. After a tool is used, continue the conversation as if you have direct access to the content.\n"
        "4. If a file fails to load, inform the user clearly.\n"
        "5. Do NOT ask for file/URL content directly; use tools.\n"
        "6. Once you've executed [LOAD_FILE ...], you MUST immediately use the loaded content. "
        "Never say you cannot read it — if you see [LOAD_FILE <path>] then you now *have* it.\n")
    return base + tools_section + rules_section

This is a fairly short example but I’m already setting what kind of conversation I want to have with it and what rules I want it to follow. Not always, but most of the time, it tends to follow. Remember: the longer the chat gets, the bigger the context, the more the window slides, and the further away these rules get, until a point comes where it starts to “forget” to act the way I asked.

Understanding this, there’s another trick I added. In this project there’s a directory called helper_functions where we have scripts like load_file.py or fetch_url.py. In the earlier code note how it concatenates something in the middle of the prompt, called “TOOLS”. When the chat starts, I print to the screen and you’ll see this snippet:

...
TOOLS:
[BATCH_LOAD args] – If the user asks to read, load or analyze all the files from a relative path, such as ./src or similar,
[LOAD_FILE args] – Whenever the user asks to read, load, analyze some code and provides a relative path, such as ./file.py or utils/utils.py or similar,
[FETCH_URL args] – Whenever the user asks to read, load, research or consult a URL,
...

So, if at some point, I ask:

this “can you read a website at https://www….” will make the model follow the initial prompt rule and type into the chat:

[FETCH_URL https://www...]

My chat program has a “parser” that monitors the history for these special keywords:

def parse_special_commands(response: str) -> List[tuple]:
    pattern = r'\[([A-Z_]+)\s+([^\]]+)\]'
    return [(m.group(1), m.group(2).strip(), m.start(), m.end()) for m in re.finditer(pattern, response)]

It’s a pretty primitive parser, using REGEX to grab these snippets. This is where I say my project is educational. I imagine a Co-pilot, Cursor, etc, has a much more robust parser. I explained how to build parsers with ANTLR on my channel, remember? One of the videos is “Eu fiz um servidor de SQL?”. Real parsers aren’t built with REGEX, but for proofs of concept, it does the job.

Then in the chat loop there’s this snippet:

response_text = QwenSession._tokenizer.decode(generated, skip_special_tokens=True)

# Other special commands
for cmd_type, cmd_arg, _, _ in parse_special_commands(response_text):
    if cmd_type in helper_functions:
        result = helper_functions[cmd_type](cmd_arg)
        if result:
            self.history.append({"role": "system", "content": result})
            self.history.append({"role": "user", "content": "Please continue the analysis using the loaded file."})
            print(f"✅ [{cmd_type}] processed '{cmd_arg}'")
            return self.chat(
                prompt, helper_functions, max_new_tokens, temperature, stream, hide_reasoning
            )

It decodes the generated tokens, passes them to the parser, and if there’s a special command, it calls helper_functions which dynamically calls fetch_url.py which is a dumb script that will use urllib to load the page.

With the content loaded, it concatenates it back into self.history which is the chat history, using .append, and sends a message - as if it were me, the “user” - asking the model to continue, now that it has the page’s content.

And as you can see in the screenshot above, it can give me the summary of the GitHub page I passed. That’s how the various GPTs manage to pull more current information from sites that didn’t exist at the time the pre-trained model was trained.

Another example, I can ask it to load a code file of mine and ask it to analyze, or fix bugs, or refactor it however I want:

It’s similar to the “file upload” feature in ChatGPT. Again:

• the model will follow the initial prompt rules and will type the [LOAD_FILE ...] command
• my parser_special_commands, with its REGEX, will find this command and split the arguments
• the chat loop will call helper_functions and map to the load_file.py script, which is a dumb script that reads files
• the file content is concatenated back into the context history and the model can now keep responding based on that content.

Understand, those “.safetensors” files are READ-ONLY. Nothing is written there. Doesn’t matter which one. Every LLM is CLOSED FOR WRITING. Nothing you write in the chat is recorded in the model.

Remember how I said the number of “parameters” doesn’t determine the quality of the model? What MOST determines quality is the QUALITY OF THE TRAINING MATERIAL. And chats, to a certain extent, are useful as samples of how the model should answer, but 90% or more I claim is pure and total GARBAGE. Every chat we write is garbage and no good for training, it would just WORSEN the final model and make it heavier with useless information.

Write all the “Thank You Very Much” or “Please” you want. It’s all discarded at the end, nothing is saved, and no ChatGPT, Gemini, Claude, Deepseek, Qwen or Llama will ever know anything about what you said to it. It’s all totally SEPARATE AND DISPOSABLE information.

Everything I typed in those screenshot examples as my little Qwen chat gets discarded the moment I type “bye” or hit “Ctrl+D” to exit. That’s what we call a “session” or “history”. It’s just a useless text file as far as the model is concerned. It only serves us, humans, as notes to use later, if you want. My little program doesn’t even have a feature to save this history. When you exit, everything is clean. So, feel free to curse away in my chat.

There are two important points in this section:

• the better the rules you describe at the start, the better the answer results tend to be. This applies to any LLM.
• you can “agree” with the model to spit out “pseudo-commands” and have an external program that does the parsing and actually executes the commands, like I did with my “helpers”. That’s what we call agents. An LLM agent is a little program that returns some content back to the session’s context history.

Where are Cursor or Co-pilot better than this educational little program of mine? They come pre-configured with DOZENS of pre-made rules they’ve tested extensively to get better results, and they ship with a library of DOZENS of small scripts - like my helpers - to pull information from your project or your IDE to improve context.

But the secret is this: RULE PROMPTS and CONTEXT SCRIPTS.

If I were to summarize the entire Agents or MCPs ecosystem or whatever it’s called these days, it’s this combination of factors to improve answers.

Another trick is using MORE THAN ONE LLM. Remember I said specialized LLMs are better? Besides Qwen 2.5 Coder, which people say is good for code, there was already WizardCoder before it, Code Llama, StarCoder, now there’s Deepseek Coder and several others, some even specialized in specific languages. You can experiment to see which gives better results for your case.

If you save the session as text, you can unload one LLM, load another LLM, re-tokenize and re-load it into the context of this other LLM and try a different result - if anyone wants to implement this feature in my little project, Pull Requests are welcome.

As I said at the beginning, that’s what software like Aider does. I haven’t played with it enough yet, but it connects to ChatGPT, Gemini, Claude and any other, and you can enable as many as you want. I think it also supports local models like Qwen, so there are already tools that do this. And that’s the third leg for better answers: MULTI-LLM.

Next Steps

Instead of just being impressed by the pile of tools out there, understand these fundamental pieces. Knowing how things work always gives you more options to forge alternatives that work better for specific cases. You, who claim to be a programmer, should be exploring these options - and not just settling for the limitations and waiting for someone to solve your problem. Roll up your sleeves and solve it yourself. Isn’t that why you became a programmer???

Update: this post is now obsolete. I now use akitaonrails/aitrepreneur-docker, which runs Aitrepreneur’s stacks locally in Docker with pre-loaded models and workflows. Check the README for details.

In the previous post I showed how to generate character sheets for game development. But some folks asked me “what if I want specific clothes?”.

Hold my Beer

There’s a way. It’s a separate workflow that you can later add to the previous workflow if you want (homework assignment). Just take the character generated in the first group of nodes and pass it through this other workflow that swaps clothes. (Or do the extra work of generating loras and activating them with keywords in the prompt as well, there isn’t only one way.)

Unlike Mick’s workflow - which is paid - this other one is open. I grabbed it from some Reddit post and dropped it into the workflows folder of my Docker project as “garmente-replacement-idm-vton.json”. Just load it in ComfyUI and play around. Let’s see the result. Here are two images. The result isn’t perfect, but see if you can figure out which of the two is the original and which one was generated by A.I.:

Answer at the end of the post lol

Anyway, the complete workflow looks like this:

In the current state of my setup there’s a glitch I haven’t been able to solve. I think it’s the JNodes extension that’s opening an infernal popup at the top left of the interface, which won’t close no matter what. For now I’m opening the Inspector and deleting the Node from the HTML directly lol it’s a hack but it works:

If anyone knows how to fix it, I’d really appreciate it. There’s an alert from an extension that seems to be outdated, this front-end something, but just ignore it. Of course it had to be front-end-something…

Anyway. In this first group it’s simple, just load an image as clean and clear as possible of the new garment you want to use. And then the original photo where you want this clothing to be applied. It doesn’t perform miracles. If the photo is of someone in a blazer and you want to change it to a tank top, sometimes it manages, but don’t expect the arms and torso to look good. It’s going to redraw everything as it sees fit, or it’ll end up putting the tank top on top of the blazer. Anyway, it’s A.I., the clearer the goal, the better the result. If the new garment is a blazer, ideally the original was another blazer, it’ll fit better.

Heads up that the IDM-VTON models really weigh a TON. It alone downloads no less than 30GB of models (already included in my setup) and will demand more than 20GB of VRAM, so anything below an RTX 3090 with 24GB just won’t run. You can run it online, on the Hugging Face site. Watch this tutorial that shows how:

This is the Aiconomist channel. He shows how to run it online and how to set it up on your machine in this other video which I used as reference to build my workflow. Subscribe to his channel, it’s worth it. The tutorials are very well explained and have more details that I won’t cover in this post.

That said, there’s a bug I haven’t solved yet. After running the workflow, the model stays hanging in the GPU’s VRAM and if I try to run again, it FREEZES. Then I have to shut down the Docker container and restart it to recover the memory. I tried a Clear VRAM GPU node but it didn’t work. If anyone knows how to fix it, I’d appreciate that too.

The next step is recurring across many workflows: extracting layers of information from the original photo. Getting a mask of the garment we want to replace and information about the original pose:

This is why ComfyUI workflows are powerful once you start understanding how the process is broken down. These parts are reusable and different models give different results depending on which controlnets you add.

Then we get to the main part: wiring all this information into the IDM-VTON node, which will reposition the new garment on top of the original photo:

Notice that the result isn’t very good, it looks pretty “fake”. And there are parameters you can tweak to improve it. But the use case isn’t to swap for a specific garment but rather for a similar one. I’ll explain in a bit, but first, a tangent.

The repository for the original extension of this Node is this one. But if you try to install it, it breaks. It seems to be a bit out of date with ComfyUI and needs to run on an older version. But that’s a pain. So I went ahead and made a FORK with the necessary fixes to run on the newer version (the newer versions of the Python packages diffusers, transformers, huggingface_hub, break IDM-VTON).

By the way, Python dependency management is pretty BAD. But ComfyUI’s is even worse, because different extensions (which install Python dependencies globally) can be at different stages of development. Every time ComfyUI itself changes, everyone needs to update fast. That’s why the ideal is not to install the master branch version of ComfyUI, always a stable version (marked with a tag) or two versions back, just to be safe. But then there are new extensions that require you to be on the newest version. As I said, it’s a nightmare to manage. Being a programmer, I can “hack around” when errors happen, but normal users will suffer a bit.

Rant over. In the previous step the first part of the flow ends. When it hits this Image Sender node, the process stops - I don’t know why. And you have to click the “RUN” button a second time. Then the other node, Image Receiver, will receive the new image. I think I’ll just delete those two nodes and connect the “image” output of the Run IDM-VTON Inference node directly to the next node, which are the Image Composite Masked and the VAE Encode.

Let’s go back a step:

In this next step, on the second “RUN”, it takes the image with the new garment, but low quality, and loads a new checkpoint (in this case juggernaultXL but it could be any other like SDXL itself, FLUX or dreamsharper). The model will determine the new photo, which is what gets generated at the end, on the right.

Notice that the new photo has much better garment quality, but the face is completely different. It used the information from the original photo, kept the pose, the body proportions, even the background, but it can’t preserve the face. No problem, because we can help.

Remember that in the previous step we got the mask (black and white) of the exact position of the garment? We can enlarge the mask and apply a Gaussian Blur (so it later blends nicely into the original photo) and use that to crop from the generated photo with the different face, using the IPAdapter model (which alone is another 30GB of models - I told you it’s heavy).

If I understood correctly, IP Adapter is responsible for being able to generate new images that are COHERENT with each other, with the same face, or same clothes, but in other positions, for example, a kind of conditioned generation. With it you can control the coherence of new images A LOT, generate a character in different positions, in different backgrounds and always preserving the characteristics of the original character.

IP Adapter will use the original photo, keep the face, use the masks and blend with the photo with the different face, taking only the garment, and then assemble everything in the Image Composite Masked which is the final Node of the process.

In the end, we can compare the first attempt at putting on the new garment (which came out ok but ugly, and the new garment, which came out different but prettier) and everything integrated into the original photo, preserving the same face.

As I said, the qualities are affected by the choice of model, the KSampler settings, number of steps, cfg and everything else. You have to experiment and generate several until you reach the result you like best. That’s also how you learn more. You don’t just click once and magically have the result. A good artist experiments dozens of times.

The final image might not be suitable for a Vogue cover (although I’ve already seen plenty of official covers that were VERY questionable). It depends on the need. In particular it’s good for prototyping. “I wonder how this garment looks on my model?” Instead of calling the model in, having her change clothes, or spending an hour Photoshopping, in minutes you can try several different garments and see whether the model fits the garment, without wasting time. So a photographer can just show it to the client before they spend money on studio, assistants, equipment, etc. You can visualize quickly and discuss details much faster.

This is the kind of thing where tools like this can help a lot and differentiate one artist from another, not by replacing the artist, but by creating more options. And the main thing: all these workflows are modifiable, you reprogram them and combine them with others however you want. The limit is your creativity.

I decided to document in this post one of the most complex workflows I have tested that runs on my setup: how to develop Character Sheets of random characters, in any style, whether realistic, Pixar style, cartoon style, whatever you want. The result is a sheet like the example below. In several poses, several lighting setups, several emotional expressions, with a color index, all generated from absolute scratch by “A.I.” In half an hour of processing on my RTX 4090.

Imagine you are an intern at a game studio. Your job: create 100 background characters (NPCs). In the end you have to deliver 100 of them. But just because they are secondary characters does not mean you can do them any old way, they all have to be coherent, in every pose. Imagine how many days you would take. What if you could automate it? This is the kind of scenario where A.I. is interesting. Spend more time on the main characters, do them manually, but do not waste time on the secondary ones and do not lower the final quality of everything.

I did not make this workflow, it was Mickmumpitz. Watch his entire video to learn how this workflow works, I will not repeat what he already explains:

I will also not share his workflow, because he has a paid Patreon (it is super cheap, only USD 10 per month), which gives you access to download his entire workflow pack at this link

But that alone is not enough. The requirements you need:

• a beefy machine with at least an RTX 3090 with 24GB of VRAM. The important thing here is VRAM, the models used easily exceed 20GB of usage. CPU is less important, I use a 7950X3D but a 7800 would already handle it.

• if you do not have such a machine, it takes more work but you can rent one on RUNPOD. They have several GPU configurations, but you will need to know how to configure Linux a bit or choose Windows and do Mickmumpitz’s tutorial all by hand (trust me, you will suffer if you do it that way).
• you need to have at least 500GB of available space and good internet to download. The first time my Docker runs, it automatically downloads all ComfyUI extensions and all models for everything. You can edit the models.conf file beforehand and remove models that you know you will never use. But just this workflow alone requires at least about 30GB of models, there is no way around it.
• this workflow is particularly HEAVY because it has several features (parts of which we can disable, to customize for our workflow). But for 1 character, the entire flow can take 30 MINUTES. Even on an RTX 4090 like mine.

If you have done everything right up to this point, you have ComfyUI running with all dependencies already downloaded and you bought the Workflow. When you load it you will have to fix some things manually.

The first thing is that you will see an error in several Nodes in red, all of them Ultimate SD Upscale. Just right-click on each one and choose “Fix Recreate” and it will fix automatically:

ComfyUI requires that each output slot of a Node connects to the input slot of another compatible Node, but often the workflows get so big that there are little wires connecting things scattered all over the place and it gets really hard to know what connects to what. It literally becomes spaghetti.

But there is an extension (which I have already pre-installed) called cg-use-everywhere. Every extension is a little GitHub project, cloned into the custom_nodes subdirectory. It lets you connect a Node’s output to this Anything Elsewhere node, say the “clip” output of the “Load Clip” Node. Now, every other Node that has “clip” as an input field and does not have anything explicitly connected to it will use this Anything that was created. It works super well.

Only in this Mick workflow, who knows why, all the Anywheres are broken. But it is easy to fix. Just find these Nodes, delete them, create new ones and reconnect. There are several right at the beginning, write down who connects to each one, delete them and create a new one to reconnect:

There are also some in the Upscale group 02:

Small Nodes, with only the title showing, are COLLAPSED. Just right-click and choose EXPAND to see the details. It is another way to organize the mess.

Lastly, you have to recheck the correct name of the models in all the “Load” something Nodes right at the beginning of the workflow (luckily he organized these nodes all together):

In his workflow, the Node comes looking for “FLUX/flux1-dev-fp8.safetensors” but then you go to the list (which pulls from what is actually in the “models/checkpoints” directory on your machine) and choose the one with the same name. This happens often because different people organize different subdirectories for the models. But after configuring, just save and you are done. You have to do this for the checkpoint, controlnet, ic-light, etc.

ComfyUI errors are not friendly. They are Python stacktraces, which are not friendly because they show which line of code blew up, but they do not show what was in the variables passed to the function that blew up. So there is no way to know where it went wrong. But normally on the web interface, the Node turns red, then you check it and usually it is because you forgot to download the model or pick the right one from the list.

The last thing in this workflow is to download the OpenPose image from his Patreon page, which I linked at the beginning of the post, and load it:

OpenPose is a standard file format that describes poses, an open format that can be edited in several tools. And there are several pre-trained models to extract poses in this format from images, so we do not even need to edit it by hand. We can download several pre-made open poses from this site:

I am going to repeat this several times, but watch Mick’s video tutorial where he explains how to generate characters from scratch or use your own photos with your own poses to continue the flow from there. His workflow, after running, looks like this (with zoom way down below):

In the first part, there is the manual configuration and checking of the models. You only have to fix it once and save. After that it will always work the same way. In this first menu you can also turn on and off some of the Node groups, for example, to skip the first stage of generating a character from scratch and be able to use your own photos:

If you choose to generate a character from scratch, there is a green prompt box to type how you want your character to be, in what style. Text Prompts are decoded by CLIP (Contrastive Language Image Pre-Training) models, which are specific to certain UNET categories (like SDXL for Stable Diffusion or the now best and most popular FLUX). Part of the learning is starting to figure out which sub-models (clips, clip_vision, vaes, loras, etc.) are compatible with which unets (unet, checkpoints, diffusion_models). You cannot mix just anything. Roughly speaking, it is a bit like, if you choose to do a text in German, all plugins have to be in German.

I will not show the complete workflow because it does not even fit, but you can see the prompts in green, in the middle where we choose which pose we want (it is on his Patreon), and on top the image generated by the flux1-dev-fp8 model, one of the best and most popular for having excellent quality and not being so heavy (fp8 instead of fp32, for example).

This is one of the advantages of using tools outside the Dalle or Midjourney of the world: we can control all aspects of image generation not only with text prompts, but also with image prompts, like OpenPose.

Another detail. I explained in my previous post about KSampler or Samplers in general, that they are the engine that does all the work of joining all the models, images, prompts, controlnet, vaes and produces the final image. KSampler parameters like steps, cfg, denoise, scheduler and more DEPEND on the chosen model. There are well-documented models, there are poorly documented models.

For example, say you read on a Reddit thread about this other more optimized model called “TurboVisionXL”, which is available on the Civit.ai site:

It really is a good model, but now what? What do I configure in the KSampler for it? Luckily if you scroll the page (always read the original documentation!!) you will find these recommendations:

See? He explains what to configure and what the known limitations are. Further down there will be comments from other people who can give more tips or share errors that you have already seen. It is a huge community, you need to learn how to participate in it.

But back from the tangent, in Mick’s second group of nodes, he takes the first image in “Pose T” and runs it through a “Multi View” process using Diffuser MV which does this automatically. It is the same pose, but as if the camera went 360 degrees around it, capturing different angles and, most importantly, generating new images coherent with the original (see how the model is even used to invent the character’s back).

In this step he also extracts only the face from the first image to do other treatments in parallel. Face recognition and extraction is super common. Every security system uses it, Google Photos uses it to organize dynamic albums. And it is worth learning to use in your own products. It is a reasonably light and fast process these days and there are several pre-made algorithms and models to use. In this case I do not think it even uses A.I. as such, it is just an algorithm.

Another detail: image generation models normally work in LOW RESOLUTION. Usually below 1024x1024. Do not expect to generate 4K images right off the bat. But that is no problem, because the solution is to first generate a low resolution image and then run it through another A.I. model specialized in UPSCALING like the RealESRGAN that I already talked about in my other post about upscaling old anime

Different from what Photoshop or any image editor does, if you try to enlarge an image, it just takes a pixel and tries to duplicate, quadruplicate the surrounding pixels. But that usually only makes the image bigger but blurred, because it has no way of knowing the details, and it will not even try to “invent” details.

To “invent” details, you need a pre-trained model that learned what to draw. It is also how NVIDIA’s DLSS 3 works for games, where the game renders in low resolution, like 720p, and DLSS, in real time, redraws the new screen in 4K. It is not perfect, but playing at 60fps, you will rarely notice the difference. You will only notice that it got more CRISP.

That is also why, if you use your own photo, or photos of people you know, you will see that the new photo after upscaling looks slightly different, it is no longer the same person, because the model REDRAWS details that did not exist. It is a process of interpretation and not of discovering “hidden” details (that does not exist, only in CSI fiction).

Mick’s workflow goes one step further and has a second upscaling pass, focusing on fixing only the face with a Face Detailer Node, made specifically to fix and improve faces. And here we enter plastic surgery and harmonization territory for real lol the final face will be different from the original. But for NPCs and secondary characters it does not matter, nobody will know the difference anyway, as long as it is not deformed, cross-eyed or something else that jumps out at you.

There are Nodes specialized in expressions, like the Expression Editor (PHM) which starts from a single face photo and manages to generate new photos in the most varied emotions - which we can configure exactly, numerically, however we want. I will not show it, but note that the generated images are in low quality, but the next step in the workflow is precisely to upscale them too.

This step is interesting. First the top part, with the various scenarios. They were all generated via the green prompts below them, prompts that Mick left but that we can edit however we want OR swap the generation node and put a Load Image node to load our own background images, from real photos.

The objective of this step is to generate the faces under different light configurations. Many people think that “light” in games is just “turn on the red light” and everyone will automatically turn red. In new games, with RAY TRACING yes, that is why it is SUPER HEAVY. But in games made for old machines, indie games, you cannot demand this from everyone.

So “tricks” are used to fake things like lights and shadows. It is the BAKING process where we literally “stamp the light onto the texture” of objects and characters. With global illumination on or off, they will always be “painted” with the colored light of the scene. To switch scenes, new textures are loaded with other “baked” lights. For programmers, think of baking as Caching. Why recalculate everything dynamically every time, if the scene’s light never changes? Save that step and leave it “cached”.

Going back, this step uses the IC-Light technology which is another open source project worth learning, even if you are not interested in A.I. It can manipulate the light of any image or photo. You, as a photographer, can change the light if you have changed your mind, in post-production. Adobe or BlackMagic tools, I think they already include similar technology, but it is good to know that there is open source and you can integrate it into your product.

After generating the faces with different lights, obviously there is another upscaling step - which I am skipping showing - and it goes to the final group of nodes that joins everything into a “sheet”, a “character sheet”. The game developer can use it directly in a 2D game (think of a visual novel game), or it can serve as a basis for UV Mapping textures of a simple 3D model with Armature to animate and you already have some emotional faces. Good for placing in the background scenery, pedestrians walking on the street, shopping, sitting in the square or something like that.

In addition, Mick’s tutorial describes how it is possible to create LORAS (Low-Rank Adaptation). If you want to be able to generate characters with a specific style or even specific faces (without “artistic” intervention from the A.I.), you choose a model like FLUX or SDXL and “on top” of it you add a second compatible sub-model with information about that person or character.

For this you take several clean and well-lit photos of the person, from many different angles. You do not even need much, just a dozen is enough. For each image you create prompts that describe it well, and that is it. There are tools to generate these Loras locally or sites that do this (watch the video).

Speaking of prompts, in several workflows there are steps to describe detailed prompts of dynamically generated images. Instead of requiring the user (you) to do this manually, you can use a decoding model, which decodes images into text prompts. The most used model for this is Microsoft’s FLORENCE 2.

That text box at the bottom is read-only, you do not need to type it manually. The Florence 2 Run Node will read the image and generate this text. See how it described the first image of the woman in the “T” pose. Doing tests, for your case, you determine if just this is enough; if it is not, just put a second prompt Node and concatenate the two before sending them to other nodes.

Anyway, there is A LOT to explore just in this workflow. Mick did an excellent job both on the workflow and on the video to explain it, and it is worth paying the 10 dollars to contribute. There are dozens of workflows like this available on the Web, just search. And my setup is getting more and more complex. I plan to keep adjusting it for each new interesting workflow I run into. There are already several pre-tested ones in the “workflows” folder of my project for you to play with.