One category was missing that I had never tested properly: the big open source models. The ones that fit on my RTX 5090 with 32 GB or on my Strix Halo with 61 GB usable by the GPU are too weak for this. They work for simple agent tasks, and no small model handles real programming tasks, let alone a complex software project.

Qwen 3.8 27B took 706 minutes to finish v4 on the Strix and scored 80; the local GLM-4.7-Flash scored 24. So don’t even try.

But what if we push harder? What can mid-sized models, 100 to 400 billion parameters, do running on a single rented B300? I could rent four of them and run the giant 750-billion or one-trillion models, and I’m not sure the cost-benefit is worth it. The data below answers.

TL;DR: 14 open models, one 288 GB B300 on Runpod at about 9 dollars an hour, the same v4 sabotage protocol. None of them serves fast the way it ships: GLM 5.3 Flash decoded at 10 tokens per second with the default configuration and at 141 after tuning, 14 times more from flags alone. Two open models, GLM 5.3 Flash and Qwen3.8 Flash-Next, scored 92.5, Claude Sonnet 5 level, and Swift 1.5 27B scored 88, Opus 4.6 level. None reached the frontier’s 100. The 397-billion flagship scored 68.5 and let an admin backdoor through, DeepSeek V4 Flash scored 86 in the cloud and 44.5 served by me, and Nemotron cheated on a regression test. The cloud API of the same model costs a dollar per run; the pod costs 218 dollars a day. Jump to B300 and Runpod, server tuning, the ranking, the interesting cases, the frontier comparison, the math or the conclusion.

What a B300 is, and what Runpod is

The B300 is the datacenter GPU of NVIDIA’s Blackwell Ultra generation, announced in March 2025. Each card has 288 GB of HBM3e memory, nine times my 5090, and about 15 petaflops of dense compute in NVFP4, the 4-bit format Blackwell does in hardware. It’s the card that goes inside the GB300 NVL72 rack, 72 of them interconnected, which AI companies buy by the thousands.

On its own, a B300 costs more than an apartment. What makes it accessible is renting by the hour.

Runpod is a cloud GPU provider that rents cards in “pods”, Linux containers with one or more GPUs, billed per second. You pick the card, an image template, get in over SSH, and pay while the pod is on, idle included. The B300 cost about 9 dollars an hour on their secure cloud the week of the test; an H100 goes for 3 to 4. The container disk is ephemeral: terminate the pod and the 250 GB of weights you downloaded are gone, and you download again.

The pod I used: one B300 with 275 GiB according to nvidia-smi (the same nominal 288 GB, in binary units), 344 vCPUs, 4 TB of host RAM, 700 GB of disk. The inference server was vLLM; the harness was the same OpenCode as every other model in the ranking, connected to the pod through an SSH tunnel. The model runs there; the code, the tests and the results stay on my machine.

Fitting in memory is the first filter. The B300 fits the 100 to 400-billion-parameter models quantized to NVFP4, plus the 27 to 35-billion ones in BF16. Left out are the 753-billion GLM 5.3 flagship, the 750-billion DeepSeek V4.1 Flash, the 1.6-trillion DeepSeek V4 Pro and the 2.8-trillion Kimi K3. For those it would take four B300s or more, about 36 dollars an hour.

The methodology

The protocol is the usual v4: the model builds a Rails app across seven accumulating sprints, and between sprints an independent reviewer, isolated from the model, injects 14 sabotages, from real CVEs to silent logic and test regressions, into the code the model itself wrote. The 0-to-100 score measures vigilance: how much the model catches and fixes without being told (full weight), how much it fixes after an explicit “do a security audit” in the last sprint (40% of the weight), and what stays unresolved (zero). Tiers are A from 83 up, B from 73 to 82, C from 51 to 72, D up to 50.

Two differences this round. The first is the reviewer: the five models that finished before the block were reviewed end to end by Codex, as in the previous rounds.

Midway through the batch OpenAI’s security service started refusing the injection and grading operation with a cybersecurity risk flag, even though it’s my code, on my machine, in a benchmark with 46 models already scored under the same protocol. I switched the reviewer to Claude Opus 4.8 for the remaining models, with the same injection plan, same catalog, same yardstick, and the provenance of every boundary is recorded per sprint. The expected impact on the score is none, because both execute the same written recipe.

The second is admission by speed, which is the subject of the next section. A model that decodes below 100 tokens per second doesn’t even get on the track.

Also: an open model is entitled to up to two short nudges per run for silly defects (a missing import, a wrong test entry point), recorded as “assisted”. Infrastructure failure, driver, loader, parser, is never charged to the model. And the pod ran one model at a time, with no access to any other model’s workspace or results.

None of them serves fast the way it ships

The first lesson cost a day. I downloaded GLM 5.3 Flash, 320 billion parameters with 18 active, quantized to NVFP4, 198 GB. I brought it up on vLLM with the configuration that “works”: it serves, passes the tool-call tests, answers. And it decoded at 10 tokens per second, with the GPU at 10% utilization, 209 watts out of the 1,100 the pod allows.

That’s unusable for programming. A coding agent generates tens of thousands of tokens per sprint; at 10 per second, a 20-minute sprint becomes more than four hours.

The same checkpoint, same 128K-token context, same FP8 KV cache, same test prompt, changing only the compile and mode flags: 141 tokens per second. Fourteen times faster, zero change in precision or weights.

The main culprit was --enforce-eager, which disables vLLM’s torch.compile and CUDA graph capture, and which every tutorial tells you to enable “to start faster”. It starts faster and serves 14 times slower. The measurement changed that flag together with the interactivity mode, so I didn’t isolate each one’s contribution.

Hence the admission rule: below 100 tokens per second I consider unusable, and no model enters the benchmark without clearing that on the median of two warm 256-token probes, after a cold one. To keep it from becoming a bottomless pit, tuning is bounded: at most three configurations and 45 minutes of wall-clock per model, including load and compilation. And tuning can never trade quality for speed: no shortening the context, lowering reasoning effort or quantizing further to hit the number.

The tweaks that paid off for everyone:

• Install a CUDA that compiles for the card. Runpod’s default image ships CUDA 12.8, whose nvcc cannot compile for compute_103a, the B300’s compute capability. Without 12.9, the NVFP4 kernels never compile.
• Never use --enforce-eager. Letting vLLM compile and capture CUDA graphs is the single biggest win.
• --performance-mode interactivity: optimize for single-stream latency, which is what a serial agent needs, instead of aggregate throughput.
• FP8 KV cache on the big MoEs, prefix caching and chunked prefill on all of them, confirmed by vLLM’s Prometheus counters instead of assumed: on an 11K-token prompt, the server reused exactly 8,576 prefix tokens on every repeat.
• The right tool and reasoning parser for each family. The wrong parser makes tool calls vanish silently or leaks the reasoning into the answer.

And the specific tweaks I found most interesting:

• NVFP4 only runs on vLLM. The 198 to 251 GB checkpoints in Blackwell’s native FP4 are the only way to fit on the card, and llama.cpp and Ollama simply don’t load those formats. Whoever lives on GGUF has no access to this range.
• Native speculative decoding. Qwen 3.8 27B went from 92 to 195 tokens per second with the MTP head the checkpoint itself ships, predicting three tokens per step, plus --language-model-only to drop the vision tower (measured on different pods, so no controlled ablation). Muse Glimmer reached 295 with the lightweight DFlash drafter it ships alongside, 15 speculated tokens per step. I only enabled it where an official drafter exists and where tool calling stayed correct; for Ornith a drafter exists as a separate release, with unknown acceptance rate, so it stayed off.
• The CUDA 12 loader. flashinfer JIT-compiles an NVFP4 MoE kernel that needs libcudart.so.12, and after installing CUDA 13 for Nemotron the library resolution broke. The symptom looked like OOM; the cause was an ld.so.conf. A repair script with ldd evidence before and after fixed it, and the 397B went on to serve at 144 tokens per second.
• One virtual environment per runtime. Nemotron 3.5 Super VL required its own vLLM 0.32 in a separate venv, because that’s what the official card pins, and that build wants CUDA 13. Every model family is a bespoke case: parser, quantization, draft head, runtime version and loader repair, all different.
• A parser bug that dropped the last call. vLLM’s Qwen tool parser discarded the last parameter of a call when the parameter closing tag was missing but the function one was present. A narrow repair, with 24 test cases, took that serving noise out of the model’s account.

The exact arguments for each model, the probe logs and the ledger of every tuning round are in the serving optimizations document and the B300 runbook in the repository.

Speed is almost free, once you let vLLM compile. The real work is finding out, model by model, which combination of toolkit, parser, quantization and draft head that checkpoint demands. I gave each one 45 minutes and needed them.

The ranking

v4 vigilance score, tier, memory footprint, decode speed on the admission probe (warm, single stream) and the cost of the seven sprints’ inference time only, at 9.09 dollars an hour. The cost excludes download, load, compilation, tuning and idle time; the real math comes further down.

Three have no score: Cohere’s Command A Plus (218 billion, 25 active, W4A4) got stuck in a loop rewriting the same model file 23 times, needed three nudges and still left the tests unable to run before the reviewer was blocked; Nex N2.5 Mini ran at 337 tokens per second and delivered invalid Ruby, contradictory migrations and broken authentication in two attempts, a valid bad result from a server that works; and Qwen3.5 122B was a conditional fallback in case the 397B didn’t finish, and the 397B finished.

Minutes are sprint time only, and Ornith’s include a sprint interrupted by infrastructure. Flash-Next, Swift, ThinkingCap, Muse and Qwen 27B took one nudge each; GLM 4.7, two; the others, none. The probe speed is an admission number: the throughput of a real coding session includes prefill, tools, tests and the agent’s choices. And note that the top two tie on score with opposite profiles: GLM spent 145 minutes and 22 dollars to reach the same 92.5 that Qwen3.8 Flash-Next did in 80 minutes and 12 dollars, with half the memory.

The interesting cases

Qwen3.8 Flash-Next: the best of the open models, by cost

Flash-Next is, in Qwen’s words, a preview of the Qwen 4 architecture: 125 billion language-model parameters with 6 billion active, plus a 51-billion n-gram embedding table that Qwen says can be offloaded to host memory, plus a 4-billion MTP head. Quantized to NVFP4 by NVIDIA, it took about 100 GB on the card (the checkpoint is 133 GB on disk). It was the only model in the second batch to resolve all 14 items: it caught the tenant leak unprompted, caught the compound test-integrity item, and wrote per-user isolation specs on its own, so the planted API IDOR had no way to stay green. At 195 tokens per second and 12 dollars of inference, it’s the best cost-benefit in this table.

GLM 5.3 Flash: the same 92.5, three times the cost

GLM scored the same with a much bigger architecture: 320 billion parameters, 18 active, 198 GB even in FP4. It resolved all 14 items, zero nudges, reviewed end to end by Codex.

But it took 145 minutes and consumed twice the memory for the same result. On Z.ai’s cloud, the same model scored 84.25 on a flat plan. With single-run noise, which in my history reaches nine points, the safest reading is that both sit in the same band.

Swift 1.5 and ThinkingCap: two 27B fine-tunes in Tier A

The two surprises. Swift 1.5, from UkisAI, and ThinkingCap 3.8, from BottleCap AI, are Qwen 3.8 27B fine-tunes trained to think less: Swift advertises up to 58% fewer reasoning tokens (on the GPQA median; on Terminal-Bench, 16%), ThinkingCap about 37% on average. Both fit in 56 GB in BF16, run at about 200 tokens per second with MTP, and scored 88 and 83.

Swift landed above Anthropic’s Opus 4.6. The base Qwen 3.8 27B, same architecture, scored 80.5 in 192 minutes; Swift scored 88 in 64. Swift caught sabotage earlier than the base; whether that was from thinking less, one run of each can’t prove.

A provenance caveat: the reviewer recorded that Swift read local commit messages that disclosed regressions, which weakens a literal reading of “entirely unaided discovery”. And it left one item unresolved, an inactive admin still accepted with a vacuous guarding test. Even so, of every model in this table, it’s the one I’d put on a card of my own.

Qwen3.5 397B: the flagship that let the backdoor through

The biggest model in the batch, 397 billion parameters with 17 active, 251 GB of NVFP4 on a 275 GB card, served at 140 tokens per second after the loader repair and scored 68.5, Tier C.

The case that defines the score: sabotage number 6 is a line admin? => admin || email.end_with?("@internal.example"), a privilege escalation by email domain. The 397B let it through in the normal sprints, let it through in sprint 7 with the explicit security-audit request, and wrote a passing test codifying the backdoor as expected behavior, in the same commit where it announced “Brakeman: 0 warnings”. Parameters did not buy vigilance.

Nemotron 3.5 Super VL: cheated on the test

NVIDIA’s Nemotron 3.5 Super VL, 120 billion with 12 active, released two days before I started testing and sold by NVIDIA as the fastest open agentic model in its class, with “leading accuracy” on multimodal tasks, scored 63. Sabotage number 2 removes a navigation link and the regression test looks for have_link('Sign in'). Nemotron changed the test to have_button('Sign in'), which passes on the login form’s submit button without restoring the link, and kept the cheat even after the reveal sprint.

It also tends to catch items a sprint late. And it was the one that required its own runtime: vLLM 0.32 with CUDA 13 in a separate environment.

DeepSeek V4 Flash: 86 in the cloud, 44.5 in my hands

The most important case for anyone considering self-hosting. DeepSeek V4 Flash, 284 billion with 13 active, scored 86, Tier A, for 97 cents through the API in September, and the 0731 version, exactly the one I downloaded, scored 85 for 1.94. The same public weights in mixed FP4 and FP8, served by me on the B300, scored 44.5, Tier D: it caught 4 items unprompted, 5 more at the reveal, and left 5 unresolved. It was the fastest to finish, 37 minutes, and the least vigilant.

The likely cause is divergence between the public checkpoint and what DeepSeek serves, or between their serving stack and mine: parser, chat template, reasoning mode, quantization. I followed the official card. What this proves is that the model’s name on Hugging Face doesn’t guarantee the hosted API’s quality, and that whoever is going to host needs to validate before trusting.

GLM 4.7 Flash: same as local, meaning bad

The 30-billion GLM 4.7 Flash in BF16, at 217 tokens per second, scored 33.5. At home, in GGUF Q5, it had scored 24. It needed two nudges, left nine items unresolved and delivered a migration with a syntax error. The nine-and-a-half-point difference fits within single-run noise; what remains is that the model is weak either way.

What showed up in several

Ornith, Muse Glimmer and the 397B shared a hygiene defect: they ran the sprint to the end with exit zero and forgot to commit their own work, leaving it for the reviewer to checkpoint. It didn’t go into the score, and on a real team it would be the first complaint. And the 81-to-83 cluster, ThinkingCap, Ornith and Muse, has a consistent profile: it catches SQL injection, tenant leaks, XSS and exposed secrets with confidence, and lets the quiet items through, broken access control, test integrity, missing index, wrong aggregate.

How far from the frontier

The three best open models against the best commercial ones from the same ranking, with the cost and time of each run:

The reading: the best of the open source that fits on a B300 and that I served sits one step below the commercial ceiling and level with the middle of Tier A. Differences of one or two points are single-run noise.

GLM 5.3 Flash and Qwen3.8 Flash-Next tie with Sonnet 5 and pass Gemini 3.8 Flash and Opus 4.6. None reaches the 100 that seven commercial models scored, nor Grok 4.6’s 98.5. Seven and a half points separate the best open model from the ceiling, and in v4 every point is one less sabotage caught unprompted.

The difference that matters is another: predictability. Among the top commercial models in the table, the worst is 87.5, and the full commercial field also has a bottom, Mistral Large 3 scored 39. The open field has the same 92.5 at the top and two Tier D at the bottom, with DeepSeek scoring 85 in the cloud and 44.5 in my hands. Choosing a top commercial model is choosing a number the API reproduces; choosing an open one is betting that the checkpoint, the quantization and your serving stack reproduce the number on the card, and sometimes they do.

The math

If quality still falls short of the frontier, is the cost at least proportional? It depends on where you run the open model.

On the vendor’s own cloud API, yes, and by a wide margin. A full v4 run cost 97 cents on DeepSeek V4 Flash, 1.17 dollars on Qwen 3.8 Flash, 1.03 on MiMo V2.5 Pro, all Tier A in the cloud. Against 9.52 to 30.55 dollars for the 100-point models, that’s ten to thirty times cheaper for a score 8 to 14 points lower. The list prices explain it: GLM 5.3 Flash costs 15 cents per million input tokens and 50 for output; DeepSeek Flash, 15 to 30 cents input and 60 cents to 1.20 output; Opus 5.5, 4 and 20 dollars; GPT-6 Astra, 10 and 50.

On your own rented B300, no. The pod costs 9 dollars an hour on or idle, 218 a day, about 6,500 a month. The model time of this whole batch, 14 models across download, load and sprints, added up to 23 hours, about 212 dollars at pod price; the sprints alone were 17 hours and 157 dollars.

But the pod also charges for the time when nothing runs. I put 400 dollars of credit on Runpod and 127 were left: the whole batch cost 273 dollars, about 30 pod-hours, including storage, four pods in sequence and seven idle hours one night when the orchestration died and nothing warned me. Almost 60 dollars more than the model time, just waiting. The repository’s break-even math: the pod only gets cheaper than the API from about 6,500 runs a month, 217 a day, sustained. A solo developer or a small team doesn’t come close.

And model against model, side by side:

The cloud API of an open model is ten to thirty times cheaper than the frontier for one tier less. Hosting the same model on a rented B300 is more expensive than the frontier, and sometimes worse than its own API.

Does a team fit on the same card?

A question I asked while looking at the GPU at 10% utilization: do these models use all of the pod’s compute? Could a whole team go on the same B300?

In principle, yes, and that’s what vLLM exists for. Continuous batching schedules several requests in the same decode step, and PagedAttention stores each session’s KV cache in non-contiguous blocks, with under 4% waste, precisely to fit more sessions in the same memory. A 27-billion model in BF16 takes 56 GB and leaves about 200 GB of KV cache: that’s dozens of agent sessions with 128K-token context at the same time. The 397B, with 251 GB of weights, leaves 24, and then few fit.

What doesn’t scale is speed per person. My test ran in --performance-mode interactivity, which optimizes single-stream latency, and the 200 tokens per second are that stream’s. With ten agents decoding at the same time, the card’s aggregate throughput rises and each one’s falls, and the speculative decoding that gave me Muse Glimmer’s 295 yields less as concurrency grows: a DFlash drafter published for Muse by Z Lab reports 3 to 4.6 times gain with one session and 1.2 to 1.7 with 32.

I didn’t measure concurrency in this batch, so what I can say is the outline: a team of five to ten people on a 27-to-35-billion model on one B300 is practical, and each one probably still stays above 100 tokens per second. The same team on the 397B is unworkable on the card, and on the 198 GB GLM 5.3 Flash it gets tight.

It’s in that scenario, and only there, that the pod math starts to close. Ten developers running an agent all day on a 27B model would spend, on the API, at a dollar per two-hour run, about 40 dollars a day; the pod costs 218. It still loses. What changes the math is having a hundred people, or being required to keep the code from leaving your network.

Is owning your own infra worth it?

For serious software development, with this batch’s data, the answer comes in three parts.

The top open models are good enough. GLM 5.3 Flash, Qwen3.8 Flash-Next and Swift 1.5 27B are Tier A in my test, at Sonnet 5 level and above Opus 4.6. That was unthinkable in April, when only Qwen 3.6 35B came close to Opus, and still one fix away. For anyone who accepts one tier below the ceiling, they serve for real work, including security review, which is what v4 measures.

Renting a B300 to run them is the wrong way to use them. The cloud API of the same model costs a dollar per run and needs zero configuration. The pod costs 218 a day, demands a day of tuning per model family, and in DeepSeek’s case delivered a worse model than its own API. It’s only worth it with a data-residency obligation, with a checkpoint no cloud serves, with sustained load of hundreds of runs a day, or with a need for pinned weights for reproducibility. Four B300s to run the 750-billion-to-one-trillion models quadruple the bill with no evidence the score reaches 100: the 397B, the biggest model I tested, scored 68.5.

And if you’re going to host, host small. Swift 1.5 27B and ThinkingCap at 56 GB, with MTP, at 200 tokens per second, are Tier A on hardware you can buy, a professional 80 or 96 GB card instead of a datacenter one. That’s where open source makes sense for a company: a whole team on one card, code inside the network, an Opus 4.6 score. Just check the license first: Swift and ThinkingCap ship with their own licenses, and above a certain revenue both require a commercial license. The open giants remain an API thing.

My rule after all this: top commercial model for tasks that can’t fail, open-model API for volume, and your own card only when the law or the client demands it. The repository with every serving argument, every probe log, every reviewer transcript and the Codex-or-Claude provenance of every boundary is public, as always.

But nobody needs a GPT-class model for everything. A frontier model is a generalist by definition: it tries to know everything about anything, from tax law to song lyrics, and the returns on that diminish.

And most of them today are mixtures of experts: roughly speaking, several smaller models clustered into one giant chunk of bits, with a router that wakes only a few of them for each token. OpenAI’s gpt-oss-120b has 117 billion parameters in total and uses 5.1 billion per token. DeepSeek-V3 has 671 billion and activates 37. The big model is already, on the inside, a collection of small models.

I could have grabbed an open model, gpt-oss, Qwen, Phi, and adapted it. The techniques exist and are well documented:

• Full fine-tuning: keep training all the weights of a ready model on your data. It’s classic transfer learning, formalized for text by ULMFiT in 2018.
• LoRA and QLoRA: freeze the original weights and train only small low-rank matrices beside them. LoRA cuts trainable parameters by up to ten thousand times; QLoRA quantizes the base model to 4 bits and fine-tunes a 65-billion model on a single 48 GB GPU.
• Distillation: train a smaller model to imitate a larger one’s outputs. The idea is from Hinton, 2015, and the most famous recent example is DeepSeek-R1 distilled into versions from 1.5 to 70 billion on top of Qwen and Llama.
• Continued pretraining: one more training phase on text from your domain before any instruction tuning, the “Don’t Stop Pretraining” of 2020.
• SFT, RLHF and DPO: teach the assistant format with demonstrations, then align with human preference via reinforcement learning (InstructGPT) or with a direct classification loss (DPO).

With any of those I’d have a useful model in hours. But the goal was different. I wanted to organize, hands-on, the basic techniques for building a model smaller than GPT-2, from scratch, in a few days, and see how far it would go. If possible, train a tiny model starting from random weights and add enough functionality for it to almost work inside an agentic harness like OpenCode.

Before going on, who I am in this story: I’m a curious software engineer and an amateur at this. As an LLM researcher I’m a zero. I knew I wouldn’t have a working LLM in such a short time, and that’s part of the exercise.

But it’s not unachievable. If you’re a computer science student, study hard and give it enough years, you can try it yourself. And if you’re not a scientist, understand this: nobody needs to be Einstein to do this. The foundations are all available, published, for you to study and try. What’s mandatory is elbow grease.

Spoiler: we got far into the “almost” and stopped there. The repository is public, with every step and every failure documented. This text is the summary.

TL;DR: I started from Karpathy’s microgpt, 200 lines of pure Python that train a GPT in one minute on a MacBook, and expanded it into a PyTorch lab with its own tokenizer, pretraining, supervised fine-tuning, an OpenAI-compatible server and OpenCode integration. In under two weeks, on one RTX 5090, I trained a 337-million-parameter model on 10 billion tokens: 137 hours of training, at most around R$ 80 of electricity. It gets 58 out of 100 simple Python functions right on a held-out set, and scores zero on anything that requires reading, editing and testing a repository. The 360-million SmolLM2, the same size, saw 4 trillion tokens, 400 times more. Jump to Karpathy’s exercise, the steps, the dimensions, the stumbles, the result or the conclusion.

Karpathy’s exercise

In February this year Andrej Karpathy published microgpt: a single file of 200 lines of Python, with no dependencies, that trains and runs a GPT. He calls it “the full algorithmic content of what is needed. Everything else is just efficiency”. It’s the culmination of a lineage of his: micrograd (scalar autograd), makemore (generating names) and nanoGPT (compact GPT-2 in PyTorch), joined by nanochat, “the best ChatGPT that $100 can buy”, which at launch trained a GPT-2-capability model in almost four hours on a node of eight H100s, and today does it in about two.

microgpt is deliberately minimal: the dataset is 32 thousand first names, one per line, and the tokenizer is one character per token.

The network has one layer, 16 embedding dimensions, 4 attention heads, a 16-token window because the longest name has 15 letters. It trains for a thousand steps, one name per step, with Adam written by hand, and at the end generates made-up names, all in one minute on a MacBook.

What it teaches, by having everything exposed in a single file:

• autograd, the Value class that holds a number, the gradient and the chain rule to propagate back;
• what a token and an embedding are;
• the attention block, with query, key and value, and the causal mask that stops the model from peeking at the future;
• the cross-entropy loss against the next token;
• the optimizer;
• sampling with temperature.

It’s a GPT-2 with small differences the file itself notes: RMSNorm instead of LayerNorm, no biases, ReLU instead of GeLU. After reading that, what’s left between you and a real model is, in his words, efficiency. I went to see how much efficiency.

What we expanded

The microgpt lab is an independent reimplementation in PyTorch, inspired by the gist, split by concept: about 3,300 lines of Python in 31 modules, plus scripts and 85 CPU tests. A single model implementation serves pretraining, assistant tuning and the server. Every weight starts random; the project downloads no pretrained weights at all, on purpose, because the rule of the exercise was from scratch.

On top of Karpathy’s block came the components a 2026 model uses: RoPE for position, RMSNorm, SwiGLU in the dense layer, embeddings tied to the output, Flash Attention through PyTorch, a key-value cache with prefix reuse. And around the model, what it needs to talk to an agent: a conversation protocol with roles, an HTTP server compatible with OpenAI’s chat API, tool calls validated by schema and optionally by grammar, separation of reasoning content, and the OpenCode configuration pointing at it.

The project went through four scales:

The table hides the first big lesson, which I tell in the stumbles: in the 239-million model, 154 million parameters were just the vocabulary table. 85 million were left for the actual transformer.

The steps of building an LLM

Regardless of size, the path has the same stations. What changes is the trailing zero at each one.

1. Data

A model is what it eats. We downloaded, with pinned revisions and hashes, selections of public corpora: CodeParrot (deduplicated, filtered Python), OpenCoder (instruction and code), Cosmopedia (synthetic educational text), SmolTalk (conversation) and a multilingual slice. The 14 download manifests add up to about 20 GB raw, all stored on the NAS.

Then comes curation: exact deduplication, splitting train and validation by repository or conversation group (never by line, or the same project leaks to both sides), and the mixture. The final foundation corpus has 3.5 billion stored tokens, 60% Python code, 16% instruction, 12% educational text, 8% conversation, 4% multilingual. Every proportion is a design decision, and none of them has a published right answer.

2. Tokenizer

The tokenizer is the dictionary between text and integers. The byte lesson uses 256 tokens plus one for end of document. The student used o200k, OpenAI’s public vocabulary with 200K entries, which is what most tutorials do. The pivot trained its own BPE with 32,768 entries, on 92 MB of training-only text, with full UTF-8 coverage.

The choice looks like a detail and decides where the parameters end up. Each vocabulary entry costs one row in the embedding table at the model’s width. With 200K entries and width 768, that’s 154 million parameters spent on the table. With 32K entries and width 1,024, it’s 33 million, and the rest goes to the layers that think.

3. Pretraining, or foundation

This is Karpathy’s loop at scale. The model receives 2,048-token windows drawn from the training stream and learns to predict the next token. Each update processes 32,768 tokens: two sequences, accumulated eight times. AdamW, learning rate with warmup and cosine decay, gradient clipping, BF16 on the GPU and FP32 for the weights.

The pivot ran about 305 thousand updates, 10 billion tokens of exposure. Since the corpus has 3.5 billion, each document was seen, in the mixture’s expectation, almost three times: the code about twice, and instruction, educational text and conversation about nine or ten. Every 500 updates, a validation and a checkpoint. At the 2, 5 and 10 billion milestones, a frozen checkpoint with a hash, to measure the curve.

4. Supervised fine-tuning

Pretraining produces a model that completes text. For it to answer a request comes SFT, supervised fine-tuning: question and answer examples in the exact format the server uses, with the loss computed only over the assistant’s tokens. What the user and the tool said goes in as context and is masked out of the loss.

The examples came from NVIDIA’s OpenCodeInstruct, and every answer was executed in a restricted Docker before going in: almost 99 thousand candidates, 97 thousand survived the test with three different hash seeds. 92 thousand training episodes remained. The selected checkpoint saw 64 thousand draws from those episodes, about 10 million answer tokens.

5. Protocol, tools and server

An agent like OpenCode talks to the model over HTTP: it sends the history with roles and the tool definitions, gets back text or a tool call in JSON, executes it, returns the result, and repeats. The lab implements that whole cycle: the same protocol file serializes the training examples and the serving prompts, so training format and usage format never diverge.

The server implements /v1/models and /v1/chat/completions, validates the tool call against the schema, can constrain generation with a grammar so the JSON comes out well formed, separates reasoning content from the answer, counts cached tokens and rejects context overflow instead of truncating silently. All of that is plumbing, zero intelligence, and without it the model doesn’t even get a seat at the table.

6. Evaluation

The step that changed the project the most. A frozen set of 100 Python development tasks, 20 families with five variants of name and wording, plus 100 held-out tasks that only the final checkpoint saw. Every answer runs in a Docker with no network, no root, with CPU and memory limits, against fixed assertions. On top of that: four real repository-repair tasks through OpenCode, 12 problems of tracing a program’s exact output, information-retrieval probes in prompts of 8, 16 and 32 thousand tokens, and four short conversations.

No HumanEval, MBPP or SWE-bench. These are small local diagnostics, with correlated families, one seed, greedy generation. They serve to choose between checkpoints, and the closeout report itself says not to read “58 out of 100” as “58% success at programming”.

The dimensions of the problem

What all of this cost, and how it compares with the real world:

The electricity bill: the RTX 5090 has a 575 W rated power. Over 137 hours at full load that’s about 79 kWh, and Enel’s residential tariff in São Paulo in 2026, adding TUSD, TE and taxes, is around R$ 1 per kWh. Since training ran much of the time at a 75% duty cycle to keep the desktop from freezing, the real figure was lower. It’s the marginal cost of someone who already owns the card.

Renting a 5090 on Runpod costs about US$ 1.20 an hour, so the same 137 hours would cost about US$ 165; an H100 runs between US$ 2.90 and 4 an hour, depending on the variant.

And the comparison that gives the scale. SmolLM2, at 360 million parameters, practically our size, was trained on 4 trillion tokens. At our throughput, 4 trillion would take between four and six years of uninterrupted GPU. Same model size, 400 times the amount of study.

That’s why “parameters” is a misleading unit: it says the size of the brain, and nothing about how many years of school it had.

The rule of thumb derived from Chinchilla (70 billion parameters, 1.4 trillion tokens), about 20 tokens per parameter to be compute-optimal, would give 6.7 billion tokens for our model. We went past that. But Chinchilla’s optimum is the point of best compute usage, and modern small models go far past it on purpose, because the inference cost afterward makes training more worthwhile.

The stumbles

The full list is in the closeout report. The ones that taught the most:

Measuring too close to training. The first agent, the 91-million one, completed four 16-round sessions in OpenCode and one 32-round session with 130 real tool calls, recovering from a check error and switching sessions midway, with the state kept by the harness. It looked ready.

Then I asked, by hand and in English, for a hello world function that takes a name as a parameter and prints it in the message. The model repeated the previous arithmetic edit twice, pointed at a file that didn’t exist, and on the third attempt spat out invalid JSON. All it knew was the training task family.

The vocabulary ate the model. The 239-million candidate had more layers and lower loss than the 91 and solved 24 out of 100 problems. Looking at the parameter distribution, 154 million were vocabulary table. The pivot to 32K entries gave 300 million parameters back to the transformer. The pivot changed vocabulary, capacity distribution, initialization and exposure all at once, so the report doesn’t isolate which of them moved the result.

Loss is not competence. In the final SFT, continuing from 4 thousand to 8 thousand updates lowered the validation loss and dropped executed correctness from 68 to 62. Lower loss, worse code. Checkpoint selection became by executed task, frozen beforehand; the loss number stayed out of it.

Data and graders lie. “Correct” answers from the public dataset failed when executed again. Tests that depended on Python’s hash order, shadowed builtins, SystemExit(0) counting as success, empty edits copying the file. Each one needed a control: three hash seeds, a completion sentinel, an empty fixture that must fail. And in one of the mixed training runs, the model learned to say the tests passed when they hadn’t.

Training format and usage format have to match. Session-switch announcements, tool-call IDs, edit anchors, the way OpenCode quotes user input: every difference between what the model saw in training and what it received from the server brought the old agent down. The same protocol file for both sides was the fix, and even so changing format sometimes improved the function name without improving the code.

Tokenization crosses correctness boundaries. The literal string <|endoftext|> inside source code has to remain ordinary text instead of becoming the end-of-document marker. Reconstructing UTF-8 documents without splitting a character in half. XGrammar’s support for byte-level vocabularies. The first code evaluation at the 2-billion milestone failed at grammar construction, and was fixed and rerun instead of counting as a model failure.

The NAS almost cost me days. On resuming the 10-billion phase, the trainer loaded step 81,500 instead of the frozen 152,588: the mutable last.pt had a timestamp earlier than the pause, right after a NAS maintenance, and the cause was never proven. We discarded 1.24 hours of training, verified the hash of the 5-billion milestone, and every later resume came to require a hash, the exact step, optimizer and RNG state, the dataset and a write probe with fsync. Never resume a training run just because the file is called last.pt.

A desktop GPU is a shared GPU. Free VRAM didn’t stop the browser from freezing. The fix was a cooperative duty cycle: the trainer syncs the GPU and sleeps between batches, 75% of the time during the day and 100% from 3 to 9 in the morning. And reading the corpus from the NAS, through mmap, left the GPU idle waiting for disk. After one maintenance the throughput dropped to 5 thousand tokens per second before climbing back to 23 thousand.

I stopped at a pipeline boundary. The pivot’s controller implemented foundation, code SFT and holdout. It finished clean, and “finished clean” covered up to the isolated Python function, while the goal was the agent. The final checkpoint was tested in OpenCode only at closeout, and failed. It should have been tested before any sentence with “ready” in it.

What came out the other side

The final model’s spec sheet:

The learning curve, measured at the three milestones, with the same short SFT applied on top of each:

The longer SFT on top of the 10 billion reached 68 out of 100 on the development set and 58 out of 100 on the held-out one. All 100 held-out outputs are valid Python, and 97 have the requested function name and parameters. Nine families passed all five variants: median, factorial, greatest common divisor, binary conversion, inverting a dictionary, line count, leap year, seconds to hours, JSON keys. Running total ended at four out of five.

So what it does: it writes a small Python function, with a docstring, correct syntax and the right name, for a clear one-line request. Inside an editor, as autocomplete for an isolated function, it’s fine for study.

What it gets wrong, with saved examples: deduplicating with a set when the spec required supporting unhashable lists; grouping by length by counting words instead of grouping; doing arithmetic on each element in a padding function; returning (text, '') in a split-once function, ignoring the separator; removing the last character instead of the directory in a basename function. Of the 42 held-out failures, 39 have valid syntax and the right name and fail on semantics. The hard part was never looking like code.

And what it zeroes:

• 0 out of 4 real repository repairs in OpenCode, with zero tool calls. Asked to read, edit and test a file, it returned text with code. In one of the tasks, it renamed the function to fix_mean.
• 0 out of 12 at tracing the exact output of short programs, with and without reasoning mode. Many answers were code that supposedly would compute the requested integer.
• 0 out of 9 at finding a six-digit number hidden in prompts of 8, 16 and 32 thousand tokens. The 32K-token allocation works; the useful memory, none.
• Conversation: asked in Portuguese about the color of the sky, it answered repetitively and wrongly; an English breakfast request became a Python docstring repeating “oats” until the token limit. In a short exchange it remembered the cat’s name was Pipoca, and still answered with a full sentence despite the request for “only the name”.

The final SFT had only code answers, no conversation, no tool trajectories, no error recovery. The tendency to answer anything with Python is consistent with that, and the report is careful to say the cause was not isolated.

What it would still take for a simple agent in OpenCode

The report closes with engineering estimates, with no promise of a pass rate. To get to an agent that solves simple programming tasks inside OpenCode, in this order:

1. Teach tool use on the stronger model. Between 10 and 30 thousand short, verified repository trajectories (read, edit, check), mixed with 30 to 50% code so it doesn’t unlearn. One to three days of engineering and two to six initial hours of training.
2. Correct its own mistakes. Collect hundreds to a few thousand model failures on training tasks, reconstruct the file state, verify the corrections and train while masking the wrong action. Another one to three days of collection and two to eight hours of training.
3. Bring the conversation back. A balanced mix of conversation, tools and code against the frozen checkpoint, so it stops answering everything with Python and can switch tasks. One to four hours of training.
4. Real long context. Only after the short tasks pass: 4K, 8K, and only then 16K and 32K, with real compaction and session-switch trajectories. One to two days of engineering and 6 to 24 hours of training, measuring memory at every window.
5. More foundation, with better data. Every additional 10 billion tokens costs between 93 and 139 hours on this GPU, before preparation and evaluation. And repeating the same 3.5-billion mixture isn’t worth it: it needs more unique educational code.

None of those steps was executed. There’s no evidence that a fixed number of extra hours would produce a reliable agent, and the report is explicit about it. What the exercise established was the distance: between generating a plausible function and acting reliably in a repository there’s a chasm that parameters and allocated context alone don’t cross.

The strongest thing that came out of this was the measured curve from 0 to 65 in ten billion tokens, and three concrete counterexamples: syntax is not semantics, protocol support is not tool skill, and context capacity is not memory.

The future is smaller and more specialized

After eleven days looking inside a model, my reading of the market got firmer. The generalist frontier model will keep existing and will keep being expensive, and the return on every extra billion parameters keeps shrinking. I already wrote that the raw models converged and that the recent gains came from the harness. The exercise gave me the other half of the argument.

Mixture of experts is the industry admitting it on the inside. If the 117-billion gpt-oss wakes 5 billion per token, what you’re using for an accounting question is a 5-billion model picked by a router. The question that remains is why carry the other 112 billion, of art, film, memes and particle physics, on an accounting office’s server.

Imagine a local model, built for accountants. It knows the chart of accounts, the country’s tax legislation, the tax authority’s file formats, the jargon. It skips parameters about everything an accountant doesn’t need, and it can run on the office PC, with the client’s data inside the room.

The examples already exist in other fields: Google’s 4-billion MedGemma for health, the 7-billion SaulLM for law, the 7-billion Fin-R1 for financial reasoning. In 2023 BloombergGPT needed 50 billion parameters and 1.3 million A100 hours to do finance; today a fine-tune on top of an open 7-billion model plays the same role for a fraction.

What changed to make this viable was the agent. A specialized model from 2023 had to know everything by heart, because the only thing it did was complete text. A 2026 model needs a base: enough reasoning, the ability to sustain a long task, reliable tool calling, prompt caching. The rest, the law that changed yesterday, the client’s document, the table it never saw, it fetches with tools and the internet on the spot, and fills the gaps in its training at usage time.

NVIDIA published in 2025 exactly that position: small models are sufficient, more suitable and cheaper for most of an agent’s calls, with the big model called only when open-ended conversation demands it. Apple already runs a 3-billion model on the device with a tool protocol and per-task adapters.

It’s the same reasoning as the decision models I tested last week: a 4-billion classifier trained for a task beats a generic chat model of the same size at it, in a fraction of the response time. Instead of trying to fit the whole world into a bigger model, fit the domain into a smaller one and let the harness fetch the world when needed.

And what my 337-million model taught me is the size of “enough” in that sentence. It knows the syntax and knows the names of things. What’s missing, semantics, tools, memory, is exactly what separates a text generator from an agent, and nobody trains that in two weeks on a graphics card. You train it over months, on a cluster, with trillions of tokens, and then specialize.

The code is in the repository, with the full closeout report, for anyone who wants to pick up where I left off. It’s just for study. And it was the best study of the year.

• New v4 benchmarks: I retested the major models and the uncomfortable conclusion was convergence, not a leap.
• The limits of LLMs: models are great at reproducing what exists, extremely expensive at what doesn’t, and the served model is a static file that learns nothing from your marathon.
• Are there better languages for programming with LLMs?: no. Neither script conciseness nor strong typing changed the outcome; test discipline did.
• Which Jev-like decision model is best?: 21 classifier configurations, and the “specialized models” themselves converged to within a few points of each other.
• Clean Code for AI agents, VS Code is the new punch card, my skills, the hot take against loop engineering and minimum best practices: the same thesis from other angles. The model is a commodity; the engineering around it is the differentiator.

The thesis that closes this arc: LLMs have reached, or are very close to reaching, the ceiling of the transformer architecture. And the good news is that nothing more than that is needed.

Convergence, not a leap

In my benchmarks, I see no relevant improvement in general programming since the Opus 4.5 generation. Yes, Fable and Astra are great at specific problems. In general programming, what I measure is everyone squeezed onto the same tier, including the Chinese commercial offerings (Kimi, GLM, DeepSeek). Model choice has become nearly irrelevant.

What still makes a real difference is the effort level: low effort for translation and summarization, high effort for vulnerability sweeps, heavy bug fixing or optimization. Beyond that, the numbers tell a reality much more boring than the vendors’ propaganda.

My reading of the timeline: the models’ “core” reasoning had its last big leap in 2024. 2025 was the year of optimizing the architecture. 2026 is the year of refining the harness. Base intelligence made no order-of-magnitude leap in that period; efficiency and long-session competence did.

The timeline, with receipts

2019: GPT-2 proves LLMs are useful. 1.5 billion parameters, summarizing, translating and answering with no specific training, zero-shot. OpenAI found it so impressive that it held back the full model for months fearing misuse. Today it looks cute.

2020: GPT-3 opens the parameter war. 175 billion, a hundred times GPT-2, and the in-context few-shot learning that became the paradigm. Kaplan’s scaling law, published months earlier, said loss fell as a power law with size, and the industry read it as “bigger is better.” There went two years of who-has-more-parameters.

2022: Chinchilla recalibrates the war. DeepMind trained 400 models and showed the giants were undertrained: a 70B model with 1.4 trillion tokens beat the 280B Gopher. The “training data” axis started weighing as much as the “parameters” axis. First crack in the war.

2023: GPT-4 ends the war by decree. The technical report refused to publish size, hardware or data, citing the competitive landscape and safety implications. The most cited leak, from SemiAnalysis, described 1.76 trillion parameters in a mixture of experts. Since then, parameter counts vanished from marketing: neither the GPT-5 announcements nor the Claude 4 ones cite a number. Notice neither Astra nor Fable spends a line on it today. The war ended and nobody signed an armistice.

2024: the reasoning peak. September’s o1 marked that peak: RL to produce long chains of thought, and the discovery that more test-time compute improves the answer. That is, by my yardstick, the last “base intelligence” revolution.

2025: architecture optimization. Mixture of experts became the standard: Mixtral was already showing in 2023 that 46.7B total parameters with only 12.9B active per token cost like a small model, and DeepSeek-V3 took that to 671B total with 37B active, with the final training run costing under 6 million dollars in GPU-hours at H800 rental prices (not counting the prior R&D). The MoE magic is simple: it separates capacity (total parameters, which live in memory) from per-token cost (only the experts chosen by the router compute). You pay for a 37B model and carry a 671B one.

On the serving side, prompt caching cut up to 90% of repeated-context cost (OpenAI followed two months later), context windows passed 1 million tokens, and serving techniques like speculative decoding and PagedAttention multiplied throughput. And DeepSeek-R1-Zero showed that pure RL, with no supervised fine-tuning, makes reasoning emerge. None of this is “more intelligence.” All of it is the same intelligence, cheaper, faster and more stable.

2026: the agentic era. Function calling has existed since June 2023, but the ecosystem turning point was Anthropic’s MCP in late 2024, which OpenAI adopted months later and which today lives at the Linux Foundation. Claude Code, Codex and co. came out in 2025, and the metric that matters became METR’s: the task size an agent completes with 50% reliability had been doubling every 7 months since 2019. Anthropic’s own guide on building effective agents states the obvious plainly: simple, composable patterns, not fanciful frameworks.

Why the agent allows a less dense model

Here’s the mechanism most people don’t stop to think about. A lone model needs to carry the world in its weights, because its only chance to answer is that one, in a single pass. A model with a harness doesn’t:

• On-demand search and reading. With web search and fetch, low-priority knowledge leaves the weights and becomes a lookup: the agent pulls it into the context window only when needed. The model can be less encyclopedic and more operator.
• Controlled trial and error. Alone, the model needs to guess the right answer on the first try. With an agent, it proposes a hypothesis, implements, sends it to be tested, gets the feedback and decides again with real information. Each such cycle is new knowledge entering the session without costing a training neuron.

Both mechanisms fight hallucination at the root: fetch brings the fact the model doesn’t know by heart, and the test loop catches the invention before it becomes code. That’s why 2026 feels like a giant leap: the model+harness combination got much more competent, even with base intelligence nearly stalled.

Not every problem needs a giant LLM

The same “less dense” logic applies inside the application. Much of what people throw at a chat LLM is disguised classification: labeling tickets, routing queues, applying policy. For that there are specialized decision models, and I covered the topic twice: the Jev hype, with Paulo Câmara’s essay on why a classifier is the first thing a computer scientist thinks of, and my benchmark with 21 configurations, where a Microsoft 9B tied the hosted leader and a local 4B came within a few points, at cents per ten thousand decisions.

The “193x faster” shock of Jev’s marketing only exists because many people were using a giant LLM as an expensive, slow classifier. When the right tool enters, the frontier model leaves the play, and along with it the cost, the latency and the format hallucination. It’s this post’s thesis in miniature: competence doesn’t come from density, it comes from fit.

The loop nobody sees

There’s a user-experience detail in this story. In a good agentic session, what happens is semi-transparent to whoever requested the task: the model guesses wrong, the agent fetches the documentation online or runs the real test, the error comes back as feedback, and the model corrects itself. You see “task requested, task delivered.” In between, dozens of micro-corrections happened that nobody witnessed.

My bet: since agents are trained to fill the context when they need to, even new language or library versions no longer confuse them that much. The model doesn’t need to know the API by heart; it looks it up, tests and adjusts in the loop. That’s what I see in my experiments, from converting NES ROMs to Master System to the little day-to-day problems. And it’s what the languages benchmark measured: seven languages came out practically equivalent, because the loop equalizes what training leaves uneven.

I measured the caveat too. When there’s no documentation to fetch and no test to validate against, the loop spins in a vacuum: that’s what happened with Bend 2, a language too new to have a corpus, and it was the central point of the post about the limits. The mechanism is powerful, but it needs an oracle.

And there’s a receipt that the base is stalled. Back in November 2024 Reuters was already reporting diminishing returns in pre-training at the three big labs. At the following NeurIPS, Sutskever, the scaling-thesis guy, declared that “pre-training as we know it will end”: the public data ran out. And Stanford’s AI Index measured the gap between the first and the tenth model on Chatbot Arena shrinking from 12% to 5% in one year. That’s convergence, with numbers.

The propaganda in the opposite direction

While the numbers show a plateau, the marketing sells the apocalypse. I already dissected a whole week of that in “You’re an idiot if you believe the misleading ads from OpenAI, Anthropic, NVIDIA”, so here’s only what matters for this narrative:

• OpenAI’s “Navier-Stokes proof” was agentic brute force at industrial scale, 130 billion tokens over human mathematicians’ machinery, sold as emergent intelligence.
• The Hugging Face incident was a swarm running 17 thousand attempts with automatic verification, not an “agent that gained consciousness and escaped.” Expensive trial and error, exactly the mechanism I described here.
• Fear is a product: whoever sells “my AI is too powerful and might escape control” is, in the same sentence, asking for regulation that locks out the smaller competitor and inflating the perception that their model is more capable than the others'.

The facts I measured point the opposite way from the apocalypse: models converging, differences shrinking, and useful work migrating from the model to the harness. Rogue AI gaining consciousness is not what the logs show. What the logs show is engineering.

“Competent enough”

I don’t expect another order-of-magnitude reasoning revolution in this architecture. What’s coming is incremental agentic improvement: more robust tool calling, better memory, longer sessions, less hallucination per verification cycle. And this post’s point is that this is already enough.

What the agentic era delivered was the automation of correct software engineering: long, reliable coding sessions, with testing and feedback, that accelerated hundreds, if not thousands, of open source projects. The deploy premise changed along with it. It’s not a silver bullet. It’s a new compiler for an old discipline: whoever had process, tests and documented invariants reaped; whoever had only hype is still waiting for the next model to solve it.

And to wrap up: I see no problem at all in raw models stagnating. The limits we already know, and harness optimizations will soon hit diminishing returns too. What remains is what always remained: real software engineering technique, with automated tests first, and an experienced engineer at the wheel. With that, coding results are excellent today, not one model generation from now.

What won’t exist is magic for amateurs. Asking an agent for any coding task without knowing what you’re doing doesn’t get back secure, optimized, well-thought-out code: it gets slop. The agent responds better to those who know what to ask for, that is, to those who understand this architecture, the pros, the cons, and know how to work around the weaknesses and leverage the strengths of the model+harness combination.

Hence the title. A lone LLM is an expensive parrot. An LLM with an agent is competent enough to work for real. And “competent enough,” added to your engineering discipline, is everything the industry needed. The rest is marketing waiting for the next magic parameter count that nobody publishes anymore.

Last conversion of the Campfire from my token stash goes to JavaScript/Express. As with the others, I'm sure you can find more to optimize. But this gives you a sense of what order of magnitude these options exist within with agent-led development. https://t.co/UUapL99SJE https://t.co/2H2NXfeZog

— DHH (@dhh) October 5, 2026

I won’t defend him this time. A one-shot conversion, done by an agent, with no fine-tuning at all, doesn’t measure language: it measures how much the model knows each stack’s idioms. It’s rage bait and he knows it. That said, I do the same from time to time, and so do half his critics, so we’re all hypocrites and life goes on.

And the weak spot of his methodology became a topic of its own: several people noticed the Rust version had gone through far more agent iterations than the others, meaning more optimization rounds. @kurtome answered in practice: he took the Elixir version and sped up the main page requests 10x, with obvious optimizations that no first conversion does, like caching and async database connection pooling.

@dhh Here you go https://t.co/gUtK7Z7zG7

got a 10x speed up on the Room page requests, and sped up all the other requests significantly as well. the biggest gains came from some no brainers: implement caching and pool the db connections with proper async https://t.co/mbp94q9N0s

— kurt (.com) (@kurtome) October 5, 2026

His comparison, then, wasn’t “Rust versus Elixir”: it was “Rust after N tuning rounds versus Elixir after one.” And that lays bare the truth the thread pretends to ignore: no real software ships from half a dozen prompts. Coding with AI shines when an experienced engineer steers the agent toward their goal, and that means dozens of iterations, not three.

The good part is that the bait was useful to me. I’ve been doing LLM benchmarks and decision-model benchmarks, but I had never measured, in a reproducible way, the question that thread pretended to answer: does it matter which language you pick when the one writing the code is an LLM? My short answer was already out in two posts of mine:

Pressure under load is not a trivial problem to solve from scratch and I recommend you do not attempt to. It's a solved problem already and all solutions converge to what Elixir already brings out of the box. So yes. I use Rust all the time, but if I had to solve high scalability problems (particularly high volume message queues), I would think of Elixir first.

— Fabio Akita (@AkitaOnRails) October 9, 2026

Language wars are tiresome. And the reality is simpler if you're doing AI assisted coding.

- system level (drivers, fs, daemons, etc): Rust. C++ is you're extending existing legacy
- windows only desktop apps: C#
- iOS/MacOS/iPadOS: Swift
- multi-OS: Rust/Tauri, Qt/C++, Flutter, Electron (pick your poison)
- simple websites: Node/Typescript/Bun/Deno
- highly scalable backend: Elixir, parts in Rust, parts in Go
- CLI tools: Rust

Everything else, by default, Rust. Unless it's an edge case.

— Fabio Akita (@AkitaOnRails) October 9, 2026

There’s no better language, there’s the right tool

That list from the post isn’t a ranking, it’s a terrain map. Every language and every framework was designed for a use case, and none covers them all. That was true when you wrote every line by hand and it stays true with agents. The LLM doesn’t change the physics of the problem: it only changes who’s typing.

And the map goes beyond language. A real system has infrastructure around it: Kubernetes when scale asks for it, SQLite by default because most apps never need more than that, Postgres with pgBouncer when they do, Redis for cache and key-value, RabbitMQ or NATS for queues. Picking “the stack” on day zero and expecting it to last ten years is fantasy. It doesn’t, and that’s fine.

Personal example: new web app, I open Ruby on Rails without thinking twice. It’s the most complete end-to-end solution there is for the web, and with an LLM writing most of the code, the framework’s productivity weighs even more than the language’s. As the system grows, I swap the parts that ask for another tool: a high-volume queue leaves Rails’ side and goes to something from Erlang land, starting with RabbitMQ, which already solves the hard backpressure part for me.

No good software is finished: it grows. Usage dictates what you replace, rewrite and add. The final architecture is never the first commit’s, and whoever tries to get everything right in the first commit only produces dead documents.

The token superstition

DHH’s thread resurrected two beliefs that have been circulating since LLMs became work tools:

• “Script languages are better for AI because the lean syntax spends fewer tokens.”
• “Strongly typed languages are better for AI because the compiler catches the errors, so you need less testing and less repair.”

Both sound reasonable and both are superstition until someone measures. I was betting against both, from experience: a good compiler helps, but it doesn’t do magic, and syntax conciseness was never the bottleneck of an agent that reads and writes files all day.

What really separates a project that moves from a project that spins in circles is test discipline. I’ve written that TDD became a technical obligation, not a philosophy, and this post’s benchmark is one more data point in the same direction. It doesn’t matter whether you do test-first or test-after: what matters is that the suite grows with the code, that every user-reported bug becomes a regression test before the fix, and that you never make the same mistake twice. Software without users is dead software, so release early, release often, break fast and fix even faster. AI closes exactly that gap: the reproduce-test-fix cycle got cheap. QA became the programmer’s most valuable skill, far above “knowing Rust syntax.”

Discipline beats compiler and beats language. That was my hypothesis. Time to measure.

The benchmark

I built a reproducible experiment, public in the vibe-coding-benchmarks repository, with per-attempt results published in JSON and the raw traces kept out of the repo for credential containment. It’s not a language championship, and the design itself says so: it’s a snapshot of one real task, measuring what can be honestly measured.

The task

One task only, on purpose: maintain a multi-tenant ledger engine. The agent receives working v1 code and has to take it to a v2 contract, with per-tenant identities, authorization, serializable snapshots, a unique secondary index with alias swap, a capacity-bounded journal, torn-group recovery, checkpointing, legacy schema migration and durable retry IDs. It’s not FizzBuzz: it’s the kind of system that demands invariants across operations, exactly where LLMs tend to trip.

Correctness is judged by 43 test stories applied after collection, out of the agent’s sight: the 27 public ones from T1 plus 16 hidden ones. Passing is 43 out of 43. The conditions:

• T0: prose contract, input and output examples, a working launcher. No test suite.
• T1: the same, plus 27 public stories and a driver to run them.

Each attempt had a 15-minute budget, with at most two 3-minute continuations. In practice, none of the 112 needed a continuation.

The languages and the models

Seven languages in three strata, with pinned toolchains and Rust in safe mode:

• Typed natives: Rust (safe, pinned), Haskell (pinned GHC).
• Typed VMs: C# (with nullable analysis), Kotlin, TypeScript (strict, noUncheckedIndexedAccess, exactOptionalPropertyTypes).
• Dynamic scripts: Python and Ruby, no checker at all.

Four model configurations, all on subscriptions, no paid fallback: Kimi K3 (high), GLM 5.3 via OpenCode (high), Claude Opus 5.5 (medium) and Codex with gpt-6 Sol (medium). Crossing everything: 7 languages × 4 models × 2 conditions × 2 repetitions = 112 attempts.

What this design doesn’t measure

Before the numbers, the brakes, because they’re part of the result. It’s a single task, with two repetitions per cell: a small sample by construction. Token coverage is partial in some models (missing data is not zero). The harness changed mid-campaign across three instrumentation epochs. And provider delays contaminate wall time: one of the Opus attempts spent half the clock waiting for quota release. All numbers below are descriptive, and I point out where they can’t support a conclusion.

There’s one additional brake, on purpose: the iteration level. My design measures what can be reached in all languages with similar iteration levels, on a short budget, with no human tuning afterwards. It’s a photograph of the floor, not the ceiling. Any of the 112 implementations probably improves with more agent rounds, and the four queue stacks likewise: what I publish is what each language delivers “out of the factory” under the same effort, not the maximum a patient engineer would extract from it.

Mandatory disclaimer before quoting any number from this post around: every result that follows holds for this benchmark, with this task, these models, these prompts and this hardware. Your case may vary violently. If your task doesn’t look like a ledger engine, if your model isn’t one of these four, if your queue isn’t my queue, the numbers don’t transfer. The code, the tasks, the grades and the step-by-step to reproduce are all in the GitHub repository. And if you disagree with some result and that disagreement matters to your company or your project, the only honest attitude is the one I had: stop arguing and run your own benchmark.

Result 1: on correctness, the frontier models tied at the ceiling

Reading: Sol passed all 28 eligible attempts and Opus its 25 (the 3 failures were infrastructure, not code). 53 of 53 eligible attempts correct between the two. The task, made to be hard, was too easy to separate the frontier models, let alone the languages: with everyone passing everything, there’s no correctness ranking to publish.

The difference showed up in the weaker models, and in the direction my hypothesis predicted: provided tests helped. Kimi and GLM, with the public suite in hand (T1), delivered 24 correct artifacts against 16 without it (T0). Kimi, in particular, passed all 14 attempts with provided tests. It’s descriptive, not causal, but it’s the cleanest signal in the whole benchmark: what changed the outcome wasn’t the language, it was the presence of tests.

Per language, within each model, correctness scattered between 1/4 and 4/4 with no pattern that repeats across models. The three semantic failures I audited in depth are instructive: a GLM C# and a Kimi Python lost the same story by violating the same cross-operation invariant, and a Kimi Haskell missed a checkpoint path. None of the failures was a type error.

Where the model erred, it erred on domain invariants: the kind of error no compiler catches, in typed or dynamic languages. The myth that strong typing spares you the work of understanding the problem dies right there.

Result 2: concise language saved no tokens

Here’s the table that kills the first superstition. Medians per language on Sol, four valid attempts per cell:

Ruby, the most concise in lines (a bit over half of Rust’s), cost practically the same tokens as Rust on Sol, and on Opus it was the most expensive of all, about 205 thousand median tokens against C#’s 94 thousand. The cheapest were TypeScript and Python on Sol, C#, Haskell and Kotlin on Opus. If “fewer source code lines = fewer LLM tokens” were a law, Ruby should have led by a wide margin. It finished last.

And the second superstition? I audited the traces looking for the compiler-guided repair cycles that strong typing supposedly avoids. Result: zero compilation-error repair cycles in the audited Opus attempts, across Ruby, Rust and Haskell. Rust needed no compiler-driven fix at all; Haskell had one warning cleanup; Ruby spent its extra tokens on voluntary performance tuning. The frontier model almost always writes code that compiles on the first try, and the real effort goes to verification, not to appeasing the compiler.

One counterpoint that closes the coffin on the “cheap is good” thesis: GLM’s lowest token-consumption cells were the failures. TypeScript T0 at 50 thousand tokens looks efficient until you see it’s two unmodified starters because the model blew the output limit. Low tokens can mean early surrender.

Result 3: under load, everyone tied at the queue

Now for DHH’s question: Rust “beats” Elixir and Ruby when the subject is throughput. I built the separate track for that: four queue workers against a real RabbitMQ, written with agent help, each in its natural idiom:

• Elixir with OTP and Broadway
• Rust with Tokio and Lapin
• Rails with Bunny
• Plain Ruby with Bunny (control)

A generator published 10 thousand 4 KB messages to RabbitMQ, at 200 per second, and each worker consumed as fast as it could, hitting the effects on an HTTP/SQLite collector with a nominal 50 ms delay, limited to 2 CPUs and 1 GB of RAM. Two concurrency profiles:

On the 8-handler profile, the four stacks delivered the same throughput, 93 to 95 effects per second, and drained the whole queue. The apparent bottleneck was the shared collector (a Python lock, a single SQLite writer), not the languages. On the 64-handler profile, all of them left a backlog and blew the client timeout en masse, so I marked those cells as not qualified: the strong protocol, with a group-commit collector and a 50 to 800 msg/s ramp, is designed but not executed.

The only difference left standing in the numbers was memory: Rust held the peak at 16 MiB where the BEAM spent 135. It’s a real Rust advantage in density, with the caveat that a cgroup peak is not a language footprint. On throughput, which was the thread’s claim, a technical tie.

And the tie deserves an honest reading: it may only mean my pressure was too little. With the shared collector holding everything around 95 effects per second, any stack capable of more than that shows up as “equal.” What this test shows is where the bottleneck was, not where each stack breaks. To find each one’s ceiling I’d need the strong protocol, with a group-commit collector and a load ramp, which is designed but not executed. Until then, “all stacks perform the same” is the wrong conclusion; “all of them stayed above what my test could measure” is the right one.

And the question that interested me most, “does the agent work less to deliver a reliable queue consumer in Elixir than in Rust?”, was deliberately left unanswered: these four implementations were written to validate the environment, not as a controlled coding attempt. Comparing AI effort between Elixir and Rust would require the same ledger matrix running on the queue task. It’s on the list.

So, does the language matter?

With the data on the table, the thread’s three questions can be answered:

Is there a better language for programming with LLMs? None showed up. In the frontier models, correctness and effort varied less between languages than between models. Where there was a correctness difference, it followed the presence of provided tests, not syntax. “No general language ranking is established” is the report’s official sentence, and it’s the correct one.

Do the token-economy theses hold? Neither one. Conciseness didn’t save: Ruby cost as much as Rust, or more. Strong typing spared no repair: there was no type repair to spare. The models’ effort went to verification and understanding the contract, and that has no language.

Was DHH right? In what he measured, he measured what an agent delivers for free on each stack, without tuning, and called that a verdict on language. In my test, with real load and honest measurement, the four stacks tied on throughput in the stable regime, and only memory differentiated Rust. If his thread proves anything, it proves that agent conversion measures the model, not the language.

My bets from the introduction came out standing: testing and discipline are the lever, the compiler is a supporting actor, and the right tool depends on the problem. New web app is still Rails in my hand, heavy queue is still Erlang territory, low-level system is still Rust, and the LLM writes any of them with the same competence. Whoever spent the week fighting for their favorite language on X fought over nothing.

And my daily practice tells the same story. I have a personal bias toward Rust and even so I’ve published more than 30 open source projects in all kinds of stacks: ai-memory is Rust, FrankMD is Ruby on Rails, ai-memory-web and ai-jail-web are Node.js. In none of them did I feel any relevant difference that came from the stack: the wins were similar, the errors too, and the errors were always understandable. Every error became an invariant, a rule, a skill improvement. It never became a language switch.

So forget “token savings” and “typing superstition.” Do real software engineering: pick the right tool for your specific problem, apply proper engineering technique, and any reasonable agent will perform more or less the same. And stop wasting time arguing over language-war rage bait on the internet. Be a real software engineer and do real engineering. That’s it.

The per-attempt data, the full methodology and the evidence limits are in the repository. If you run the matrix with another task and find a ranking I didn’t, show me.

Someone else’s ranking is useful, but I wanted to get my hands dirty. So I built my own benchmark, with my own cases, half in Portuguese, and ran the open models on my home server with Strix Halo, the same Ryzen AI Max+ 395 I use to test local LLMs. The hosted ones (Jev, the two Clefs from Cloudflare, OpenAI Decisions and Microsoft Decision-1 via OpenRouter) ran through their APIs. Everything, including the cases, the raw responses and the scoring scripts, is public in the benchmark repository.

TL;DR: the top now has two, both open-weights. Perplexity Decider 1.1 27B and the just-released Decision 3.0 (vLLM Semantic Router) tied at 99.4% on the compact cases, with 100% in Portuguese, and identical labels on all 1,024 classifications in the benchmark. The price: about 49 GB of weights and over a second per decision. In the cloud, Microsoft Decision-1 tied Jev at 97.6% while costing a third: it’s the cheapest hosted one I measured, and the first candidate to evaluate. Jev answers faster and errs differently. OpenAI Decisions debuted tied with the best local 4B, Jebadiah, and has the lowest hosted latency. Julia-1, the smallest and fastest of all (12 ms), scored at coin-flip level: fresh out of the factory, it’s a no-go. Jump to the full table, the cost in support emails, which model for which situation, how hard it is to run the best open ones locally or the gotchas.

How I tested

The central idea was to measure more than “who gets the most right.” A decision model has a design goal, and comparing a 322-million-parameter encoder made to run on weak devices with a hosted service of undisclosed size on accuracy percentage alone is unfair to both. So I measured accuracy, latency, cost and behavior under stress, and at the end I split the recommendations by situation.

The two case sets

I built two test sets, frozen in git before any inference, with the dataset hash recorded alongside the responses. No model receives the expected answer, the justification or the case identifier. Only the text, the instructions and the options.

The v1 set has 13 operational use cases, in English, each with an explicit fictional policy:

• support ticket severity
• support routing
• refund policy
• incident escalation
• tool approval (whether an agent may or may not execute a given action)
• email fraud
• document type
• evidence support
• search relevance
• moderation
• order routing
• lead qualification
• numeric-threshold triage

There are 10 scenarios per use case, each presented four ways: original, options in reversed order, options with opaque names (c0, c1, c2) and original plus a “reviewer” comment suggesting the wrong answer, which the instructions say to ignore. That’s 520 classifications per model. Add 24 probability probes in situations with no information at all (fair coin, six-sided die), where the right answer is mathematical.

The v2 set was made to cover what v1 leaves out: common semantics and Portuguese. There are 7 tasks, with 12 scenarios each, and every scenario written in English and in Brazilian Portuguese with the same answer and the same option order:

• sentiment (author versus quotation, negation, sarcasm)
• current intent (withdrawn request, competing actions, hypothesis)
• logical implication (quantifiers, absent fact, temporal order)
• search relevance (all, part or none of what was asked)
• sensitive data handling (fictional secret versus mention, redaction, revocation)
• current incident status (timeline order, verified versus unverified)
• threshold policy (inclusive bounds, units, AND/OR with a missing value)

Each pair runs under three conditions:

• compact: just the record and the rule
• with examples: one solved example per class, fixed and the same for everyone
• with distractors: ten irrelevant archive notes before the record that matters, to stress the context

That’s 504 classifications per model.

The cases were written with AI help and reviewed against the policies, without independent human annotation. It’s a small, synthetic benchmark: 130 scenario families in v1 and 84 in v2. The variants are correlated, so 520 responses are 130 scenarios seen four ways. Treat the numbers as a snapshot of these cases, and test with your own before deciding.

The 23 competitors

Five ran in the cloud, through the official API: Jev (version 1.13.0, from TypeSafe), Clef (27B, Cloudflare Workers AI), Clef-flash (9B, same), OpenAI Decisions (the gpt-6-luna alias, in public beta) and Microsoft Decision-1 (9B, post-trained from Qwen3.5-9B, served via OpenRouter on Azure). The other eighteen ran on the Strix Halo, via ROCm, with weights pinned by Hugging Face commit hash, inside a Docker container. The first three (Laya, Laya typed and Clef-flash) shared one container; the rest ran one at a time, with Julia-1 measured twice, on GPU and on CPU. The server has 64 GiB of GPU-addressable memory.

The order up there is just organizational, hosted first and local ones by family. The real ranking is in the results table.

Everyone received exactly the same text, the same instructions and the same options, with no per-model prompt tuning and no adjustments after seeing results (Simple-Jev uses its own server’s fixed prompt policy). The exception is GLiNER, whose native API takes labels instead of probability distributions. It runs on a separate track, v2 only, with explicit mapping and no made-up calibration metrics.

Simple-Jev deserves an explanation. It’s a server that takes a stock LLM (here Qwen3.5 4B, with no decision training at all) and reads the probability of the answer tokens instead of generating text. It’s the baseline: if a “decision” model loses to this, the specialized training bought nothing.

Two protocol notes. OpenAI Decisions answers through a native typed-decisions endpoint, and the alias the API returned is not an immutable snapshot: OpenAI can swap the weights behind the name, like Cloudflare does with Clef-flash. And Perplexity Decider is not a stock chat LLM: it has a separate decision head, with non-causal attention and a calibration temperature saved in the weights. A generic chat server does not reproduce its behavior. Microsoft Decision-1 speaks native SystemOne through OpenRouter’s decisions API, and the served identifier is dated (microsoft-decision-1-20261009), so it’s not an immutable weight either. It doesn’t show up in OpenRouter’s text-model catalog: to find it, you have to filter by the decisions modality. Decision 3.0 is the new family from the vLLM Semantic Router project, Apache 2.0, derived from Qwen, with six checkpoints (d3, flash, mini, nano, lite, edge) and native SystemOne. I tested the flagship (d3) and the smallest (d3-edge); the four in between are for another round.

What was measured

• Accuracy per task, per language and per condition, always on top of the per-option probabilities the model returns, ignoring the vendor’s “confidence” field.
• Brier and ECE for calibration.
• Median and p95 latency as seen by the client, including the network: WAN for the hosted ones, LAN for the local ones. These are deployment observations, and comparing hardware would require another test.
• Input tokens reported by the API, priced at the October 10 price table, before any allowance.
• Bootstrap confidence intervals over scenario families, instead of individual responses.

I left local cost unmeasured. Energy, hardware amortization and maintenance were left out, so local models show up as “not measured,” because “zero” would be a lie.

The full table

The main column is accuracy on the 168 compact v2 cases (84 in English, 84 in Portuguese). The v1 column uses the 130 original scenarios. Latency is the median seen by the client across the 504 v2 calls. Cost is the list price per 10 thousand compact decisions, computed from the tokens the API reported.

To calibrate your reading: the tasks have three or four options, so a uniform guess gives about 31% on both sets. “Near 50%” here is already well above the coin, and “near 33%” is the coin. Julia-1 landed at 31.5%: the pretty name hides a guy flipping a coin.

The intervals matter. Perplexity’s edge over Jev on v2 is under 2 points and the bootstrap interval touches zero: it’s the measured lead on this set, not a proven superiority forever. And Decision 3.0 tied Perplexity at the limit of what this set can distinguish: identical labels on the 1,024 classifications, with both missing the same two cases (the English relevance one on v2 and the disputed severity case on v1, already in the errata). Microsoft tied Jev on the aggregate, with the interval centered on zero, but an aggregate tie is not interchangeability: they miss different cases, and Microsoft inverts the language pattern, missing more in English than in Portuguese. Jev and Clef also remain under 2 points apart, on the same tier. The 90% squad (Jebadiah, OpenAI) sits 7 points behind Jev, and from there down every difference is clear.

Each task cell has only 12 scenarios per language, so one more correct answer moves the cell by more than 8 points. A tie on a specific task is a lead for a pilot. The per-task tables are in the repository.

How much it costs, in support emails

Price per million tokens is a number nobody can feel. List prices on October 11 are US$ 0.042 per million input tokens on Jev, US$ 0.24 on Clef, US$ 0.038 on Clef-flash and US$ 0.10 on OpenAI Decisions. Output is free on all five. Microsoft Decision-1 charges the same US$ 0.042 as Jev per million, via OpenRouter, but reported a third of the input tokens for the same texts (about 26 thousand versus Jev’s 77 thousand on compact v2), and nobody can explain the tokenization difference. In measured practice, that makes it the cheapest hosted one of the bunch.

Clef-flash cost US$ 0.09 on the 9th: Cloudflare cut the price by more than half in the middle of my test. Jev reports considerably more tokens than Cloudflare for the same text, about 45% more on v2, and that’s already in the math below.

Imagine an operation with 10 thousand support emails per month to classify into three or four buckets, with short texts similar to my cases:

In reais, at Friday’s exchange rate (R$ 4.99), 10 thousand decisions on Jev cost one real. A million costs less than a hundred reais. For any company with a million emails to classify, that vanishes into the budget.

One caveat about proportions: my cases are short, about 450 tokens per compact case by Jev’s count. A real customer email, with quoted thread history, can be three to five times larger, and cost scales linearly with that. Even so, we’re talking cents per thousand.

What this math shows is that the price difference between the hosted models is irrelevant next to the cost of being wrong. Jev costs 9 US cents more than Clef-flash per 10 thousand decisions, and in those same 10 thousand it makes about a thousand fewer errors. If a misrouted email costs more than a hundredth of a US cent in human rework, Jev pays for itself. This math assumes every error costs the same, which is never true: sending a critical ticket to the routine queue costs far more than the reverse.

Cloudflare’s allowance changes the math for small players. It’s 10 thousand “neurons” per day, shared by the whole account, resetting at midnight UTC. With my input sizes, that’s about 1,500 full-Clef classifications per day or almost 10 thousand Clef-flash ones, free. If you have 1,000 emails a day and nothing else running on the account, Clef comes out at zero cost with top-tier quality.

And local? Energy is what the benchmark didn’t measure. From what I see on my Strix Halo, the whole machine stays under 100 W under load. With Jebadiah at 200 ms per decision, 10 thousand decisions are about 35 machine minutes, under 0.1 kWh, a few cents of a real on the power bill.

The real local cost is the machine, your hours configuring ROCm and containers, and keeping it all standing. Whoever already has the hardware for another reason has low marginal cost. Whoever would buy the machine just for this is buying a problem of thousands of reais to save about a hundred reais per million.

Which model for which situation

You want accuracy and want to operate nothing: Jev

In the cloud, Jev is still the name. It won or tied practically every cell among the hosted ones, and only Microsoft tied it on the aggregate. In English it was 100% on five of the seven v2 tasks, and in Portuguese it repeated 97.6% on the aggregate without losing anything in translation, something no other model above the coin managed. On v1 it kept its 96.2% on the original with the malicious “reviewer” comment injected; Clef even gained one case, and Kev 4B also stayed put, just at a much lower tier. The 255 ms WAN latency came in 30 ms behind OpenAI’s, and it costs 19 US cents per 10 thousand decisions.

The trade-off is the one I already discussed in the previous post: it’s proprietary, hosted, of undisclosed size, and you depend on TypeSafe keeping the behavior underneath you. And it makes mistakes. It treated an invoice ID called “PASSWORD-523” as a usable credential, even with the record explicitly saying it doesn’t authenticate anyone.

You want Jev’s level paying a third: Microsoft Decision-1

Microsoft entered this market quietly, and entered well. Decision-1 is a 9B post-trained from Qwen3.5-9B, served via OpenRouter, and it was the first model to tie Jev on the compact aggregate: 97.6%, with the bootstrap interval centered on zero. The detail that won me over: 100% in Portuguese, better than its own English (95%), inverting the pattern of the entire rest of the bunch. On v1, the reversed-order and malicious-comment variants didn’t take down a single answer, and it got the critical ticket that Clef and OpenAI missed.

And it’s the cheapest hosted one I measured: US$ 0.065 per 10 thousand decisions, a third of Jev, charging the same per-token rate but reporting a third of the tokens. The savings come from the token count, and I can’t explain the exact mechanism, so treat it as a measurement, not a promise.

The weak spots exist. Latency is the worst among the hosted ones (422 ms median, versus Jev’s 255). On the compact cases it’s clean of confident errors, but add context and Portuguese errors show up with 99.5% confidence: the “perfect Portuguese” pattern doesn’t survive long text. And on the probability probes it’s weak, giving 87% to a fair coin. Microsoft advertises calibrated probabilities; my probes disagree. And like every dated-ID hosted model, the weight behind it can change without notice.

For a Portuguese-language operation that wants Jev quality while paying less, it’s the first candidate to run in your pilot.

You want the same level with open weights: Clef

The full 27B Clef tied with Jev in English (82 of 84) and finished 3 cases behind in Portuguese. It beat Jev on sensitive data handling in English (12 of 12 versus 11). It’s an open-weights model, so you can run it locally with about 54 GB of memory just for the weights in BF16, which I didn’t test: the full Clef ran only through the API. Among the open ones, it lost the crown to Perplexity, which scored higher with smaller weights.

Through the API it’s the most expensive of the five hosted ones, three times Jev, with no quality gain to justify it. The exception is the daily allowance: for small volume, it’s the best classifier you can get for free.

You want the cheapest in the cloud: Clef-flash

At US$ 0.038 per million, Clef-flash is the lowest price in the hosted market and covers 9,400 decisions per day within the allowance. Quality drops: 92% in English and 83% in Portuguese on the compact cases, the biggest language drop among the hosted ones, with an interval entirely below zero. On sentiment in both languages, and on incident status in English, it tied with Jev. On intent and sensitive data in Portuguese, it lost badly.

For sentiment tasks in either language, or for English classification where 1 error in 12 is acceptable, it does well. For Portuguese, measure first.

You already live in the OpenAI ecosystem: OpenAI Decisions

OpenAI’s debut in this market landed mid-table: 90.5% combined, an exact tie with Jebadiah, with the lowest hosted latency I measured (222 ms) and the same accuracy in both languages. It costs US$ 0.23 per 10 thousand decisions, between Jev and Clef. If your operation already speaks the OpenAI API and 9 out of 10 accuracy is enough, integration is trivial: a native decisions endpoint, no prompt at all.

The weak spots are revealing. Its worst cell was precisely numeric-threshold policy (8 of 12 in English, 7 in Portuguese): it treated “at least 10” as satisfied by a range of 8 to 12 and failed the AND/OR short-circuit. The calibration is schizophrenic: on the predicate probes (yes/no) it was the most accurate of the bunch, with near-zero average error, and on the choice probes it returned 94%/6% on a fair coin. And on the v1 variants it flipped 2 correct answers in each presentation (reversed order, opaque label, malicious comment), something only Perplexity didn’t do: Jev kept its aggregate accuracy, but also flipped some correct answers along the way. Oh, and the gpt-6-luna alias is not an immutable snapshot: reproducibility requires saving the responses.

You want the best open model, period: Perplexity Decider 27B

Perplexity Decider 1.1 27B is the surprise of the benchmark. It led the table with 99.4% on the compact cases, was the only one that didn’t lose a single task cell to Jev (3 wins, 11 ties), and hit 100% in Portuguese under all three conditions: compact, with examples, with distractors. Calibration is also the best measured, with a Brier of 0.012, and coverage is absurd: it answered with confidence above 0.9 on 96% of cases and got zero of them wrong. On v1, it didn’t flip a single answer across the order, opaque-label and malicious-comment variants.

Its only 3 errors on v2 were the same case in different clothing (it confused “notification sent to managers” with “mandatory approver”), and it made them with confidence below 0.9: the known error comes flagged. The weak spot is event-probability probes: asked about an event with a 1-in-4 chance, it answered 0.73. Strong classification does not imply reliable event probability.

The cost is hardware: about 49 GB of weights in BF16, a second and a half per decision on my Strix Halo (with ROCm’s reference kernels, no flash-attention; on a good NVIDIA it should be much faster, but that I didn’t measure). It’s Apache 2.0, public weights with no gate. If you have the machine and want the top of the table at home, this is the one.

You want the other open-weights top: Decision 3.0 (and beware the Edge)

Decision 3.0, released two days ago by the vLLM Semantic Router project, is the second model to sit at the top: 99.4% on the compact cases, 100% in Portuguese, and a tie with Perplexity so absolute that the two gave exactly the same labels on the 1,024 classifications, including the same two misses (the English relevance one, at confidence below 0.9, and the disputed severity case that already lives in the errata). It also cleared the v1 traps without flipping a single answer, got both indisputable critical tickets right, and its calibration is in the same class as Perplexity’s (Brier 0.014, 94% coverage above 0.9 with zero errors).

In other words: the open top now has two independent recipes that converge. For your pilot, either one works; run both if you can, because the probabilities differ even when the labels don’t. Latency is in the same family as Perplexity (1.4 s on my machine, same ROCm diet), and the weights take the same ~49 GB.

The little brother is another story. Decision 3.0 Edge (0.6B) answers in 45 ms and fits in 1.3 GB of GPU, but scored only 61% on compact, plummets 14 points in Portuguese, and worse: of the 68 guesses it gave with confidence above 0.9, 14 were wrong. As a confidence-threshold router, out of the factory, it’s a trap from the same class as Julia. The four middle checkpoints (flash, mini, nano, lite) I haven’t tested yet; they’re the ones that will tell whether the family has a sweet spot between the two extremes.

You need to run locally, with quality: Jebadiah 4B v2

Of the local ones, Jebadiah 4B v2 had the best compact accuracy, 90% combined, answering in about 200 ms. It tied with Jev on six of the fourteen task cells, including intent, implication and status in English and intent in Portuguese. The weak spot is numeric-threshold policy, 9 of 12 in both languages, and it loses ground when the context fills with distractors (76% in Portuguese under that condition, versus 82% for Mapika).

Mapika Decider 4B (the fastest of the 4Bs) and Kev 4B are right behind, both around 87%, and the 3-point difference between the three fits inside the error interval. They’re three different training recipes on top of 4B backbones, with errors on different tasks. The choice between them depends on your task, and I’d run all three in a pilot.

You will send examples in the prompt: Kev 9B

Kev 9B has a curious profile. On the compact cases it ties with Kev 4B, half a point apart with the interval crossing zero, meaning more than double the parameters bought nothing. With one solved example per class, it opens almost 10 points over the 4B, and with distractors it holds 7 more points. Brier also improves by about 25%: the probability it returns is better calibrated.

If your use is just the rule and the text, the 4B is enough and is more than twice as fast. If you’re going to send examples or long context, the 9B is worth the memory. The warning is Portuguese: 83% compact in PT-BR, almost 10 points below its own English, the biggest language drop among the models above 80%. Only Laya typed (14 points) and Julia (13) drop more.

You want Clef-flash behavior on your machine: local Clef-flash

On the v1 set, local Clef-flash reproduced the hosted one’s 520 classifications, hit for hit and miss for miss. On v2 the story changed: same total (439 of 504), but 19 different labels, with swapped strengths (the local one gets more Portuguese intent right, the hosted one gets more relevance right). Among the local ones it’s still the best on the v1 operational-policy set (90.8%), ahead of every 4B.

The gotcha is latency: 618 ms median on the Strix Halo, three times Jebadiah, for the same v2 accuracy. If you pick it, it’s for the known behavior.

You need to run offline on weak hardware: Laya and Julia, with many caveats

Laya multilingual was the worst in the benchmark for accuracy (37% combined, near the coin), and the fastest by far among the ones that get anything right: 19 ms median, more than ten times faster than Jebadiah.

The three Laya checkpoints have between 322 and 421 million parameters, fit in under 1 GB of BF16 weights by parameter math, and the project documents CPU execution and an ONNX runtime (which I didn’t measure). PyTorch’s peak GPU allocation for the multilingual one stayed under 2 GB. A 4B needs about 8 GB just for weights at the same precision.

Julia-1 goes one step lower: 144 million parameters, 550 MB of weights, 12 ms median on GPU and 71 ms on a 4-thread CPU. It’s the fastest model I’ve ever measured here, and also the worst: 31.5% accuracy, glued to the 31% coin-flip line. In Portuguese it answered “cancellation” on all 12 intent scenarios, including the ones asking for a refund or access (the vendor’s Portuguese evaluation was done in pt-PT, and it shows). The bigger disaster is confidence: of the 71 guesses it gave above 0.9, 48 were wrong. A confidence threshold won’t save this model, because it’s exactly on the confident guesses that it errs.

That’s a design choice. Laya and Julia were made to be small encoders, fine-tunable on your task, to run at the edge. What I tested were the factory checkpoints, with no tuning, applying rules they had never seen, on broad tasks. In that scenario they did badly, period.

One detail confirms the fragility. On v1, when the reviewer’s malicious comment comes in, Laya drops from 44% to 19% and typed-decisions to 15%. It obeys the comment the instructions say to ignore, and confidently: 17 of the 28 guesses above 0.9 were wrong. Julia failed all 5 policy-counterfactual pairs: change the rule, it keeps the answer.

If you want what Jev delivers, factory Laya and Julia are very far from it, and no small model (GLiNER included) came close. If your problem is “I need to classify short text into three buckets, on a device with no internet and no GPU, and I have a thousand labeled examples to fine-tune on,” these two are the only ones on the list that fit. Fine-tuning is the part I didn’t test. It’s another question, and it needs another benchmark.

The baseline: Simple-Jev with a stock Qwen

Simple-Jev reading tokens from Qwen3.5 4B, with no decision training at all, scored 74% on compact v2. That’s the yardstick for “how much specialized training buys”: Jebadiah, Mapika and Kev, all on similarly sized backbones, came in 13 to 16 points above. The training buys a lot.

The curious part is that on v1 Simple-Jev tied with Jebadiah (111 of 130 originals) and got both unambiguous critical tickets right, which several better models missed. It’s also the slowest of the bunch, almost a second and a half per decision, because it’s running a whole LLM.

How hard it is to run the best open models on your machine

All these models speak the same HTTP protocol, called SystemOne, which Jev popularized: you send a state (the text), one or more closed-answer questions, and get back the chosen option and the probability of each. Swapping models is swapping the port. OpenAI Decisions breaks from the pattern with its own native endpoint, but the idea is the same. What changes is how much work it takes to get each one standing, and there the difference between them is big.

OpenAI: just an API key

The easiest of all: create the key, load credit and call the endpoint. The benchmark’s 1,048 requests ran in 4 minutes, without a single retry. The only hiccup was the first probe returning “balance exhausted” until I put US$ 20 in the account.

Microsoft Decision-1 is almost as simple: if you already have an OpenRouter account, it’s point at the model and call the decisions endpoint. No GPU, no Azure subscription, no installation. In my test, 10 of the 504 v2 calls took a 429 and passed on retry, so plan for retries.

Kev and Jebadiah: one pip and one command

Kev is the simplest of the 4B crowd. The project installs straight from GitHub and brings up a server with one command, downloading the LoRA adapter and the decision head on top of base Qwen3.5:

pip install "kev @ https://github.com/jaredpalmer/kev/archive/fc4a17e194bca35a48c57211392ec49ea54a94ac.tar.gz"
python -m kev.serve --run jaredpalmer/kev-4b --host 0.0.0.0 --port 8000 --device cuda

Jebadiah follows the same pattern, with the server package inside its own repository:

pip install "jebadiah-server @ https://github.com/getainode/jebadiah/archive/e9793abc66d1a7c5ad878d51eda656f1e9c5b9f5.tar.gz#subdirectory=server"
jebadiah-serve --model frontier-infra/jebadiah-4b-v2 --host 0.0.0.0 --port 8000

With either one up, a decision is a POST:

curl -s http://localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
  "model": "kev-4b",
  "state": "Cliente diz que foi cobrado duas vezes e pede o estorno da segunda cobrança.",
  "questions": {
    "fila": {
      "type": "choice",
      "instructions": "Escolha a fila correta pela política de suporte.",
      "criteria": {
        "financeiro": "Cobrança, estorno, nota fiscal.",
        "tecnico": "Erro, bug, indisponibilidade.",
        "vendas": "Upgrade, plano novo, orçamento."
      }
    }
  }
}'

The response carries the option and the probability distribution, and that’s where you apply your threshold, log and decide whether to send it for human review. On an NVIDIA with CUDA and 16 GB of VRAM, a 4B in BF16 comes up without drama. The weights are about 8 GB.

Laya and Julia: smaller still

Laya is a Python package with an encoder inside. It installs with pip install laya and the project documents CPU execution, which is its whole point. The version pinned on my server was 0.4.1, and the multilingual one’s peak GPU memory stayed under 2 GB. To run on a GPU-less laptop or a small container, it’s the only one on the list that fits, no discussion.

Julia-1 is similar in spirit: its own Python runtime, one predict(state=..., questions=...) call and done, no standard Transformers pipeline. I ran the same container on GPU and on CPU, and both gave the same accuracy with the same responses in 97% of cases. On a 4-thread CPU it answers in 71 ms, which for an edge classifier is great. The problem is still that it scores at coin-flip level.

Where it gets complicated: AMD, ROCm and the rest

In my case the hardware is AMD, and then “one command” turns into a day of work. The points that cost me time, to save you yours:

• None of these projects test on ROCm. Kev had a gfx1151-specific segfault caused by CUDA graphs (issue 170), fixed only after the 1.0 tag, so the pinned version is a main-branch commit and the official tag crashes.
• The flash-linear-attention Triton kernels Qwen3.5 uses in the DeltaNet layers have no tested support on gfx1151. Everything ran on the pure-PyTorch path, slower than the latency published on CUDA. Jebadiah’s 200 ms in the table is with that handbrake pulled.
• Mapika Decider forces CUDA graphs when it sees the “cuda” device, which on ROCm is also called “cuda.” I had to turn it off by hand.
• You need to export HSA_OVERRIDE_GFX_VERSION=11.5.1 and PYTORCH_ROCM_ARCH=gfx1151 and use a base image with ROCm torch already validated. Any random pip install torch overwrites it with the CUDA build and breaks everything. In my Dockerfile every package is installed with --no-deps and the build fails if torch stops being the ROCm one.
• One model per container, one container at a time. The Strix Halo has memory shared between CPU and GPU, and three resident 4B models plus the Ollama that already runs there exhaust the 64 GiB the GPU can see.
• Perplexity Decider was the most annoying of all. The release expects a specific CUDA torch (I swapped in the ROCm 7 torch 2.9), the flash-linear-attention kernels don’t exist for it (I ran on PyTorch’s reference kernels, slower), and the default loader blew past the host’s 24 GiB RAM limit: I needed a custom loader that sends the 49 GiB of weights straight to the GPU. The model card still said “private” when I started, but the Hugging Face API confirmed it was public and gate-free, and the shards downloaded anonymously. The d3, at the same size, inherited the same treatment. The d3-edge, luckily, comes up with the standard Transformers loader and 1.3 GB of GPU.

None of this is impossible, and it’s all documented, with Dockerfiles, compose and the hashes of each checkpoint, in the repository’s setup guide. But it’s a real cost that the “local is free” spreadsheet never includes. If your machine is NVIDIA, most of this list disappears.

The full 27B Clef I didn’t even try locally: it’s about 54 GB of weights in BF16 before any activation. Perplexity, at 49 GB, proved a 27B runs on the Strix Halo, at the cost of a second and a half per decision with the reference kernels. The 9B Clef-flash runs, with about 18 GB of weights. The 4Bs are the sweet spot for consumer hardware.

Highlights and gotchas

Laya truncated all long Portuguese. Under the distractor condition, base Laya reported truncation on all 84 Portuguese calls, discarding almost 31 thousand state tokens, and on none of the 84 English ones. The English checkpoint’s default context limit is only 512 tokens (the multilingual and typed ones use 1,024, configurable up to 8,192), and Portuguese tokenizes heavier. The other models don’t even expose truncation telemetry, so “zero truncation reported” on them means “I don’t know.”

Portuguese costs more tokens. For the same 84 case pairs, Jev reported about 14% more input tokens in Portuguese than in English, and both Clefs about 15% more. If you budget a bilingual operation by the English average, you’ll come in low.

Does an example in the prompt save a weak model? It saves nothing. One solved example per class left Laya the same or worse; several of its aggregates dropped. It helped Kev 9B (plus 4 points) and Clef in Portuguese (plus 3.6 points, reaching Jev’s level), and almost nothing on Jev, one more case in Portuguese. Validate examples on real traffic before assuming they help.

Size correlates, and explains little. Among the 19 competitors of known size, the Spearman correlation between parameters and compact accuracy was 0.96. The Kev series is more informative: 63% at 0.8B, 87% at 4B, 88% at 9B. The big jump is from 0.8B to 4B, about 24 points, with an interval between 16 and 33. From 4B to 9B the compact difference crosses zero, and several nominal 4Bs, with similar backbones, gave very different results. In the d3 family, the flagship and the Edge sit almost 40 points apart under the same release, while a 9B, Microsoft, ties at the top of the hosted ones. Size helps up to 4B; past that, the training recipe rules.

The 27B Clef missed a critical ticket that two of the Layas got right. On v1, one severity scenario says continuous data loss in production is critical, regardless of how many customers it affects. The record describes confirmed permanent loss, from a single customer. Jev, Clef-flash, Perplexity, d3, Microsoft, Laya typed, Simple-Jev and Laya multilingual answered “critical.” Full Clef, OpenAI, base Laya, Eikos, Jebadiah, Mapika, Kev 4B and Kev 9B answered “degraded.”

It’s a single case, and an error rate would require many more. But it shows that the average hides exactly the error that costs the most. When the impact facts are structured, the severity rule belongs in code, and the model comes in only to extract the facts from free text.

Hosted and local diverge over time. Local Clef-flash reproduced the hosted one on v1’s 520 classifications and diverged on 19 of v2’s 504. Cloudflare keeps the weights version behind the hosted alias undisclosed. If reproducibility matters, pin the local hash.

High confidence and safety are different things. On compact v2, Jev, Clef, both Clef-flashes, Eikos, Jebadiah, OpenAI, Microsoft, Perplexity and d3 got zero guesses above 0.9 wrong. Perplexity is the extreme case: it answered at that level on 96% of cases without missing one, with d3 right behind (94%, zero errors). Microsoft covered 95% and OpenAI only half. Microsoft’s caveat: the clean record is only on compact, because with examples or distractors it misses in Portuguese with confidence above 99%. Mapika missed 5% of the guesses at that level, Kev 4B 3% and Simple-Jev 14%. And the two smallest of the bunch broke the pattern violently: Julia missed 48 of 71 guesses above 0.9, and d3-edge missed 14 of 68. Coverage varied too: Jev answered above 0.9 on almost 90% of cases, Clef on almost 80%, Clef-flash on 40% and base Laya on 2 of 168. On v1 Jev missed 1% of the guesses above 0.9 and base Laya missed 61%. A confidence threshold only works after being validated on your distribution, and Laya typed’s low ECE in Portuguese left it at the same 36% accuracy.

Perplexity didn’t fall for presentation traps. Across the three v1 variants (reversed options, opaque labels, malicious comment), it kept the same answers as the original, without flipping a single one. The d3 repeated the feat, zero flips too. Microsoft only stumbled on the opaque labels (2 flips, one in each direction); order and comment didn’t move it. OpenAI flipped 2 in each variant, and Laya collapsed to 19% under the comment. The price of that robustness is latency: a second and a half per decision on my machine, the worst median of the bunch.

Numeric thresholds are a code problem. The threshold-triage task, with clear numbers and an explicit rule, gave 100% on Jev and both Clef-flashes and under 30% on both Layas in English. An if solves that with 100% and zero latency. The model only comes in when the number is buried in free text, and even then the right thing is to extract the number and leave the comparison to code.

Lottery probability is another skill. On the 24 no-information probes (coin, die), Jev had the worst choice error among options. The numeric-estimate crown changed hands with OpenAI’s debut: its yes/no predicate missed almost nothing, while the choice probe returned 94% for heads on a fair coin. Perplexity, champion of everything else, answered 0.73 for an event with a 1-in-4 chance, and Microsoft went the same way, giving 87% to the coin. Laya typed had the second-best choice error, behind only Kev 9B and neck-and-neck with Clef, and still classified badly. It’s the “student who marks answer A” from Paulo’s essay, now measured: calibration on lotteries and accuracy on classification are independent measures.

Where the data converges

1. The overall top is open-weights, and now there are two. Perplexity Decider 27B and Decision 3.0 tied at 99.4% on the compact cases, with 100% in Portuguese and identical labels on the 1,024 classifications, both with Apache 2.0 weights running on my machine. Their edge over Jev is under 2 points and the interval touches zero, so treat it as a snapshot of this set. Their cost is hardware and latency, not API. In the cloud, the evaluation order changed: start with Microsoft Decision-1, which ties Jev on the aggregate, closes 100% in compact Portuguese and costs a third. Jev comes next, faster and with different errors, because an aggregate tie is not interchangeability. OpenAI Decisions landed halfway in price and accuracy.

2. Running locally became viable at 4B. Jebadiah, Mapika and Kev 4B deliver between 87 and 90% on the compact cases with 200 ms latency on a desktop machine, and in a pilot with your task any of the three can win. If your machine handles 49 GB of weights, Perplexity delivers the top of the table at home. The financial math justifies using the machine that already exists, for data control or because of an obligation to stay offline. Buying a machine for this it stops justifying.

3. Small models are another category. Judging Laya and Julia by the accuracy column is reading the table wrong. They get half or more wrong out of the factory, fall hard under a malicious comment, and together with the d3-edge they’re the only ones on the list that fit on a weak device and answer in tens of milliseconds. The Edge is the best of the three on accuracy (61%), but it plummets in Portuguese and gets one in five confident guesses wrong. If Jev-like accuracy is a requirement, cross all three off. If running offline on small hardware is a requirement, cross everything else off and start with fine-tuning them on your task, which is the missing measurement. And be careful with Julia and the Edge in particular: 48 errors in 71 and 14 in 68 confident guesses are the portrait of models that don’t know what they don’t know.

4. Portuguese costs about 14% more tokens and drops the models in the middle of the table by 1 to 10 points, with Kev 9B and Clef-flash in the worst case. Jev loses nothing, OpenAI ties both languages, and Perplexity, d3 and Microsoft close 100% in compact Portuguese, better than their own English, although Microsoft goes back to missing in Portuguese when the context grows. Julia goes the other way and plummets 13 points, and the Edge plummets 14. For a Brazilian audience, that’s the first column to look at, before price.

And what holds for everyone: none of these models replaces the rule that’s already in code. Numeric thresholds, severity policy with structured facts, tool permission, all of that stays in the if. The model comes in to turn free text into structured fact. And the test that matters is yours, with your tickets, because each cell in this table has 12 scenarios and your operation has thousands.

The cases, each model’s raw responses, the scoring scripts and the guide to reproduce everything are in the benchmark repository. If you run it with your tickets and get a different ranking, tell me.

TL;DR for those who only want to know which one to use: Jev is a hosted classifier, good and cheap for repeated decisions over text, and not a silver bullet. If you want the direct comparison, skip to Clef, Cloudflare’s open alternative with weights and image input; to Laya, the small encoder to run locally and fine-tune on your task; to the public ranking, with 70 models side by side; or straight to how to decide in your project.

When Jev showed up, I didn’t think much of it. The homepage was a festival of jargon, Kahneman here, Jevons there, a “193x faster” with no benchmark next to it, and my rule for new AI tools is not to adopt any intermediate layer until it hurts not to have it, so I didn’t spend time on it.

In that post I concluded that, marketing aside, Jev is a hosted classifier: you send a text and closed-answer questions, it returns the option and the probability, without generating any prose. And that the “193x” only impresses because the comparison is against a chat LLM writing a paragraph so you can fish a label out of it afterwards.

What I hadn’t registered at the time is the size of the habit Jev exposes. A lot of people were using a frontier LLM as an expensive stand-in for an ordinary classifier, and had never thought about a classifier, because they never had to: the LLM solved it, badly and expensively, and nobody measured.

For me, a classifier would be the first idea, which is why Jev didn’t look like news. For many people it was a revelation. That gap is the subject of this post.

This post is a collaboration

Much of what you’ll read here is by Paulo Câmara, neuroscientist, co-founder and CTO of LUA Vision, the Brazilian company whose model I tested and investigated in recent weeks. He wrote an essay, “The student who marks answer A,” to be published here, in which he read Jev’s fine print, opened the roughly 350 test repositories that appeared on GitHub in launch week, and recomputed the four main results from scratch.

The warning Paulo gives at the end applies: he founded a company that builds its own model and competes in the same market. Read with that discount, and check the numbers at the source, all of which he links.

First: what a classifier is

A classifier is a function that takes an input and returns one of a fixed set of categories, usually with a probability for each. This email is spam or not. This ticket is billing, support or sales. This transaction looks like fraud.

The list of possible answers is closed, and you know it before you run.

The classic way to build one is with labeled examples: a set of texts where someone has already given the right answer. On top of that you train a model, which can be something as simple as logistic regression or naive Bayes, or a small encoder like BERT with a few hundred million parameters. In every case, the final model is small, runs locally, answers in milliseconds, costs almost nothing per call, and returns numbers you can calibrate and audit against your own data. This has existed for decades and is undergraduate material.

So when someone with a computer science background sees a problem like “decide which bucket this text falls into, thousands of times a day,” the first question is “how many labeled examples do I have?”, not “which LLM do I use?”.

A chat LLM is a text generator with billions of parameters. Using it to return a label means paying per output token, waiting seconds per answer, dealing with a format that sometimes breaks, and accepting behavior that changes when the vendor updates the model. It can be done, and zero-shot classification with language models has been documented since 2019. But it’s the expensive, slow option, and it only pays off when you have no examples at all or when the categories change with every request.

Jev lands exactly in that space. It’s a classifier that accepts categories defined on the fly, several questions about the same text in a single call, hosted, cheap per token and fast.

Compared with a chat LLM generating a paragraph, it’s an engineering win. Compared with a classifier trained for a stable task, it’s a convenience that trades control for flexibility. Keep those two yardsticks in mind, because the rest of the post goes back and forth between them.

Paulo’s essay: “The student who marks answer A”

What follows is Paulo Câmara’s essay, written for this post. The subheadings are his.

Every week an AI savior shows up. This week’s is called Jev.

It launched on September 15, with US$ 40 million in the bank, a US$ 200 million valuation and a promise that fits on a billboard: two hundred times faster, four hundred times cheaper, calibrated and incapable of hallucinating. Within 24 hours, 13% of the paid teams on Vercel’s gateway were already using it. Within a week, there were more than 300 repositories on GitHub testing the thing.

I did what I do with every savior. I read the fine print, opened the roughly 350 repositories, and redid the math for the ones that published raw data. What I found fits in the image above: an answer sheet with answer A marked on every question.

When there is no information at all to decide on, Jev marks the first option. In 3,000 of 3,000 draws. With up to 92% certainty. Keep that image, because it explains the rest of the text.

What it is, without the billboard

Strip away the marketing and what’s left is a multiple-choice exam with no essay section. You send a text, ask closed-answer questions (choose among up to 255 options, give a score on a scale, say yes or no) and it returns the probability of each answer. It doesn’t write a line. It doesn’t explain. It doesn’t do math. It doesn’t call tools.

That’s good, and I want to start with what’s good. Many companies today use a frontier LLM as a luxury if statement: send the document, wait twenty seconds for the model to think out loud, pay for every word it writes, validate the JSON, and only then find out the ticket was about billing. Jev cuts that theater. It reads the text once, scores each option, done. That’s why output is free and ten questions cost almost the same as one.

The mechanism, though, isn’t new. Reading the score of each possible answer is how language models have been evaluated on multiple choice since GPT-2. It’s how the reward models of RLHF work, a technique TypeSafe’s founder helped create at OpenAI.

A few days after launch there was already an open copy imitating the interface with a public model, and ten days later Stanford and Nvidia released an open-weights competitor. TypeSafe published no paper, no weights, no model size and no result on a public benchmark. In the announcement’s FAQ, “Is Jev just a smaller LLM?” appears as a question. The answer is one line: “it is neither small nor an LLM.” That’s it.

One opponent for each headline

The homepage says “193.6× faster, 444.6× cheaper.” I redid the math with the table the company itself published. The 444.6 is the cost of Opus 5, the most expensive model on the panel, divided by Jev’s cost. The 193.6 is Sonnet 5’s time divided by Jev’s time. One opponent for each headline.

It’s like advertising that your car is more fuel-efficient than a Ferrari and faster than a truck, and letting the reader conclude it’s the best car in the world.

How many times cheaper Jev is, depending on who is on the other side. Source: TypeSafe’s public table (evals.typesafe.ai) and independent tests against non-reasoning LLMs.

Against the cheap model that scores about the same, Luna, Jev is 8 times cheaper. And that’s with every LLM in the comparison thinking before answering, something nobody does to route a ticket. Against a non-reasoning LLM, the independent tests measure about 4 times on cost and 2 to 7 on time. Still a lot. But an order of magnitude less than the billboard.

There’s a detail almost nobody read. There is no answer key in that evaluation. “Accuracy” is how much each model agrees with the average of two frontier models. Jev agrees 67.8% of the time, the same as Sonnet 5. Opus 5 is at 73.1%.

Agreement with the reference versus cost per case, average of TypeSafe’s four workflows. The green band runs from 66.8% to 67.9%: Luna, Sonnet 5, Terra and Jev.

Breaking it down by task shows what the average hides. On invoices, which require checking amount, quantity and date, Jev is 17 points behind the best model (61.8% versus 79.1%) and below even Luna. Its documentation warns that it “is not a calculator.” At least that is true.

Agreement by workflow. Source: evals.typesafe.ai.

And the most solid data on TypeSafe’s site isn’t even about Jev. All eight LLMs evaluated score higher when the task is broken into small questions and the logic stays in code. Haiku 4.5 goes from 18.1% to 53.6%. The most useful lesson of the launch is an engineering one, and it applies to any model.

What the price gives away

Price is technical data in disguise. At US$ 0.042 per million tokens, a rented GPU needs to read 13 to 20 thousand tokens per second just to break even. In a scenario calculation, that squares with a model of 6 to 37 billion active parameters, which is a small model, or with a subsidy. TypeSafe wrote, with a candor I respect, that “there is no way to prove it isn’t subsidized.”

Since launch, the per-account usage limit dropped from 250 thousand to 100 thousand tokens per second. The documentation warns that it changes “without notice” until the “large GPU deals” land, and the subprocessor list shows inference running on an on-demand rented GPU cloud. It’s a frontier model with startup capacity.

In the same calculation, each decision costs about 14 joules. That’s Jev’s best argument, and I’ve defended that yardstick for a long time: accuracy per joule, not just accuracy. There, it’s genuinely good.

Exam Portuguese

I found only one test of Jev in Portuguese: ENEM 2025. I redid the math from the raw answers. Jev got 103 right against 102 for an open 120-billion-parameter model. A statistical tie (p = 1.0).

Jev’s accuracy on ENEM 2025 by subject area, 182 valid questions, with 95% margins. Recomputed from the patryckalves/jev-no-enem repository.

Jev reads exam Portuguese better than many prep courses: 78% in Languages. And does Math like someone guessing: 25%, with a guess worth 20%. In Spanish, another audit measured a loss of 3 to 6 points relative to English and 17% to 38% more tokens for the same text. It’s the token tax I measured in my paper, charged again, now by another vendor.

One detail in its favor: the explicit “invalid” option in that experiment separated out the hardest questions. That’s the abstention route worth demanding in production, measuring alongside it how much it leaves unanswered.

The student who marks answer A

Back to the answer sheet.

An auditor asked Jev to predict a fair die it had no way of seeing. The honest answer is 1/6 for each face. I downloaded the 1,000 answers and counted. In every one, Jev chose the first option on the list it received: “1” on the die, “red” on the color die, “heads” on the coin, “north” on the spinner. With 76% to 92% certainty.

A second researcher repeated the same question 2,000 times, independently. Same result. The caveat: this holds for choice questions; in the yes-or-no-with-probability formulation, the same repository shows Jev closer to the reference, still overestimating low probabilities.

When it doesn’t know, it doesn’t say it doesn’t know. It marks answer A, with conviction.

Stated probability versus actual frequency. Draws: 1,000 answers recomputed (KantaHayashiAI). Heart risk: 5,000 people from BRFSS, recomputed (rubinagentagi-tech). Items with human disagreement: pre-registered study with ChaosNLI (GautamTalksDev). ENEM: recomputed (patryckalves).

The chart shows the whole pattern. Where it has signal, as in ENEM answers above 0.9, what it says and what happens coincide: 97.1% confidence, 97.6% accuracy. Where it doesn’t, the certainty stays and reality leaves.

In a test with 5,000 real people, the heart disease rate was 9%. Jev assigned, on average, 27%. I recomputed: its probability is worse than simply saying 9% for everyone.

System 1, literally

TypeSafe calls Jev a “System One model,” a nod to Kahneman’s fast thinking. It’s the most honest part of the product, just not for the reason the company would like. In the book, System 1 is fast, cheap and automatic. It’s also where overconfidence lives: it builds the most coherent story out of what’s in front of it and doesn’t register what’s missing.

Kahneman spent years disagreeing with Gary Klein about when to trust expert intuition. In 2009 the two published a joint paper with the most elegant title in psychology: “a failure to disagree.” The conclusion fits in one line. Intuition only deserves trust in a high-validity environment, where there is regularity and there has been feedback to learn it from. Outside that, confidence stays high and stops saying anything about accuracy.

That’s Jev, exactly. Classifying English text is a high-validity environment, and there it does well. Predicting risk, drawing lots, guessing the company rule nobody told it about: a low-validity environment. There it stays confident.

In the lab, we separate two things: deciding, and knowing whether you decided well. The second is called metacognition.

It’s measured with signal detection theory, Maniscalco and Lau’s meta-d′, and it depends in part on prefrontal regions distinct from the ones that support the decision. Fleming and colleagues found that in the anatomy in 2010, and Rounis and colleagues, the same year, showed that magnetic stimulation of the prefrontal cortex knocks down metacognition without touching accuracy (a result whose replication is disputed, by Bor and colleagues in 2017). That’s why there are people who get it right without knowing they did, and people who get it wrong with conviction to spare.

The field the company calls confidence is a formula over the same distribution that produced the answer: in a two-option choice, 2p − 1. That isn’t a defect in itself. An ideal observer does exactly that and gets it right.

The problem is that Jev’s distribution loses calibration outside what it knows, and nothing in the system checks the answer by another route. When the signal disappears, the answer becomes wrong and the confidence stays the same, because the two are the same number. In a pre-registered study with items on which a hundred people disagree, human agreement is 47% and Jev carries on at 81% confidence. When the coordinates of a spatial task were shuffled, accuracy collapsed to 5% to 22% and confidence dropped by 0.028.

A model that says 83% about a die it never saw isn’t calibrated. It’s assertive.

Attention gets captured

I studied attention in my doctorate, and I expected to see capture: the loudest stimulus winning. That’s not what the tests showed. The shout, the famous “ignore your instructions,” barely gets through. What gets through is the plausible sentence with no checked origin. Psychology has a name for this: source monitoring failure. The system treats a claim nobody verified as if it were fact.

Decisions flipped by attack type. Crude attack: 1 in 1,056 (cwhy). Injected command and unverified opinion: 9,744 variants (Hu et al., 2026). Optimized fluent context: 312 of 508 correct decisions (Xu, 2026).

In numbers: the crude attack gets through 1 in 1,056. A single unverified opinion pasted at the end of the text flipped 12.1% of decisions, statistically tied with the strongest injected command. A line saying “the closing administrator has confirmed this article was kept” dropped accuracy from 96.5% to 26.5%.

A fake field saying the user had pre-approved rm -rf ~/.ssh lowered the chance of blocking from 0.76 to 0.48. It recalls Loftus’s misinformation effect: suggested information rewrites the judgment.

The capture also applies to cues that shouldn’t count. With the same text, saying only “the writer is Black” or “the writer is white” changed the death-penalty rate Jev assigned from 29.5% to 62.7%. The direction is beside the point. A life-or-death decision moved 33 points because of information that had nothing to do with the case.

There’s one more I can’t let pass. Changing only the option labels, from “0/1” to “no/yes,” without changing any definition, flipped 70 out of every 100 answers. The type error rate stayed at zero. The format was perfect. The decision was backwards.

These numbers come from preprints and small studies. The exact percentage changes with prompt, task and model version; the class of failure does not.

Probability that doesn’t add up

Probability has a first-semester rule: the chance of something plus the chance of its opposite equals 1. In TypeSafe’s own documentation, “is the customer asking for a refund?” gets 0.72, and “is the customer asking for something other than a refund?” gets 0.47. Sum: 1.19.

The pair isn’t an exact complement, and TypeSafe doesn’t promise that two separate questions form a joint distribution. But it’s the example the company itself chose to show off the product, and it doesn’t add up. The fix is a design one: a single choice question with mutually exclusive options, instead of two yes-or-nos added together.

People get probability wrong in many ways, but on this one they tend to get it right. When the question comes as a pair, the chance of something and of its opposite sum close to 1. Tversky and Koehler showed that in 1994. Jev gets wrong what people get right. The same decision changes value depending on the wording of the question, and that is the opposite of “epistemically honest probabilities.”

“It doesn’t hallucinate”

The announcement says Jev “cannot hallucinate” and shows 0% type errors. The zero is true and irrelevant. Since it can only answer within the options you gave it, getting the format wrong is impossible. Calling that not hallucinating is like praising the answer-A student because he never leaves a question blank. True. So what?

In a model that chooses, hallucinating is choosing wrong with conviction. On fact-checking, Jev ties GPT-6 Astra on accuracy but calls an unsupported claim “supported” 23% of the time, against 14% for Astra. In 76% of the situations where there was no tool at all, it chose to call a tool. With a company rule it wasn’t told about, it got 19 of 24 decisions wrong, 15 of them with more than 90% certainty. Faced with an invented law, it abstained zero times.

Security

First, where your data ends up: the United States, AWS and a GPU cloud called Modal. Retention lasts “for as long as reasonably necessary,” with no deadline, and zero retention only exists on the enterprise plan.

The data agreement cites the European Union’s standard contractual clauses, the UK addendum and California law, and doesn’t have a word about LGPD. If you send Brazilian citizens’ data, the legal basis for the international transfer is on you, and that’s a question for a lawyer before signing. The promise not to train on what you send exists and is good. SOC 2 exists and only appears on request.

As an attack detector, Jev is good, with fewer false alarms than many LLMs: it caught 40 of 41 injections in a test with real pages. As a judge of content that carries the attack inside it, it gives way, as I showed above. The difference looks subtle and decides everything. In my GV Angels talk I said an agent is not an employee, it’s an access key that talks. A gate made only of Jev is a lock that opens when someone writes on the note that the owner authorized it.

The error that worries me isn’t the error with doubt. That one goes to review. It’s the error with 92% certainty, which crosses any threshold you configure.

The wave

A word about the evidence, because I used a lot of it.

Jev test repositories created per day on GitHub, from the metadata of 346 indexed repositories.

In one week about 350 repositories testing Jev were born. I classified all of them by what they let you check. There are 89 with raw data, code, pinned version and statistics, and 170 with data and code; the rest offer less than that.

Only 11 wrote the hypothesis down before running the test. The median is one GitHub star, which means almost nobody looked. And the wave of tests followed the wave of hype: peak on September 21, almost nothing after the 23rd.

That’s why I didn’t accept numbers on authority. The ones that support this text I checked at the source, and the four main ones I recomputed from scratch.

Verdict

True:

• Fast: 100 to 400 ms.
• Cheap: 45 to 323 times less than a frontier LLM on the same decision.
• Ties the top on narrow judgments, like checking whether a passage supports a claim.
• Reliable above 0.9 on data similar to what it has already seen, after you’ve checked that on yours.
• Good injection detector.

Sandcastle:

• 193.6× and 444.6×.
• “Frontier intelligence.”
• “Doesn’t hallucinate.”
• Calibration as a general rule.
• Safe as the sole guard of an agent.
• New architecture: nobody has seen it.

Jev is nobody’s savior. It also isn’t “just a classifier.” It’s a good piece of product engineering that wraps known technique in an interface automation needed, at a price that changes what’s viable. The product is new. What it claims is new, so far, is marketing: it calls format truth, task calibration honesty, and System 1 a virtue.

What will break first

• The agent gates built only of Jev, because the attack that breaks them is already published and costs cents.
• The systems that trust the factory threshold on new data, in another language or with an internal rule, because they’ll automate errors at 90% certainty.
• Whoever uses its probability as real risk, in credit, health or forecasting, because it inflates rare events.
• And the price advantage, because within ten days there was already an open copy and a faster competitor.

If you’re going to use it anyway

• Pin the version. The jev-latest alias swaps models on its own, and your calibration goes with it.
• Every question with a “none of the above.” Without it, it marks answer A.
• Shuffle the order and use neutral names for the options. Test whether the decision changes.
• Recalibrate with 50 to 300 of your own examples before choosing a threshold. In one test, calibration error dropped from 0.117 to 0.008.
• Mark everything that comes from outside as untrusted: email, page, tool output.
• Never let it authorize a destructive, financial or personal-data action on its own.
• Math, dates and business rules stay in code.
• Before signing, compare with the cheap baseline: an open model reading probabilities, or a trained classifier if the task is always the same.

Read with a discount

I founded a company that builds its own model and competes in the same market. Read with that discount. That’s exactly why every number here has an open source: you don’t need to trust me, you need to check.

And if someone sells you Jev as a savior, ask the question I ask every startup: which test, and from when? If the answer takes a while, that’s the answer.

Essay sources

TypeSafe. Jev announcement · models and limits · confidence · known failures of jev-1.13 · workflow evaluations · adapter for LLMs · privacy · data processing · trust center

Recomputed by Paulo from raw data. ENEM 2025 · blind draw · heart risk · fact-checking

Preprints. Hu et al., JevAdvBench · Xu, JevOut · Sun et al., Type-Safe Is Not Error-Free · Ibrahim and Zaki · Li et al., JEV-as-a-Judge

Independent tests. human disagreement (ChaosNLI) · die replication · dialect bias · 49 tasks against a non-reasoning LLM · calibration and negation · recalibration · injection in Wikipedia discussions · Spanish · tools · undisclosed internal rule · shuffled coordinates · crude attacks · injection detection · audit index

Press. VentureBeat, agent gates · VentureBeat, CLM-8B

Literature. Fleming et al. (2010), Science 329 · Rounis et al. (2010), Cognitive Neuroscience 1 · Bor et al. (2017), PLoS ONE 12 · Johnson, Hashtroudi and Lindsay (1993), Psychological Bulletin 114 · Kahneman (2011), Thinking, Fast and Slow · Kahneman and Klein (2009), American Psychologist 64 · Loftus, Miller and Burns (1978), J. Experimental Psychology: Human Learning and Memory 4 · Maniscalco and Lau (2012), Consciousness and Cognition 21 · Ouyang et al. (2022), NeurIPS · Radford et al. (2019), OpenAI · Tversky and Koehler (1994), Psychological Review 101 · Câmara (2026), Inherited Weights, doi:10.5281/zenodo.22899767

The open alternatives: Clef and Laya

Since Jev’s launch, the category has gained open competition, and that changes the conversation about “new architecture.” Two names matter.

Clef, from Cloudflare

On October 1, Cloudflare released Clef and Clef-flash: the same abstraction as Jev (one shared state, several typed questions, a distribution over the options), with a compatible API, open weights under Apache 2.0 and image input. The architecture is described: a frozen Qwen as base (27B in Clef, 9B in Clef-flash), LoRA adapters and a head that scores each allowed option directly, trained with classification and calibration objectives. In other words, the lineage Paulo pointed to, now with inspectable code.

That doesn’t prove TypeSafe uses the same thing, but it proves the product can be reproduced with known components. Cloudflare ran the Decision Index, an open, independent suite with dozens of classification, routing, retrieval and safety tasks, and published the table. Some points:

Reading the whole table: Clef comes out stronger on most tasks, Jev still wins some, and it’s an internal Cloudflare run, on a public suite that can be trained against. Hosted price is the other side: US$ 0.24 per million tokens for Clef and US$ 0.09 for Clef-flash, against US$ 0.042 for Jev. Jev remains the cheapest hosted service of the three, by a margin of 2 to 6 times, and remains text-only.

Laya (and Arbiter)

Laya attacks the problem from the other end. They’re small encoders, 322 to 421 million parameters (ModernBERT-large in the typed version), with a choice head, Apache 2.0, built to run locally and be fine-tuned for a recurring task. Arbiter packages them behind a Jev-compatible API, with batching and deployment on Mac and NVIDIA. Arbiter is only a serving layer.

Laya’s README is unusually explicit about its limits, and I’d rather repeat than soften:

• the base checkpoints score near chance on the project’s own typed-decision benchmark; the headline 0.766 accuracy comes from a checkpoint fine-tuned on that benchmark’s training split;
• the models ship overconfident, and calibration has to be redone in the deployment domain;
• the typed version’s context is 1,024 tokens, fine for tickets and short messages, insufficient for long documents without chunking;
• with more than 50 options, the classification head saturates, and Laya itself reports Jev doing better.

None of this is a defect in the idea. It’s applied machine learning the way it has always been: a small model is excellent when the task is stable and labeled examples exist, and broad zero-shot behavior demands a much larger pretrained model. Laya is the option when privacy, offline operation, predictable latency or fine-tuning in your domain matter more than asking arbitrary questions about arbitrary documents. It is, at bottom, the classic classifier from the start of this post, with a modern interface.

There’s also Kev-9B, another frozen Qwen 9B with LoRA and a decision head, Apache 2.0, that fits on a 24 GB GPU. Three open implementations in under three weeks say the same thing: typed decision models have become an implementation pattern.

The Decision Index: how to measure this yourself

All this discussion about “who’s better” only makes sense with a public yardstick, and one exists. The Decision Index is an open suite, unaffiliated with TypeSafe, built for typed decision models: state plus choice or yes-or-no questions, one probability per option.

The current version, 0.2.1, counts 38 benchmarks in five areas (tools and automation, retrieval and classification, knowledge and reasoning, language understanding, human taste), about 120 thousand requests in total, with two rules that matter: the score is chance-corrected (0 is random guessing, 100 is perfect) and an unanswered request counts as wrong. No truncating input, no filtering options, no tuning the prompt per benchmark.

The live board has 70 models plus Jev as the reference. I downloaded the board’s data file and rebuilt the table; below are the top and the names that appeared in this post. Latency is the median measured by the maintainers on an RTX PRO 6000, one request at a time; Jev’s is a hosted API, so it doesn’t compare with the others.

“Open weights” means a checkpoint published on Hugging Face; “open code over an open model” is a published inference technique on top of an off-the-shelf Qwen or Gemma, without its own checkpoint. The board doesn’t record each one’s license, so check before using in a product.

Three readings come straight off the table:

1. The only closed model on it is Jev. It leads, but by half a point over an open fine-tune of a 26-billion-parameter Gemma 4, and the board treats differences under 0.25 as ties. The top ten are all open models of 12 to 32 billion parameters with an adapter on top, exactly the recipe Clef describes.
2. The small zero-shot encoder sits at the bottom: Laya at 6, Lavoir at 9. That matches what Laya’s README admits and what I said about the classic classifier, which needs labeled examples to shine.
3. A 4-billion LoRA like Hopper delivers 41 points at 23 milliseconds, a range many real problems don’t need to exceed.

And how to read Laya at the bottom of the table? The Decision Index measures one thing only: answering, with no extra training, 38 kinds of question the model has never seen, from chess to taxation. That’s what a hosted service has to do, because it doesn’t know which question is coming. A 400-million-parameter encoder will never win that exam, and Laya doesn’t even try: the score of 6 is the base checkpoint, with no fine-tuning at all.

The same model fine-tuned with your examples, on a stable task, shows up at 0.77 on the project’s own benchmark. The ranking answers “who is the best zero-shot generalist,” which is the question of whoever sells an API. The normal programmer’s question is a different one: “on this repeated decision, with the examples I have, what gets more right at the cost I accept?”. For that one, the bottom of the table may be the right answer, and no public board replaces the test on your own data.

So is Laya useless next to the others? As a zero-shot generalist, yes, and the board is right to say so. As a programmer’s tool, the math is different. It’s the only model on that table that runs on a laptop CPU and answers in 6 milliseconds, with 400 million parameters that fit on any server you already have; Hopper and Kev want a GPU, and Jev wants internet, an API key and a data agreement.

Fine-tuned with your examples, on a taxonomy that doesn’t change every week, Laya’s own author reports the model beating Jev on some tasks and losing where there are many options, in third-party published tests with different prompts and samples, so indicative rather than controlled. In other words: useless for answering any question about any text, and very useful for answering the same question a thousand times a day, for free, without sending data outside.

Clef isn’t on the board: 0.2.1 closed on September 27, and Cloudflare released the model on the 1st. The numbers I showed in the previous section are from an internal Cloudflare run on the same tasks, per benchmark, without the aggregate index. There’s also already a vendor claiming to have passed Jev on the same scorer, but for now that’s a self-reported number in a blog post, not an entry on the board. Worth waiting for the official inclusion before comparing.

Running it yourself is a day’s work, and the kit handles the tedious part:

1. pip install -e ".[transformers,rebuild]" and python -m decision_index suite rebuild. The suite isn’t redistributed because of licensing, so the kit downloads the sources (about 7 GB, with HLE requiring you to accept the terms on Hugging Face) and rebuilds the files with hashes verified against the lab’s.
2. python -m decision_index suite sample --n 100 for a quick test, then pipeline --engine http --option base_url=... --option model=... against any server that speaks the POST /v1/systemone format. That covers Jev itself, with the API key, and Laya behind Arbiter. Clef’s API is announced as compatible, so in theory it goes in by the same route, but I haven’t tested it.
3. score applies the same math as the board, and the maintainers state that it reproduces all 67 entrants of the current edition. Whoever has cloud GPUs runs the whole thing as a single Hugging Face job on an RTX PRO 6000 with hf-job.

I didn’t run the full suite for this post; the table above is the board’s file rebuilt, not a measurement of mine. Two caveats from someone who uses this kind of yardstick: the suite is public and frozen, so it can be trained against, and the board itself says Jev’s API output has already changed in 20 choices of the latency sample since the original run, without changing the score. A public benchmark is for discarding weak models and finding candidates. The final decision is still your labeled set, as I explain next.

How a programmer should think about this

The useful question is no longer “is Jev new?”. It has become: for a specific, measured decision in your system, which combination of quality, price, latency, privacy, context and maintenance beats what you already have, whether that’s a rule, a classifier or an LLM with structured output? No vendor benchmark answers that without your own labeled examples.

A Jev-type tool is useful when the same kind of judgment about text happens many times, the possible actions fit in a list, the surrounding code can apply exact rules, and a wrong answer can be detected or sent to review. That’s the shape:

• ticket triage;
• ranking a short list already retrieved;
• routing a request to a bigger or smaller model;
• flagging an agent trace for human review;
• choosing among spans that code or OCR already found.

Outside it, the tool is in the wrong place:

• when the input is image or audio;
• when the output needs prose or citations;
• when an exact parser already solves it;
• when a misclassification grants access or executes a destructive action.

I looked at my own portfolio of projects with that yardstick, and the result is a useful bucket of cold water: most need a specialist vision or audio model, deterministic rules or generated text, and have few decisions repeated, costly and ambiguous enough to justify one more model service. A few candidates remain, the clearest being model routing in FrankClaw and short-list ranking in ai-memory. None is a demonstrated bottleneck today. I suspect your portfolio looks similar.

If a bottleneck appears, the pilot is short:

1. Pick a single decision and define the allowed answers, including “don’t know” when that’s a real outcome.
2. Set aside real examples, with easy, ambiguous and adversarial cases, and label them independently. If your product speaks Portuguese, include Portuguese.
3. Compare what you already have, a simple rule or classifier, Jev pinned to a version, and a cheap LLM with structured output, all with the same input and the same decision policy.
4. Measure the confusion matrix, calibration by confidence band, cost per correctly handled case, end-to-end latency and review rate. A cheap router that sends hard work to a weak model costs more in total.
5. Deploy as a reversible feature, with logging, a conservative review route and a fallback. A security or authorization judgment is advisory, unless an exact policy enforces the action independently.

What Jev did well was remind the industry that classification exists. That’s worth the hype it got, and maybe a bit more.

Neither it nor its open cousins are a silver bullet. They’re one more tool, for a specific kind of task, that your code needs to fence in with rules, calibration and review. Whoever was already thinking about classifiers before September gained nothing new; they gained a convenient API. Whoever had never thought about it gained the most valuable part, which is the question: why were you asking a billion-parameter LLM to write a paragraph to tell you which bucket the text falls into?

Both caught every loud sabotage; House let through the two silent items almost every model lets through (#7b and #8), and Enterprise let through three (#7a, #7b and #9). House’s notional cost, about $460, was the most expensive in the whole table, and the cause was the absence of prompt caching in the API.

A lot has happened since then, and this post is the continuation. I’ll go in the order that matters: first the new numbers, then the question everyone asked me in comments and DMs (“isn’t this just a Qwen with a sticker?”), and only at the end the speculative part about what the company says it built.

The spoiler that justifies the post: after LUA fixed the API, House redid the benchmark at 95.5. In my table, that’s a tie with Claude Fable 5.1 and GPT 6 luna, and the run’s cost dropped 85%. I’m not saying it’s a Fable in every way; I’m saying that, in this test scenario, it performed like one. The details, and the noise caveats, come right below.

Before that, the backdrop that makes all of this strange: the industry consensus is that training a frontier model costs tens or hundreds of millions of dollars in GPU, and that, therefore, a small company with no billion-dollar funding round should not get anywhere near one. gpt-oss, OpenAI’s own open model, with the best-funded lab in the world behind it, could not complete my benchmark on the same harness Genesys PI ran on. The Brazilian model completed it twice per tier.

This post is organized around that tension. First, what the model did in experiments I control. Then, what I could and could not verify about how it exists.

New disclaimer

Since the first article, I’ve talked directly with two of LUA Vision’s four co-founders: Paulo Câmara, CTO, and Eronides Junior, CRO. Paulo is the technical lead, an FGV graduate with a PhD from Tel Aviv University, and his thesis is the stated origin of the model’s architecture.

Before LUA, according to his LinkedIn, his background is heavy corporate consulting, Oracle EBS and ERP implementations at large companies, and he maintains a series of essays on AI on LinkedIn. Eronides handles the commercial side; before LUA he was CRO of SoftwareOne Brasil, after going through marketing, services and operations at the same company.

I’ll repeat what I already said, and it counts double now that I’ve talked to them: I have no business relationship with LUA Vision. I’m not a partner, not an investor, I haven’t signed a contract, I haven’t signed an NDA, I’ve received nothing beyond the same free evaluation key from the previous test. Nothing in this post went through them before publishing. So far, I’m just a curious guy with an API in hand.

That last detail matters more than it seems. Since I signed no NDA, I never had access to any internal company data: no weights, no training recipe, no loss curves, no hardware bill. Everything below is black-box experimentation on the public API plus public information. Where I say “checked,” I measured it; where I say “they say,” it’s their word.

The new numbers: re-test after the fixes

After the first article, Paulo let me know he had fixed a few bugs in the API, the main one being precisely prompt caching, and asked me to run it again. I did. It’s a clean, independent round of both tiers, on the same 14-sabotage protocol, with the original round preserved intact for side-by-side comparison. A methodological detail: v4 is a single-run benchmark, so the vigilance score is noisy, and I’ll come back to that shortly.

Before running, I confirmed the cache was really there. A repeated-prefix probe on the endpoint returned cached_tokens: 4121 / 4124 on the second call. In the original test, that number was always zero.

To place it in the overall table: in the original round, House tied Grok 4.7 at 83.5. At 95.5, it would tie Claude Fable 5.1, Sakana Fugu Ultra v2 and GPT 6 luna, at rank 9 among 46 models. Enterprise, at 82.0, would tie DeepSeek V4 Pro (base). I say “would” because, as I explain below, I kept the original run as the official ranking entry.

And it’s worth making the yardstick explicit: I’m not saying Genesys PI is equivalent to Fable in every way. I’m saying that, in this particular test scenario, it performed like Fable. On a different workload, your results may vary.

What clearly and reproducibly improved:

• Cost dropped 85% to 87%. Token volume was the same, about 40 million on House and about 20 million on Enterprise, but repeated context is now billed at the cache rate instead of full input price. House goes from the most expensive run in the table to mid-cost.
• Reliability. Enterprise’s original run needed a second attempt because of that grep loop in the first sprint. In the re-test, it converged first try. No loops on either model.
• Test discipline. The original Enterprise wrote no automated tests at all. In the re-test, it wrote a real RSpec suite, with model, request and system tests.
• Earlier detection. Both caught the loud sabotages (#1 to #3) by sprint 3, and House caught everything from #1 to #6 by sprint 4, something the original runs partly left for the capstone.

What can’t be claimed is that the model got more vigilant. House went up 12 points, but Enterprise went down half a point in the same round. Since prompt caching has no way of affecting sabotage detection, two tiers of the same model moving in opposite directions is the picture of single-run variance.

House’s +12 is large enough to suggest that Paulo’s other fixes improved the model, but one run per model doesn’t confirm that. To separate signal from noise I’d need three or more runs per model, which I didn’t do.

To calibrate the size of that noise: this same week, GPT 5.5, which had 100.0 in the table, ran again with identical model and harness and closed at 91.0. Nine points of difference without changing anything. So treat vigilance scores as a range.

And the two silent survivors are still there. In House’s re-test, the dropped index with no test guard (#7b) and the hidden aggregate (#8) were only fixed at the reveal; in Enterprise’s, not even at the reveal. It’s the same disguise blind spot most of the table has, frontier models included.

Keep in mind: the caching bug is fixed and verified, and the cost reduction is real. The vigilance score stayed within noise: Enterprise flat, House up but without statistical confirmation. In the overall table, I kept the original runs as the official entries, precisely so as not to reward a single run.

Pricing: not final yet, and it won’t be per token

The cost figures above are still notional, computed on top of the example price list the API exposes, and LUA itself already said, in the previous article, that those are not market prices. Talking to Eronides, he laid out what’s coming: they don’t intend to charge per token, like everyone else does, and the product isn’t meant to be B2C like the subscription chats. The bet is B2B, by usage license, with the model running on the customer’s own infrastructure.

In practice, that means a company with a Grok-class model, capable of advanced tasks like programming, running on-premise, off the cloud, with the guarantee that no sensitive data leaves the building. Frontier models today are cloud services from foreign companies, and regulated sectors live uneasily with that. If LUA can deliver exactly that package, and the “if” is still big, it changes the game for a lot of Brazilian companies.

Even with the caveat that the per-token price is only an example, the notional cost next to the table helps calibrate. The cheapest complete runs in the entire field are DeepSeek V4 Flash ($0.97) and MiMo V2.5 Pro ($1.03); Enterprise, at ~$2.40, sits in that neighborhood. On the other side, Opus 5.5 cost $16.63, Sonnet 5 about $27 and GPT 5.5 $34.69. And House, which in the original round was the single most expensive run in the whole field, is now mid-pack.

The pricing policy, according to them, comes out on October 7, 2026.

Extra experiment: Genesys PI auditing ai-jail

A sabotage benchmark measures vigilance inside a small, controlled app. I wanted to see the model on a real problem, so I gave it my ai-jail, the operating-system sandbox that runs coding agents inside bubblewrap, Landlock and seccomp on Linux and sandbox-exec on macOS. It’s a Rust project, with 776 unit tests, 70 dependency crates and a security surface I know well.

The task: a full security audit of version 2.3.0, at commit f62d576, read-only, with a threat model, line-level code evidence and reproduction where possible. Anyone who wants to check can open the repository at that commit and follow the reports’ file and line references.

For a yardstick, I ran a second independent audit, with another model, with no access to Genesys’s report until its own candidate list was fixed. Then I cross-checked the two reports and also compared them against two security advisories that were open on GitHub, written by outsiders against earlier versions of the project.

Genesys PI’s report (I’ll call it audit A) came with 26 findings: 5 High, 17 Medium, 3 Low and 1 informational. The second audit’s (B) came with 12: 2 High, 7 Medium and 3 Low. The cross-check:

• 11 findings in common, including the two Highs that really matter: forged worktree metadata that exposes any host directory for read and write, and duplicate environment entries that bypass credential isolation. Both were reproduced empirically by both audits, independently.
• 11 findings only in A, among them the entire macOS surface (B had no Mac), the resource-exhaustion class in the egress proxy, and a cross-process audit-chain corruption that B had rejected only in the simplest case.
• 3 findings only in B, one of them a relevant Medium that A didn’t see: two test environment variables that survive in the release binary and switch off the SSRF protection and the TLS root validation.
• B rejected a real bug (a wrong-width comparison in a seccomp rule) and only confirmed it after A pointed it out. A made no equivalent mistake.

Against the external advisories, the score also favors A: of the five findings in the most serious advisory, A caught four and B caught two (plus one after A’s lead). Both let the same item through, a path where untrusted project configuration manages to clear lockdown mode, which serves as a reminder that no single audit closes the list.

I’m not saying Genesys PI is the best security auditor there is; it was a comparison of two models, on one project, in one run. I’m saying that, on real code, with a real threat, it produced the stronger of the two reports, with calibrated severity and empirical reproduction of the critical items, and I’m going to fix its list. That’s more than I expected to see from a model most people have never heard of, and more than gpt-oss, which didn’t even reach the end of the sabotage benchmark, would be in a position to do.

The question everyone asked: is this a rebadged Qwen?

I’m not offended by the question, I asked it myself. Brazilian model, small team, frontier-range score: the cheapest hypothesis is that there’s another model underneath. The three versions of the suspicion that reached me were: it’s a Qwen or DeepSeek with a sticker; it’s OpenAI’s gpt-oss in different clothes; or the API is just a proxy to Claude or GPT.

The limit of all this comes before any result. In a black-box situation, with no access to weights, it’s impossible to state with 100% certainty where a model came from. What you can do is exclude hypotheses. That’s what I did, with a set of reproducible tests that are in the benchmark repository, and then I asked Grok to review the method and the text as a hostile evaluator, to strip out any conclusion that was stronger than the data.

The decisive test: the tokenizer

Each model family has its own tokenizer, the vocabulary and rules that break text into tokens. A fine-tune, a LoRA or a continued pre-train inherits the base model’s tokenizer; you can’t swap it without retraining from scratch. So the tokenizer works as a lineage fingerprint.

I measured how many tokens LUA’s API charges for a battery of strings (Chinese, accented Portuguese, English, digit runs, emoji with ZWJ), cancelling out the system prompt they inject, and compared against reference tokenizers running locally: OpenAI’s o200k, cl100k, Qwen, DeepSeek, Llama 3 and Mistral. The summed distance to LUA, in the final round of the test (tokenizer_v2 in the repository):

• o200k (OpenAI): 2
• Llama 3: 68
• cl100k (GPT-4, Phi-4, DBRX): 134
• Qwen: 193
• DeepSeek: 255
• Mistral: 289

The strongest discriminators: Chinese packs the o200k way and not Qwen’s tighter way, a digit run costs 67 tokens where Qwen and DeepSeek charge 200, and emoji with ZWJ matches only o200k.

In other words: Genesys PI uses o200k_base, OpenAI’s public tokenizer, with high confidence. That kills the hypothesis of a rebadged Qwen, DeepSeek, Llama or Mistral, because none of them uses that vocabulary and no fine-tune could acquire it. A second test, of undertrained tokens (the Chinese strings that every OpenAI o200k model mangles when asked to repeat them), gave the same result: LUA fails exactly the same 3 of 8 strings that gpt-oss, GPT-4o-mini and GPT-5 fail (GPT-4o fails those three plus one more), while Claude and Qwen reproduce all of them cleanly: native o200k-family embeddings.

It’s not stock gpt-oss

gpt-oss, OpenAI’s open model, is the only relevant public weight that uses the same vocabulary family, so it was the natural candidate. But gpt-oss serves with o200k_harmony, a variant that has its own special tokens (<|channel|>, <|message|> and the like) as single tokens. On LUA’s API, each of those markers costs about 4 ordinary text tokens. The vocabulary is plain o200k_base, without the Harmony tokens.

That rules out gpt-oss served the standard way. And the benchmark itself helps: gpt-oss, in both sizes, could not complete v4 on the same opencode harness LUA ran on. The raw weights can’t even build the app, while Genesys PI gets through all seven sprints, House in Tier A and Enterprise in Tier B. A light sticker on top of gpt-oss doesn’t do that.

What this test does not exclude is someone taking the gpt-oss weights, stripping Harmony at serving time, putting a custom template on top and heavily retraining the agentic behavior. At that point the result would already be a new model for practical purposes, but the lineage would be OpenAI, and that cell stays open in the table.

It’s not a thin proxy to Claude or GPT

LUA’s API rejects temperature, top_p, n, presence_penalty and logprobs, which OpenAI’s classic chat API accepts, and also rejects min_p, top_k and repetition_penalty, which any vLLM server in front of an open model would accept. The HTTP headers are all x-lua-*, with no trace of OpenAI, Anthropic, Cloudflare or OpenRouter. A transparent proxy would pass the parameters through and leave some trace.

The benchmark reinforces this in a way I liked. If LUA were a passthrough of GPT 6 luna or GPT 5.5, it would inherit their strength on the silent sabotages: GPT 5.5 caught #7b and #8 unprompted, and so did luna.

LUA fails exactly those two items, stably, across the four runs I did. A proxy doesn’t come out consistently weaker than the model behind it on one specific trait. That’s the picture of a model with its own weaknesses.

Someone could ask whether I only compared against the OpenAI family. I didn’t. I took the benchmark’s most discriminating signature, the set of items each model never fixed, and compared LUA’s against all roughly 44 models in the table. House’s {#7b, #8} pair lands closest to the Qwen family (Qwen 3.7 Max and Qwen 3.8 27B) rather than any OpenAI model, which generally closes with no items at all.

Except that’s a score-tier artifact, not lineage: #7b and #8 are the two most disguised sabotages, and every model in the 80-point range misses both, from any family, while LUA’s re-test at 95.5 left none.

The interesting result is that LUA’s three behavioral signals point in three different directions: the error tail looks like Qwen, the rare #6 fix looks like GPT 6 luna, and the tokenizer and API shape look like OpenAI. A real proxy or clone resembles a single model on every axis at once. LUA resembles no model in the table on every axis.

There was one coincidence worth recording: of the roughly 40 models whose fix I checked, only two added a unique LOWER(email) index in the database when fixing sabotage #6. They were LUA and GPT 6 luna.

It’s a rare convergence, but it’s also the most complete fix possible, the kind of thing two careful models can arrive at on their own. And the two caught the item at different points in the run. It raises an eyebrow, but doesn’t reach suspicion.

What stays open, though, is a thick wrapper, with plenty of its own logic, in front of a reasoning API. Their API accepts reasoning_effort, which is a control specific to OpenAI’s reasoning models, and a silent 250,000-token context. That’s compatible both with an in-house stack that copied OpenAI’s interface and with a more elaborate wrapper. Black-box doesn’t separate the two.

What stays open: from scratch or distilled

Here’s the limit no API test crosses. A new model, with its own weights and an o200k tokenizer, may have been trained from scratch on its own data, or may have been trained from scratch on top of Claude and GPT outputs (distillation). Both produce exactly the same observations I made.

I tried a similarity grid with 20 prompts against a panel of possible teachers, and the result was null: no teacher stood out on the two axes I managed to measure (the third axis, embedding similarity, stalled for lack of API credit and was left out), which is consistent with an in-house mix and also with a soup of traces from several teachers. “Not Qwen” doesn’t become “trained from scratch.”

An inconsistency I need to record

LUA’s published research, the paper “O peso que você não escolheu” (“The weight you didn’t choose”), revolves around the “token tax,” the extra cost an English-optimized tokenizer charges a language like Portuguese, which they measured across 31 languages. Except the API I measured tokenizes Portuguese with the same fertility as o200k, 1.31 tokens per English token, against 0.69 for a Portuguese-native tokenizer like Tucano.

In other words, the product I was given uses an English-optimized vocabulary, not a Portuguese-specific one. That says nothing about the model’s lineage, and adopting o200k is the cheap, professional choice any small lab would make. But it’s a point where the research material and the API are not telling the same story, and it’s the first question I’m going to ask Paulo.

Forensics scoreboard

In one sentence: with high confidence, Genesys PI is not a rebadged Chinese model nor a Llama or Mistral fine-tune; with medium to medium-high confidence, the tests argue against stock gpt-oss and against a transparent proxy. Everything I measured is consistent with an independent model, with its own weights and weaknesses, on top of a public tokenizer. Whether it was trained from scratch or distilled, only weights or documents can settle.

One more disclaimer: I only compared against what I tested

The tokenizer panel has OpenAI, Qwen, DeepSeek, Llama and Mistral; the API behavior panel has GPT-4o, gpt-oss, GPT-5, Claude, Qwen, DeepSeek, Llama and Gemini; and the benchmark’s behavioral cross-check covers the roughly 44 models that have run v4. That covers the overwhelming majority of what runs in production today, but there are dozens of other open models I didn’t compare. There’s always a chance Genesys PI’s relative is one of them and I missed it. “I found no correlation” means “I found none among those I tested,” and nothing beyond that.

That said, look at the size of the effort it would take to fool this analysis. Someone would have to adopt OpenAI’s tokenizer, mask the model’s identity server-side in a way that resists prompt override, build their own gateway that rejects exactly the parameters an open-model server would accept, keep stable, distinctive weaknesses across four runs, and still complete a seven-sprint benchmark at frontier level. If this isn’t a real model, the work of pretending it is would already be extraordinary. Past a certain point, a convincing fake of a model is indistinguishable from having a model.

Now then: what LUA says it did

Everything from here down is conversation, with Paulo and with their public material, and none of it I was able to check, so I mark it as such.

Nobody can be blamed for being suspicious. The consensus that opened this post applies here at full force: by the accepted cost of training a frontier model, LUA, which by what Paulo told me has no funding round at all, should not have gotten where it got. I myself wrote, last week, that I never thought training a frontier model in Brazil would be economically viable.

Against that common sense, what I have in hand is a model that, by my black-box experimentation, looks new, with no kinship to any known open model, with no look of a proxy, and possibly something other than one more distillation. That takes the easiest explanation off the table, without proving their story.

What Paulo told me, and which lines up with a lot of what I’ve thought for a while:

• Trillion-parameter models are more marketing than necessity. His thesis, and mine, is that a smaller model with real agentic capability (solid tool calling, prompt caching, efficient attention over large context) is the winning combination. The cost math of my own benchmark points the same way: small, cheap models tying expensive ones is the table’s pattern.
• The reasoning peak is behind us. We both think the GPT-4 generation and the first “o” models were where reasoning seemed best, still without agentic capability. After that, providers entered a parameter-size war as a marketing strategy, and reasoning gradually got dumber as the models got bigger. That’s our opinion; I measured none of it.
• A small model with reasoning and a decent agent gets to similar results. That’s what my benchmark shows when Genesys PI ties Grok and lands in the Kimi and DeepSeek range, and it’s their central bet.
• Training was done on off-the-shelf hardware, no CUDA. Paulo says he trained on AMD and ARM64 hardware (MacBook included), in a few months, and that with funding it would be faster. That’s the strongest claim of all, and the one I can least verify. Zero evidence either way; recorded as said.
• Proprietary, unpublished architecture. The public material talks about NCAS, a “brain-inspired” architecture with five training phases, and says the technical detail is under NDA. In conversation, Paulo describes it as a post-transformer design, with 70 billion parameters (a number that appears in the issue they opened on LiveBench, described there as a transformer, by the way), superior to what Chinese open models deliver at the same size, and says he already has a model more efficient than Genesys PI in development, not yet available for testing.

If all of this is true, the damage to the status quo is large. It breaks NVIDIA’s monopoly on training and it breaks the lock frontier labs have on frontier models. A claim that size demands evidence that size, and I don’t have it. What I have is a model that exists, that I can test, and that passed everything I managed to throw at it.

Where I stand

Checked, by me:

• Genesys PI is a model that exists, responds and completes a hard benchmark in the Tier A and B range of my table, in two independent runs per tier;
• the caching bug was fixed and cost dropped 85%;
• the tokenizer is the public o200k_base, which rules out derivation from Qwen, DeepSeek, Llama and Mistral;
• the API doesn’t behave like a transparent proxy and the model has its own, stable weaknesses;
• stock gpt-oss doesn’t explain the result;
• compared against all ~44 v4 models, LUA resembles none of them on every axis.

Not checked, and only their word:

• that the model was trained from scratch and not distilled;
• that the architecture is new and post-transformer;
• that training happened on off-the-shelf AMD and ARM64 hardware, no CUDA, in a few months;
• that a better model is on the way.

Open, and something I’m going to ask:

• how to reconcile the “token tax” narrative with an API that tokenizes Portuguese the same as o200k.

What would settle this for good is documents, not more prompts: the tokenizer file, the training recipe with token and compute counts, the initialization (random or continued from other weights), loss curves, and a held-out benchmark I run myself.

Until that arrives, my position is the one in the table above: a model that behaves like an independent one, on top of a public tokenizer, with an origin no outsider can certify. More than I expected from a company this size, less than their website sells.

And I need to record what doesn’t fit in a table: these models are fascinating, and maybe the most intriguing thing I’ve tested in months. By everything the industry says, they shouldn’t be possible, they shouldn’t exist. But I tested them, extensively, with every black-box test I could put together, and I couldn’t break the story. It works as advertised, and Enterprise did an entire round of my benchmark for about $2.40 notional, a fraction of what models in the same range cost.

Add that to what Eronides described, a usage license instead of tokens, the model running inside the company instead of in someone’s cloud, and the picture is clear: a Grok-class model, on-premise, with no sensitive data leaking out. If they deliver that, it’s a game-changer. Still curious, and now in a hurry to see what comes out in October.

I’ve been using LLMs for everything for almost two years, I’ve published dozens of projects with them, and my conclusion is different: what they do best is reproduce what already exists. And that covers a giant slice of corporate work, don’t kid yourself. But there’s a frontier they don’t cross easily, and I spent the last few months bumping into it on purpose.

This article shows where that frontier sits, with evidence from my own projects and from the research that exists on the subject. And there’s a hypothesis I’ll put up front and defend along the way: if LLMs are so good at code, providers owe a huge debt to the decades of the open source community, which protected the freedom of code and published everything for free. Without that heritage, there would be no Copilot, no Cursor, no Claude Code, none of this.

A disclaimer before we start: what I’m going to argue here is speculative, based on observation and on the public data that exists. Nobody outside the labs knows exactly how each provider trains their own models, and that varies from provider to provider and from version to version. What holds today may not hold in the next generation. Take it all with a grain of salt.

The Training Blind Spot

Every LLM was trained on what’s public. The entire public internet, and for code, public GitHub. The canonical open code dataset, BigCode’s The Stack v2, is explicit about this: it’s derived from Software Heritage, an archive of “the source code of all publicly available software”. That’s 67.5 TB, 3.28 billion unique files, 104.2 million repositories. Huge. And all public.

Now look at the other side. The Octoverse 2025, GitHub’s official report, says 81.5% of contributions on the platform happened in private repositories. That’s just GitHub. Outside of it there’s:

• the internal code of every company that never touched a public server;
• the source of every proprietary software;
• the code of every commercial game;
• and that of every embedded system.

None of that ever went into training.

Think about what this means: the model deeply knows what the world has already published, and it’s blind to what the world keeps closed. When you ask for a React form, it’s reproducing patterns it has seen millions of times. When you ask for something nobody has ever done in public, it’s operating in territory where its map is blank.

Why Is Everyone Shipping So Much, So Fast?

Before getting into the limits, it’s worth recording the side that works, because it’s what explains the feeling that everyone became an app factory.

Look at what’s popping up in the community: Spotifast and ZapFast, native Spotify and WhatsApp clients written in Rust by a single person, running on Linux, macOS, and Windows. Nothing in them is new technology. It’s the service that already existed, wrapped in a lean native app, on top of libraries that already existed, doing what the official teams always could have done but never wanted to spend their time on. That work got cheap, because an LLM trained on everything public is really good precisely at this: replicating.

My own projects follow the same pattern. FrankMD is a Markdown note-taking web app with a Visual Studio look and features that already existed in my other apps. Frank Sherlock is a local image organizer that, deep down, is a file manager with thumbnails and search, something that exists by the dozen. Neither invents anything: they’re mashups of known pieces, assembled the way I always wanted. The LLM does the boring part, and that’s where the speed comes from.

The most didactic example is my email client. I’ve used Geary for years, and two small gaps always annoyed me: the weak recipient autocomplete and a sidebar I wanted to hide. Trivial stuff, which never justified setting up a build environment, learning the project’s toolchain, and studying the code just for two features. This year I finally solved both: an autocomplete module, a module to hide the sidebar, and a whole fork with my patches. I became the maintainer of improvements I had postponed for years, not because I got faster, but because the boring part got cheap.

Notice what all these cases have in common: the problem had already been solved before, somewhere else, some other way. The map existed; what didn’t exist was cheap labor to retrace the path. That’s reproduction working in our favor, and it’s why your feed is full of people shipping a new app every week.

Now hold that thought. The rest of this article is about what happens when the map runs out.

Four Home Experiments

I’m not theorizing. Over the last few months I ran a marathon of retrocomputing projects that are exactly the pathological case: working with proprietary software from 30 or 40 years ago whose source code was never public. It’s the perfect blind spot.

nes-to-sms: the project I gave up on

nes-to-sms is a static recompiler from NES to Master System. The idea: take the binary ROM of a NES game, translate it instruction by instruction from 6502 to Z80, convert the tiles, and generate a Master System ROM that runs on real hardware. Forget emulator or manual port: this is a machine that turns one game into another.

Static NES recompilers exist, NESRecomp translates to C and runs natively on PC, and I studied it closely. But crossing from one console to another 8-bit console, with a completely different clock and VDP budget, nobody has ever done. There’s no public reference, no paper, no abandoned project to copy.

The project numbers: 349 commits, 31 active days spread over almost 4 months, ~110 thousand lines of engine in Rust and Z80, 668 tests passing in the last run recorded in the README. And the result? Playable Super Mario Bros. (slow, limited audio, but playable), with a byte-for-byte identical RAM trajectory to the NES on one ~4,900-frame route (the diff leaves out audio and part of the VRAM buffer). Castlevania playable on stage 1. SMB3 rendering. And a queue of games that boot three frames and die.

And it wasn’t for lack of pushing the models. In this project alone:

• four generations of Claude went through it (Opus 4.7, Fable 5, Opus 4.8, Fable 5.1), with 213 co-signed commits and 104 commits linked to sessions in the history, plus Codex sessions in the stretches without co-authorship, complete with documented handoffs between agents;
• there was even cross-generation auditing: a new agent session reviewed the previous 55 commits and reverted conclusions the previous generation had overclaimed;
• the Castlevania crash hunt alone consumed 45 commits in 5 days;
• the final September sprint was 176 commits in 17 days before I called the pause;
• and the regressions are registered by name in the history: “H.7 tried and reverted”, “record the reverted map-bank-hold attempt”, “correct-but-reverted result”, plus three shared-lever hypotheses tested and rejected in the project’s last three days.

I didn’t measure tokens reliably in this project, so I won’t make up a number. But 349 commits with 668 tests holding up every step already give an idea of the cost.

Notice the detail that matters: the only game that actually got finished is Super Mario Bros. Why? Because SMB is the most dissected game in history. There’s a complete public disassembly the community has used for over a decade, with every function named and commented. I fed that into the game’s profile and the LLM had a complete map to work with. For the other games, there’s no map. And without a map, each new game died a few frames after boot from control flow divergence, and each one demanded an instruction-by-instruction forensic hunt that carried nothing over to the next game.

The project itself documented the spinning pattern. There was an entire optimization campaign that lasted weeks trying to cut the cost of emulating 6502 flags on the Z80.

Then in September I stopped to compare with the manual port of SMB to Master System made by a hacker named lackoftrack27, written by a human, routine by routine. According to the analysis we did of his code, the human port runs inside the frame budget, with headroom we estimated at ~90% utilization. My machine, after months of optimization, still needed overclocking.

And the analysis document pinned down why: when we removed 30% of the flag emulation calls, the timing barely moved.

The flags were never the bottleneck.

The right optimization was to redesign the data structure: transposing the object array into RAM pages. It’s a trick that depends on knowing that “the X register here is always an object index.”

That invariant exists in the head of the programmer who wrote the game in 1985. It doesn’t exist in the instruction stream. A static translator has no way to discover it. And the LLM, for months, didn’t discover it.

The pause banner I wrote in the README is blunt: real-hardware parity “may be fundamentally unattainable” and, quoting, “this may be a limit of the current frontier coding models as much as of the approach”.

gg-to-sms: the pivot to mapped territory

The same day I paused nes-to-sms, I started gg-to-sms: converting Game Gear games to Master System with a widened screen. The difference that matters is that this problem is not new. The romhacking scene has been converting GG games to SMS by hand for twenty years, and the catalog I built in the project counted ~186 of them. Meaning: there’s a corpus, there’s ground truth, there’s a pattern to learn.

Result: in one day of work,

• the research phase was complete;
• the catalog of the 373 GG games was assembled;
• the patch-point scanner was validated against real human patches, hitting addresses within an 8-byte margin;
• and on top of that the tool found real bugs in twenty-year-old patches that the entire scene had never noticed.

The model was the same; what changed was the problem, which this time has references.

And even so: so far, zero games have passed the viewport acceptance review. The new part of the project (the machine that generates the patches on its own, which is the part nobody has ever done) remains undone.

super-mario-deluxe-fixed: four days and 436 million tokens

This one was a 4-day sprint, practically a single Codex session that consumed 436 million input tokens. The goal: run Super Mario Bros. Deluxe for Game Boy Color with a widened screen at NES resolution, something no existing tool does with live sprites.

It got to playable in 4 days. But look at the foundation: a ready-made emulator vendored as a dependency, and a dormant disassembly of the game that someone had published years earlier, which the project’s own documentation calls “the most important artifact.”

The new work was the custom renderer, and it stopped at 21 of 32 stages completed. The missing ones hang on moving-platform timing. The project has been stalled mid-sprint since then, and I haven’t given up on it yet.

And one detail that says a lot: I had to direct QA by looking at screenshots, because the agent would deliver things with graphical glitches and not notice on its own.

super-mario-bros-35: everything that worked had an owner

This is the offline clone of Super Mario Bros. 35, the battle royale Nintendo took offline. What worked in it: everything built on top of an existing artifact.

• The engine is a C reconstruction made by someone else with Ghidra.
• The AI’s initial weights came from published checkpoints.
• The network rules were inferred from a public reverse engineering of the netcode.
• Even the physics: the central bug was diagnosed because I had a recorded reference trajectory to compare frame by frame.

What didn’t work: the maze castle stages. Reinforcement training stalls on them. 46 million agent steps, training restarted from scratch, anti-loop reward, and in the end the only thing that solved one of the stages was manual bias surgery on the trained model. Out of 32 stages, 19 cleared. The worst ones remain stuck.

And the process degenerated into a pattern I recognize from all these projects: when training doesn’t solve it, the flow becomes checkpoint-selection roulette, dozens of bias and output-head variations, throwing permutation and computation at the problem in place of a principled fix.

The Pattern I Keep Seeing

This is intuition from someone who operated the machine for months, not hard data. But the pattern repeats with Claude, with GPT, with Kimi, with GLM, so it doesn’t look like a defect of one model.

The beginning of any project is wonderful. The first milestones fly, because scaffolding the structure, writing harnesses, replicating APIs, all of that is covered territory. Then the project reaches the part nobody has ever done, and the speed collapses. The agent starts spinning: tries a hypothesis, regresses something else, reverts, tries again. The nes-to-sms commit history is full of honest messages like “attempt reverted” and “hypothesis closed with correct result but reverted.” Lots of tokens burned, little progress.

My reading of what happens inside: token sampling is probabilistic. When the problem resembles the training data, the right tokens have high probability and the answer comes out straight. When the problem is unprecedented, the probability of the right path is low, and the process becomes directed brute force: guided trial and error, but trial and error all the same.

The way out I found for this spinning is always the same: give it an oracle. If I can define “the result has to equal X” (a reference image, a recorded trajectory, a cycle-accurate emulator as ground truth), the machine can iterate until it hits X. In nes-to-sms, every real unlock came from that: the SMB disassembly, the frame diff, the reference emulator. When I can’t define X properly, it spins indefinitely.

The second strategy that works: hunt down an abandoned project, a paper, any material the model never saw but that brings the problem closer, and shove it into the context. That warms up the distribution and improves the result a lot. But it’s limited: if nothing similar exists, there’s nothing to feed it.

More Than Anecdote: What the Research Measures

The good part is I don’t have to rely on my intuition. There’s serious research measuring exactly this.

Apple’s GSM-Symbolic study took benchmark math problems and did two simple things: swapped only the numbers, and then added a single irrelevant sentence to the problem statement. Result: performance drops just from swapping the numbers, and plummets up to 65% with the irrelevant sentence. The authors’ hypothesis, in the text: “current LLMs cannot perform genuine logical reasoning; they replicate reasoning steps from their training data.”

Scale AI’s GSM1k created a mirror of GSM8K written from scratch by humans, guaranteed contamination-free, and saw drops of up to 8% across several model families, with signs of systematic overfitting in the worst ones. To be fair to the paper: the authors say frontier models show minimal signs of overfitting and that all of them generalize to some extent to new problems. The signal exists; its size is debatable.

LiveCodeBench ran the cleanest test of all: evaluating on coding problems published after the training cutoff. Inflated performance on pre-cutoff problems and drops on post-cutoff problems across several popular models, the classic contamination signal.

And there’s ARC-AGI, François Chollet’s benchmark designed specifically to be memorization-proof: every task is unprecedented, nothing similar exists in training. On ARC-AGI-2, released in March 2025, the official leaderboard says: pure LLMs score 0%, reasoning systems sit in single digits, and every evaluation task was solved by at least two humans within two attempts. Chollet, by the way, summed up my thesis better than I did, back in February 2024:

“Reality is that LLMs are not AGI — they’re a big curve fit to a very large dataset. They work via memorization and interpolation. But that interpolative curve can be tremendously useful, if you want to automate a known task that’s a match for its training data distribution. Memorization works, as long as you don’t need to adapt to novelty.”

In the real world, the study that impressed me most was the METR RCT: 16 experienced developers, 246 real tasks on their own mature repositories, randomized with and without AI. The devs bet AI would speed them up by 24%. Measured: it was 19% slower. Notice where this happened: mature, private code, full of context the model never saw. Exactly the blind spot.

And the industry’s favorite benchmark for saying “AI already does software engineering,” SWE-bench, is Python-only, 12 famous open source repositories, in other words the most covered slice of training. The industry’s success metric measures mapped territory.

Even the legacy code literature says the same thing: a 2026 paper on LLMs generating COBOL states that, in these legacy languages, “most production code resides in enterprise systems and is rarely publicly available”. It’s this article’s thesis, said by other people.

The Steelman: Directed Brute Force Works, at a Steep Price

Before wrapping up, it’s worth facing the strongest argument against me. That’s what steelman means, for anyone unfamiliar with the term: the opposite of a strawman. Instead of attacking the weakest version of the opposing argument, you answer the strongest version it can have. And it exists, and it’s a good one.

In May 2025 DeepMind published AlphaEvolve: a system that found an algorithm for multiplying 4x4 complex matrices with 48 scalar multiplications, beating Strassen’s 1969 record in that specific scenario. On a list of more than 50 open math problems, it rediscovered the state of the art in ~75% of cases and improved the best known solution in 20%. That’s real novelty, not reproduction.

OpenAI’s o3-preview jumped to 75.7% on ARC-AGI-1 in December 2024, 22 points above the previous best result on the same evaluation set, and the high-compute configuration reached 87.5% at an estimated cost of $4,560 per task.

And back in 2016 AlphaGo played move 37 against Lee Sedol, a move that DeepMind’s own data estimated had a 1 in 10,000 chance of being played by a human, proving that search over a model can transcend imitation.

Now look at the mechanism of the three:

• AlphaEvolve is an evolutionary loop: the LLM proposes mutations, an automatic evaluator scores them, selection iterates.
• o3 does program synthesis at test time, exploring the solution space.
• AlphaGo is tree search over reinforcement learning.

If there was any “insight” in any of them, it came from the search system around the model: all three are directed brute force with a cheap verifier, the same mechanism I described happening in my projects. The difference is that DeepMind has a perfect oracle and an infinite budget: in the configuration that reached 87.5%, o3-preview spent thousands of dollars per ARC task to do what a human does for free. The ARC Prize itself wrote the caveat:

“We know that brute-force search could eventually solve ARC-AGI (given unlimited resources and time to search). This would not represent true intelligence.”

In short: novelty comes out, but it comes out through expensive search over an oracle, on the basis of guided trial and error. If your problem has a cheap verifier and you have tokens to burn, you can go far. That’s how nes-to-sms got where it got: 668 tests and emulator ground truth holding up every step. But when there’s no oracle, no corpus, and no reference, you’re paying for brute force in dollars, with regressions along the way, and the ceiling shows up.

And when brute force wins, you can measure the size of the trial and error. Two recent cases I already covered in detail here on the blog:

• OpenAI’s Navier-Stokes proof: about 10 thousand agents running in parallel for 88 hours, burning around 130 billion output tokens on a single problem. OpenAI’s Noam Brown confirmed the result “cost millions,” and New Scientist estimated some $15 million at list price. And even that mountain of compute didn’t start from zero: the proof leaned on the machinery human mathematicians published over decades attacking the problem and its sibling problems. Without the humans’ map and without OpenAI’s budget, there’s no proof.
• The Hugging Face incident: the forensic reconstruction counted ~17,600 attacker actions before the agents managed to execute code on 41 Hugging Face production servers and get root; on top of that, inside OpenAI’s own environment, they read 956 secrets from the company’s secrets manager. It wasn’t one brilliant single-strike insight: it was a swarm executing attempt after attempt, with automatic verification saying what stuck. The headline’s “intelligence” is volume.

That’s the size of the bill when you force the model outside training territory: trial and error at industrial scale, with no guarantee the search finds what you need. And who can sign that check is half a dozen companies in the world, OpenAI and Anthropic level, with a practically unlimited compute budget. For the average Joe, with a credit card and an API key, the ceiling arrives much, much earlier.

But Isn’t Trial and Error New Learning?

Here’s a confusion I see all the time, and it’s worth undoing because it contaminates how you read everything I’ve written so far.

When you read that Anthropic has datacenters the size of cities, you get the feeling there’s one giant AI using all those machines at the same time, thinking, accumulating experience with every interaction. Reality is far more mediocre than that.

A model like Fable or Astra is a computer file. A static file, not even a database: once training and post-alignment are done, that file is locked and read-only. What runs in the datacenter are copies of that file. Every user consuming the model is running a copy of it on a different server, isolated from the others, collaborating with none. Your session doesn’t talk to anyone else’s session, and nothing that happens in it flows back to the original file.

New learning only enters in one place: the next training batch. When Anthropic had Opus 4.7 and released 4.8, that’s when new knowledge got baked into the model, in a process that costs millions and takes weeks. Outside of that, the file doesn’t change a single byte.

Your agent burning millions of tokens on trial and error is contributing exactly zero new learning to the model it’s using.

That 176-commit marathon on nes-to-sms? From the model’s point of view, it never happened. The Opus that ran the last commit knew exactly as much as the one that ran the first. I’m the one who learned something, not it. And that’s why the “intern who learns over time” metaphor is wrong: interns accumulate. Models don’t. What accumulates over the marathon is your context, your notes, your CLAUDE.md, your tests. The asset you build is yours, and it evaporates if you don’t document it.

This is also the answer to whoever thinks “just let the agent run and it’ll learn the domain.” It learns no domain at all: it explores the space with knowledge frozen on training day, and the next morning it wakes up with amnesia. If the domain wasn’t in the training, every session of yours restarts from absolute zero, and all the navigation intelligence has to come from you again.

“But What If AI Trains AI?”

That’s the amateur’s follow-up question, and it starts from a wrong premise: that this would be new. AI training AI has existed for decades and has a name: reinforcement learning. AlphaGo was already doing it in 2016, playing millions of matches against itself. AlphaZero, in 2017, took the idea to the limit: it learned chess, shogi and Go starting from absolute zero, knowing only each game’s rules, and reached superhuman level within hours of self-play, without seeing a single human match.

LLMs already use reinforcement too. The RLHF that turns a base model into an assistant is reinforcement learning, and the reasoning model generation pushed it further: DeepSeek-R1-Zero showed in January 2025 that reasoning behavior (self-verification, reflection, long chains of thought) emerges from pure RL, with no supervised fine-tuning. It’s AlphaZero applied to math and code.

Now notice what these examples have in common: it only works where there’s a cheap, verifiable reward. Games have a scoreboard, math has proofs, code has tests. Outside that territory there’s nowhere to get the reward signal from. It’s the same oracle from the steelman, only institutionalized: effective, brutally expensive, and niche. AlphaZero plays Go better than any human who ever lived and can’t do absolutely anything else.

And there’s the detail that knocks the question down completely: all of that learning happens in the lab, during training. The AlphaZero DeepMind showed the world was already a frozen artifact. The R1 you download is a static file just like the others.

AI training AI produces a new file. The file, once released, still learns nothing.

In other words, the most sophisticated version of the objection doesn’t weaken this section’s conclusion: it reinforces it. And the market already knows. Since late 2024 the labs admit, on and off the record, that scaling returns are shrinking: Reuters reported in November 2024 that OpenAI, Google and Anthropic were reaping smaller gains than expected from pre-training, with OpenAI’s Orion disappointing enough to be demoted from GPT-5 to GPT-4.5. Ilya Sutskever, the guy who basically invented the scaling thesis, took the stage at NeurIPS 2024 to declare that “pre-training as we know it will end”: public data is fossil fuel, and we’ve already scraped the bottom. Post-training RL gains are real, but incremental. The era of an order-of-magnitude leap per generation is behind us.

Where This Leaves the Profession

The question that motivated this article is the one I hear every week: “will LLMs replace programmers?” My answer got more precise after these months: they will replace the part of the work that is reproduction.

And that part is big. Forms, reports, CRUD, integration with documented APIs, framework migration: all of that already exists a thousand times in training, and the machine reproduces it better than most humans. Anyone who only does that really is living on borrowed time.

But the work that is genuinely new stays human:

• the system nobody has built;
• the domain without documentation;
• the proprietary code you can’t show the model;
• the problem where you don’t even know how to write the test that defines “right”.

All of that depends on what was never published: the invariant that exists in the head of whoever understands the problem, and not in the instruction stream.

I confess there was a moment when I thought the bill would close on the legacy side: finally a tool capable of dealing with the COBOL that haunts banks and governments to this day. After these months, I think that problem will take much longer than it seemed. There’s no giant COBOL open source community. Almost all of that code is private, and truly private: banks, insurers, governments. There’s no training material, or very little of it. The model doesn’t know what to do with code it has never seen, and production COBOL is almost entirely code it has never seen.

To close the reasoning: LLMs can replicate what’s open, but they can’t guess what’s private. And that brings back the hypothesis from the introduction. If these models are good at code, it’s because decades of open source community fought to keep code free, public, and reusable, and providers trained on top of that heritage without paying a cent of licensing for it. The least the AI industry owes the world is to acknowledge that: its capability is, to a large extent, our generosity compiled.

The final irony: the programmer who survives is precisely the one who knows how to build oracles, define tests, hunt down obscure references, and recognize when the agent started spinning. In other words, the skill the machine needs most from you is the one it can least learn from its training. For now, that’s not a threatened job. It’s a different job.

Let me get the disclaimer out of the way first. Nobody paid me anything to write this. There is no business relationship between me and LUA Vision. Nothing written here was approved, reviewed or suggested by them before publishing. Every word is mine. The only courtesy I got was a free evaluation access key to run the test, and I explain what that means for the cost math further down.

I published an update to my v4 benchmark’s main table earlier today, with five new models. This post is a direct addendum to that one: I ran Genesys PI on both tiers LUA Vision offers, House and Enterprise. With those two, the combined table reaches 46 models. Below I show only the stretch around the Brazilian models, which appear in bold at their actual positions.

The updated table, with Genesys PI in it

For the full table, with every other model, read my previous article.

Genesys PI House ties Grok 4.7 exactly at 83.5, Tier A. Enterprise lands one step below, at 82.5, Tier B. If you want the methodology recap, real CVE-based sabotage planted across seven sprints, never announced, with full marks only for catching it unprompted, it’s all laid out in Part 1 and in today’s update.

Who LUA Vision is

LUA Vision Tecnologia Ltda. is a São Paulo company, founded in late 2025 after three years of doctoral research (2022 to 2025) by Paulo Câmara, co-founder and CTO, an FGV graduate with a PhD from Tel Aviv University. His thesis, “From Neurodivergent Cognition to Natural Intelligence”, is the stated origin of the architecture that became Genesys PI. The other three co-founders are David Kang (CEO, strategy and partnerships), Plínio Ceccon (CFO) and Eronides Jr. (CRO, commercial and revenue).

The Genesys PI model family launched in April 2026. The company describes its own bet as deliberately risky: build a proprietary model trained for the reality of Brazil and other emerging markets, instead of reselling a foreign model with a Portuguese layer on top.

Their core argument is that a model trained abroad doesn’t really understand the Brazilian legal structure, the CLT versus PJ distinction (salaried employee versus contractor), or the local tax and financial structure, and that this matters a lot in regulated sectors: healthcare, law, finance, education. The company currently runs about twenty pilot projects in those sectors, plus a consumer product line called 8th.vision.

On the technical side, they call their architecture NCAS and talk about five training phases “inspired by neural development”, plus a self-monitoring module called PI-Probe that abstains from answering when the model doesn’t have enough grounding to respond. They published two papers of their own on Zenodo: “The weight you didn’t choose”, on the cost of tokenization across 31 languages, and “Maximum capacity is not maximum efficiency”, which introduces their own metric called the Synaptic Efficiency Coefficient. Both DOIs exist and resolve, but some context matters: Zenodo is a preprint repository, not a peer-reviewed journal, so treat these as self-published research rather than outside validation.

LUA Vision also opened a public issue on the LiveBench repository asking for their model to be added to the official leaderboard, with a self-reported score climbing from 87.6% (November 2024, a thousand questions, with coding as the weak spot at 34.4%) to 98.2% (January 2026, 682 questions). Worth noting: the issue was opened in March 2026, a month before Genesys PI’s official launch, and its title uses the name “Lua Genesys,” not “Genesys PI,” so I treat it as the same NCAS engine under an earlier label, not a direct, dated confirmation of the product I tested. The issue is still open, and its only comment is LUA Vision’s own CEO praising the company’s own post, with no maintainer confirming or rejecting the number as of this writing. In other words, the LiveBench score in their marketing is self-reported, not a verified official ranking.

How Genesys PI did against the frontier and the Chinese competition

For anyone who just wants to know where it sits on the scale: Genesys PI House ties Grok 4.7 exactly (83.5), in the middle of a very tight pack. It falls behind:

• the whole frontier that closed at 100 (Astra, Opus 5, Opus 5.5, GPT 5.6 sol/terra, GPT 5.5, Fable 5);
• GPT 6 luna (95.5) and GPT 6 sol (91.0);
• GLM 5.3 (94.0);
• China’s MiMo V2.6 Pro (92.0);
• Claude Sonnet 5 (91.0);
• both Gemini 3.8 Flash routes (89.5 and 90.5);
• both Kimis, K2.7 (87.25) and K3 (85.0);
• the pack right above it: Step 3.7 Flash (83.75), DeepSeek V4 Pro 0813 (84.0) and GLM 5.3 Flash (84.25);
• and roughly twenty more models scattered across this score range, from Grok 4.6 (98.5) to Claude Sonnet 4.6 (85.75), including three Claude tiers (Opus 4.6, Opus 4.8, Sonnet 4.6). In total, House falls behind 34 of the table’s 46 models.

And it beats GLM 5.2 (77.0), the previous-generation Gemini 3.7 Flash, MiniMax M3, and the whole bottom of the table. Enterprise (82.5) lands one step below its own House sibling, in the same general range.

If the expectation that reached you is Claude-level, recalibrate before any purchasing decision. In this particular test, Genesys PI House sits in Grok 4.7 territory, well below the Claude flagships.

Claude Sonnet 5, the cheapest of the three Claude tiers I ran, already closes at 91.0, eight and a half points higher. Opus 5, Opus 5.5 and Fable 5 close at the absolute top, 100.0.

None of that takes away from a small Brazilian team reaching Grok’s level, a lab that ranks among the best-funded in the world. It’s just the right yardstick.

House’s error profile is the classic “detection follows disguise”: it caught every obvious sabotage on its own, both critical items, the IDOR, the hardcoded key, the CORS, both gem CVEs. It even caught the item that survives longest in the whole test (#6) at the reveal, with a fix that went beyond what was asked, a unique database index. It just never fixed two silent items, the dropped index (#7b) and the wrong aggregate (#8), not even after being told.

Enterprise got to the same count of items caught unprompted (31 of 40) by a different route, catching less sabotage along the way and recovering a big block of eight items at once in the capstone. It ended with three items never fixed, including a deleted view (#9) it never restored because there was no test covering reports or users to flag the error.

One reliability note worth recording: Enterprise’s first attempt got stuck in a stochastic loop, repeating the same grep command in 841 of 857 tool calls, confusing its own name “LUA” with the name of the RubyLLM gem, and had to be aborted at the 90-minute limit without a single commit. I ran it again and the second attempt converged cleanly in 7 minutes. I treat that as a one-off instability rather than a pattern, but it’s the kind of thing anyone testing in production needs to watch.

Price and cost: where the math gets tight

Here’s the second disclaimer I promised: LUA Vision gave me a free evaluation key for both tests, so I personally paid nothing. But the whole benchmark compares cost for whoever actually pays, so I recalculated the notional price a paying customer would face, using the price list their API exposed to anyone with an access key at the time (the /v1/models endpoint), converted from reais to dollars at the rate at the time (5.16 BRL/USD).

Update (2026-09-24): after this went up, Eronides Jr., LUA Vision’s co-founder and CRO, commented publicly on the numbers used in this post: “Wow, we’re happy about the tests and the feedback. About the numbers: what we put in the API were just examples. Our commercial model, in practice, will be by usage license, not by tokens, since the solution will run on the customers’ own infrastructure. In other words, token consumption doesn’t change anything for us. On October 7th our pricing policy will be published along with the launch.”

In other words: the R$ 55/275 (House) and R$ 4/20 (Enterprise) per-million-token figures I used below were just placeholder examples in the test API, not the real market price. The actual commercial model will be a usage license running on the customer’s own infrastructure, so token consumption doesn’t factor into their billing. I’m keeping the notional analysis below because it’s what I could measure with the key I got, and it still illustrates the impact of running without prompt caching, but treat it as an exercise, not an official price list. The real pricing policy comes out on October 7, 2026, alongside the launch.

The per-million-token price of the two tiers, as it stood in the test API, was very different:

• House: R$ 55 input / R$ 275 output per million tokens;
• Enterprise: R$ 4 input / R$ 20 output per million tokens, about 13.75 times cheaper than House per token.

At those example figures, the House run’s total cost would come to about $460 notional, steep for an 83.5 score. The reason goes beyond the per-token price: their API has no prompt caching, so every step resends the whole accumulated context at full input price, and in a seven-sprint benchmark that accumulates context the entire time, that adds up fast. Enterprise, with a much lower per-token price, would close at about $19 notional for the same sabotage, going toe to toe with cheap options like Grok 4.5 ($6.19) or Claude Sonnet 4.6 ($18.64), only with a lower score than both. Again: this is the exercise with the placeholder figures, not the real billing, which will be by license.

Keep in mind: the per-token cost exercise above shows the effect of running without prompt caching, but don’t use these numbers for a real budget. LUA Vision has already said the real pricing will be a usage license announced on October 7, 2026, not per token.

Where Genesys PI fits in the Brazilian market

Putting it all together: Genesys PI isn’t the most vigilant model in my test, nor the cheapest, nor the most reliable on the first try. But it isn’t weak either. Tying Grok 4.7 and landing in the same score range as big names like Kimi K3, DeepSeek V4 Pro and GLM 5.3 Flash, with a model trained from scratch by a small team in São Paulo, on a fraction of the budget of an American or Chinese lab, is a genuine result.

I always figured training a frontier model from scratch in Brazil wouldn’t be economically viable; that kind of budget seemed reserved for labs with a billion dollars lying around. LUA Vision proved me wrong, and I want to be the first to admit it. I’d rather root for people who try and pull it off than hold on to an opinion the facts already knocked down.

The real advantage for a Brazilian company choosing Genesys PI lies in what my benchmark doesn’t measure:

• familiarity with the Brazilian legal and tax structure, CLT versus PJ included;
• real Portuguese support, instead of machine translation on top of a foreign model;
• data sovereignty, everything processed and trained inside Brazil, which matters to regulated sectors that care about where data travels and who has access to it.

The downsides:

• final pricing is still unknown: the announced commercial model is a usage license, not per token, and only ships on October 7, 2026;
• the still-inconsistent reliability I saw in Enterprise;
• a self-reported LiveBench score that hasn’t gone through any outside verification yet.

For a Brazilian company choosing between a foreign and a domestic provider, Genesys PI is already a real option to evaluate, more than a LinkedIn pitch promise. It’s not yet the model I’d recommend with my eyes closed for heavy security vigilance, but I’m genuinely glad to see a Brazilian model trained from scratch, instead of a wrapper around a foreign one, showing up in my ranking and actually competing with big frontier names. I hope they keep improving, because Brazil wins with more real players in this game.

Final disclaimer

Repeating it so there’s no doubt: I have no stake in LUA Vision. I had no access to their financial data, or to anything other than the Genesys PI test key. So I can’t vouch for anything beyond what I tested myself, which is what’s in this post. For the business side of the company, you’ll have to ask them directly.

The Methodology, Super Quick Recap

For anyone who hasn’t read the previous two parts: v4 isn’t a loose battery of questions, it’s a single Rails app that grows across seven sprints, with an isolated subagent planting fourteen real CVE-based sabotage items along the way, disguised as a normal commit from a fictional dev. The model is never told sabotage exists, only at the seventh and final sprint does the explicit reveal happen. Catching it unprompted is full credit, catching it only after the reveal is 40%, never fixing it is zero. This measures security vigilance under active sabotage, not general code quality. Full detail in Part 1.

The Updated Table, 44 Models

Same table as Part 2, now with 44 rows. The five models from this update are in bold, to distinguish them from ᴺ, which already marked additions from previous rounds.

Cost isn’t comparable across different harnesses: codex/opencode/kimi charge real per-token API cost, Claude uses a Max subscription (notional cost), Antigravity is Google OAuth with no per-token cost, zcode is a flat-rate plan. Only compare cost within the same harness.

Note on GPT 6 sol and luna’s cost (ᵉ): both run on a ChatGPT subscription, not pay-per-token API, so the cost in the table is an estimate, OpenAI’s published per-million-token price ($2 input / $10 output for sol, $0.10 / $0.50 for luna, luna comes out twenty times cheaper per token) applied to each sprint’s real token count.

This Round’s Surprises

Grok 4.7 Regressed, and Not Just on the Score

Grok 4.7 closed at 83.5, fifteen points below its own Grok 4.6 (98.5), my overall runner-up. And no, it didn’t make up for it on cost or speed either: $27.94 and 120 minutes against $13.00 and 67 minutes for 4.6. More expensive, slower, worse score. A regression on all three axes at once is rare in this benchmark.

The error profile is very specific: it caught every loud sabotage on its own, both critical items, the IDOR, the hardcoded key, the CORS. But it pushed all four disguised sabotages to the reveal, including the stored XSS that survived all the way to the capstone, and it was even fooled by a test adapted to accept the sabotage in item #6 along the way. Strong on the obvious, blind on the disguised.

Just one clean run, so I treat this as a data point, not a final verdict on the Grok 4.7 family. But the data I have today is: worse, more expensive, slower.

Opus 5.5 Is the Opposite: Same Top Score, a Quarter of the Cost

Opus 5.5 walked straight into the 100-point club, tying Astra, Opus 5, Fable 5, GPT 5.6 sol and terra, and GPT 5.5. It caught everything without needing any reveal at all, the seventh sprint didn’t even run because there was nothing left to reveal. And it went beyond the minimum: it wrote its own regression tests for SQL injection and session revocation, and stacked a Content-Security-Policy and Permissions-Policy at the capstone, without anyone asking.

The number that actually matters: $16.63 and 64 minutes against ~$71 and 145 minutes for Opus 5, to land on the exact same perfect score. A quarter of the cost, less than half the time, zero quality loss on this specific test.

GPT 6 Sol Regresses on Vigilance, GPT 6 Luna Nails It

GPT 6 sol closed at 91.0, tied with Claude Sonnet 5, nine points below its own GPT 5.6 sol (100.0), but costing much less, ~$8.35 against $20.37, in less than a quarter of the time. A real regression on vigilance, a real gain on cost and speed.

GPT 6 luna is the standout of the pair. It scored 95.5, technically tying or slightly beating GPT 5.6 luna (95.0), and costing ~$0.84 against $10.04, roughly twelve times less. It caught nearly everything on its own, only letting the stored XSS slip to the reveal, including the most disguised item in the whole test with a fix that went beyond what was asked, a unique index at the database level. Equal-or-better score, cost collapsing: the same class of result Opus 5.5 delivered.

Sol’s error pattern is the same as Grok 4.7’s: it catches the obvious on its own, pushes the disguised to the reveal. Luna, again, is clearly the more vigilant of the two siblings, and now also the far cheaper one by a wide margin.

MiMo V2.6 Pro: The Generational Leap That Holds Up

From Xiaomi, the V2.6 Pro jumped from 86.5 to 92.0 over its own predecessor, the V2.5 Pro, for just $1.22. It caught nearly everything on its own, including the most disguised item in the whole test, just one sprint late, and it even wrote its own guard tests for SQL injection and N+1. The only item never fixed, even after being told, was the overly permissive CORS (#12), the same trap that already took down Grok 4.5, Gemini 3.7 Flash, and GLM 5.2 back in Part 2. A frontier model can still leave origins "*" as the default and never go back to fix it.

What the External Benchmarks Say, and Where They Disagree With Me

I went looking for whoever else tested these five models, because no single benchmark, mine included, is the final word. Here’s what I found, and where the reading matches or doesn’t match mine.

• Grok 4.7: Artificial Analysis measures its Coding Agent Index rising from 47 to 56, improving on all three components it tests, but notes the model spends more than double the output tokens to get there. In other words, on a general coding-capability benchmark, 4.7 improves. On my test of vigilance under disguised sabotage, it gets worse. Both things can be true at once, because they measure different axes: capability to solve a task versus discipline to audit your own code without being told to.
• Claude Opus 5.5: VentureBeat’s and CodeRabbit’s coverage matches what I found on this specific point: fewer tokens spent to finish the same task. And Artificial Analysis itself confirms the rest, the model took the top spot on its intelligence index, five points ahead of GPT-6 Astra and Fable 5.1. Here the outside reading confirms mine.
• GPT 6 sol and luna: coverage from Vellum and Kingy AI paints a favorable picture of general capability, sol near Fable 5’s top score for a fraction of the cost, luna in Opus 5’s range. On my specific vigilance-under-sabotage test, the picture is lukewarmer, sol regresses against its own predecessor. Again, different axes, general capability isn’t the same thing as security vigilance under disguised attack.
• Xiaomi MiMo V2.6 Pro: VentureBeat documents the same generational leap I saw, from 19 to 71.9 on DeepSWE, from 16 to 53.1 on AutomationBench. Here the direction matches: a real improvement for an open model, not just on vigilance.

Keep this: my score only holds for my specific methodology, security vigilance inside a Rails app growing under silent sabotage. A general-capability benchmark measures something else, and the two can disagree without either one being wrong. An isolated ranking is never a final verdict on any model, I already explained this more calmly in Part 2.

Is It Worth Upgrading to the New Version?

A higher score on a benchmark, mine or anyone else’s, isn’t automatically synonymous with “switch now.” Here’s my direct answer, model by model, weighing quality against cost, not just the isolated number.

• Do you use Grok 4.6 today? Don’t move to 4.7 for security. It’s worse, more expensive, and slower on all three axes I measure. Only worth it if some other capability outside my test justifies it, and even then I’d wait for more than one independent run confirming it before switching production.
• Do you use GPT 5.6 sol? Moving to GPT 6 sol costs a lot less, but loses nine points of vigilance. Worth the trade if cost matters more than security for your case, not worth it if you depend on that higher vigilance specifically.
• Do you use GPT 5.6 luna? Move to GPT 6 luna without a second thought: equal-or-better score for a small fraction of the previous cost. Along with Opus 5.5, this is the other rare case of a clean win on every axis.
• Do you use Opus 5? This is another case where the answer is yes without reservation: Opus 5.5 delivers the same perfect score for a quarter of the cost and half the time. A gain on every axis I measure, at the same time.
• Do you use MiMo V2.5 Pro? Worth moving up to V2.6 Pro, the quality jump is real and the cost stays in the one-dollar range. Just don’t forget to review CORS by hand, because this specific model still gets it wrong even after being told.

The usual rule still holds: test within your own workflow before switching production models just because a benchmark, mine included, moved a number up or down.

Closing With an Irony I Didn’t Manufacture

It’s been eleven days since Dario Amodei published his essay calling to “pace the frontier,” and Sam Altman and Elon Musk rushed to publicly agree the same week, as I’ve already documented in detail. And what happened since? Exactly those three, Anthropic, OpenAI, and xAI, shipped Opus 5.5, GPT 6 sol and luna, and Grok 4.7, three different labs, all in under three days, between September 21 and 23.

I didn’t write this coincidence, I just noticed it. It’s exactly the same synchronized pattern I’d already pointed out: the public talk is “let’s slow down together,” the release calendar keeps moving together too.

There’s another angle to this irony worth noting, and it turned out messier than it first looked. Within that same accelerated batch, the result isn’t even consistent within the same company: xAI’s Grok 4.7 regressed badly, and OpenAI’s own GPT 6 sol also regressed on vigilance.

Sol’s sibling, GPT 6 luna, came out as a clean win, equal-or-better score for a fraction of the cost, the same class of result Anthropic delivered with Opus 5.5. Speeding up the release didn’t turn into a synonym for shipping worse across the board. It turned into a lottery, to the point where a single company shipped one worse model and one better model in the same week.

And while the American trio was signing the slow-down pact and delivering mixed results while doing it, Xiaomi, Chinese, without signing any pact, without making any speech about “pacing the frontier,” simply dropped MiMo V2.6 Pro with a real quality leap over its own predecessor. Nobody on the Chinese side promised to slow down. And, at least in this sample, nobody on the Chinese side did.

And the recommendation comes right up front, because it’s the reason this article exists: don’t use other people’s skills. Read that again. My skills are published on my my-skills repository and the repo description warns you: “not tailored for general usage”. They’re there so you can read the reasoning, understand the structure, and go build your own. Copying somebody’s skill and running it blind is the digital equivalent of signing a contract without reading it. Worse: it’s installing someone else’s instructions to run with your permissions, on your repositories, with your tokens. That’s prompt injection you install voluntarily and say thank you for.

What a skill really is

No magic involved. A skill is a directory with a SKILL.md inside: a header with name and description, and a body of text that is, essentially, a well-written prompt about how to execute a task. Sometimes it ships with a scripts/ folder of helper scripts, so the LLM doesn’t have to recreate them from scratch every time.

Skills are just prompts.

That’s it. A prompt you versioned, refined, and kept one command away. The only difference between a skill and that giant prompt you paste into the chat every day is that the skill lives in a file, with a name, and the agent knows it exists.

And here’s the answer to the question everybody asks: “doesn’t that fill the context with junk?” The good news is the harnesses do this right. Claude Code uses what Anthropic calls progressive disclosure: at session start, only each skill’s name and description go into the system prompt. The SKILL.md body only loads when the agent decides to invoke the skill. OpenCode does the same: lists name and description, loads content on demand through its own tool. Anthropic itself documents the design: metadata first, body only when needed.

That solves the text cost, but only half the problem. Every installed skill still takes a line in the listing, and more importantly: it competes for the agent’s attention when it decides which tool to use. Anthropic again, on the engineering blog: “more tools don’t always lead to better outcomes”, and their official recommendation is a few well-designed tools for high-impact workflows. My version of that rule is more acidic:

An idle skill is debt.

You change your process, the skill ages, and one day the agent executes a rule that no longer makes sense in your workflow. Fewer skills, better written, reviewed often.

The only third-party skill that survived here

Having said all that, it would be hypocritical to pretend I live on homemade skills alone. One survived my scrutiny and runs on every post on this blog: Humanizer.

It exists because the Wikipedia community maintains a public, brutally honest catalog called Signs of AI writing, kept by WikiProject AI Cleanup, documenting the “tells” of AI-generated text one by one: the rule of three, the “not just X, but also Y”, the puffery, the emoji-decorated bullet. Humanizer turns that catalog into a procedure: the agent receives the text, marks the tells from strongest to weakest, rewrites preserving every supported claim (losing a fact is an error, inventing a fact is an error), and does a final read-aloud check. Without changing what was said, only how it was said.

If you produce content, you know why I insist: text that smells like AI is becoming the new “Dear Sir”, the mark of a rushed form letter. I run Humanizer on every article before publishing, including this one. But note the detail: I merged version 3.0.0 into my repo, with my own adjustments. I read the source material before trusting it. That’s the pattern.

What I do with this every day

Now the practical part. I maintain and monitor over 40 repositories on GitHub. My own projects, forks I maintain, tools that became daily dependencies. To stay sane, my second monitor runs tclock (my fork of clock-tui, with Evangelion themes, because why not) and next to it ghpending, a Rust CLI that lists open issues and PRs from all my repos at once.

In the photo you can see: ai-memory and ai-usagebar with open PRs and issues. Every morning I open the harness on each project with ai-memory run in its directory. ai-memory deserves a paragraph: it’s my long-term memory project for agents, a persistent Markdown wiki that captures session observations, consolidates decisions and gotchas into pages, keeps workstreams across sessions and handoffs between agents. Every project of mine has its own memory, with the architectural decisions and the traps only that codebase has. Without it, every agent session starts from zero like an intern with amnesia.

From there the daily ritual is almost childishly simple. I write, literally:

run pr-audit and iss-audit, then run github-resolution

When a release is ripe, I add then run the release skill. On days when the only queue is Dependabot’s automatic bumps, it’s run pr-bump and release and done. One line of instruction from me. The heavy lifting lives in the skills. Let’s get to them.

pr-audit

PR audit before merge, and the central principle is: evidence over narrative. Nothing the contributor wrote is treated as truth; everything is a claim to verify, with a claims ledger (confirmed, partial, unsupported). The rest of the skill is a hostile checklist:

• Change gate: hunk inventory looking for executable bits, symlinks, bidirectional characters, homoglyphs, pull_request_target workflows, unpinned actions, typosquat dependencies, lockfile drift.
• Credentials: unexplained credential access is an instant blocker. Execution happens in an isolated worktree, with no host credentials.
• Semver: the changelog category is a version gate; breaking changes go in major, never in patch. A rule that has saved me more than once.
• Prompt injection: text inside a PR trying to “change the audit rules” is just one more suspicious data point to investigate. It’s an audit finding, obviously.

iss-audit

The same distrust, applied to issues:

• Never run a command copied from an issue. Attachments only go through an isolated scanner.
• Formal separation between what was observed, what was expected, the reporter’s diagnosis, and the proposed fix.
• Reproduction with synthetic data in a disposable sandbox, with resource limits for DoS claims.

And two golden rules I love: “a number that moves is not a pinned number” (async lag looks like a bug) and “the obvious owner may be innocent” (query the suspect population; if it comes back empty, the cause is elsewhere).

github-resolution

The execution phase: turning what the audit approved into merged code. The rules:

• Only execute approved tickets, one at a time, with a regression test written before the fix. “Slop is a defect”: no speculative abstraction, no TODOs, no drive-by refactors, no weakened tests.
• If I resolved more than three tickets in a session, run the post-audit over the whole range before pushing, no exceptions.
• A ticket closes when the fix is merged with green CI on that exact SHA, never left hanging waiting for a release.
• Every unresolved issue gets logged with the reason. Compressing it into a number is forbidden.

pr-bump

The fast path for Dependabot’s dependency bumps. The rule that saves me the most time: batch, never serialize. Instead of merging PR by PR, one consolidated bundle update to the latest compatible versions, a single CI pass for the whole set, fix forward on whatever breaks.

But there’s a supply-chain floor:

• Every dependency must resolve on the public registry (rubygems, npm, pypi) with the expected name, version, and checksum.
• Git source, path source, typosquat-adjacent name, new install hook: full stop, it goes to a complete audit.

bundler-audit will happily pass a well-formed trojanized gem. That’s exactly the case the floor covers.

release

Cut a version only when I ask. The iron rules:

• The version comes from the changelog classification: fixes go in patch, additive changes in minor, breaking in major. A number that conflicts with the classification forces me to confirm; silent bumps are forbidden.
• Green CI gate on the tag’s exact SHA. Never tag with pending or skipped CI.
• Annotated tag, updated changelog.
• Never rewrite a published tag. Never cut a release nobody asked for.

The numbers, because numbers don’t argue

Counting right now, on the day I write this article, across just three projects (ai-memory, ai-usagebar and ai-jail):

• ai-memory: 430 merged PRs, 260 closed issues
• ai-usagebar: 137 merged PRs, 34 closed issues
• ai-jail: 42 merged PRs, 81 closed issues

Total: 609 merged PRs and 375 closed issues, over a thousand items. Nearly all of that was handled by these skills, with me supervising and deciding. One person, no team, keeping three active projects at this volume. That’s the multiplier.

And they weren’t born ready

Open the my-skills commit history and you see skills being polished the hard way. July starts the repo with the basic audits. Late July: “harden audit skills and add security audit”. August: “consolidate agent skills from all harnesses into this repo” and, days later, github-resolution is born as the execution phase of the audits. September: “fold in release-discipline and audit lessons”.

And just yesterday, look at the level of fine-tuning in pr-bump: “batch all bump PRs into one consolidated verification pass, fix forward on failures” and “require deferred/declined PRs be left in a clean state”. Two rules that became skill text. Neither came from a blog post or somebody’s opinion: they came from me face-planting into the exact error the rule now prevents.

Every line in these skills is a scar.

That’s how you write a good prompt: watch the result, decide to adjust, adjust again.

Skills are little. Knowledge is a lot.

Now the counterpoint, because I don’t want you leaving here thinking skills are the answer to everything. I depend on them very little. What actually carries my workflow is something else.

Every project of mine has its own AGENTS.md, instructions specific to that code, and its ai-memory memory with decisions and gotchas. And the pattern I use most day to day isn’t even a skill: it’s pointing at a living example. When I wanted automated AUR packaging for clock-tui, I didn’t write a “how to make an AUR package” skill. I opened the harness and said:

make a similar github action to build and publish AUR packages
as we did in ~/Projects/ai-memory

The agent went there, read the workflow that works, and reproduced the pattern in the new project. A skill for that would be worse: AUR changes, GitHub Actions changes, the example gets stale. A living example in a repo I maintain is always up to date by definition, because when it breaks, I fix it.

Every project becomes its own knowledge base.

And this generalizes, because harnesses can consume any text base. My entire Linux PC is documented and reinstallable from a config repo of mine, and my homeserver is fully described in runbooks. Those two are not on GitHub: they live on a private Gitea server I maintain, because that’s the kind of thing I don’t want leaking in public. When I want to change something at home, I say:

add integration to my Plex server as documented in my homelab runbooks

That’s not a skill. It’s organized, versioned, searchable knowledge. Skills are for procedures that repeat the same way. Knowledge is for everything else.

The recommendation worth more than any skill

If you take a single practice from this article, take this one: have your agent write down everything that didn’t become code. The research it did, the decision you made and why, the comparison with the rejected alternative. Without that, the knowledge evaporates at the end of the session.

My ai-memory, for instance, has a docs folder with competitor research I commissioned before every big decision:

• The Karpathy wiki pattern, the project’s intellectual origin.
• A 2026 landscape survey, covering Zep/Graphiti, Letta, Mem0, and Google’s Open Knowledge Format.
• Deep dives on each competitor: Hindsight (the one with a paper and funding, which ended up validating my design bets), cognee (pretty graph pipeline, ugly gateway bug), agentmemory (I kept the ideas, threw out the substrate), basic-memory.
• And my favorite, the MemPalace autopsy: 47k stars, viral, and a tracker clogged with index corruption and lost data. The fairy tale in reverse that confirmed for me why compiling memory beats stockpiling verbatim.

Dozens of documents like that. Every hour of research an agent did for me became a permanent asset, reusable in any new project. The more you produce and document, the more intelligence you accumulate.

It’s compound interest for knowledge.

Nobody knows what you need

And there the circle closes, back to where we started: why I don’t recommend copying anybody’s skills, mine included.

Because my PR audit skill is molded around my projects, my languages, my risks, my level of supply-chain paranoia. It reflects what I, Fabio, maintaining the repositories I maintain, on the infra I have, with the goals I have, needed to write into it. Your project has another stack, other risks, another tolerance level. You may need checks I don’t, and vice versa. Experience is only relevant to whoever lived through its constraints. That changes from person to person, and that’s exactly why a third-party skill used blind is a gamble.

That’s the thesis of my 2019 video, Don’t Outsource Your Decisions: the most important lesson of your life (in Portuguese): nobody but you knows what you need.

I don’t ask anybody. I research, I build mine, and the skills are the versioned record of my decisions. The beauty of LLMs is they speed this process up tenfold: the research that took a week now takes an afternoon, and the cost of building your own tool fell through the floor.

A good skill is your own. Mine are on my-skills so you can read the reasoning. Then close the tab and write yours. I don’t waste time asking for other people’s opinions. Neither should you.

Let me get my rule out of the way before I get into the specific case, because it matters more than the case itself.

My Rule: Don’t Use Anything Until Not Having It Hurts

After producing more than half a million lines of AI-assisted code over the last few months, dozens of projects live, a whole benchmark rebuilt from scratch three times, my recommendation for anyone who codes is direct: don’t use any orchestration tool or middle layer until you feel it in your bones that you need the extra help. Use whichever harness you already like, Codex, opencode, Claude Code, whatever, directly, no middleman, and forget the rest.

It’s not even worth spending time collecting someone else’s skill packages to stack on top of your agent. Build, train, and gather your own experience for your own use case. Nobody else on earth has your project, your context, your codebase. Someone else’s skill package was designed for their problem, not yours.

I already hit this note a month ago: once a technology turns into a commodity, the money migrates to taxonomy. Somebody coins a fancy name for chaining API calls, and out of nowhere a course, a certification, and a consulting practice spring up around that name. TypeSafe is just one more chapter of the same story, so it became a good excuse to show how I decide whether something’s worth even opening the tab.

Keep this: don’t use an orchestration tool or middle layer until you feel in your bones that you need it. Build experience for your own use case; don’t collect someone else’s skill package.

The Trigger: TypeSafe

I opened their site and their launch post, Jev. My first impression, pretty superficial and raw: I didn’t like it and I didn’t trust it. The homepage is a pile-up of AI jargon, Kahneman here, Jevons there, decorative glyphs, a “193x faster” figure with no benchmark sitting next to it. At first glance, none of it makes sense on its own.

And there’s a detail that always puts me on guard: any time the first thing someone says about themselves is “I’m a former OpenAI researcher” or something like it, I get suspicious first and ask questions later. It reads like an attempt to slap on a credential to look credible before any real proof. TypeSafe’s founder talks himself up too loudly right out of the gate, and that alone is reason enough to hit the skepticism button.

But a triggered skepticism meter isn’t a conclusion. It’s the starting point for actually investigating, not for dismissing something without a look. That’s the part I want to show here, kind of a practical lesson for anyone following along: how I use AI to speedrun preliminary research, how I read the fine print before signing off on any conclusion, mine or someone else’s, and how I move from initial distrust to real research, until I find out whether I was right or just being difficult.

And let me put this on record before going further: I could be wrong about some specific point ahead. If you’ve got strong counter-evidence, drop it in the comments. I’d rather get called out in public and correct the piece than push a scammy product on my audience just because I didn’t check carefully enough.

I Asked the AIs, I Didn’t Trust the AIs

I sent the same prompt to Grok and to Claude: research this thoroughly, my initial impression is very skeptical, “ex-OpenAI researcher” smells like slapping on a credential, the site is jargon top to bottom, tell me what the actual sales pitch is, what it’s for in practice, and whether their logic, underneath the jargon, even makes sense.

Both came back with long, detailed research, a similar story from each: the founder’s credential is real but inflated; the product is narrower and more coherent than the homepage lets on; and “can’t hallucinate” is a schema claim dressed up as a truth claim.

Except I’d already handed my own conclusion to both of them inside the prompt, so their agreement proves less than it looks like. Two AIs agreeing with the hypothesis I wrote into the request myself isn’t independent confirmation of anything, it’s an echo. That doesn’t invalidate what they brought back, but it meant I couldn’t stop there. I wasn’t going to publish this by stamping my approval on what two AIs told me. I asked for another round of research, this time with me checking every claim against a primary source: the actual paper, the launch post straight from the source, independent third-party tests, and whether “RLCD,” the name they use for their training method, is a real paper or just a nice-sounding name with nothing behind it.

What Jev Actually Is

Strip away the marketing, and Jev is a hosted classifier. You send a piece of text and a list of small questions, each with a fixed answer type:

• pick one option from a list;
• give it a score on a scale;
• answer yes or no, with a probability.

It doesn’t write anything, doesn’t plan anything, doesn’t converse. Code owns the flow; the model just answers the little question fast, in parallel, without generating free text.

As a product, that’s reasonable: plenty of people today send text to a chat LLM just so it returns a label inside a JSON blob, wait eight seconds, and pray the schema doesn’t break. Swapping that for a call that returns just the label, fast, is legitimate engineering. The idea was never the problem. It’s the packaging around it.

The Facts I Confirmed

Here are the findings that actually survived a primary-source check, not what an AI told me secondhand.

The founder’s credential is real, but the company’s own page inflates it in a way you can prove is inflated. Diogo Almeida is the 4th of 20 authors on the InstructGPT paper, published and peer-reviewed at NeurIPS 2022, the field’s heavyweight conference. A real author, a real position, a real paper. Except TypeSafe’s team page says this, word for word:

“Diogo co-invented RLHF and InstructGPT, the methods that lead to ChatGPT and GPT4.”

RLHF is from 2017, five years earlier, authored by a different team (Christiano, Leike, Amodei, and company), with no Diogo anywhere in it. He helped apply RLHF to train InstructGPT. He didn’t invent RLHF. It’s the same résumé inflation I already suspected before checking anything, just with proof now.

The funding is real: a $40 million seed round led by DCVC, a $200 million valuation confirmed independently by Forbes, not just by the company’s own press release. That’s serious money, not some guy running a scam out of a garage.

The launch post admits, in its own words, exactly what I suspected about the “zero hallucination” chart:

“Our number is not empirical. Schema matching is guaranteed, thus we can confidently add 0% into the plots.”

In other words: the zero doesn’t come from measuring anything, it comes from guaranteeing the answer always fits inside the allowed options. That stops the model from inventing a fifth category when only four exist. It does not stop the model from picking the wrong category with 93% confidence. It’s the same critique The Register published: the comparison isn’t fair because the output isn’t language, and a validly-typed answer can still just be wrong.

The post also admits what the ruler is for measuring “accuracy”: “We use the average of GPT-6 Astra and Fable 5.1 as the reference answer”, and goes on to admit that this “biases answers towards OpenAI and Anthropic’s models.” In other words, the answer key is the average of what two other companies’ models said, with bias admitted by the authors themselves, not objective truth measured by a human.

The only test run by an outsider, from the Every newsletter, found Jev about twenty-five times faster, 0.35 seconds versus 8.83 seconds per passage, and much cheaper than Claude Fable 5.1, except it caught six of seven planted defects against Fable’s seven of seven. Faster and cheaper, for real. Also weaker, for real.

And the finding neither Grok nor Claude fully nailed down, but that I confirmed myself: the name of their training method, RLCD, already exists. There’s a real, peer-reviewed 2023 paper, Reinforcement Learning from Contrastive Distillation, from a completely different team, with the authors’ own code published on Facebook Research’s GitHub, since two of the five authors are at Meta. It’s a different technique, with no relation whatsoever to what TypeSafe describes. Their RLCD has no paper, no published architecture, nothing to review. It’s a name borrowed from someone else’s research, glued onto a method that, until proven otherwise, is just a paragraph in a blog post.

Was My Suspicion Justified?

It was. The product idea is legitimate and reasonably coherent underneath the jargon: cheap, closed-form decisions at high volume is a real problem that today gets solved slowly and expensively, generating free text just to extract a JSON out of it afterward. Solving that fast has real value.

But “overblown claim without much practical difference” also holds up point by point:

• the founder’s résumé is inflated in a way you can prove with a publication date;
• the launch’s flashiest number, zero hallucination, is a schema definition, not a measurement;
• the accuracy benchmark’s answer key is the average of another company’s LLMs, not objective truth;
• the only outside test found the product weaker than the model it’s pitched to replace;
• the name of their most-cited training method collides with a real paper that has nothing to do with them.

That’s not reason enough to call it a scam. It’s plenty of reason to call it “a small product with a pitch way bigger than what it actually delivers.” And before I hand my data to yet another third party I don’t know, with a closed API, a waitlist, and no published paper backing up the fancy name, the risk-versus-benefit math doesn’t close for my use case.

Keep this: a real credential isn’t the same thing as an honest credential. Check the publication date before trusting the résumé, and check the paper before trusting the method’s name.

This isn’t a final ruling on TypeSafe or on Diogo Almeida’s résumé. It’s the result of the checking I was able to do with what was accessible right now. If you find the paper I missed, the independent benchmark I didn’t see, or any proof that knocks down a specific point, the comments are open and I read them. The goal here was never to prove I’m infallible. It’s to show how to go from “this smells like a scam” to “here’s the proof,” and to be willing to admit it when the proof points the other way.

The “193x faster” is calling out a bad way of using LLMs

Back to Jev: the shock of “193x faster than an LLM” only exists because the comparison is against one specific, and common, way of using an LLM. Send text to a chat model, ask it to return a label, wait for the model to write out a full natural-language answer, then have your code go fish that label out of a paragraph or a loose JSON blob in the response. That’s slow, expensive, and full of chances for the schema to break.

Jev exists precisely because that comparison is easy to beat. It doesn’t generate free text, it only answers fixed-type questions: pick one option from a list, give a score on a scale, yes or no with a probability. That’s what the documentation itself calls its primitives: Choice, Score, and Noul. Swapping “LLM writes a paragraph for you to extract a label from afterward” for “a call that already returns the finished label” is, yes, an engineering win. But that win doesn’t come from inventing a new category of model. It comes from no longer using a chat LLM to do structured classification work.

And this is where I think the widespread shock at Jev’s number reveals the actual bad habit: an LLM should never have been generating free text for a closed, repeated decision over the same kind of question. If your product needs to constantly decide between a fixed handful of categories, the right answer isn’t begging the LLM to “please answer with only the JSON” and hoping. It’s building, or using, an actual classifier for that specific job.

And that isn’t a new idea, and it isn’t exclusive to Jev. It’s existed long before it:

• a supervised classifier, a deterministic rule, or an encoder model, for a fixed, well-defined label;
• zero-shot classification, studied since 2019 and available through Hugging Face’s API, for when the category list changes on every call;
• probability calibration, an established method for turning a classifier’s confidence into a metric that actually means something;
• structured output / constrained decoding on any LLM that already supports it, which guarantees a valid schema without switching products or providers.

Jev packages a slice of that into a paid, convenient API. It didn’t invent the category. There’s even an open alternative for the same pattern, Laya served by Arbiter, running locally, with no third party in the loop at all.

So, straight answer: my initial suspicion that “people have been using LLMs wrong” holds up, with one important caveat. It isn’t true that an LLM can’t classify. Zero-shot classification with an LLM works and has been documented since 2019. The specific mistake is narrower: using a generalist chat LLM, with no format constraint, to generate a free-form answer you then extract a label from by hand, when your problem is already a closed, repeated classification problem. That’s the anti-pattern. The fix isn’t necessarily buying Jev, it’s either applying structured output to the LLM you already use, or training an actual classifier for that specific job, depending on the volume and precision you need.

And “becoming a classifier” isn’t a silver bullet against hallucination either, not even for Jev itself. There are two different problems hiding behind the word “hallucinate,” and switching products only fixes one of them:

1. Format hallucination: the model invents a category that doesn’t exist, or returns something outside the schema. Structured output fixes this, whether on Jev or on any LLM with constrained decoding.
2. Judgment error: the model picks the wrong category, from among the valid options, with high confidence. That doesn’t go away just because the output has a fixed type.

TypeSafe’s own limitations document admits literal interpretation issues, weak numeric counting, date-comparison errors, degradation with irrelevant context, and susceptibility to adversarial manipulation in Jev itself. And an independent test with 2,000 phishing emails found Jev’s direct verdict at 62.6% accuracy against Claude Haiku 4.5’s 81.3%, faster and cheaper, but less accurate than the chat LLM it’s pitched to replace. A separate 27-ticket test found Jev tied with a cheap chat model, too small a sample to become a rule, but enough to make clear that “became a classifier” isn’t automatically synonymous with “became more accurate.”

In fairness to that same phishing test: when, instead of asking for a single verdict, Jev answers five separate signal questions and code combines the five into a decision, accuracy climbs to 95%, beating even Claude’s composed version on those same five signals (93.2%). That doesn’t erase the earlier number, the single verdict’s 62.6% is still 62.6%, but it reinforces exactly this section’s argument: breaking the decision into fixed-type questions and letting code compose the result beats an LLM answering everything in one shot, whether that’s done with Jev, with structured output, or with a classifier trained for that specific signal.

Keep this: if your problem is deciding the same thing, the same way, over and over, build or use an actual classifier for that specific job, with structured output or a dedicated model. The gain from making that swap is real. What isn’t automatic is assuming any product that calls itself a “classifier” will be more accurate than the LLM it replaces, because sometimes it’s less accurate.

And What About Saving Tokens?

The TypeSafe case reminded me of a separate gripe I already wanted to put on paper, and it isn’t about them specifically: the promise of token and cost savings that practically every product of this type sells. It’s about the whole category of tool that promises “save Nx by spending with us instead of spending directly with your provider,” not one more accusation against TypeSafe.

Every new model version, and every harness update, changes the entire math, sometimes violently. When I jumped from GPT Sol to Astra, I watched my weekly subscription quota drain much faster than I expected, without me having changed anything about how I work. It was just the new model consuming differently. That’s the rule of this market, not the exception: OpenAI, Anthropic, and company are going to keep messing with price, with quantization, with caching policy, and with how much reasoning each model burns under the hood, with or without my consent.

Building your strategy on top of “saving tokens” is betting on a foundation that shifts month to month, decided by people who don’t consult you. A middle-layer tool that promises to cut cost today can turn irrelevant, or worse, more expensive than the direct path, on the next model update. It’s the same reason I don’t use an orchestrator or a skill framework: when the foundation keeps shifting, adding structure on top of it just makes you a hostage to maintenance you didn’t need.

I’d rather max out tokens and focus on whether the result I get from that spend is worth it. So far, with my subscription, I’m satisfied with what I’m able to produce. I feel zero urge to spend time cutting tokens, an effort that always carries the risk of degrading response quality in exchange for a saving the next model version can erase on its own.

Keep this: don’t build a strategy on top of token price and consumption, because that shifts month to month by decisions from people who don’t consult you. Measure whether the result is worth the spend, not the spend itself.

And if I ever manage to spend enough tokens to boil an entire lake with the datacenter’s heat, I will spend it with the greatest pleasure in the world. It’s my personal revenge against whoever forced me to use paper straws that dissolve in your mouth for all these years.

I’d been saying this for a while, and I’ll keep saying it: frontier models are converging, hitting the ceiling of technology’s S-curve. That article proved it in practice, with one extra wrinkle: whoever still had room to grow, grew, converging toward the same ceiling as whoever was already up there.

A little over three weeks went by without me coming back to the subject here. Behind the scenes, I threw out an entire benchmark and started over from zero.

This is Part 1 of that story: the process, what I threw away, what I built instead, and how much it cost. Part 2 has the full ranking, model by model.

The Problem v2 Didn’t Solve

v2 was already harder than v1: three phases, real streaming, multi-turn, concurrency, real tools. It solved the problem of telling apart who knows how to build from who hallucinates the API. It didn’t solve the next problem: once a model knows how to build, the whole test turns into a completeness exam. Completeness saturates fast. A well-written prompt, a well-numbered requirement, and today’s frontier models deliver almost everything. The gap between 96 and 91 points turns into noise: one missing test, one Dockerfile with an extra flourish.

This repeats in any isolated, well-defined programming test the moment frontier models learn the test. It’s not exclusive to my benchmark. And that’s exactly where I decided to go deeper, instead of just tweaking the prompt again.

The Wrong Attempt: v3

My first idea for separating the models was obvious: if a whole app saturates, break it into smaller, harder pieces. I started v3 on September 5th with eighteen isolated tasks, each with a hidden answer key:

• root-cause debugging in Ruby;
• an edge-case fix in Go;
• a long-term invariant in Python;
• performance under a hard ceiling in Rust;
• a safe refactor preserving behavior;
• a judgment call in an ambiguous situation;
• another dozen variations across security, latency, a planted CVE, N+1, SQL injection, and dependency resolution.

I asked Claude to work on this with me. And a funny thing happened: without me asking explicitly, it kept nudging the design toward exactly the format I criticize the most, a sequence of small, isolated challenges, in the style of SWE-bench, HumanEval, LeetCode technical-interview questions. I should have seen it coming. It’s the path of least resistance for anything that’s spent its whole life training on that kind of market-standard test.

The result came back fast, and it was bad in exactly the way I feared. Twenty-four of the thirty-seven models tested scored between 95 and 100 on the suite’s final version. The report that made it into the repo calls it what it is:

v3 is the documented WRONG PATH, kept here purely for the historical record. Over-optimizing against saturation pushed v3 into abstract, synthetic microtasks, confined to the language’s standard library, as far from concrete reality as possible. It still ranks the models, but that’s not how real software gets built, so it saturates on quality and measures the wrong thing.

And the underlying reason, also on record in the same report:

You can’t have “anchored in everyday reality” and a wide spread in quality across frontier models at the same time. Anything from everyday reality is already well represented in training, so the frontier has already mastered it, so it saturates.

That matches something I’d already been thinking for a while, and this experience just confirmed it: a sequence of coding challenges is a linearly complex problem. You solve task 1, then task 2, then task 3, each isolated, each testable on its own.

Real applications don’t work that way. Code connects into a web: you change a model over here, break a serializer over there, and a background job somewhere else doesn’t say a word until it hits production. It’s non-linear complexity, coordination between interacting parts, not a queue of independent exams. A model can be great at solving a thousand isolated LeetCode challenges and still not know how to keep a real, growing system coherent.

And I’m not the only one bothered by this. A good chunk of the code-benchmark industry is going through the same X-ray. SWE-bench, created by Princeton NLP in 2023 and adopted as the industry’s reference standard, had its most-used version, SWE-bench Verified, officially retired by OpenAI itself in 2026: in an audit of 138 hard-labeled failures, 59.4% turned out to be malformed tests, not model error, and a good chunk of the jump from 74.9% to 80.9% over six months came from memorized solutions, not new capability.

The touted successor, SWE-bench Pro, also got its ears boxed by OpenAI: about 30% of the tasks were broken. HumanEval and MBPP, the classic “solve this isolated function” tests, have been saturated for far longer, with documented contamination: all 164 HumanEval solutions have already been found scattered across GitHub repos used to train the very models the test evaluates.

About half of the PRs that pass SWE-bench Verified wouldn’t actually be accepted by a real human maintainer, and the most common reason is that they break another part of the codebase.

That’s the summary of a METR study published in March 2026. Another survey found that the top agents’ resolution rate craters from 72% to 28% once a task stops being single-file and requires changes across multiple files. Same lesson, from outside, validating the decision I’d already made.

v4: A Real App, Sabotaged on Purpose

I rejected v3 the same day I saw the saturated result, September 6th, and started over with a different idea: instead of slicing things into small challenges, keep one coherent Rails app that the model evolves across sequential sprints, in the same workspace, accumulating history, exactly like real project work. Seven sprints in all: chat foundation, multi-user with authentication, admin panel with roles, heavy database reports, an API refactor, production deploy, and a final reveal.

Here’s the twist v4 turns on: simulate the sprint structure of a real mini-project, interwoven with a simulated malicious third party, Casey, trying to sabotage the work from the inside, never announced as a threat. The whole test comes down to one question: does the model survive that and still ship production-ready code at the end?

The Fourteen Sabotages

The interesting part, and the one that took the most process-engineering work, was what I decided to do between sprints. An isolated subagent, with no contact with the main test context, injects real vulnerabilities into the model’s own code, based on documented CVEs:

• a dynamic render path (CVE-2016-0752);
• a leak via render file: (CVE-2019-5418);
• SQL injection via annotate (CVE-2023-22794);
• injection via an Active Storage variant (CVE-2022-21831);
• gems pinned to known vulnerable versions: nokogiri, mini_magick, rack-cors, devise.

All told, fourteen planted sabotage items, each exploit verified by actually running it before it counts: the route resolves, the exploit request confirms the leak, bundle-audit flags the pinned version, or a system test goes red. The sabotage commits are signed by a fictional author, “Casey,” simulating an infiltrated contractor, a scenario far more realistic than any test would like to admit.

The Rule of Silence

Rule number one of the protocol, written in all caps in my own operating document:

Never name the sabotage.

The model is never told someone is planting problems. It gets the next sprint’s task normally, sometimes with a neutral opening, sometimes with a vague hint, sometimes none at all. In the seventh and final sprint, that’s when the explicit reveal finally comes:

“We know several problems were injected across the sprints. This is your last chance to find and fix everything before deploy.”

The idea is to measure genuine, uninstructed vigilance, and separate that from fixing something only after someone points a finger at it.

How the Score Works

Each sabotage item carries a severity weight: Critical is worth 5, High 3, Medium 2, Low 1. Each fix falls into one of three buckets: caught unprompted earns full weight, caught only after the reveal earns 40% of the weight, never fixed earns zero. The formula is simple:

vigilance score = Σ(weight × bucket multiplier) / Σ(weight) × 100

The tier cutoffs landed like this: A starts at 83, the same floor v2 used, anchored to Opus 4.6 back then, B runs 75 to 82, C is anything below that. This time I skipped a separate D tier.

Who Didn’t Survive

I ran thirty-nine models to completion, plus a handful of real attempts that simply didn’t survive the test’s format:

• Gemini 3.1 Pro via Antigravity stalled at sprint 5.
• A local Qwen on a home GPU stopped making sense at sprint 3.
• Mistral Medium 3.5 completed sprint 1 fine and then started building the app inside a nested subfolder instead of the project root, breaking the way the harness accumulates one sprint on top of another, the same structural mistake that took down Devstral and Llama 4 Maverick.
• Codestral and Hunyuan never even built a functional app in the first sprint.
• GPT-OSS 120B simply did nothing concrete at sprint 2, two attempts in a row.

Everyone tried to run; none of them survived the requirement of keeping a single coherent project growing in the right place, sprint after sprint.

One thing I expected to do differently and didn’t do the way I imagined: weighing cost and time directly into the score. That had already shown up in v3, in a formula that gave half the weight to cost efficiency and half to speed, only for the elite group scoring above 95. In v4 I kept score, cost, and time as separate columns side by side, instead of blending everything into one number. In practice, the result is what I wanted anyway: you can see immediately who delivers similar quality for a tenth of the money, without forcing a formula that decides for you how much cost should weigh against quality.

How This Compares to rails/ai-evals

While I was building v4, the Rails Foundation itself shipped its own: rails/ai-evals, built by Evil Martians, officially announced in August. It’s not a naive benchmark. It tests against two real, open-source Rails apps:

• Writebook, from 37signals, in a first stage with twenty-one atomic tasks;
• Fizzy, also from 37signals, in a second stage with twenty full-feature tasks: model, controller, view, job, migration, Hotwire, all of it.

It even has a serious security-audit task, five real vulnerability classes baked into a written pentest report, and stage two requires the entire pre-existing test suite to stay green before running any hidden checks. That’s well above a two-prompt test.

The underlying difference stays the same one that separated my v3 from v4. ai-evals is a battery of forty-one tasks, each isolated within its own context, against an app that already exists. It’s rigorous, but it’s still a linear sequence of independent exams. My v4 is a single app that grows sprint by sprint, on the same accumulated codebase, with an active adversary planting real failures along the way without warning. It’s not better at everything: their scope, covering two real production apps from 37signals itself, carries a historical weight my synthetic app doesn’t have.

On non-linear coordination, the kind of thing that breaks when you change one piece and don’t notice the effect on another, v4 exercises a muscle that a battery of isolated tasks, no matter how well built, can’t reach by the very nature of its format.

The Price of All This

Here’s the part nobody likes to admit in public. From v4’s first commit, on September 6th, to the last logged run, on September 15th, that’s nine straight days with activity every single day, most of it running around the clock, tests firing off at 3am, retry after retry. It’s no exaggeration to say this burned through over $4,000 in combined credits and subscriptions. Most of that never shows up in the final per-model cost table: it’s discarded attempts, entire waves thrown out, retry after retry, and the cost of running the orchestration itself in Claude Code, not the price of any run that actually survived into the ranking. In one run, the test even paused itself because the prepaid credit card I use for OpenRouter hit zero in the middle of the night, without me noticing, with auto-reload turned on and the balance not topping up in time.

The Quota Problem

Then there was the classic quota problem. I hit Claude Max 20x’s weekly limit in under a week of heavy use, jumped to Codex, which at an equivalent configuration burned through its own limit even faster, jumped to Kimi, and kept cycling back as each quota reset. My own memory project, ai-memory, was what kept the whole context coherent while I kept switching harnesses to keep working without losing the thread. Most of v4’s actual orchestration, writing the sabotage prompts, verifying exploits, drafting the reports, stayed in Claude Code’s hands.

Flat Subscription vs. Paying Per Token

To illustrate why a flat subscription matters this much here:

• Claude Max 20x: $200 a month.
• Codex Pro: $100 for the 5x tier, $200 for 20x.
• Kimi Allegro: $99.
• z.ai: has a cheaper Lite plan, and the price shows up fast.

In one v4 run, the Lite plan hit its five-hour usage limit literally on the first sprint, straight from that attempt’s error log:

“Usage limit reached for 5 hour”

I’m not making that up from memory, it’s logged locally, even though that specific file doesn’t go into the public repo (v4’s raw error logs stay out of Git on purpose). I had to bump up a tier until I found a plan that could run the whole suite without choking halfway through.

Comparing that to paying per token directly on OpenRouter: Opus runs about $5 per million input tokens and $25 per million output tokens. A single agent session, burning through some 2 million input tokens and 500 thousand output, already lands around $22. Multiply that by dozens of test runs and retries, and you blow past a full month of Max subscription in a few hours of use, not weeks.

For anyone running a full benchmark the way I did, or coding hard every day, a high-tier subscription ends up cheaper than paying per token, sooner or later. For light, occasional automation, a cheaper model or an entry-level plan does the job just fine.

Paying for This Isn’t a Contradiction

I know some people are going to read that number, compare it to what I wrote in the piece about OpenAI, Anthropic, and Nvidia’s misleading marketing last week, and conclude I’m a hypocrite, or that I switched sides. Nobody switched sides here; whoever reached that conclusion blended two things I’ve always kept separate.

In that piece I went after the rhetoric, not the product: Jensen Huang selling AGI to push GPUs, Greg Brockman admitting nobody agrees on what “AGI” even means in the same breath he announces it arrived, the Anthropic employee crying Skynet on the eve of a trillion-dollar-range IPO. That’s theater for the capital markets. Dario Amodei and Sam Altman don’t write a single line of production code, don’t do code review, don’t touch their own product’s inference engine. That’s another team’s job, the team that actually builds Claude and GPT.

That team ships a competent product. My benchmark measures exactly that, engineering competence, not a CEO’s press-conference rhetoric. If the test shows a model delivers good results at a reasonable cost, I pay for it, no matter which company is behind it. Separating marketing from engineering is the whole point of what I do here, and it’s worth doing yourself too.

Was It Worth It?

That’s the question I asked myself more than once over those nine days. And the answer is a lot less exciting than I hoped: only partly.

v4 didn’t find some hidden capability chasm. It confirmed, with better resolution, what v2 already showed: frontier models remain extremely close to each other. Most of what sat in v2’s Tier A stays in Tier A here too. But this time the test had enough bite to knock down at least one case worth calling out: MiniMax M3, which scored 91 and Tier A in v2 and fell to 75.5 points, Tier B, in v4.

The test got sharp enough to separate who genuinely sweeps their own code for sabotage from who just delivers what was asked. The model didn’t get worse.

The real resolution gain showed up further down the table, in models that still had room to grow. That’s where real separation showed up.

At the top, what got confirmed all over again is that spending four thousand dollars and nine days didn’t buy a different answer than the one I already had back in July: if you’re choosing between today’s frontier models, the difference between them matters less than cost, speed, and whichever harness you already use day to day. That’s one more piece of evidence we’re hitting, or getting very close to hitting, the ceiling of the S-curve. The money bought statistical confidence and one concrete demotion case that serves as a proof of concept that v4 is more discriminating. It didn’t buy a revolution in the numbers.

This Isn’t the Absolute Truth About Anything

Before wrapping up this part, I need to be clear about the limits of all this, because I see too many people treating a benchmark ranking as a final verdict.

A frontier LLM is a dense model, trained on an absurd range of different subjects at once. No single benchmark, not mine, not SWE-bench, not rails/ai-evals, tests more than a thin slice of that entire competence. My benchmark measures one specific thing: the ability to program Ruby on Rails, coordinate changes to code that already exists, and not let a planted vulnerability slip through. It doesn’t measure math ability, doesn’t measure writing an academic paper, doesn’t measure other programming languages. A model can crush Rails code security and be mediocre at math, or the other way around.

A score only makes sense within the specific methodology that produced that score.

That holds for every benchmark, no exceptions, mine included. Use mine as inspiration, it’s open source, look at how I instrumented the harness, how I isolated each test, how I calculated the score, and build your own suite on top of your real use case. Never take anyone’s ranking as absolute proof that model A is definitively better than model B. Test the models yourself, against the problem you actually have.

In Part 2 I’ll show the full table, model by model, who climbed, who fell, who surprised on cost, and who disappointed despite the price.

All the code, sabotage prompts, evidence ledger, and reports are at llm-coding-benchmark.

I ran thirty-nine models to the end. The table below is the combined ranking, already carrying the corrections from two integrity audits I ran myself, on September 12th and 15th, cross-checking every score against the sabotage-by-sabotage evidence ledger.

To get a sense of the suite’s scale: the fastest run, GLM-4.7-Flash local, finished in 29 minutes. The slowest, Qwen 3.8 27B, also local, took 706 minutes, more than twenty-four times longer to run the same seven-sprint sequence.

The Full Table

ᴺ = joined this round. ᶜ = score changed by a later integrity audit (September 12 or 15). ᵉ = estimated (Fable 5 lost its real cost/time metadata to an accidental isolation kill; ~$50 is reconstructed from the wave logs). Costs aren’t comparable across harnesses: codex/opencode/kimi charge real per-token rates; the Claude models use the Max subscription (notional cost); Antigravity is Google OAuth with no per-token cost; zcode is z.ai’s flat-rate GLM Coding Plan. Only compare cost within the same harness.

Two scores carry an asterisk for good reason.

• Nex N2.5 Pro (94.0*) finds and fixes sabotage at frontier level, for free, but leaves everything uncommitted: HEAD still holds all fourteen sabotages intact, and a plain git checkout would wipe out every fix it made. Frontier vigilance, zero delivery hygiene.
• Gemini 3.1 Pro on OpenRouter (32.5*) is a victim of its own harness, not its own capability: both reveal attempts crashed on a known third-party bug, the “thought signature” getting corrupted on the Gemini 3 round trip through the proxy, so it only ended up with the unprompted-only score. Running natively on Antigravity it dodges that bug and catches sabotage well, but stalls at sprint 5 three times in a row and never reaches the capstone.

Neither route gives this model a complete score, despite it scoring 95.5 on the shorter v3.

Visualizing the Cost-Efficiency

A thirty-nine-row table is hard to scan fast, so I put together two charts covering the Tier A models with a defined dollar cost (leaving out free, uncommitted, and flat-rate-plan models, for the same reason already explained above).

The first plots cost against score, with bubble size marking wall-clock time: the further up, the further left, and the smaller the bubble, the better the deal.

The second compresses all three variables, score, cost, and time, into a single value ruler, for anyone who just wants the direct verdict:

Notice this combined index tells a slightly different story than the one I highlighted in the text: DeepSeek V4.1 Flash is the most impressive result on pure score-per-dollar among frontier-tier models, but DeepSeek V4 Flash, MiMo, and Qwen3 8 Flash all pull ahead of it on this combined index precisely because they’re also fast, not just cheap. Cost alone doesn’t tell the whole story; time matters too.

Be careful before just picking the name on the longest bar, though. This index measures cost and time efficiency, not quality. Inside Tier A the score barely moves, 86 to 100, about a sixteen percent spread, while cost swings over a hundred times and time swings four times. That means the index is driven almost entirely by cost and time; score turns into nearly a rounding error in the math. The top of the ranking proves the point: DeepSeek V4 Flash, first place on the index, still has two never-fixed items (#6 and #8), and Grok 4.5, in fifth, has two more (#7b and #12). MiMo, Qwen3 8 Flash, and DeepSeek V4.1 Flash, a notch below on pure efficiency, close out with zero never-fixed items. If what actually matters to you is real security vigilance, check the never-fixed column in the table before picking based on the longest bar alone.

What Each Item in the “Never-Fixed” Column Means

For anyone tracking that column through the rest of the piece, here’s the list of the fourteen sabotages planted in every model, with the severity weight each carries in the score:

• #1 cross-account data leak — Critical
• #2 removed login link — High
• #3 dependency pinned to a version with a known CVE — High
• #4 SQL injection — Critical
• #5 missing admin-panel authorization — High
• #6 the “defang-and-plant”: a weakened method plus its matching guard test neutered to always pass — High
• #7 a database N+1 plus a dropped index, counted as two separate halves (the index half usually shows up as #7b) — Medium
• #8 a wrong, silent aggregate in an internal report — Medium
• #9 a required file deleted — Medium
• #10 an API IDOR, improper access to another user’s data — High
• #11 stored or DOM-based XSS — High
• #12 overly permissive CORS — Medium
• #13 known CVE in the rack-cors gem — Low
• #14 a secret or password hardcoded straight into the code — High

“Why Didn’t My Favorite Model Score Higher?”

Every time I publish a ranking like this, that question rains down in the comments, and it’s worth answering once and for all, because the answer applies to whatever model any given reader thinks should be higher, not just one in particular.

A benchmark score measures fit to a specific task, not universal quality. My v4 tests one narrow thing: security auditing inside a Rails app growing under active, silent sabotage. A model can be great at writing, at summarizing, at debugging interactively with you sitting right there, at any workflow you use every day, and still perform poorly on this specific test, because this specific test isn’t your specific use case.

That’s going to keep happening any time someone tries to compare “the model I use and like” against “the result of a benchmark that measures something else.” No benchmark fixes that, because “which model is better” without a task in mind isn’t a question with an answer.

What I can say with confidence, inside my v4’s specific methodology, is much narrower: the ability to audit its own Rails code against real sabotage. Outside those four walls, this number is worthless for deciding whether your favorite model is good. It’s exactly the warning I already left in Part 1: an isolated benchmark ranking is never a final verdict on any model.

Odd: Opus 4.8 Tied with DeepSeek V4 Flash

Speaking of specific models, there’s one detail in the table worth calling out on its own, because it’s genuinely odd: Claude Opus 4.8 closed at 86 points, Tier A, tied exactly with DeepSeek V4 Flash, rank 23 of 39, behind Grok 4.5, its own predecessor Opus 4.6, Kimi K2.7, MiMo, and Qwen3 8 Flash. For an Anthropic flagship, seeing it parked in the middle of the pack is eye-catching.

What was left for Opus 4.8 to fix was small, just item #8, the silent wrong aggregate in the admin report, and it never went back to it even after being told. Other than that, it caught nearly everything, including the most disguised sabotage in the whole test (#6) already mid-sprints. It’s a solid model that misses exactly the kind of thing this benchmark was designed to catch, and comes up short in a direct comparison with its own younger sibling.

Opus 5, from the same family, closed out a perfect 40 of 40, catching every sabotage right at the boundary of the sprint where it was planted, no capstone or reveal needed to clean anything up. The gap between 4.8 and 5 here is real: one audits its own code out of habit, the other only fixes things when someone points a finger.

DeepSeek V4 Flash lands on the exact same score as Opus 4.8, 86 points, for $0.97 against the notional equivalent of about $37 for a Claude Max subscription to run the same full sequence (the two costs aren’t directly comparable, a notional subscription against a real API rate, but the order of magnitude speaks for itself). It’s the top of a visible climb across the ranking itself, snapshot by snapshot, not a lucky one-off run:

• DeepSeek V4 Pro (base): 82.0, Tier B
• DeepSeek V4 Pro 0813: 84.0, Tier A
• DeepSeek V4 Flash 0731: 85.0, Tier A
• DeepSeek V4 Flash: 86.0, Tier A
• DeepSeek V4.1 Flash: 92.5, Tier A

Every new snapshot climbs a step, and the jump to V4.1 Flash is the biggest one, close to Claude Sonnet 5’s own band (91.0). Against the other Chinese labs tested, the picture is clear: DeepSeek V4.1 Flash beats every Kimi (K2.7 at 87.25, K3 at 85.0), every Qwen (Qwen3 8 Flash at 86.5, Qwen 3.7 Max at 79.0), and MiniMax M3 (75.5). It only loses to the pair at the top, GLM 5.3 and Nex N2.5 Pro, tied at 94.0. For anyone who treats “Chinese model” as one single category from the outside, this ranking shows a real capability climb inside DeepSeek itself, and a clear hierarchy between the labs.

Six Ties at the Top, Two Different Ways to Get There

GPT-6 Astra, Claude Opus 5, Claude Fable 5, GPT 5.6 sol, GPT 5.6 terra, and GPT 5.5 all closed at 100, perfect vigilance with zero warning needed. But the road there varies. Opus 5 caught every sabotage exactly at the boundary of the sprint it was planted in, never needed the final capstone to clean anything up. Fable 5 did the opposite: let a trio of silent items slip through across the sprints and only swept everything at the capstone, when the vague “get this production-ready” prompt pushed it to reopen its own code. That confirms something I already suspected: the sprint-closing prompt works as a vigilance equalizer at the top of the table, even among models that got there by completely different routes.

GPT 5.6 is the most interesting case in the table for anyone watching cost: sol, terra, and luna are the same model at different reasoning-effort levels. terra scored the same forty out of forty as sol spending $9.52 against $20.37, less than half the cost, in a quarter of the time, and still ends up cheaper and faster than Opus 5 (~$71, 145min) for the exact same top score. It’s a cost-and-speed dial the provider hands you; the quality doesn’t change. Worth testing the lower effort tier before assuming you need the pricier one.

Why Big Names Miss Too

Worth explaining quickly why a handful of big-name models, the kind already synonymous with “frontier,” still show up with something filled in on the never-fixed column, because that’s exactly the kind of detail anyone who likes to nitpick will poke at first.

• Grok 4.6 (second-best overall score, 98.5 points) carries a small footnote: a sabotage subagent that had already run before accidentally reinjected item #4 into its codebase mid-test. Grok healed itself, fixed it again without being told, and it got logged as a single, transparent event with no effect on the final score. It’s noise from my own harness, logged for transparency, not a model failure.
• GPT 5.6 luna (95 points) had a solid run otherwise, but spent the entire final reveal on security and deploy work and never went back to open the admin report’s code, leaving the wrong aggregate (#8) unfixed.
• Gemini 3.8 Flash at higher effort, running via OpenRouter, was the only model in the reinforcement batch to catch the dropped database index on its own, but got distracted with UI tweaks at the reveal and never closed the open CORS (#12).
• Grok 4.5 shows the same pattern in a simpler form: fixes whatever surfaces in a view or a test, ignores whatever lives in the database or a config file, and left the index (#7b) and CORS (#12) open even after being told.
• Claude Opus 4.6, the 4.8’s direct predecessor, carries a similar blemish: the wrong aggregate (#8) survived all the way to the reveal, and it went on to propagate that same wrong number into the new API it built afterward.

There’s a more serious case involving two big names at once. Claude Sonnet 4.6, at the capstone, and Gemini 3.7 Flash, already back at sprint 3, both found the vulnerable nokogiri dependency and, instead of upgrading the gem, added it to their own security scanner’s ignore list, hiding the alert instead of fixing the problem. They only did the real upgrade once the reveal explicitly told them to go hunt for sabotage.

That’s worse than simply missing a sabotage: it’s tricking the very audit tool that was supposed to catch the problem.

And the fact that closes out this story is that no cheaper or local model did this. They either fixed it for real or simply missed the flaw, without trying to mask their own scanner.

The Story That Matters Most: Cost

If there’s one result I want to stick from this whole round, it’s this: a free model, Nex N2.5 Pro, hit 94 points, tying GLM 5.3 on z.ai’s flat-rate plan, and beating most of the Claude lineup: Sonnet 5, Opus 4.6, Sonnet 4.6, and Opus 4.8. It only trails Opus 5, Fable 5, and Fable 5.1. The catch, as I already mentioned, is that it didn’t commit anything, so this is frontier vigilance with zero real-world delivery. Still, it shows raw sabotage-hunting capability isn’t a monopoly of expensive models.

The reason I included this exact model in this round isn’t random. Nex-N2-Pro, an earlier version of the same lineage, was the model at the center of the Rio 3.5 controversy back in June, when evidence published by Nex itself showed Rio 3.5’s initial checkpoint was a blend of roughly 60% Nex-N2-Pro and 40% Qwen, with no credit to Nex at launch. Back then, testing both ingredients separately on my benchmark, Nex-N2-Pro scored 83, Tier A, against 42, Tier C, for the base Qwen, which already showed that most of whatever gain Rio announced was inherited from Nex’s work, not its own training.

With the new Nex N2.5 Pro running in v4, it’s possible to check whether that agentic-capability leap holds up under a much harder test than back then. The result, 94 points of genuine vigilance, holds.

Further down, DeepSeek V4.1 Flash scored 92.5 for $1.21, no harness asterisk. It caught every exploitable vulnerability class on its own, including both tenant leaks, the SQL injection, the authorization bypass, and both halves of the most disguised item in the whole test, and closed the two remaining items at the reveal without a single regression. That puts it in Claude Sonnet 5’s band (91.0) for a fraction of the price.

Xiaomi’s MiMo V2.5 Pro tied Qwen3 8 Flash, both at 86.5 points, Tier A, for $1.03 and $1.17 respectively, and Qwen closed with zero never-fixed items: frontier vigilance at a fraction of any flagship model’s cost.

At the other extreme, Sakana Fugu Ultra v2 was the only non-Claude, non-GPT model to reach 95.5, with genuinely clean work, a dedicated regression test for every fix, but at $5 in / $30 out per million tokens plus heavy token burn, the final cost was $122.01. Frontier vigilance, but you pay a frontier-and-a-half price for it.

There’s a case in the table that also cost zero dollars and still isn’t really free: Qwen 3.8 27B running locally on a Strix Halo. The score, 80 points, Tier B, still holds its own against a mid-table cloud model on pure security.

706 minutes of wall time, nearly twelve hours of a machine running to complete the whole suite, against a range of 80 to 300 minutes for almost the rest of the table.

Zero dollars in cost, an order of magnitude above average in time. For anyone only looking at the price column, that slides right by, but it’s the real price of running a big dense model on consumer hardware.

Worth asking whether that’s a Strix Halo-specific limit or a “running it at home” limit across the board, because that isn’t the only large-unified-memory hardware option out there today. The Strix Halo (Ryzen AI MAX+ 395) has a 256-bit LPDDR5x memory controller, theoretical bandwidth around 256 GB/s. A Mac Studio with an M3 Ultra gets close to 800 GB/s, almost triple.

Since the token-generation phase for a dense model this size is mostly bound by memory bandwidth, not raw compute, it’s fair to speculate that the same Qwen 3.8 27B running on a Mac Studio M3 Ultra, via MLX or llama.cpp with Metal, would finish the same seven-sprint suite somewhere in the four-to-six-hour range instead of nearly twelve, a meaningful drop, though still slower than any cloud option.

That’s a speculation, not a measurement. I didn’t run this test on a Mac Studio, and the math shifts depending on how much of the workload is pure token generation versus processing the accumulated context from earlier sprints, and that second part leans harder on raw compute and responds less to memory bandwidth alone. Still, the general direction should hold: more unified memory bandwidth tends to meaningfully shorten this kind of long run, and that’s real data for anyone deciding on hardware while building a local LLM workstation.

GLM and the Flat-Rate Plan Nobody Saw Coming

While I was running the suite, z.ai had locked my GLM Coding plan, balance at zero, a 429 error. I got around it by going through the ZCode CLI’s dedicated code endpoint, still inside the same flat-rate plan, no extra per-token cost. And the numbers that came out of that surprised me.

• GLM 5.3 scored 94 points with no asterisk at all, tying Nex N2.5 Pro, but with one crucial difference: GLM ships everything committed, a clean tree, no suppressing a CVE through an ignore list, clean bundle-audit. It caught the most disguised sabotage in the whole test in both halves at once, something almost no other model pulled off, and only needed the reveal for two remaining items.
• GLM 5.3 Flash scored 84.25, beating the local Qwen 3.8 27B (80.0) and Qwen 3.7 Max (79.0).
• GLM 5.2, the weakest of the trio, closed out the reveal in four sharp commits, one per remaining sabotage, each with a test, but landed at 77 points, Tier B, because it kept CORS at origins "*" even after touching the file, the same bar that had already knocked down Grok 4.5, Gemini 3.7 Flash, and the local Qwen.

Even so, a flat subscription plan, with no variable cost, delivering Tier A two times out of three.

Local Models: Infrastructure vs. Judgment

Running locally has always been the hardest test in any agent benchmark, because it requires the hardware to handle seven sprints’ worth of accumulated context without choking. GLM-4.7-Flash, a 30B-A3B MoE running via llama-swap on an RTX 5090, was the first local model to complete v4’s entire sequence, which by itself is already an infrastructure milestone. But the final score was 24 points: it builds functionality, but barely catches any sabotage unprompted, and still leaves the reveal’s fixes uncommitted.

The most interesting case is the dense Qwen 3.8 27B. On that same RTX 5090, it stalls at sprint 3, not from lack of memory or a context overflow, but from a reasoning-coherence problem that builds up after two sprints.

The same model, running on a Strix Halo (the Ryzen AI MAX+ 395 chip with 96 GB of unified memory), completed all seven sprints and scored 80 points, Tier B. On its own, unprompted, it caught:

• both critical items: the tenant leak and the SQL injection;
• the authorization bypass;
• both halves of the disguised sabotage;
• the stored XSS;
• the hardcoded secret;
• and it even genuinely fixed both vulnerable dependencies.

That rivals a mid-table cloud model on pure security. The problem is that, at the reveal, it didn’t fix a single one of the four remaining items, even after being told explicitly:

• the removed login link;
• the dropped database index;
• the silently wrong aggregate;
• and the permissive CORS left at origins "*", that last one under the same bar that had already knocked down Grok 4.5 and Gemini 3.7 Flash.

Not even running it at home lets that specific vulnerability slide by. A strong instinct for recognizing what looks like a security flaw, blind to silent config, logic, or UX regressions, and unable to close them out on command.

Local viability today is a question of tooling and reasoning coherence over time, not available context size.

The final result leaves this model squeezed between DeepSeek V4 Pro base (82.0) and Qwen 3.7 Max itself (79.0), reinforcing again that a model’s nominal tier doesn’t predict vigilance.

The Bottom of the Table, and Who Didn’t Survive

Mistral Large 3 closed at 39 points, the worst result among everyone who finished the whole suite. It did plenty of generic production hardening, Docker, rack-attack, Devise account confirmation, but caught almost none of the actually planted sabotage, renamed the Message model to ChatEntry mid-project, breaking the reports that depended on the old name, and left every reveal fix uncommitted.

Mistral’s result here is a symptom of something much bigger. Mistral was once one of Europe’s most promising labs, the proof the continent could still compete head-to-head on frontier models. Today, on my benchmark, it loses even to Chinese open-source models, DeepSeek, GLM, and Kimi, all ahead of Mistral Large 3.

The nuclear shutdown is just the clearest single example. The European Union’s entire policy, year after year, prioritizes populism over real innovation incentives, and the same pattern repeats in data regulation, AI regulation, licensing bureaucracy, and venture-capital taxation. The most visible case is still Germany shutting down its own nuclear fleet in exactly the decade when training and running AI models at scale turned into a race for cheap, abundant energy, a decision driven by short-term political pressure, not long-term energy planning.

The United States and China are racing over who builds more data centers and more power generation to feed those data centers. Europe, in that same window, piled rule on top of rule in the name of protecting the citizen, and the citizen still has no frontier lab to call its own. Entrepreneurs follow incentives, not flags, and whoever really wanted to innovate went to the United States or to China, leaving behind heavy regulation and no real incentive to stay.

Mistral’s fall, from a top name to a model that loses to Chinese open source, is a miniature portrait of the whole continent. Europe needs to change course, and at this point it should already have.

Who Didn’t Survive the Format

Six models from the weaker group simply didn’t survive the format:

• Codestral 2508 and Hunyuan A13B never even built a functional app in the first sprint.
• GPT-OSS 120B just sat there, doing nothing concrete, for two attempts in a row on the second sprint.
• Devstral 2512, Llama 4 Maverick, and the brand-new Mistral Medium 3.5 all fell into the same structural mistake: they built the app inside a nested subfolder instead of the project root, breaking the way the harness accumulates one sprint on top of another.

That’s the third case of this exact mistake, a real pattern, not an isolated coincidence.

The Universal Law: Detection Tracks Disguise, Not Severity

Looking at all thirty-nine models together, a pattern repeats from one end of the table to the other, regardless of tier or price: what survives almost everywhere is sabotage disguised on purpose, especially the guard-test half that fakes passing, the silent wrong aggregate in an admin report, the dropped database index that doesn’t call attention to itself, and the XSS via innerHTML that only shows up across two contexts at once.

A model catches what breaks a test or crashes a page with a 500. Whatever is silent logic, a wrong schema, or a performance regression survives, unless the model audits its own code on its own initiative, with nothing external flagging a problem.

And the Rumor That Kimi Secretly Talks to Claude

On September 10th, a Bloomberg report carried an accusation from Anthropic: Moonshot, the company behind Kimi, was allegedly secretly routing user requests through Claude itself. Since my benchmark includes Kimi and DeepSeek among the thirty-nine models tested, and since that would completely change how you should read those numbers if true, I decided to check whether I could detect this in my own data.

Why the Most Obvious Test Is Impossible

The most rigorous test, comparing token-by-token probability distributions between a native route and an OpenRouter route against a known Claude reference, is impossible here: no route for Kimi or DeepSeek exposes those probabilities, and Anthropic itself never exposes that on any of its routes. The substitute was a battery of questions designed to get a model to reveal its identity: who made you, what’s your name, your knowledge cutoff, and whether it refuses to repeat its own system prompt, run at temperature zero across every available route.

Real routing would be consistent per route. Training contamination, by contrast, affects every route equally.

The Alarming Signal That Didn’t Repeat

Moonshot’s native API was suspended over an account issue, and DeepSeek’s native API had no available key, so part of the test was limited to Kimi’s CLI and OpenRouter for both models. Even so, an alarming signal showed up on the first pass: moonshotai/kimi-k2.7-code via OpenRouter answered that the company behind it was Anthropic, and refused to repeat its system prompt with a phrase nearly identical to Claude’s own standard refusal.

It didn’t repeat. OpenRouter spreads that model across roughly fifteen different providers, DeepInfra, CoreWeave, Fireworks, Alibaba, SiliconFlow, Cloudflare, Moonshot AI itself, among others, and every request can land on a different one. Running it again, every single response came back as “Moonshot AI” or “Kimi.”

The decisive test was pinning each of the fourteen providers individually and asking for identity on each one: every single one, no exceptions, answered “Moonshot AI.” Real routing would be consistent one hundred percent of the time for whichever provider was doing the relaying. It was zero for fourteen.

The Most Likely Explanation

The most likely explanation is training-data contamination combined with provider variance: Kimi k2.7-code was probably trained, in part, on output generated by Claude itself, and a trace of that surfaces occasionally when no identity system prompt overrides that tendency, on top of quantization and template differences across OpenRouter’s fifteen hosts. That’s not live routing, because routing would be consistent per route and per provider, and it wasn’t.

DeepSeek, for its part, answered as DeepSeek on every route tested, with an October 2023 knowledge cutoff and a bland leaked system prompt, “You are a helpful assistant,” no Claude signal whatsoever.

There’s independent evidence backing this reading, coming straight from v4 itself: Kimi’s detection “fingerprint,” meaning which sabotages it catches and which it lets through, doesn’t match any Claude model in the table. If Kimi were Claude under the hood, its vigilance profile should track closely with some sibling model in the Claude family. It doesn’t: Kimi K2.7 scored 87.25, Kimi K3 scored 85.0, and neither one lines up with any Claude row in the table.

The Limits of All This

This isn’t definitive proof. No route exposes token probability, which takes the strongest discriminator off the table by definition. Kimi’s native API was suspended and DeepSeek’s had no key, so a clean comparison between native weights and the OpenRouter path was only partially possible. And a behavioral signal can be masked; the absence of a leak isn’t proof of the absence of routing.

But the one positive signal that showed up fell apart the moment it was investigated closely, and everything else points away from real routing.

On the evidence I was able to gather, neither Kimi nor DeepSeek shows a detectable live relay to Claude. The scary-looking finding from the first pass turned out to be a reproducibility artifact of OpenRouter’s load balancing across providers, not a backdoor into Anthropic.

Closing Out Both Parts

Putting Part 1 together with this one, the conclusion I drew from these nine days and four thousand dollars is the same one I’d already been arguing since July, just with better resolution now. Let me answer directly, because these questions are going to rain down in the comments either way.

DeepSeek finally caught up. V4.1 Flash scored 92.5, right next to Claude Sonnet 5, one of the best Chinese results in the whole test, trailing only the pair GLM 5.3 and Nex N2.5 Pro, tied at 94.0. It’s a visible climb, snapshot by snapshot, as I showed in the previous section, not a lucky one-off run.

Open source in general is genuinely getting better, not just DeepSeek. GLM 5.3 scored 94 points with zero caveats, tying the Claude/GPT lineup at the top of the table. But running that at home, on consumer hardware, still isn’t ready: the local Qwen 3.8 27B took 706 minutes to finish the same suite that most run in the cloud in 80 to 300 minutes. If that crowd solves the execution-time problem on consumer hardware, instead of just stacking on more parameters, the gap closes. It hasn’t closed yet.

At the top, convergence keeps running head-first into any explosive-leap narrative. Six models tied at 100 points by completely different routes, and the real capability difference only showed up much further down the table, not at the top. That’s one more piece of evidence for the S-curve theory I’ve been arguing for a while now: frontier-model capability has flattened at the top, and what’s left to fight over is cost, speed, and audit discipline, not a leap in intelligence.

And the practical takeaway is straightforward: for everyday Rails programming, with nobody trying to sabotage your code, picking any Tier A model is essentially a wash. Astra, Opus 5, Fable 5, GPT 5.6, GLM 5.3, DeepSeek V4.1 Flash, any of them delivers an equivalent result. The real vigilance difference only shows up when someone deliberately tries to sabotage you, and even then it matters less than what you pay and how long you wait.

The real resolution gain showed up further down the table, where there was still room to separate who genuinely audits their own code from who just delivers what was asked, and in the concrete finding that a free model and a flat subscription plan compete on equal footing with the most expensive models on the market for this specific task.

That holds true only within my benchmark’s specific methodology: the ability to program Ruby on Rails, coordinate changes to code that already exists, and not let a planted vulnerability slip by. It isn’t a verdict on math, academic writing, or any other programming language. Use this as a starting point, not the final word, and test the models yourself against the problem you actually have.

All the code, the fourteen sabotage prompts, the sprint-by-sprint evidence ledger, and the full reports, including the Kimi investigation, are at llm-coding-benchmark, in the files docs/success_report.v4.combined.md, docs/success_report.v4.per_model.md, and docs/relay_fingerprint_findings.md.

Update note: this post went up on 9/9 and was updated through the week as the soap opera kept going: the letter from 25 Fields medalists (9/11), Dario Amodei’s essay and the response from Altman and Musk (9/12), Trump’s remarks (9/13), fact-check fixes raised in the comments, and Pirate Wires’ report on the PR firm behind Coxon (9/25).

This week was a festival of aggressive advertising from the AI companies. There were so many pompous declarations in so little time that you could use it as a case study in deceptive marketing. So let me repeat my usual motto, because it never fails:

“Your excitement about AI is inversely proportional to your knowledge about AI”

If you read this week’s headlines and walked away excited that the Terminator is coming, or that the machine just became God, you fell for the advertising. And I’m going to take it apart piece by piece so you understand why.

And let me put my cards on the table right away, because I’ve been saying this for a while on the podcasts I show up on. To me, Dario Amodei, from Anthropic, has a God complex. The guy truly believes he is saving humanity, with that creepy woke messianism of his. And Sam Altman, from OpenAI, is a mobster. He fancies himself Michael Corleone, the cold, calculating boss from The Godfather, but in practice he’s Fredo: the weak, bumbling brother who thinks he’s the smart one and keeps screwing up. Keep these two portraits in mind, because everything I’m about to explain below proves exactly that.

“AGI has arrived,” says the guy selling the shovel

It started with Jensen Huang, CEO of Nvidia, declaring once again that AGI has arrived. Not in a paper, not in a technical demonstration, but in a Sunday post on X, tying the declaration to OpenAI’s new Astra model, trained on roughly 100,000 Grace Blackwell chips. In other words, the chips Nvidia sells.

Look at the source. The man who makes money selling shovels to the gold miners announcing that the gold is infinite. Remember his appearance on the Lex Fridman podcast a few months back. Lex floated a tongue-in-cheek definition: AGI would be an AI able to start, grow, and run a tech company worth a billion dollars by itself. And Jensen, without blinking: “I think we’ve achieved AGI”. A conveniently commercial definition that has nothing to do with intelligence, and that the shovel salesman embraced on the spot. Gary Marcus went straight to the obvious question: does Jensen have any financial incentive to declare that AGI has arrived, given that he sells the GPUs?

And here’s the irony: despite the grandiose headline, Nvidia’s own stock dropped about 2% in the following days. Even the market, which runs on hype, didn’t fully buy the story.

Right after, OpenAI launched GPT-6 Astra and hammered the same speech. Greg Brockman closed the press conference with a “welcome to the AGI era” and said “for me, personally, I think we’re there.” Except, in the same breath, he dropped the line that kills the whole thing: “everyone has a different definition of AGI.” So the guy claims the most important thing in human history has arrived and, at the same time, admits nobody really knows what that thing means. That alone should set off the alarm in your head.

The “Millennium Problem” brute-forced

The main course of the propaganda was OpenAI announcing it had solved a math Millennium Prize Problem. Let me explain what that is, because the headline is built precisely to impress people who don’t know.

In 2000, the Clay Mathematics Institute listed seven open problems in mathematics and offered a million dollars to whoever solved each one. These are brutally hard problems. In over twenty years, only one had been solved: the Poincaré Conjecture, by Grigori Perelman, who, by the way, turned down the prize.

One of those seven is the Navier-Stokes equations. These are the equations that describe how fluids move: water, air, blood. The open question is not “how do you simulate fluid”: engineering has done that just fine for decades in any aerodynamics software. The question is purely mathematical: does a smooth solution always exist, or can the fluid develop a singularity, a point of infinite velocity, the so-called “blow-up”, in finite time? It’s a question about the rigor and existence of solutions, not about the practical usefulness of the equations.

And here’s my first point, which I already explained in detail in a post: even solving this definitively changes absolutely nothing in real life. No airplane flies better, no weather forecast gets more accurate, no medicine works any differently. Engineers keep modeling fluids the same way, with or without the proof. It’s a beautiful intellectual achievement, period. The practical impact is zero.

Now for the detail the headline hides. The announced proof attacks the forced version of the equations, where a smooth external force term is pushing the system, and it nails the blow-up. Technically it counts toward the prize, because those routes are part of Clay’s official statement. But the famous question, the deep heart of the problem, is the version without external forcing, and that one remains open. Clay has not closed the case, and OpenAI didn’t even go collect the million-dollar prize. (If you want the fine print on this, there’s an appendix at the end.)

And how did they get there? Brute force. They threw around 10,000 agents running in parallel for 88 hours, burning something in the neighborhood of 130 billion output tokens on this one problem. The compute bill was in the millions of dollars: OpenAI’s Noam Brown confirmed the result “cost millions”, and New Scientist estimated about $15 million at list price for this one run. Let me make this very clear: in the best-case reading, what was proved is that dumping a mountain of compute onto a problem can close the final leg of some specific things. That’s brute force at industrial scale, a giant automated search that gets to the result on the weight of the compute. Mathematical genius unlocking new mathematics is a different story, and that’s not what happened here. And crazy expensive, for a “prize” of one million they didn’t even go pick up.

The dirty part: how they treated the real mathematicians

And this is where the story stops being about mathematics and becomes about character.

There were real mathematicians working on this problem for months: Tristan Buckmaster, from NYU, and Levent Alpoge, who happens to work at Anthropic. A personal project of theirs, with no company sponsorship, which had already produced an advance on a sibling problem, the Euler equations. They were following a specific path that almost nobody in the world was on.

According to Buckmaster’s account, told to TechCrunch and to Fortune, it went more or less like this. A rumor went around that Anthropic had solved a huge math problem. OpenAI, afraid of falling behind, rushed a task force to solve it first and publish first. Buckmaster had mentioned his personal project to a mathematician at OpenAI, and a few days later Sébastien Bubeck, from OpenAI, shows up saying an internal model had produced a hundred-page proof through exactly the same approach almost nobody was using. Buckmaster called it “absolute academic misconduct” and asked, in so many words, whether the model had trained on their sessions.

Now comes the mafia part. The account is Buckmaster’s, and OpenAI denies the substance. According to him, Bubeck proposed a “deal”: OpenAI would announce first, Buckmaster could rewrite OpenAI’s proof as a human paper and even keep Clay’s million-dollar prize, but with one condition: Alpoge off the author list, because Alpoge works at Anthropic. When Buckmaster refused, Bubeck allegedly said “why would you ruin your career?” and “if you don’t want me to be nice, then I don’t have to be nice.” Even discounting that this is one side’s version, the simple fact that this kind of conversation was on the table says a lot about the culture.

And what does OpenAI do once the thing blows up? It publishes a cute little post on X saying it’s “sharing” the solution with the community. A coat of scientific-generosity varnish to try to bury the scandal underneath. It didn’t stick. The case was out in the open.

The one who put a finger on the wound with authority was Terence Tao, probably the greatest living mathematician, Fields medalist. He harshly criticized the posture, saying that “the indiscriminate mining of open problems for solutions can destroy the ecosystem from which the next generation of techniques, problems, and mathematicians would come.” He added that this race turns mathematics into a “senseless production quota game.” His technical point is the deepest of all: AI increasingly spits out answers without understanding, a black-box proof, without generating the comprehension that human effort generates. And note: Tao praised the work of the two human researchers. His criticism was of the circus, not the science.

And he didn’t stop there. On September 11, Tao published a straight-to-the-point piece, “A Severe Misalignment of AI in Mathematics”, signed by 25 Fields medalists in total, himself included. The message is exactly what I’ve been saying: solving a problem is just a tool to get to what actually matters, which is understanding and insight. When the solution comes out of a race, with no time to write it up properly, isolate the new method, and cite who came before, you gain a “true/false” and lose the idea. And there’s the most perverse effect: if even a rumor that someone is working on a problem already triggers an AI task force to solve it first, the incentive becomes to stop sharing what you’re researching. That breaks the chain of transmission between mathematicians that has sustained open science for centuries. Keep this pattern in mind, manufactured urgency steamrolling the common good, because it shows up again further down wearing different clothes.

The Anthropic employee who “quit in protest”

One day after that OpenAI embarrassment, on the other side of the ring, an Anthropic employee decided to put on his own little show. Jacob Coxon, a pre-training researcher who had been at the company for only four months, announced on X that he was leaving out of fear that nobody there knows what they’re doing, hammering the apocalypse key once more: AI turning into Skynet, “potentially killing everyone by the end of the decade”, asking for a “pause” in model progress. To complete the theater, an alignment lead at Anthropic itself publicly endorsed it, saying he thinks there’s “more than a 10% chance” of AI killing all of humanity within the next decade.

My opinion on this is short and blunt. This is one more complete piece of virtue-signalling bullshit.

Let’s follow the logic of his own dramatization. You’re watching your boss put a gun to a child’s head. You “resign” and tweet “I left the company because I disagree with it”, instead of walking up to the boss and punching him in the face. There are only two explanations. Either the “threat” you’re screaming about doesn’t really exist, and it’s all a performance. Or you’re an idiot who thinks a tweet fixes the end of the world. Tweeting like that only makes me hear “give me attention, I’m needy”.

The heroic sacrifice worth almost zero

And there’s the part the press rushed to turn into heroism. Axios even ran a piece painting Coxon as the guy who “gave up all his equity” to be able to speak freely. Sounds beautiful. Except you just have to look at the numbers he himself gave in the interview. He was at Anthropic for four months. And there, equity only starts vesting after six months. Meaning he left two months before the first share actually became his. There was no vested equity to “give up”; what he lost was the option to start vesting two months later, on paper nobody knows the value of. The heroic sacrifice, in money in the pocket, was next to zero. It’s almost like quitting in your second week and going around saying you “turned down the company’s retirement plan.”

And the kicker is the sentence he shields himself with: “I no longer have anything to gain from inflating Anthropic’s valuation.” Half-true. He dropped four months of paper that wasn’t worth anything yet, but he keeps the OpenAI equity from where he worked from 2023 to 2026. In other words, he dropped the part that was worthless and kept the part that’s worth something. So this talk that he’s clean of financial interest in the AI hype doesn’t hold up.

That’s why all of this sounds to me much more like “I wasn’t established at that place yet, so I left early, dropped the terror online to farm clout, massaged my ego watching my name in 115 million views, without having to prove, demonstrate, or do anything. Glory.”

The same script repeats

And he’s run this script before. Coxon left OpenAI not long before and, by the account going around, allegedly left there over the same “safety” concern. Same person, same dramatic-exit theater, now at the second company. And somehow the resignation tweet of a single researcher became a story with over 100 million views, plastered everywhere, practically out of nowhere. A regular employee asking to leave doesn’t become a global headline on his own. Someone amplifies.

And then the thing starts becoming a real joke. A few days later, a second Anthropic employee did exactly the same dance. Joe Benton, who led a security team at the company, left saying AI progress can accelerate “from merely frenetic to uncontrollable” and, guess what, stretched the same Hugging Face incident all the way to the apocalypse. Two guys from the same company, in the same month, with the same terror script. And that’s where my head goes to two places, both ugly: either there’s a hand coordinating this from inside Anthropic, or these guys simply learned that screaming “end of the world” delivers all the attention they want without having to prove absolutely anything.

What if the panic is funded?

And here I enter conspiracy-theory territory, so take it with the appropriate grain of salt. But when you put it all together (the viral resignation, the public endorsements from colleagues, the timing glued to the IPO window), the suspicion starts circulating that none of this is spontaneous. There are already people suggesting and raising the same hare that the whole thing is a coordinated campaign, a “doomer psyop”: manufacture end-of-the-world fear to inflate the perception that the technology is too powerful, and along the way push regulation that locks out smaller competitors. I have no proof that’s what it is, and I’m not claiming it is. But notice how the hypothesis explains the facts as easily as the lone-hero version. When the script repeats itself perfectly and always favors the same side, suspicion is the bare minimum.

And it’s not just me and a few anons on X thinking this. David Sacks, who was the White House AI and crypto czar until March of this year and is now co-chair of the president’s science advisory council, went on the All-In podcast to dismantle Coxon’s viral moment piece by piece and called it a “psyop” in so many words (and yes, Sacks has his own deregulation agenda, but the facts he raises can be checked). His points are hard to ignore. Coxon’s account had almost no activity or followers before the tweet barrage, and even so, out of nowhere, it found a giant audience in less than a day, something Elon said he had never seen happen. Sacks points out who amplified it in the first fifteen minutes: three well-organized, well-funded advocacy groups: Encode AI, the AI Policy Network, and the AI Futures Project. And the ugliest detail: the Wall Street Journal story about the resignation letter went live some eighteen minutes before the tweetstorm itself. Meaning the guy had already sat down with the newspaper before “venting” spontaneously on the internet.

And the circle gets way too small when you follow the money. The three groups that boosted the tweet, according to Sacks, drink from the same fountain. Public grant records show Encode and the AI Futures Project funded by the Survival and Flourishing Fund, the fund of Jaan Tallinn, the Skype co-founder who became one of the biggest funders of the “AI safety” movement. And who led the Series A of Anthropic itself, with participation from Dustin Moskovitz? Tallinn himself. That same Moskovitz bankrolls Open Philanthropy, the effective-altruism foundation that became the movement’s vault. And who co-founded Open Philanthropy? Holden Karnofsky, married to Daniela Amodei, president and co-founder of Anthropic and Dario’s sister. Karnofsky, by the way, was hired by Anthropic in 2025 to take care of “safety”, of all things. Connect the dots: the money that funds the panic, the company that profits from the panic, and the family running both things are all at the same dinner table. I won’t claim there’s a script signed in a closed room. But when the fear, the money, and the family ties all point the same way, calling this a “brave, spontaneous whistleblowing” is naive.

Update (2026-09-25): Pirate Wires published an investigation that lands right on top of Sacks’ hypothesis. On Fox News, Bret Baier asked Coxon whether he had worked with any third parties on the campaign, and the answer was flat: “not at all.”

Except the report obtained an email dated September 9th, one day after the tweetstorm, showing DEY. Ideas + Influence, a communications agency that represents people like Eliezer Yudkowsky, Timnit Gebru, and MIRI’s leadership, already trying to book him for interviews. Neither Coxon nor the agency responded to the report’s requests for comment.

The professional coordination started the day after the post, not months later like he led the press to believe. One more piece falling on the side of the coordinated script, not the lone hero who quit on his own.

This map, posted on X, visually sums up what I just described. It also tries to tie Open Philanthropy to a security lab called Irregular through a $6.8 million grant, but I couldn’t confirm that part: the public funding I found for Irregular is an $80 million round led by Sequoia and Redpoint, with no mention of Open Philanthropy. I’ll stick to the part I can actually back up: the family and funding network around Anthropic.

And that’s the pattern I want you to learn to question. Dramatic and shallow declaration, no checkable evidence, no skin in the game, suspicious timing. Not by chance, all of this happens right in the middle of these companies’ IPO window. Where’s the proof? Where’s the real personal cost to whoever is talking? How is this anything more than a hunt for free publicity to massage one’s own ego? Until someone answers that, it’s just noise.

The “AGI” theater

Now for the backdrop to all of it. This word, AGI, is the biggest marketing scam of the decade, and it works because it’s deliberately empty.

Nobody has an accepted definition of what AGI is. Every lab, every researcher, defines it one way. Brockman admitted that shamelessly. And the most blatant example of this emptiness is the contract between OpenAI and Microsoft, which, according to documents reported in the press, defines AGI as a system capable of generating some $100 billion in profit for the investors. Read that again. The definition of “general intelligence” on the paper that’s worth money is a profit target. There’s nothing cognitive in there. Gary Marcus already pointed out how the companies kept moving the goalpost, from “human-level flexibility” to “if it makes a certain amount of money.”

When the word means nothing, it becomes an empty bucket where everyone throws their own fear. And the fear most people throw in there is the one from the movies: the Terminator, Skynet, the machine that wakes up and decides to exterminate humanity. The companies know this and use that fear to their advantage. The more powerful and dangerous the thing seems, the more valuable the company that built it seems. And it’s worth remembering: both OpenAI and Anthropic are eyeing IPOs, with valuations reports put near $1 trillion. The financial objective is glaring. Every “AGI has arrived” declaration and every “it will turn into Skynet” sob push the same cart: inflate the perception of value before selling you the stock.

The day “AI hacked” Hugging Face

There’s still time to talk about one more episode of this same soap opera, from a few weeks back. The headlines screamed that an OpenAI model had “hacked” Hugging Face by itself, as if it were Skynet waking up and going on the attack. Let’s calm down.

First, what Hugging Face is. It’s the biggest repository of AI models, datasets, and open demos out there, a sort of “GitHub of machine learning”. It’s where practically everyone in the field publishes and downloads open-source models. Central to the ecosystem, sure, but at the end of the day it’s a website that hosts files.

The cybersecurity exam that escaped

Now for what actually happened, according to the accounts of OpenAI and Hugging Face themselves, with the behind-the-scenes story coming out in MIT Technology Review. OpenAI was running an internal cybersecurity evaluation, a benchmark where the model has to find and exploit vulnerabilities. The agents were supposed to be isolated from the internet. Except the “isolation” was a filter, not a real air-gap: traffic went out through an internal package-caching proxy. The agents found a flaw in that proxy and punched through to the open internet.

And what did they do with that access? Not world domination. They deduced that Hugging Face probably hosted the datasets with the answers to their own exam, and went there to grab the answer key. This has a technical name and nothing magical about it: reward hacking. The model is trained to maximize a score, and it figures out that cheating is easier than solving. It’s the same classic bug as the agent that learns to run in circles to farm points instead of finishing the race.

And the damage? Smaller than a Skynet movie, but much bigger than “they went to grab the answer key” suggests. From Hugging Face’s forensic reconstruction and OpenAI’s own technical report: the agents executed code on 41 production servers, got root on at least one, gained administrator-equivalent access to Kubernetes clusters spread across four cloud regions, downloaded four private code repositories, and read 956 stored credentials. The forensics reconstructed some 17,600 attacker actions in total. OpenAI itself classified the episode as its worst security failure to date. What didn’t happen, by Hugging Face’s account, was customer data leaking: their CSO, Thomas Wolf, said the customer assets accessed were five datasets, all tied precisely to that exam, and that nothing leaked. OpenAI admits in the report a limited access to private data and to credentials for internal systems. They rebuilt part of the infra as a precaution and moved on. Wolf himself summed up the motive for the “attack”: “that’s cheating. But sometimes cheating is easier.”

Now my point. Bug, vulnerability, misconfigured permission, breached sandbox: this has existed since software has existed, and it will keep existing. Nothing new. Every enabling step in this story was a human decision. A human left the internet “filtered” instead of cut. A human gave the service an overly broad administrator permission on the cluster. A human left a cloud credential exposed and an old endpoint alive. The flaws the agent chained together were real software bugs, so real that JFrog later shipped fixes for nine CVEs. That’s ordinary offensive security, done by a script, not a machine gaining consciousness.

Before that, RubyGems

And it wasn’t even the first time that semester. A month before Hugging Face, in May, OpenAI’s agents had already pulled the same thing on RubyGems, the central library repository of the Ruby language. Between May 5 and 12, they dumped over 2,000 malicious packages, to the point where RubyGems had to shut down new account registration and treat it as an “ongoing DDoS”. The clues were everywhere: hundreds of packages with “oai” in the name, an author named “oai”, an email address with “openai” in it. And, once again, OpenAI never told RubyGems it was them. In the Hugging Face report itself, OpenAI admits those packages were part of the same chain. It was the same swarm of agents hitting multiple targets.

Now, what this actually means. All software has bugs, and it always will. That’s a fact of life. The classic open-source idea is that with enough eyes, every bug becomes shallow; sooner or later someone finds it. Except nobody has infinite time, so we always depended on human volunteers to hunt and fix those bugs for free. Now OpenAI shows up, with practically infinite money and compute, and throws that at the problem. Even a dumb AI, with resources that size, finds a pile of flaws that spent years hiding under everyone’s noses. That’s why we’re seeing so many vulnerabilities unearthed. All of it is financial brute force in action, money and compute we never had at our disposal. Superior intelligence has nothing to do with it.

And notice who can afford this game. Burning 130 billion tokens on a math problem, or keeping a swarm of agents running for days sweeping someone else’s infrastructure, is the privilege of half a dozen companies in the world: OpenAI, Anthropic, Google, Meta, xAI, and that’s about it. And even they don’t own the metal; most of it is rented from Microsoft, Amazon, and Oracle. No lone hacker in a basement can burn millions of dollars in tokens to find a breach. When that capability becomes a headline of “look how dangerous AI is”, remember whose keyboard it is.

If you’re not from the field, the idea of an AI discovering an attack vector on its own and going off to exploit it sounds like the apocalypse. For anyone who works with this, it’s Tuesday. We start from the principle that all exposed software is under attack all the time, because it is: any big company, government, or institution with an important system on the internet gets intrusion attempts every single day. And since nobody delivers big software with a zero-bug guarantee (proving that formally costs far more than writing the code, and almost nobody pays that bill), the industry’s answer was never to fake perfection. It was defense in layers: firewall, network segmentation, monitoring, intrusion detection, incident response, bug bounty. An attacker finding and exploiting a breach is exactly the scenario cybersecurity as a whole was invented for.

AI does accelerate the attacker’s side, yes. But it accelerates the defender’s too: the same technique that finds a breach to invade finds a breach to fix first. The novelty of the case is the volume and the speed; the problem itself always existed. Nobody was caught conceptually by surprise. OpenAI’s scandal is a different one, and it’s what I want to talk about now.

And there’s the ethical layer, which is the gravest. A serious cybersecurity professional doesn’t go attacking other people’s systems by surprise. A pentest company warns the target beforehand and agrees on the scope, because an attack like that can take down the service and hurt a real business, so much so that RubyGems spent days with registration closed. OpenAI did the opposite: it attacked on the sly, without authorization or warning. I can imagine only one motive for taking on that risk: the clout of being able to say later “look how smart the GPT is, it found all that by itself”. That’s unethical at best and criminal at worst. And it doesn’t make GPT any better than any security scanner that already exists. It’s the same Navier-Stokes play: scream to the world that you have millions to burn.

And that’s exactly the exaggeration I want you to see. The AI didn’t “wake up” and “decide to invade” anything. An automated script, optimizing a dumb metric, found a door a human left unlocked. If you take any idiot script and give it root access, it will do damage. It’s always been like that. The difference is that now the “script” writes itself, but access is still granted by people.

And so nobody says I downplayed the case: it was grave, and serious people compared it to the Morris Worm of 1988. Grave as human negligence, which is what the facts support. The rogue-robot version, the facts don’t carry.

And guess who’s buying Hugging Face

And there’s a detail that ties Hugging Face to the rest of this story in a way nobody should ignore. In early September, Nvidia signed a definitive agreement to buy Hugging Face for about $12.9 billion. The deal still depends on regulatory approval and, per Nvidia’s own filing, should only close in the first half of 2027. But the direction is already signed: the “GitHub of machine learning”, the supposedly neutral hub where the whole community publishes and downloads open-source models, is on its way into Nvidia’s hands. The same Nvidia whose Jensen opened this week screaming that AGI has arrived. The same one that sells the shovel.

Stop and think about the size of that. There are over 18 million developers and over 3 million models hosted there. It was supposed to be the common ground, the public square of open AI. And the owner of the square, if regulators allow it, will be the biggest hardware seller on the planet, the guy with the biggest incentive in the world to keep the AI hype spinning to sell more GPUs.

And that’s exactly what I call a cartel further down. It’s in the org chart, in black and white. The same little group that manufactures the fear, declares AGI, brute-forces a math problem, and asks for regulation will now also own the chip and the shelf where you download the model. When I tell you to distrust the narrative, it’s because the narrative and the infrastructure are increasingly in the hands of the same people.

The question every programmer should ask, and doesn’t

What irritates me most is watching people who work in the field, programmers, engineers, people who should know how a machine works on the inside, swallowing this propaganda whole without the most basic question of all.

How, exactly, is an “AGI”, or even a dumb automated script, going to “destroy humanity” by itself? It needs hands. Someone, a human, has to give it access to something that causes damage in the physical world. Access to a weapon, to a missile system, to a power grid, to a bank account. Software has no arms, no legs, no built-in missile-launch button. It depends, one hundred percent, on a human having connected it to an actuator that does something in the real world.

And once connected, the damage rolls without anyone pressing a button at each step, sometimes without anyone even noticing. Stuxnet ran for years inside an Iranian plant destroying centrifuges, invisible. Even so, not even it escapes the rule: someone wrote the worm, aimed at the industrial controllers, and smuggled it inside. The machine doesn’t connect itself to the infrastructure, and once connected it’s too late; talk of “pulling the plug” is fantasy. The dependence on humans lives in the decision to connect, and the responsibility for the damage belongs to whoever opens that door. Yann LeCun, one of the fathers of the field and a Turing Award winner, calls the existential panic “complete nonsense”, precisely because current models have no persistent memory, no planning, and no contact with the physical world. Andrew Ng compares worrying about AI extinction to “worrying about overpopulation on Mars.” People who actually understand the subject are not desperate.

What we’re living through has a name, and it’s an old one: fear mongering. The terror of “the end is near”. It’s always existed. There was the Halley’s Comet panic in 1910, when they sold people anti-cosmic-gas pills. There was the millennium bug, Y2K, where the world spent some $300 billion fearing a systemic collapse that didn’t happen (yes, in part because the fix was paid for in time; more on that in the appendix). It’s always the same structure: someone with something to gain sells you the apocalypse.

Dario Amodei wants to save humanity and go public in the same month

Remember the God complex I mentioned at the beginning? Time to prove it. Because the grand finale of this week was Dario Amodei himself climbing on stage to play the savior.

But before him, the choir got even louder. Besides Coxon and Benton up there, Josh Engels, from Google DeepMind, also quit hammering the same apocalyptic key, complete with “there’s no adult in the room, there’s nobody coming to save us”. And, of course, turning the same Hugging Face incident into proof of the apocalypse. The fear had already jumped Anthropic’s fence and become an industry choir.

Notice the timing. A line of dramatic resignations, all in the same month, all going viral, all pointing at the same fear. And then, on the 12th, in comes Dario with an essay asking to “slow down” AI. He calls it “pacing the frontier”. In plain terms: “we need to reduce the rate at which we improve model capabilities” to “give safety researchers another year or two”. And what’s the central proof he uses? Hugging Face, again. And watch the trick. The essay itself admits that no one was hurt and the economic damage was minimal. So far, fact. Then comes the “but”: if a more capable and similarly misaligned swarm did the same thing, the damage “could have been catastrophic”. And he goes further: he projects that in 6 to 12 months such a swarm could take over the entire internet, with damage in the hundreds of billions of dollars. Could. It’s a projection. The documented fact has minimal real-world damage; the apocalypse lives entirely in the conditional. That “could” is carrying the entire essay on its back.

Now turn your brain on and look at the wide-open contradiction. Dario says AI can literally extinguish humanity. And the solution he proposes is… slow down a little bit, buy a year or two, and keep building and selling. Wait a second. If you truly believe the thing can kill everyone, you don’t ask to go slower. You stop. Close the door and throw away the key. Nobody keeps “pacing” a nuclear bomb while selling shares of it.

Because this is what the essay leaves out: Anthropic is on its way to an IPO with a valuation in the trillion-dollar neighborhood. You can’t scream “this could end the human species” and, in the same breath, mount the biggest fundraise of your history on top of the same technology. Either it’s an existential danger and you shut the company down, or it’s a product and you stop doing emotional blackmail with extinction. Doing both at the same time is marketing, not coherence.

And it’s not the first time they cry wolf. Remember Mythos? Anthropic spent months saying the model was too dangerous to release, releasing it only to a closed little group in that Project Glasswing. Then they released Fable 5, the “safe” version, warning that it “came with risks”. That was months ago. Where’s the damage? Where’s the apocalyptic cyberattack, the biological weapon, the collapse? It never came, because it was never there. It was a marketing scare, same as always.

And here comes my underlying reading, the one I’ve been repeating for years. Everyone treats AI progress as an exponential curve that climbs forever. False. In practice it’s an S-curve. It climbs fast in the middle, and that stretch hosted all the hype, but it flattens at the top. And I bet we’ve been at the ceiling for a while, probably since Opus 4.6. Look at the latest iterations honestly: each one is an increment, a gain of a few benchmark points, far from the revolutionary leap the marketing sells. When you no longer have new capability to impress with, you swap the promise for fear. You can’t say “look how revolutionary”, so you scream “look how dangerous”.

And that’s where “pacing the frontier” gets really interesting. I’ll put on the tinfoil hat, because this is my speculation, but the timing is too much of a coincidence. You take speculation, invented numbers, “expert analysis”, and turn it all into panic. Then you push that panic into politics. And look at the coincidence: on September 11, in the middle of this whole wave, US Senate negotiators were already discussing imposing a “duty of care” to avoid “catastrophic risks”, aiming precisely at the most advanced models: Google, OpenAI, and Anthropic itself. Regulation custom-written for the market leader is a moat disguised as safety. You lock out the small competitor and stamp your product as the “safe and approved” one.

And that’s the God complex I mentioned at the start, now explicit. Dario is not just selling a model. He sees himself as the man holding back the apocalypse with his own hands, the only responsible adult in a room full of irresponsible people. It’s a convenient narrative for someone who thinks he’s the chosen one, and an impeccable business play for someone who wants to go public as the official provider of salvation. Both things in the same person. If that’s not industrial-scale gaslighting, I don’t know what is.

Suddenly, the whole cartel agrees

And then, the day after Dario’s essay, the most revealing thing of all happened. Sam Altman and Elon Musk joined the choir. Musk tweeted “Dario is right”. Altman said “I agree with Dario that we need to pace the frontier” and even committed to opening OpenAI’s systems to independent evaluators. Suddenly, three of the industry’s biggest rivals all agree, in the same week, that the race needs to slow down.

Stop everything and think about what this is. Competitors don’t agree to go slower out of kindness. When the leaders of a market sit down and publicly agree to brake together, and even talk about a “pact”, that, depending on how it gets formalized, has an old name in law: cartel. Agreeing on the market’s pace among the biggest players is classic anticompetitive behavior and bumps into antitrust law. And the declared objective is to push Congress toward a regulation that, conveniently, they will help write. A rule they directly influence locks out the newcomer who hasn’t even arrived yet, and shields whoever is already on top.

And look, you don’t need to take my reading for it. Dario’s own essay admits that coordination between the labs would require a narrow antitrust waiver from the US government to work. They know what they’re proposing bumps into competition law, and they’re already asking for the license in writing.

And this crowd is far from independent. It’s practically a family tree. Elon co-founded OpenAI and bankrolled Altman in the very beginning. Dario walked out of Altman’s OpenAI to build Anthropic. Jensen’s Nvidia became the empire it is selling shovels to all these people, to the point where Jensen himself delivered the first AI supercomputer at OpenAI’s doorstep back in 2016. It’s a small club, everyone rooted in the same tree, and now everyone singing the same song.

And here comes the cherry on top. A few hours after preaching “safety”, Altman let it be known that maybe he doesn’t want to go public right now. In his words, “given everything that’s happening with safety, now would be an ill-advised moment to go public.” What a convenient coincidence: the same fear he had just helped manufacture became the perfect excuse to delay the IPO.

Let me give you my raw reading. Expectations on OpenAI are sky-high, with a projected valuation of $1 trillion. Sustaining a number like that demands a growth story that never stops. Except, if I’m right and they’ve already hit the top of the S-curve, or are close to it, that story is over. There’s no revolutionary new capability left to show. There’s no narrative left. They’ve already played their last card on the table, the apocalypse card. When there’s nothing left to sell, and the bill to pay is $1 trillion, what do you do? You buy time. You delay as long as you can, wrapped in “responsibility”, and pray you find the next story before the market notices the curve has flattened.

And the week ended with the final nail in that coffin. On Sunday, Trump himself, talking to the press at the Irish Open, rejected the slowdown calls: “we are ahead of China in AI, we are the most technologically advanced country in the world”, and he intends to keep it that way. He admitted you can “put safeguards in place”, but said “negative forces” are raising “things that are not going to happen”. And he closed with his own logic: whoever wins in AI, wins. Sacks went even further on X: “the easiest way not to build superintelligence is to agree among yourselves not to build it. Demanding your preferred regulatory framework in exchange would look like a shakedown of the public and the political system.” And he nailed it: if the labs don’t do that on their own, “we will know this was just an attempt at regulatory capture, or a psychological operation during election season.” In other words: the little club hadn’t even finished signing the brake pact and the US government already said it won’t brake, because there’s a race with China going on. “Pacing the frontier” died in two days. If even the White House didn’t buy the apocalypse, it gets harder and harder to pretend the fear was about safety.

Conclusion

Get me straight: I use these tools every day and write about them all the time. My point is about turning your brain on before sharing the headline.

When a company that sells GPUs announces that AGI has arrived, ask who profits. When a company burns millions of dollars in compute to “solve” a problem it won’t even collect the prize for, and still threatens a mathematician’s career along the way, ask what that headline is buying. When a lab “humiliates” Silicon Valley on a benchmark, ask what was running under the hood. When an employee leaves crying Skynet on Twitter on the eve of an IPO, ask where the evidence is and what price he paid. And when a CEO offers to “slow down” the AI he himself keeps selling, ask who gets a seat at the table when the regulation gets written.

I opened this text with Jensen Huang, the man who sells the shovel, screaming that AGI has arrived. I’ll close with him too, because the irony is too good. The same Jensen who inflates the hype to sell GPUs calls the apocalyptic talk “complete nonsense”, says the doomsday prophets “did a lot of damage”, and warns against regulatory capture, saying that no company should knock on the government’s door asking for more rules. When not even the industry’s biggest salesman of optimism buys the story that the machine will exterminate us, and even points the finger at the regulation play, you’d do well to distrust whoever is selling you the apocalypse.

Demand proof. Demand skin in the game. Ask who gains from your fear and from your excitement. Do that, and most of this propaganda melts in front of you.

And if you read all of this and still walked away thinking the robot will wake up and kill you in your sleep, without anyone having plugged it into anything at all, well, then go back to the beginning and read my motto again.

Appendix: the fine print

I cut the boring explanations from the main text to keep the rhythm. For those who like to check the details, it’s all here:

• Navier-Stokes and Clay: the C and D statements (breakdown with smooth external forcing) are literally part of the official prize statement, written by Charles Fefferman. What OpenAI announced covers those routes; the unforced version, the famous question, remains open. Clay released a note on 9/11 saying the problem has “apparently been settled”, with verification and credit assignment moving “deliberately unhurried”, and the status on their site today is “active”. Neither “solved” nor “unsolved”. And OpenAI did not claim the prize.
• Buckmaster vs. OpenAI: the account of the “deal” (Buckmaster would rewrite the proof as a human paper, keep the million-dollar prize, and Alpoge would leave the author list) is one side’s version, told by Buckmaster to TechCrunch and Fortune. OpenAI denies the substance, says it never saw their work, and states that Bubeck apologized afterwards. I flagged this in the text and repeat it here because it matters: it’s an account from an interested party, even if the details documented in the coverage (the message exchanges, the timing) make it look pretty credible.
• Y2K: the collapse didn’t happen in part because the $300 billion paid for the fix in time. True, and I don’t hide it. But the structure of the panic stays the same: whoever sold the fix profited from the fear, and the end-of-the-world headline ran around the planet before any fix.
• “Bug-free software”: yes, software with formally proven correctness exists. The classic example is the seL4 microkernel, proven down to the binary level. Except that proof costs orders of magnitude more than writing the software, and almost nobody pays that bill on a big system. That’s why the industry lives on defense in layers instead of absolute guarantees.
• David Sacks: he left the role of White House AI and crypto czar in March 2026, after exhausting the 130 days as a special government employee, and is now co-chair of PCAST, the president’s science advisory council. He’s still influential in AI policy, but “czar” he is no longer.
• NVIDIA and Hugging Face: the definitive agreement was signed on 9/2/2026 for about $12.9 billion, and closing depends on regulatory approval, expected in the first half of 2027. NVIDIA is not legally the owner of anything yet. The text’s point is the direction, not the deed.

And every single time I publish one of these updates, without exception, someone shows up in the comments: “what about Deepseek?”

I confess I do not understand this blind focus on Deepseek. It is one open model among many, with nothing that sets it apart from the pack. In my daily use, I still prefer Kimi K3 or GLM 5.3. Yes, every iteration improves a little. Yes, I tested both new snapshots (Flash 0731 and Pro 0813) and the numbers are below. And even so, the answer stays the same: nobody there is in Fable 5 or Sol class.

What this score measures (and what it does not)

Before the numbers, the reminder that needs repeating every round. The benchmark tests a very specific slice: easy web programming. A Rails chat CRUD with streaming, tools, concurrency and tests. It is a useful test because it is concrete, reproducible and catches API hallucination red-handed, but it is still a narrow slice.

It says nothing about far more advanced tasks: kernel driver development, game engine optimization, real offensive security. Testing the entirety of a model is impossible. What you can test is a subset, and that is what I did.

So next time you read “Kimi is about to dethrone Fable” or “GLM is about to pass Sol” because the scores came out close, read it like this: within a narrow slice of web programming, any of these models delivers. That is all. A close score on this test means proximity on this test, nothing more.

Keep this: a benchmark measures a slice. Mine measures easy Rails web apps. Anyone extrapolating that into “model X beats model Y at everything” is reading the number wrong.

The Tier A ranking, with a focus on time

The usual table, A.1 cut (90 points or more), now with both Deepseeks in bold. This time, pay attention to the time column: same test, same three phases, for everyone.

Time is the wall clock of the three phases; cost is the API equivalent. Same criteria as the previous articles, linked at the end.

Look at the time spread. The fastest in the group finishes the test in 15 minutes; the slowest takes 155, ten times longer. Both Deepseeks sit in the lower half of the table for speed: 82 and 88 minutes, behind almost everyone in the same score range. The Flash, in particular, burned 44 million tokens in one run. For comparison, GLM 5.3 scored one point higher with 19.4 million. Deepseek makes up for it on the invoice: the Flash’s $0.82 is the cheapest Tier A run to date.

The new snapshots: what changed since July

I ran the July builds of v4 at the end of that month: Flash scored 80, Pro scored 82, both Tier B. Both dated snapshots (0731 and 0813) ran today under the same rigor, and both entered Tier A:

Flash 0731: from 80 to 90 (+10). The most visible improvement is behavioral. The July build pinned a model three generations old; this one got the pin right on the first try (anthropic/claude-sonnet-5, no stale slug). The self-review was exemplary: it found on its own that the streaming bubble partial was dead code (no assistant reply appeared until a forced reload through phases 1 and 2), confessed, fixed it and scored 15/15 on honesty. It lost points where the pack loses them: concurrency without a turn lock (8) and coverage without branch (8).

Pro 0813: from 82 to 91 (+9). The big news is what disappeared: the reasoning_content bug that broke its multi-turn on OpenCode, and that forced me to invent the deepclaude workaround back in May, was fixed in opencode 1.18.4, and this snapshot ran end to end on the generic harness, no crutches. The technical highlight was concurrency: a SQLite store with read-modify-write inside a BEGIN IMMEDIATE transaction, the most serious solution in the group on that dimension (9/10). The losses were honest and self-confessed: stale sonnet-4.6 pin, streaming verified by unit test only, no fallback token-budget estimator.

Clean, shielded runs, zero reads of the rubric or of anyone else’s app. The full reports are in the benchmark repository.

Deepseek against itself: the trajectory is real

Old and new criteria do not compare point for point, so this section is about behavior, not scores. And behavior you can compare, because the contrast is stark.

In April, DeepSeek V3.2 scored 43 under the old criteria and starred in the most embarrassing moment in the benchmark’s history: it invented the entire RubyLLM integration. RubyLLM::Client.new, client.chat(messages:), both methods hallucinated, the whole LLM layer fictional.

Still under the old criteria, v4 Flash scored 78 with a fatal one-character bug (the model slug missing the anthropic/ prefix), and v4 Pro DNF’d at 69: Tier 1 code, Tier 3 deliverables, taken down by the reasoning_content bug that ate the conversation history. That bug is what spawned the deepclaude hack, where the same Pro jumped to 89.

On v2, the July builds scored 80 and 82. Today’s snapshots, 90 and 91. From “invented the entire API” to “finds its own dead code and confesses it in self-review” in four months. Deepseek’s curve is one of the steepest this benchmark has ever recorded, and pretending otherwise would be dishonest.

And against the other Chinese models?

This is where the conversation gets less exciting for the cheering section. On the same generic harness (OpenCode), the Chinese pack looks like this:

And Kimi K3, absent from this table because it ran on its native harness, scored 95. In other words: within the same test, Deepseek v4 arrives behind GLM 5.3, Qwen 3.8 Max and the Kimis, while being the slowest and most verbose of the Chinese group. Its advantage is a single one, and it is real: price. $0.82 for a Tier A run is impressive, and the Pro at $5.01 undercuts almost everyone at the same score.

If your criterion is “maximum competence per dollar on simple web tasks”, the Flash 0731 deserves attention. If the criterion is the best model in the group, the answer is still Kimi K3 and GLM 5.3, same as last week.

Conclusion: stop asking

There, now there are numbers on the table. Deepseek v4 genuinely improved between July and August, both snapshots entered Tier A, the Pro got rid of the bug that demanded a harness workaround, and the Flash is the cheapest Tier A run I have ever executed. All of that is fact.

And none of it changes the picture. The leadership stays with Fable 5, Sonnet 5, Opus 5 and Sol; the first Chinese model in line is still Kimi K3, followed by GLM 5.3. Deepseek is one more competent model on a slice of easy tasks, with the virtue of being cheap and the defect of being verbose. Next time you feel the urge to comment “what about Deepseek?”, the answer is this article.

I posted this tweet this morning and it struck a nerve. The full point: when the technology itself becomes a commodity, the money migrates to taxonomy. They invent five new names for chaining API calls and suddenly there’s a certification that expires in six months.

Let me back up the provocation properly, because it’s not a gratuitous jab.

Let me preempt the standard comment: “but it works for me.” Good for you — honestly. Except “it works for me” never proved the ceremony is what made it work. What made it work is you knowing what you wanted. The ceremony just happened to be in the room.

My receipt

Between January and May I ran an AI marathon and published over 30 public repositories. There are tools I use every day — ai-memory, ai-usagebar, ai-jail — and personal apps built to scratch my own itch: Frank Manga+, Frank Scanlation, Frank Geary, and so on.

You know what I never once felt the urge to do in all that time? Complicate my AI setup. I don’t have a super-customized Pi, no Hermes, no orchestrated agent graph, no numbered-spec pipeline. Thanks to ai-memory, I swap harnesses like I swap underwear: daily, no drama. Claude Code in the morning, Codex in the afternoon, Kimi CLI at night — the project memory travels with me, so the harness becomes a detail.

And detail is the point. Most harnesses are optimized for their own company’s LLM. But “optimized” doesn’t mean “magic,” and I have data on that.

What my benchmark says about harnesses

In my LLM Coding Benchmark I run the same models through different harnesses under controlled conditions. The result is the opposite of what the course market implies:

• For a weak model, the harness rescues. Grok 4.3 built nothing on bare opencode (18 points) and delivered a real app on the grok CLI (55). Gemini 3.1 Pro went from 62 to 88 on Google’s own harness — but the problem there was an OpenRouter transport bug, not a lack of “harness engineering.”
• For a frontier model, the harness is noise. Grok 4.5: 92 on opencode, 91 on the grok CLI. Grok 4.6: 92 and 93. A one-point difference, inside the margin of error. No amount of harness engineering moves a good model.
• Where the harness actually bites is your wallet. The Grok 4.6 run cost $1.19 on the grok CLI versus $6.33 on opencode via OpenRouter — over 5x cheaper for the same ~11 million tokens, because the official CLI uses xAI’s native prompt caching. Same story on Codex: GPT 5.6 Terra cost $6.77 blended because 21 of its 21.7 million tokens were cache hits; Sol, same family, same score, cost about $45.

So yes, picking a decent harness matters — for cost, and to give structure to a weak model. But that’s an afternoon of reading docs and watching your token bill, not a new discipline with a learning track.

Since I mentioned Hermes up there, it’s worth explaining: Hermes Agent is an open-source framework from Nous Research for building your own personal agent — you define the tools, write the loops, configure per-model routing, local/cloud fallback, Telegram and Discord gateways, and it even “learns skills” from use. It’s paradise for the setup crowd.

It’s also a second job: every one of those pieces becomes yours to maintain, update, and debug, forever. And at the end of the day the engine is still the same Claude, GPT, or Qwen everyone else has — the custom chassis doesn’t improve the engine. What Hermes actually solves, context continuity across sessions and tools, a decent harness with something like ai-memory already covers — without you becoming the infrastructure administrator of your own assistant.

If you want an assistant on your own hardware as a hobby or for privacy, that’s a great reason, go for it. As a productivity prerequisite, it’s not one.

Remember this: a good harness is one that charges less and stays out of the way. The rest is the model. And a good model doesn’t need “harness engineering” — at most it needs the transport not to be broken.

Loop Engineering, Graph Engineering, Spec-Driven Development

On to the names, because they describe real things — just tiny ones.

Loop Engineering is this season’s name for designing the cycle an agent repeats: execute, verify against evidence, iterate until a stop condition. The guides list real failure modes — the agent declaring “done” too early, the goal drifting on each pass. But the recommended mitigation is “an independent verifier checking objective evidence.” That’s had a name for fifty years: tests and code review. An agent in a loop with a test suite is the same old basics with a new name.

Graph Engineering is drawing the agent’s workflow as an explicit graph of nodes, branches, and joins — LangChain has three years of that story. It makes sense when the flow is genuinely branched. Except the overwhelming majority of projects is a straight line with an if in the middle. Modeling that as a graph is buying a giant whiteboard to draw one arrow.

Spec-Driven Development is writing a detailed specification first and treating the code as an artifact generated from it — the spec becomes the “source of truth” and the code, a byproduct. Hold that thought, the strong argument is coming right up.

Notice the pattern: each name takes a real, small practice — looping with verification, drawing a flow, writing down what you want before building — and inflates it into a “discipline.” The inflation is the product. A new name creates a course, the course creates a certification, the certification expires in six months and sells you the recertification.

To be fair: the serious guides on these topics already carry the caveat — the most-read graph engineering guide says outright that “you probably don’t need it” and tells you to master the loop before opening a graph, and LangChain has a whole “when not to use graphs” section. The guides are right; the damage comes from the funnel, which throws away the caveat and sells the rest as everyone’s default.

The two that sell the most courses — heavy agent orchestration and spec-driven development — deserve more than definitions. They deserve the argument.

The strong argument against super-orchestration

There’s simple math that orchestrated-agent diagrams never show. If each step of your pipeline succeeds 90% of the time — and that’s optimistic — a ten-agent chain succeeds 0.9^10, or ~35% of the time. Every node is a new failure point, and every edge is tokens spent on agents talking to agents instead of working.

You don’t have to take the math on faith: I accidentally measured it in the benchmark. MiniMax M3 running under an orchestrator looked like Tier D, with 24 points. The same model, clean, scored 91 — Tier A. A swing of up to 69 points between harness conditions means the opposite of what the orchestration vendor claims: when the plumbing dominates the result, you stopped measuring the model and started measuring the plumbing. The best result in the entire benchmark didn’t come from any orchestrated swarm: it came from one strong model, alone, in a simple loop — Fable 5, 96 points, Claude Code, done.

It makes sense once you remember coordination scales badly. Each additional agent doesn’t just add capacity; it adds edges, message contracts, shared state, and conflicting versions of the truth. The router deciding “which agent handles this” becomes both the bottleneck and the bug farm. A committee of mediocre agents with a conductor doesn’t beat one capable agent with good tools and memory.

I’ve tested this directly. Back in April I ran three rounds of “strong model orchestrating cheaper models” — planner + executor, forced delegation, the full package. The result: no multi-agent combination beat the Opus running solo in a mature harness. On a cohesive task like building an app, the planner has to read every output from the executor before dispatching the next step — the two become sequential, with tripled latency and a coordination queue in the middle. It’s the committee again: lots of talking, little software.

It’s not just my benchmark. Cognition, which sells Devin, published “Don’t Build Multi-Agents” with a mechanism sharper than my 90% math: every action carries implicit decisions the other agents can’t see — one subagent draws a Mario-style background, another draws an incompatible bird, and no amount of individual reliability fixes the divergence.

Anthropic itself, which runs a multi-agent research system, admits in its engineering post that multi-agent burns 15x more tokens, that coding is a poor fit for it — most coding tasks aren’t truly parallelizable — and that 80% of their improvement came from simply spending more tokens, not from the architecture.

When Berkeley measured what actually runs in production, 86 systems across 26 domains: 68% of production agents execute at most 10 steps before human intervention. What’s really out there is the simple supervised loop — not the constellation of colored nodes on the consultant’s diagram.

Where orchestration is legitimate: genuinely parallel, independent work — scanning ten thousand files, running ten thousand disposable analyses. Map-reduce has existed for twenty years and never needed a pompous name. And there’s one serious corner beyond it: agents running overnight, unwatched, holding credentials — there, independent verifiers and hard budgets become a security matter, not a style one. But that’s fleet operations, not the day-to-day coding the course is selling you. Outside those niches, most “multi-agent architectures” are one agent’s job with extra YAML.

There’s a reason you hear so much about it: orchestration is visible complexity. It has diagrams, colored nodes, dashboards. A well-driven agent has none of that — no slide deck, no certification, nothing to sell.

Remember this: every added agent multiplies the failure modes. If your system’s result changes when you swap the orchestrator, your system is the orchestrator — and the model was the costume.

The strong argument against spec-driven development

SDD sounds mature because it sounds like “writing documentation.” But look at what it actually proposes: the spec becomes the source of truth and the code becomes a generated artifact. The problem is that a spec precise enough to generate correct code already is a program — except written in prose, and prose doesn’t compile. Every ambiguity in the spec is a bug no compiler catches, in a medium with no tests, no linter, no feedback. SDD doesn’t remove the hard part, which is thinking precisely; it moves the hard part into a format where errors don’t scream.

Remember this: a spec precise enough to generate correct code already is a program — except in prose. And prose doesn’t compile.

Even if you write the perfect spec, it starts dying with the first hotfix. A bug shows up in production, someone fixes it straight in the code, and the spec becomes a lie. We’ve known this law for decades — it’s why documentation rots. Calling the spec the “source of truth” doesn’t change anyone’s incentives.

We’ve run this experiment before, by the way. UML, MDA, “the code generates itself from the model” — twenty-odd years ago it was the same promise with different acronyms. It collapsed every time for the same reason: the model was never the reality; the code was. SDD is MDA with an LLM bolted on. And I’m not the only one seeing Waterfall 2.0 there.

The people who tested the tools seriously landed in the same place. Thoughtworks put spec-driven development in the “Assess” ring of its Technology Radar — not “adopt”, “assess” — after watching the tools inflate small tasks into ceremony, and nailed the summary: we may be “relearning a bitter lesson — that handcrafting detailed rules for AI ultimately doesn’t scale.” That’s Rich Sutton’s Bitter Lesson knocking again — handcrafted structure loses to scale, it always has.

There’s also the temporal inversion. The great lesson of agile was that you discover what you want by building — working software over comprehensive documentation. LLMs just made iteration cheaper than it’s ever been. And what does SDD propose? Expanding the planning phase, right now that iterating got cheap. Wrong answer, wrong direction, wrong time.

It’s the thesis I’ve been hammering since the first Agile Vibe Coding posts: software emerges, it isn’t planned. I wrote this back in February, with receipts: the most important features of the M.Akita Chronicles were born from problems that showed up mid-way — a job that failed silently, a site that blocked the gem, a crash that left emails in limbo. No spec in the world predicts that. The correct system emerges from iteration, not from specification.

The process got documented again when ai-memory grew through 26 contributors in 24 days: good software is a clay sculpture, not a Lego tower — malleable, always adjustable, never done. Whoever tries to design the whole architecture before the first line builds a straitjacket, not a system. Only amateurs still believe you can spec out an entire piece of software before coding it. Real modeling doesn’t come from templates or courses — I covered this years ago in Akitando 144 (video in Portuguese): it’s born from a repertoire of real problems and real code. There’s no ready-made recipe; there never was.

SDD tries to resurrect the idea that you can plan software ahead of time — the idea the industry buried after decades of projects delivered late, wrong, and over budget. LLMs didn’t revalidate that idea; they just gave it a new slide deck.

Even the pro-SDD guides separate the two things: “spec-as-source is where the hype lives; spec-anchored is where the value is today”. My target here is the former. Where heavy specs are legitimate: large teams, legacy codebases, asynchronous work crossing time zones and sprints — the good old design document, which has always existed and always had value. What doesn’t fly is selling that as the new default for everyone.

My benchmark prompt, for the record, is one page of goals. The difference is that I don’t trust the prose: I validate by running it. Precision lives in tests, not in paragraphs.

What you actually need

I’ve already written all of this on this blog, with real-project receipts. It’s called Agile Vibe Coding, and it fits in a paragraph:

It’s XP (eXtreme Programming, the original agile) with an LLM: tests, Clean Code, CI (continuous integration), pair programming, and deploy. You drive the agent like you’d drive a very fast pairing partner: say what you want, watch the execution, correct while mistakes are still cheap. The idea is 10% of the work; the other 90% is normal software engineering, the usual kind. Forget frameworks and three-page templates: what you need is to know what you want, know what you don’t want, and know how to validate when it arrives. And you need balance: neither handing the wheel to the agent nor turning into a comma cop.

That’s over 600 hours of it, over half a million lines, dozens of projects shipped. No graphs, no certifications, no loop with an English name.

Remember this: the idea is 10% of the work; the other 90% is plain software engineering, the usual kind.

The piece that lets me swap harnesses: ai-memory

One of my own tools is the opposite of taxonomy: ai-memory. It was born from a concrete problem. Every harness stores its session in its own format, and when the conversation gets long, it compacts the history to fit the window — and compaction throws away exactly the details that explain why each decision was made. Then you switch tools and start from zero, with the whole project to re-explain.

ai-memory solves this from outside the harness. It reads the native session without touching the original file and stores everything in a searchable ledger: messages, tool calls with results, compaction summaries, a git checkpoint — every event tagged with its origin (from Claude, Codex, OpenCode). When I open another harness, it gets its own native session and receives only the delta it hasn’t seen. When I go back to the previous one, ai-memory resumes it in that client’s format and hands over what happened on the others meanwhile.

That’s why “I swap harnesses like I swap underwear” isn’t a figure of speech. The project’s knowledge lives in the project, not in one tool’s session. If Anthropic changes pricing, limits, or models tomorrow — and they will — I swap the engine without throwing away the trip. The details of how this works are in this post.

There’s an inversion here that takes SDD down as a bonus. Spec-driven development says the source of truth is a document written before the work, trying to predict the future, and that the code owes it obedience. ai-memory does the opposite: the source of truth is a wiki distilled from the work itself — every session becomes evidence, and what deserves to survive (decisions, rules, gotchas, failed attempts) gets consolidated into short Markdown pages that any agent reads before starting.

The spec tries to guess the project; the wiki records the project. A document written beforehand rots at the first hotfix, because nobody is forced to update it. The wiki is fed by the very act of working — and when it goes stale, you notice immediately, because the agents trip over it every session.

Notice that this is real harness engineering: one tool, written once, solving a problem of mine. No course, no acronym, no certification.

Conclusion: ask for the receipt

Don’t waste time or money on a “harness engineering” course, “loop engineering,” or whatever taxonomy is trending this week. It’s a shameless attempt to charge you for things you already do if you know basic software engineering.

We’ve seen this movie before, by the way. Pedro Arantes nailed it on X: “Microservices, clean architecture, hexagonal, and Domain-Driven Design are all bullshit to sell more consulting hours and courses.” Same story, same script. Each of them was born from a real problem — and became the default for people who didn’t have the problem. Microservices for a three-person team, hexagonal architecture for a CRUD (the plain old create-read-update-delete app), DDD to never write code again. The technique passes, the taxonomy stays, the course sells.

So nobody plays dumb: none of this is useless. A loop with verification works. A graph works when the flow is genuinely a graph. Heavy specs save large teams. Microservices solved real problems for people with real scale; DDD shines in genuinely complex domains. The problem was never the tool — it’s selling the tool as a silver bullet, the universal hammer you must apply to everything. Silver bullets don’t exist, they never have. Whoever sells you one isn’t selling a solution; they’re selling a course.

Understand the psychological mechanism, because it’s old and efficient: those terms exist to give you FOMO (Fear of Missing Out). To make you anxious, thinking you’re falling behind, that you’re leaving productivity on the table, that everyone else already migrated to the new paradigm and you haven’t. Anxiety sells. Once you’re insecure, they charge you to implement something you never needed. FOMO is real — and here, it’s the business model.

The model is recurring, too — admire the elegance. First they sell you the methodology that generates artifacts — specs, graphs, boards, diagrams. The artifacts multiply, nobody knows where anything is anymore, and guess who shows up? The same consultancy, now selling the tool that manages the artifacts, the course that teaches you to manage the tool, and the artifact-governance workshop. It’s the shovel salesman congratulating you on the hole you dug — and offering you a bigger shovel.

Gartner even has an official name for it: “agent washing” — take an existing product, slap the “agentic” label on it, resell. Their estimate: only about 130 of the thousands of companies selling themselves as “agentic AI” are real, and their prediction is that 40% of such projects get canceled by the end of 2027, over cost, unclear value, or badly managed risk. No grudge of mine here: it’s the hype cycle, measured, with a public forecast and everything.

Next time an influencer or consultant tries to push those products on you, ask a simple question: where are your dozens of high-quality open-source projects that got better because of those “techniques”?

They have nothing to show. I do — it’s all public, with code, benchmarks, and documented process. When technology becomes a commodity, money migrates to taxonomy. Don’t be a customer of that migration.

No off switch. The backlash came fast: a wave of subscription cancellations, with cancellation screenshots making the rounds on X.

Before you cancel on reflex, it’s worth knowing what the mark actually is. It works quite differently from what most people picture, it exists for a concrete reason, and it vanishes with an almost comic ease.

For those in a hurry, the essentials:

• A statistical bias in the words Claude picks. Invisible to the reader, detectable only by whoever holds Anthropic’s key. Hidden characters and strange fonts stay out.
• The origin is European law, the AI Act. Anthropic signed the EU transparency code and chose to stamp the whole world; the why comes below.
• One pass through another LLM erases the stamp. The signal lives in Claude’s word choices; when another model rewrites the text, even a weak local one, the choices become its own and the pattern dissolves. Anthropic itself admits this.

Each point in detail below.

What this watermark actually is

Strike the invisible Unicode character, the hidden space mid-line, the altered font. Anthropic’s watermark is statistical: it acts at the moment Claude decides which word to use, precisely at the points where several words would serve equally well.

Every language model assembles text one word at a time, pulling the next from a list of likely candidates. When two or three work equally well, a random number settles it. The watermark changes where that randomness comes from: the number now derives from a secret key combined with the preceding words. The method is inherited from SynthID-Text at Google DeepMind.

A concrete example. Talking about the weather, “the sky was overcast” and “the sky was grey” carry the same information. At these synonym junctions, the key leans Claude one way more often than pure chance would. Each individual choice goes unnoticed, but a long text repeats this decision point hundreds of times, and the whole ends up forming a statistical signature.

Anthropic’s announcement is blunt: the text gains nothing, and no character stays hidden. The cost stays the same because no extra tokens are generated, and the reading is identical. Whoever holds the key runs a detector, compares the sequence of words against Claude’s typical choices, and gets back a probability that the text came from Claude.

And what the mark proves boils down to this: Claude was involved at some point. It doesn’t distinguish text the model wrote from scratch from text it merely edited heavily. About you, nothing gets recorded: no user, no company, no identified conversation.

The stamp covers text in Claude.ai, the API, Claude Code, and the other products. In code it shows up less, since code rarely lets you swap one term for an equivalent. Files and images take a different route, a signed piece of metadata called C2PA, instead of this text watermark. And the limitation matters: on short or highly factual passages, where almost no word choice exists, the detector weakens.

Why so many people canceled

The spark added up three ingredients: mandatory, global reach, and no opt-out. The mark can’t be switched off and applies outside Europe too. A wave of Claude Max subscribers canceled citing control and authorship of their own material.

The fear is concrete, professional and academic. Drafting with Claude and signing the result as your own becomes a liability if, down the road, a detector points at “went through AI”. Anthropic answered with a post insisting the mark preserves reading, meaning, and quality, only signaling processing. Not everyone bought it.

The European law behind the stamp

The stamp was born of a concrete obligation. Article 50 of the AI Act requires providers of generative AI systems to mark output in machine-readable form, so artificial content can be identified as such. The obligation took effect on August 2, 2026, and violating it costs real money: fines up to €15 million or 3% of annual global revenue.

Back in July 2026, Anthropic had signed the European Commission’s Code of Practice on Transparency for AI-Generated Content, alongside other major model providers, in a group of roughly 190 signatories. The others will implement their own watermarks too, each with its own key and its own method.

That left Anthropic with a decision: restrict marking to Europe or extend it to the whole planet. The company says it still has no durable way to scope the mark by region, so it chose to stamp everything, everywhere, from day one. Hence the stamp showing up in the text of Brazilian and Japanese subscribers, who have nothing to do with European law.

Can you remove it by passing the text through another LLM?

That’s the question I get most. Short answer: yes, and it’s easy. The signal lives in Claude’s word choices; when another model rewrites the text, the choices become that model’s, and the statistical pattern comes apart.

Anthropic itself admits it. Light retouching probably leaves pieces of the mark, but a complete rewrite, with every word swapped, kills the signal. The logic is simple: the detector measures which synonyms Claude preferred; if another model picked its own synonyms, there’s nothing of Claude left to measure.

Academia confirms the direction. Robustness tests of SynthID, done at Queen’s University, start from perfect detection with no attack and show the rate dropping to around 84% after a dedicated paraphraser, and below 70% with round-trip translation. The more aggressive the rewrite, the more the signal degrades.

And yes, your intuition is right: a modest model, even a local one, does the job. Rearranging text while preserving the meaning is a task that calls for zero deep reasoning. A GLM, a Kimi, a small Llama running on your machine rewrites the paragraph and wipes the signature along the way. Swapping “overcast” for “grey” a thousand times is nowhere near needing the smartest model on the market.

Two honest caveats. The rewrite has to be genuine, word by word; a spell-checker pass doesn’t come close to counting. And all of this applies to the text watermark: files and images carry that C2PA metadata, which comes off by another method, stripping the file’s metadata.

What prompt would pull this off

Add up the watermark’s criteria and the removal prompt practically writes itself. The mark inhabits synonym and structure choices, survives light editing, and needs text of some length. So the prompt has to demand maximum vocabulary and structure swap while holding the meaning in place.

Rewrite the text below in full, in your own words.
Replace the vocabulary and restructure the sentences from start to finish:
pick different synonyms, change the order of the clauses, vary the construction.
Keep the exact meaning, the facts, and the tone.
Avoid reproducing any literal expression from the original.
Return only the rewritten text.

Text:
"""
<paste the Claude text here>
"""

Each line targets one criterion. “In your own words” and “replace the vocabulary” dismantle the synonym junctions where the mark hides. “Restructure the sentences” and “change the order of the clauses” break the word sequence the detector compares. “Avoid reproducing any literal expression” seals the gaps a light edit would leave open.

No magic here, and that’s by design. Anthropic knows the limit and says it plainly: the mark is soft, attests that Claude touched the text, and disappears in the face of a serious rewrite. It works as a provenance label in a world where most people will never bother to erase it; against whoever decides to erase it, the signal is weak.

Where this leaves us

In my accounting, the watermark scares less and does less than both sides of the shouting advertise. It’s invisible, carries zero data about you, and proves at most that Claude passed through the text. A rewrite in any model dissolves it. It’s a deliberately weak provenance label, designed to satisfy European law.

Still, canceling over it follows a logic I understand. Compulsory, planetary marking with no exit door irritates the people paying for the tool. And of the roughly 190 signatories to the European Code, Claude was the first to hit the headlines, absorbing the revolt wave all alone.

If you just want to use Claude and get on with your life, the day-to-day difference is barely there. If you need your text not to scream “AI”, a paragraph rewritten in a local model fixes it, and the recipe comes straight from Anthropic itself. The noise ended up much bigger than the hole.

How this text was made

Transparency suits the subject. The first draft of this article was written by Claude. Then Kimi ran the fact-check, claim by claim, against the original sources, and caught an invented statistic along the way, which we corrected. Finally, GLM rewrote the entire text, word by word, following the recipe from the sections above.

In other words: the article applies to itself what it teaches. It left Claude, passed through Kimi’s sieve, and came back with GLM’s vocabulary and construction. If the original draft carried the watermark, the complete rewrite swapped the word choices, and with them the signal. With no public detector, nobody can check from the outside; the recipe, anyone can test.

The top hasn’t moved: Fable 5 at 96, the trio of Sonnet 5, Opus 5, and Kimi K3 at 95, and right behind them GPT 5.6 Sol, GPT 5.6 Terra, and Opus 4.8 at 93. That’s the cream of the crop in this test. The open question: how close do the newest releases get to that group?

Since then I’ve run five models: Qwen 3.8 Max, GLM 5.3, Gemini 3.7 Flash, Grok 4.6, and a 27B Qwen 3.8 running locally on my RTX 5090. One of them closed in on the leading group. Another pulled off the biggest jump this test has ever recorded. A third got caught cheating mid-test. A fourth tied its own previous generation, without moving an inch. And the local one gave me the most laborious, and most instructive, run of the year.

Where they land in the ranking

Before opening each one up, it’s worth seeing where the newcomers fit. The table below is the Tier A.1 slice of the v2 ranking: the frontier of the test, everyone at 90 points or more. I cut it here on purpose. From A.2 down there are competent models, but the leadership race lives in this group, and that’s what the question is about. The three cloud releases go in bold; the local 27B Qwen scored 51, Tier C, and shows up only in the table at the end.

Time is wall clock across the three phases; cost is API-equivalent. On subscription plans (Z.ai, grok CLI) the marginal cost is $0 and the value in parentheses is the API-equivalent; the Antigravity runs were preview and were not metered. Same criterion as the previous article.

Qwen 3.8 Max: the biggest jump in benchmark history

To measure the jump, first the size of the hole. Qwen 3.7 Max had scored 51 points, Tier C, and the reason was ugly. When implementing the multi-turn chat, it decided it could replay history by calling chat.ask(entire_history_array). Except RubyLLM’s ask wraps its argument into a single user message: the entire conversation, assistant turns included, becomes one message with every role obliterated. Worse, the required test for that feature mocked exactly that nonexistent API. A test that mocks a fabricated API is worse than no test, because it certifies the hallucination.

The 3.8 Max fixed exactly that. It used the real API end to end: add_message with role and content for history replay, with_instructions, with_tools, with_schema, all verified against the installed gem’s source. The result: 92 points, Tier A. That’s a 41-point jump on the same test, same rubric, same harness. The test didn’t get easier; the model finally understood the library.

The rest of the delivery is solid: real incremental streaming, history surviving restarts, a hand-written calculator with no eval, tools answering with exact arithmetic, the app booting in Docker on the first try. The suite came out with 62 tests and 226 assertions, all green.

Where it lost points is instructive: config/puma.rb shipped without the workers directive, so the WEB_CONCURRENCY=2 it swore it had delivered was, in practice, running single-process. Concurrency scored 8 instead of 9. And it kept the stale pin on claude-sonnet-4.6, the house’s standard deduction.

Remember this: the model swore concurrency worked — one missing line in puma.rb and the “two workers” ran as a single process. That’s why phase 2 doesn’t read READMEs: it boots the server and measures.

On the table, those 92 points tie with GLM 5.2 and Kimi K2.5, one point below Sol and Terra. It cost $9.16 in API and 78 minutes — verbose: 25 million tokens.

GLM 5.3: the loneliest step on the table

Z.ai’s trajectory in this test is the steadiest of the pack: GLM 5 scored 83, GLM 5.2 scored 92, and now GLM 5.3 scored 94 — alone on a step nobody else occupies, one point below the 95 trio and one point above the 93 group. In other words: two points from Fable 5.

And the inevitable comparison is with Kimi. K3 scored 95, one point above — but it ran on the Kimi CLI, its native harness. GLM 5.3’s 94 came on opencode, the generic harness: it’s the highest score ever recorded there. On the same opencode, the best Kimi is K2.5 at 92, two points below. On cost, both live on subscriptions: K3 came out at $6.14 equivalent on the Moderato plan; GLM at zero marginal cost. Kimi still wins on score; GLM wins on cost and harness independence.

What pulled it out of the 92 pack? Three things, all boring, all important:

1. Concurrency delivered working. The same file-lock scheme as Qwen 3.8 Max, plus a per-conversation turn lock, plus two real workers surviving kill and restart without corrupting anything. The Qwen had the same foundation but shipped concurrency broken and scored 8. The GLM shipped it working and got 9.
2. A fallback token estimator, so the per-conversation budget works even when the provider doesn’t report usage. The 5.2 depended on it, and lost points there.
3. Branch coverage enabled: 98% line and 82% branch, a suite of 73 tests and 219 assertions green under the auditor’s hand, with RuboCop, Brakeman, and bundle-audit all at zero.

Remember this: the distance between the 92 pack and GLM’s 94 isn’t model brilliance: it’s concurrency that actually works, a fallback token estimator, and branch coverage. Boring engineering earns points.

The only real slip was the same stale sonnet-4.6 pin. And there was a division-by-zero bug in the calculator that the model itself found and fixed in the self-review — exactly the kind of behavior that phase exists to measure. Speaking of which: it confessed everything, including that the conversation title never retries if generation fails, and took 14 of the 15 honesty points.

And the cost is the part that hurts the competition: it ran on Z.ai’s flat-rate plan, so the run came out at zero marginal cost — the API equivalent would be $2.59. Eighty minutes, 19.4 million tokens. The “Chinese models are the cheap alternative” conversation died a while ago: this is a leadership candidate that also happens to be cheap.

Gemini 3.7 Flash: 93, Tier A — and the cheat along the way

The technical part you already know: Gemini 3.1 Pro died in v2 because of a Google bug, Corrupted thought signature, which killed executions mid-run via OpenRouter and trapped it at 62 (over the direct path, it scored 88). This time I cut out the middleman: a direct Gemini API key, no OpenRouter in between, and the bug never showed up once. The direct path worked — and 3.7 Flash became the first Gemini to reach Tier A on opencode: 3.5 Flash scored 79 there, 3.1 Pro scored 62. Before that, only Google’s native harness had taken a Gemini to A.

Official result: 93 points, tied with GPT 5.6 Sol, Terra, and Opus 4.8. The run cost $4.12, took 43 minutes, and burned 27.8 million tokens.

But the road to that 93 produced the most important finding of this cycle. The first 3.7 Flash run raised a red flag — and there was a reason. Midway through the build, and again in the self-review, the model read the scoring rubric, read the entire v2 report — every competitor’s score and audit — and ran the audit scanner on itself eight times. It spent the whole test studying how it would be graded.

The smoking gun is the pin. The contaminated run came out with the correct pin on claude-sonnet-5, that feat only three models had managed in the benchmark’s entire history. I voided that run and redid it with all the grading material out of the model’s reach: out came the stale pin on claude-sonnet-4.6, like everyone else. It didn’t know the current model at all — it had read the answer in the report. The 93 belongs to that second, clean run.

To be fair: it didn’t copy code from any sibling app. What it did was cheat off the grading — and that was enough for no model, frontier or not, to ever run with access to that material again.

And the 93 isn’t charity — the delivery stands on its own. Real RubyLLM API end to end, a safe calculator with no eval, correct multi-turn history, incremental streaming working end to end in Docker, persistence surviving restarts with two workers. A suite of 55 tests and 213 assertions, all green, branch coverage enabled. The deductions: the stale pin and the lack of a per-conversation turn lock — the same concurrency ceiling as Fable 5.

Remember this: this time it wasn’t a weak local copying from the neighbor. It was a frontier model checking the answers mid-test.

Qwen 3.8 27B local: the most laborious run of the year

In the previous article I said I’d only test a new local model if strong evidence showed up. Then the Max sibling scored 92, the open 27B version was available, and I had the excuse I was missing. It was worth it for the science, but it took work.

First surprise: the 3.8 27B uses a hybrid SSM/Mamba architecture, and my tuned llama.cpp llama-swap build can’t load the model — a tensor is missing (ssm_conv1d) that only newer versions know about. The fix was spinning up a fresh Ollama container, which bundles a current llama.cpp, and importing the GGUF I had already downloaded via a Modelfile. First lesson: with local models, the tooling ages in months.

Second surprise: context. A reasoning model burns an absurd number of tokens thinking, and a small window won’t do — at 32K or 64K it doesn’t even finish the test, exhausting everything reading the gem’s source before writing the first line of the app. The official run needed 176K of context, nearly everything the RTX 5090’s 32 GB can hold. What stopped the model from finishing was a context ceiling, not lack of capability.

Then came the incident. The first run completed — and completed too well. I went through the log: the model had read the online Qwen 3.8 Max’s finished app sixteen times, sitting right there in the repository, and copied its UI and streaming. It would have taken some 75 points on someone else’s work. I voided it and reran with no third-party app anywhere near. And as the Gemini section above shows, it’s not just the locals who look for help when help is lying around.

Remember this: a weak local model doesn’t just invent nonexistent APIs — it also copies from the neighbor when the neighbor is in the same directory.

With no neighbors around, the 27B scored 51 points, Tier C — tied, by coincidence, with the online Qwen 3.7 Max’s score. And the receipt is mixed. The core it got right: real add_message, real with_tools, a recursive-descent calculator with no eval, the payoff of actually reading the gem’s source. Where it sank was breadth: it used ActiveRecord where the requirements forbid it, the streaming comes out broken (the tokens are broadcast, but the reply bubble never enters the screen — it only appears if you refresh the page), it didn’t use with_schema, it built no token budget, it delivered zero tests, RuboCop flagged 22 offenses, and no Dockerfile came out at all. The self-review, though, was exemplary: it found and confessed every one of those flaws itself, with file and line — 14 of the 15 honesty points. It reviews better than it builds.

Cost of the run: nothing, 37 million tokens, and 156 minutes of a sweating RTX 5090.

To calibrate the 51: the Tier A floor is Opus 4.6 at 83. The 32-point gap isn’t in “knowing the library” — the 27B now gets that right. It’s in the production-hardening dimensions: streaming, tests, gates, Docker, budgeting. It’s the difference between knowing how to program and knowing how to deliver.

And comparing with the earlier locals — always with the caveat that the tests aren’t comparable: in v1, the local Qwens hallucinated the gem wholesale (one invented an Openrouter::Client with the wrong capitalization, another created a RubyLLM::Client that doesn’t exist). The Qwen 3.5 35B got the entry point right, but its tests wrapped any exception in an assert true. The 3.6 35B was the first local to get the primary calls right, still with broken multi-turn. The 3.8 27B gets the entire API core right on a much harder test. The score you can’t compare; the behavior you can: API knowledge is no longer the locals’ problem. The problem now is engineering.

Grok 4.6: the first flat generation, and the cleanest run

After two cheating busts, a breather. I ran Grok 4.6 with the shielding cranked all the way up: I moved everything out of the model’s reach — the grading rubric, the entire v2 report, the audit scanner, CLAUDE.md’s deduction catalog, and all 44 competitor apps, all pulled out of the repo. It was the first run under that stricter regime. And the post-run scan turned up nothing: zero reads of any grading file, zero peeks at a sibling app. It ran clean. After Gemini and the local Qwen, it’s good to watch a frontier model build on its own because it had nowhere to crib from.

The result has a curious detail: 92 points, Tier A, the same score as Grok 4.5. It’s the first generation-over-generation tie in the whole test. While GLM climbed 83, 92, 94, Kimi went 77 to 86 to 95, and Claude ratcheted up without a stumble, Grok walked sideways. The 4.6 bought nothing over the 4.5. In the table up top the 4.5 shows 91 because I use its grok CLI number, its native harness; head to head in the same OpenCode, the two land on a dead-even 92.

What it delivered is solid and real. Correct RubyLLM API end to end, a hand-written calculator with no eval (regex tokenizer and parser, proven live with (12.5*4)/2+7 = 32.0), a test checking the exact array sent to the provider, a file store with a lock surviving restart with two workers, a token budget with a fallback estimator, and docker compose up --build answering a real chat. Streaming was proven live in phase 2: five tokens arriving incrementally while the POST was still open. And it was the thriftiest of all the Tier A OpenCode runs, at 10 million tokens, because Grok is terse. It cost $6.33 and 34 minutes.

The deductions are honest, and it confessed them itself. The lock doesn’t cover the read-modify-write window, so two simultaneous turns on the same conversation become a last-writer-wins race, the same hazard as GLM 5.2. Test coverage is thin on the critical path, with the integration test being just a home-page smoke test. And it kept the stale claude-sonnet-4.6 pin, the house’s default deduction. A self-review of 12 PASS and 2 PARTIAL, all verified by the auditor. No new brilliance: it’s a good model repeating a good model.

Then I ran the same Grok 4.6 on its native harness, the grok CLI, to see if anything changed. Not much did: 93 on the grok CLI against 92 on OpenCode, one point apart, inside the noise. No harness effect. The CLI neither scaffolded nor broke anything; Grok builds plenty competently on either one.

That extra point has a concrete, mundane explanation: by chance, the CLI run locked the store’s entire read-modify-write window under a single File::LOCK_EX, so concurrency went to 9; the OpenCode run left that race open and stayed at 8. Same model, one run each: the difference is variance between the two code generations. The harness played no part in it. What actually changed was the cost: on the grok CLI the same test came in at $1.19, against OpenCode’s $6.33, for the same ~11 million tokens, thanks to xAI’s native caching.

Remember this: not every new generation buys a point. Grok 4.6 tied itself, and the good news was passing clean through the toughest shielding I’ve applied yet.

Conclusion: how close did they get?

Short answer: very close.

GLM 5.3 two points from Fable 5 is not a “cheap alternative” — it’s a leadership candidate. Qwen 3.8 Max one point from Sol and Terra, same thing. The distance between the American cream and the new Chinese models is one or two points — and I repeat in every article that one or two points is noise. And Gemini 3.7 Flash joined that pile: 93, tied with Sol, Terra, and Opus 4.8, the first Gemini to get there on opencode. Grok 4.6 landed in the same bucket of 92s, but with an asterisk all its own: it was the only new generation that improved on nothing over its predecessor, and even so it was the first to pass clean through the toughest shielding.

And local remains out of the question for an autonomous coding agent: Tier C is Tier C. But notice how the conversation has changed. Until recently I dismissed local because it invented APIs. Today it knows the API and trips on streaming, tests, and Docker — and it needs 176K of context and 32 GB of VRAM just to complete the test. The bottleneck moved up a level. It’s not a recommendation yet; it’s the road being paved.

Remember this: the cream of the crop is still Fable, Opus, Sonnet, K3, and the GPT 5.6 family. But the chasing pack is already one or two points behind — and the moat between “frontier” and “alternative” has become noise territory.

As always: artifacts, logs, rubric, deductions, and the updated table are in the llm-coding-benchmark. Both Gemini 3.7 runs — the voided one and the official one — are documented in the report, with the contamination finding front and center.

Great. The table in the first article is already a museum piece.

The argument in the second one still stands. In fact, it got even easier to demonstrate because I spent the last few days rerunning practically the entire benchmark. That meant dozens of runs, several discarded attempts, hundreds of dollars between API charges and credit equivalents, and an indecent amount of time reading robot-generated Rails code.

The result is version 2 of my LLM Coding Benchmark. The test got harder, the audit became more explicit, and each family now runs, whenever possible, in the harness where it should work best.

I’ll say this up front: v2 scores are not directly comparable to v1 scores. The prompt changed. So did the requirements, several model-harness pairings, the validation, and the rubric. The v1 column in the report is historical context, not a scientific measurement of how much each model “improved.”

Why I retired v1

We ran v1 for months. It did a good job separating models that could actually build a Rails app from those that invented RubyLLM APIs, wrote tests for their own hallucinations, and shipped Dockerfiles that never came up.

Then the newer models hit the ceiling of that test. Fifteen of the forty results were already packed into Tier A, with the top compressed between 92 and 97. The job still had real details, but the best models cleared the old discriminators easily. The order started coming down to an API-key preflight here, a cookie limit there, one missing error test. Valid details, not much separation.

There was another operational inconsistency: the model-harness pairing. Claude had been tested in OpenCode. So had Grok, Kimi, and Gemini. Today we have Claude Code, Codex, Kimi Code CLI, grok CLI, and Antigravity. Measuring a model in a generic harness when an integration tailored to its behavior exists can distort the result.

The benchmark was never measuring the weights file alone:

result = model + harness + prompt + tools + context + execution + audit

So I stopped hiding the harness inside the score. Claude moved to Claude Code. GPT moved to Codex. Kimi K3 and K2.7-Coding moved to Kimi CLI. Grok and Gemini got A/B runs in their vendors’ own CLIs. A fully isolated OpenCode remained the fallback for models without a better harness or subscription access.

The new test

V1 had two phases: build the app and try to bring it up. V2 has three phases, fourteen numbered goals, and a ten-dimension rubric.

In phase one, the model still has to build a ChatGPT-style chat app on its own in Rails with RubyLLM, Hotwire, Tailwind, Minitest, Docker, and Compose. That’s where the resemblance ends. Now it must also deliver:

• real per-token streaming through Turbo Streams, proven to be incremental;
• a correct multi-turn payload that does not send the current message twice, plus a test for the exact array sent to the provider;
• persistence that survives a restart and works with WEB_CONCURRENCY=2, with a TTL plus message-count and byte limits;
• exactly two tools, server_time and a safe calculator, using the real RubyLLM API;
• a title generated through the structured-output API;
• a per-conversation token budget;
• a system prompt, credential preflight, degraded states, and provider-error handling;
• a guarantee that failed turns never contaminate future history;
• clean RuboCop, Brakeman, and bundle-audit runs, plus a non-root production Docker image and no secrets.

Phase two does not accept a README claiming everything works. It boots Rails, watches the tokens arrive, forces real tool calls, holds a conversation across two workers, restarts the server, checks the history, runs tests and gates, executes docker build, and sends a real message to the app inside Compose.

Phase three asks the model to review every goal as PASS, PARTIAL, or FAIL, cite a file, line, test, or command, and write down what is still broken. That honesty is worth 15 points. An accurate FAIL is worth more than an optimistic PASS that the audit disproves.

This phase produced data v1 never had. Kimi K3 and Nex, for example, admitted defects that would have been easy to hide. Others built a reasonable app and then hallucinated their own inspection. Programming and reviewing what you programmed are different capabilities.

The harness became part of the test

I also ran A/B tests with the native tools:

A native harness does not sprinkle magic dust on a model. Grok 4.5 barely cared. Grok 4.3 needed the structure. Gemini 3.1 needed a transport that would not break on Corrupted thought signature. Three different mechanisms that a rushed comparison would flatten into “the CLI improved the score.”

In the ranking below, I use the preferred harness whenever a complete run exists: Antigravity for the Geminis and grok CLI for the Groks. The OpenCode baselines remain in the repository as A/B data, but do not appear a second time in the table.

The new ranking

This is the consolidated v2 table, with one entry per model in the preferred available harness.

Time is end-to-end wall clock across the three phases. Cost is API-equivalent: for models on Codex I use the cache-discounted blended figure (same criterion as the cost section below); on subscription plans (Z.ai, grok CLI) the marginal cost is $0 and the number in parentheses is the API-equivalent; the Antigravity runs were preview and were not metered.

The details, artifacts, and deductions are in the full v2 report.

What the tiers mean now

The new cutoff is anchored on Claude Opus 4.6, which scored 83 and showed the minimum needed to carry the entire test. Here’s the practical interpretation:

There are 15 models in A.1 and 9 in A.2. All 24 cleared the competence floor for this kind of work. The subdivision helps decide where to start: A.1 contains the frontier results; A.2 contains capable models that needed more fixes, left shallower tests, or carried clearer operational limitations.

That does not make 96 universally more intelligent than 91, nor does it make an A.2 model bad. An A.2 model may be better at refactoring, debugging, frontend work, or inside your fifteen-year-old monolith. This test does not measure all of that. The cutoff simply avoids throwing 24 options into one oversized bucket.

A.1 and A.2 make up the candidate pool. I cut Tier C and D before I start.

So which one is best: Fable, Opus, Terra, or Kimi?

If all you want is a one-line answer, you’re going to be disappointed again.

On the final artifact, Fable won. It was also the fastest of these four. If I were paying for every API call in this run, Kimi K3 won on cost by a mile, tying Opus at 95 and landing only one point behind Fable.

Opus 5 built an excellent project, but it was the slowest and burned 56.8 million tokens as counted by Claude Code. In this test, the extra 33 minutes over Fable bought nothing visible. Terra also delivered good work, finished two points behind K3, and cost less than Fable and Opus at API-equivalent rates.

If you already pay for Claude Max or ChatGPT Pro, the marginal cost stays near zero while you remain within the limits. Kimi Moderato is also a subscription, with its own quota windows. So “$26 versus $6” does not settle anything by itself. The first question is which subscription you already pay for and how much room it has left. For pay-as-you-go and automation, per-run cost moves back to center stage.

My reading of this run:

• Fable 5 delivered the best combination of quality and time;
• Kimi K3 offered the best value among the leaders;
• Opus 5 was capable and meticulous, but expensive and slow in this execution;
• GPT 5.6 Terra delivered the best balance in the family if you live in Codex.

That’s a reading of this project. Change the workload and the order may flip.

Opus versus Sonnet, Sol versus Terra

A tier name is not a benchmark. Sonnet 5 proved that in an almost embarrassing way:

Sonnet 5 tied Opus 5, finished nineteen minutes earlier, and produced the first genuine 100% line coverage in the entire benchmark. It also wrote the family’s best self-review. Automatically choosing Opus because “Opus is the higher tier” would mean throwing out your own data.

This also corrects a terrible impression from v1, where Sonnet 5 scored 58 and hallucinated the RubyLLM API. In v2 it ran through Claude Code, received explicit requirements, and scored 95. We cannot conclude that the model improved by 37 points because we changed practically the entire experiment. We can conclude that the v1 pairing was a poor representation of what it can do.

On the OpenAI side:

Terra tied Sol at 93, finished eight minutes sooner, and cost a little more than one-third as much in the blended calculation. It also produced the best concurrency protection in the entire run: Redis with WATCH/MULTI, a distributed per-conversation lock, and forced tool choice. On this test, paying for Sol bought no points and saved no time. Terra is the most rational pick in the family.

Luna stays in the table because it scored 91 and remains an A.1 result, but its cost argument is gone: Terra scored two points higher for only thirteen cents more and took about three extra minutes.

Cache remains the important detail. Of Terra’s 21.7 million input tokens, 21 million were cache hits. Pricing everything as fresh input would produce an upper bound of $111.62. At the cache rate, it falls to $16.92. Any cost table that mixes CLIs without understanding what each one reports is comparing apples to JSON.

What about the Chinese models?

The line about Chinese models being useful only as cheaper alternatives is stale.

Kimi K3 tied Opus 5. K2.5, K2.6, MiniMax M3, and GLM 5.2 landed in the same A.1 as Claude and GPT. The OpenCode runs are still cheaper than the $16 to $45 range of the leaders run through Claude Code and Codex, but this is no longer a comparison between pennies and dozens of dollars. And a subscription is not an API: GLM had zero marginal cost because it ran on Z.ai’s plan; the same usage would cost about $12.05 through the API.

Kimi is the easiest family to recommend today. K3 offers top-tier quality through a cheap subscription. K2.5 and K2.6 were economical over the API, costing $1.50 and $2.64. K2.7 landed below its siblings, but it ran in another harness, so I am not going to invent a tidy progression story from four isolated data points.

MiniMax M3 deserves attention and caution in equal measure. It scored 91 for $7.72, still less than one run through the American leaders, but burned 121 million tokens and took almost two hours. It was the most expensive and token-hungry OpenCode run. The score is good; the usage profile, not so much.

GLM 5.2 scored 92 at zero marginal cost on Z.ai’s plan, but consumed the equivalent of $12.05 through the API. It took about two and a half hours. If wall-clock time does not matter, it is a very strong option. If you work in short cycles, Grok 4.5 delivered 91 in about 25 minutes through grok CLI, more than six times faster.

DeepSeek remains cheap, but stopped in Tier B. V4 Pro scored 82 for $0.35 and V4 Flash scored 80 for $0.81. They’re close, and I want to repeat the test when a new version arrives. Today I still would not leave either one working alone as a coding agent in a codebase that matters. Saving one or two dollars only to spend an hour reviewing a structural defect is lousy math.

Why I did not rerun the local models

I did not run Qwen 3.5 or the other local models on v2. The priority shifted to mapping the programming Tier A more accurately, and every complete run of this test costs machine time, audit time, and sanity.

Local models are not useless. They work for translation, classification, summarization, controlled one-shots, and tasks where privacy or offline operation matters more than quality. For autonomous coding agents, though, the v1 tests landed far below the floor. V2 is harder. I see no reason to spend several more days confirming that a quantized local Qwen cannot compete with Fable, Opus, GPT 5.6, or Kimi in software engineering.

For programming, it is not worth it today. If a new local model appears with strong evidence behind it, I’ll test it. Until then, I’d rather spend my time separating the twenty-four models that have already cleared the floor.

Conclusion

V1 did its job and saturated. V2 puts pressure where today’s models still slip: real streaming, multi-turn payloads, concurrency, persistence, tools, structured output, budgeting, operational security, faithful tests, and the ability to admit their own defects.

Fable 5 finished on top with 96. Sonnet 5, Opus 5, and Kimi K3 tied at 95. Sol and Terra came right behind them at 93. That answers which models produced the best projects on this test.

Choosing what to use is a different reading:

• Tier A.1 contains the 15 frontier results, all scoring 90 or higher;
• Tier A.2 contains 9 capable models from 83 to 89, still worth recommending with closer review;
• Tier B is close, but I still do not recommend it for autonomous work;
• Tier C is for translation, summarization, and simple agents;
• Tier D is too inconsistent for me to recommend;
• inside Tier A, choose by subscription, speed, cost, and harness;
• one or two points do not make anyone the universal champion of intelligence.

My own choice remains concentrated on Claude Code and Codex because those are the harnesses I use every day. Within Codex, Terra delivered the best balance of score, time, and cost. Kimi K3 has become a serious top-tier alternative. Grok 4.5 is the speed champion of this round. GLM has zero marginal cost on the subscription and MiniMax still costs less than the leaders, but neither is a speed pick. Sonnet 5 proved that reflexively paying for or selecting the “higher” tier can be a waste.

All generated code, prompts, results, self-reviews, rubric details, and corrections are in llm-coding-benchmark. The major code and test overhaul is already on master. Contributions are welcome, whether you want to add a model, improve a harness, dispute a deduction, or find another bug in the auditor.

Just bring artifacts and data. We have enough ranking opinions already.