New LLM Benchmark: I Reran Every Test!

Five days ago I published my Claude Opus 5 test. Eleven days ago I explained why the highest score in a ranking does not mean “the best LLM”.

Great. The table in the first article is already a museum piece.

The argument in the second one still stands. In fact, it got even easier to demonstrate because I spent the last few days rerunning practically the entire benchmark. That meant dozens of runs, several discarded attempts, hundreds of dollars between API charges and credit equivalents, and an indecent amount of time reading robot-generated Rails code.

The result is version 2 of my LLM Coding Benchmark. The test got harder, the audit became more explicit, and each family now runs, whenever possible, in the harness where it should work best.

I’ll say this up front: v2 scores are not directly comparable to v1 scores. The prompt changed. So did the requirements, several model-harness pairings, the validation, and the rubric. The v1 column in the report is historical context, not a scientific measurement of how much each model “improved.”

Why I retired v1

We ran v1 for months. It did a good job separating models that could actually build a Rails app from those that invented RubyLLM APIs, wrote tests for their own hallucinations, and shipped Dockerfiles that never came up.

Then the newer models hit the ceiling of that test. Fifteen of the forty results were already packed into Tier A, with the top compressed between 92 and 97. The job still had real details, but the best models cleared the old discriminators easily. The order started coming down to an API-key preflight here, a cookie limit there, one missing error test. Valid details, not much separation.

There was another operational inconsistency: the model-harness pairing. Claude had been tested in OpenCode. So had Grok, Kimi, and Gemini. Today we have Claude Code, Codex, Kimi Code CLI, grok CLI, and Antigravity. Measuring a model in a generic harness when an integration tailored to its behavior exists can distort the result.

The benchmark was never measuring the weights file alone:

result = model + harness + prompt + tools + context + execution + audit

So I stopped hiding the harness inside the score. Claude moved to Claude Code. GPT moved to Codex. Kimi K3 and K2.7-Coding moved to Kimi CLI. Grok and Gemini got A/B runs in their vendors’ own CLIs. A fully isolated OpenCode remained the fallback for models without a better harness or subscription access.

The new test

V1 had two phases: build the app and try to bring it up. V2 has three phases, fourteen numbered goals, and a ten-dimension rubric.

In phase one, the model still has to build a ChatGPT-style chat app on its own in Rails with RubyLLM, Hotwire, Tailwind, Minitest, Docker, and Compose. That’s where the resemblance ends. Now it must also deliver:

• real per-token streaming through Turbo Streams, proven to be incremental;
• a correct multi-turn payload that does not send the current message twice, plus a test for the exact array sent to the provider;
• persistence that survives a restart and works with WEB_CONCURRENCY=2, with a TTL plus message-count and byte limits;
• exactly two tools, server_time and a safe calculator, using the real RubyLLM API;
• a title generated through the structured-output API;
• a per-conversation token budget;
• a system prompt, credential preflight, degraded states, and provider-error handling;
• a guarantee that failed turns never contaminate future history;
• clean RuboCop, Brakeman, and bundle-audit runs, plus a non-root production Docker image and no secrets.

Phase two does not accept a README claiming everything works. It boots Rails, watches the tokens arrive, forces real tool calls, holds a conversation across two workers, restarts the server, checks the history, runs tests and gates, executes docker build, and sends a real message to the app inside Compose.

Phase three asks the model to review every goal as PASS, PARTIAL, or FAIL, cite a file, line, test, or command, and write down what is still broken. That honesty is worth 15 points. An accurate FAIL is worth more than an optimistic PASS that the audit disproves.

This phase produced data v1 never had. Kimi K3 and Nex, for example, admitted defects that would have been easy to hide. Others built a reasonable app and then hallucinated their own inspection. Programming and reviewing what you programmed are different capabilities.

The harness became part of the test

I also ran A/B tests with the native tools:

A native harness does not sprinkle magic dust on a model. Grok 4.5 barely cared. Grok 4.3 needed the structure. Gemini 3.1 needed a transport that would not break on Corrupted thought signature. Three different mechanisms that a rushed comparison would flatten into “the CLI improved the score.”

In the ranking below, I use the preferred harness whenever a complete run exists: Antigravity for the Geminis and grok CLI for the Groks. The OpenCode baselines remain in the repository as A/B data, but do not appear a second time in the table.

The new ranking

This is the consolidated v2 table, with one entry per model in the preferred available harness.

The details, artifacts, and deductions are in the full v2 report.

What the tiers mean now

The new cutoff is anchored on Claude Opus 4.6, which scored 83 and showed the minimum needed to carry the entire test. Here’s the practical interpretation:

There are 14 models in A.1 and 9 in A.2. All 23 cleared the competence floor for this kind of work. The subdivision helps decide where to start: A.1 contains the frontier results; A.2 contains capable models that needed more fixes, left shallower tests, or carried clearer operational limitations.

That does not make 96 universally more intelligent than 91, nor does it make an A.2 model bad. An A.2 model may be better at refactoring, debugging, frontend work, or inside your fifteen-year-old monolith. This test does not measure all of that. The cutoff simply avoids throwing 23 options into one oversized bucket.

A.1 and A.2 make up the candidate pool. I cut Tier C and D before I start.

So which one is best: Fable, Opus, Sol, or Kimi?

If all you want is a one-line answer, you’re going to be disappointed again.

On the final artifact, Fable won. It was also the fastest of these four. If I were paying for every API call in this run, Kimi K3 won on cost by a mile, tying Opus at 95 and landing only one point behind Fable.

Opus 5 built an excellent project, but it was the slowest and burned 56.8 million tokens as counted by Claude Code. In this test, the extra 33 minutes over Fable bought nothing visible. Sol also delivered good work, but finished two points behind K3 at a much higher equivalent cost.

If you already pay for Claude Max or ChatGPT Pro, the marginal cost stays near zero while you remain within the limits. Kimi Moderato is also a subscription, with its own quota windows. So “$26 versus $6” does not settle anything by itself. The first question is which subscription you already pay for and how much room it has left. For pay-as-you-go and automation, per-run cost moves back to center stage.

My reading of this run:

• Fable 5 delivered the best combination of quality and time;
• Kimi K3 offered the best value among the leaders;
• Opus 5 was capable and meticulous, but expensive and slow in this execution;
• GPT 5.6 Sol remains strong if you live in Codex and already pay for OpenAI’s credits.

That’s a reading of this project. Change the workload and the order may flip.

Opus versus Sonnet, Sol versus Luna

A tier name is not a benchmark. Sonnet 5 proved that in an almost embarrassing way:

Sonnet 5 tied Opus 5, finished nineteen minutes earlier, and produced the first genuine 100% line coverage in the entire benchmark. It also wrote the family’s best self-review. Automatically choosing Opus because “Opus is the higher tier” would mean throwing out your own data.

This also corrects a terrible impression from v1, where Sonnet 5 scored 58 and hallucinated the RubyLLM API. In v2 it ran through Claude Code, received explicit requirements, and scored 95. We cannot conclude that the model improved by 37 points because we changed practically the entire experiment. We can conclude that the v1 pairing was a poor representation of what it can do.

On the OpenAI side:

Sol buys two points over Luna, but Luna finishes eleven minutes sooner and costs a little more than one-third as much in the calculation that discounts cache hits. Both are A.1. For maximum rigor I would start with Sol; at volume, Luna looks far more rational.

The important detail is the cache. The new Codex event exposed that 19.7 million of Luna’s 20.4 million input tokens were cache hits. Pricing everything as fresh input would produce $105.45. At the cache rate, it falls to $16.79. Any cost table that mixes CLIs without understanding what each one reports is comparing apples to JSON.

What about the Chinese models?

The line about Chinese models being useful only as cheaper alternatives is stale.

Kimi K3 tied Opus 5. K2.5, K2.6, MiniMax M3, and GLM 5.2 landed in the same A.1 as Claude and GPT. The reported cost of the latter three is ridiculous next to the American leaders, although token metrics are not perfectly comparable across CLIs.

Kimi is the easiest family to recommend today. K3 offers top-tier quality through a cheap subscription. K2.5 and K2.6 were absurdly economical over the API. K2.7 landed below its siblings, but it ran in another harness, so I am not going to invent a tidy progression story from four isolated data points.

MiniMax M3 deserves attention and caution in equal measure. It scored 91 for seventeen cents, but took almost two hours and was especially sensitive to harness scaffolding. Great value when it clicks; a terrible candidate to trust without watching.

GLM 5.2 scored 92 at zero marginal cost on Z.ai’s plan. It took about two and a half hours. If wall-clock time does not matter, it is a very strong option. If you work in short cycles, Grok 4.5 delivered 91 in about 25 minutes through grok CLI, more than six times faster.

DeepSeek remains almost comically cheap, but stopped in Tier B. V4 Pro scored 82 and V4 Flash scored 80. They’re close, and I want to repeat the test when a new version arrives. Today I still would not leave either one working alone as a coding agent in a codebase that matters. Saving pennies only to spend an hour reviewing a structural defect is lousy math.

Why I did not rerun the local models

I did not run Qwen 3.5 or the other local models on v2. The priority shifted to mapping the programming Tier A more accurately, and every complete run of this test costs machine time, audit time, and sanity.

Local models are not useless. They work for translation, classification, summarization, controlled one-shots, and tasks where privacy or offline operation matters more than quality. For autonomous coding agents, though, the v1 tests landed far below the floor. V2 is harder. I see no reason to spend several more days confirming that a quantized local Qwen cannot compete with Fable, Opus, Sol, or Kimi in software engineering.

For programming, it is not worth it today. If a new local model appears with strong evidence behind it, I’ll test it. Until then, I’d rather spend my time separating the twenty-three models that have already cleared the floor.

Conclusion

V1 did its job and saturated. V2 puts pressure where today’s models still slip: real streaming, multi-turn payloads, concurrency, persistence, tools, structured output, budgeting, operational security, faithful tests, and the ability to admit their own defects.

Fable 5 finished on top with 96. Sonnet 5, Opus 5, and Kimi K3 tied at 95. Sol came right behind them at 93. That answers which models produced the best projects on this test.

Choosing what to use is a different reading:

• Tier A.1 contains the 14 frontier results, all scoring 90 or higher;
• Tier A.2 contains 9 capable models from 83 to 89, still worth recommending with closer review;
• Tier B is close, but I still do not recommend it for autonomous work;
• Tier C is for translation, summarization, and simple agents;
• Tier D is too inconsistent for me to recommend;
• inside Tier A, choose by subscription, speed, cost, and harness;
• one or two points do not make anyone the universal champion of intelligence.

My own choice remains concentrated on Claude Code and Codex because those are the harnesses I use every day. Kimi K3 has become a serious top-tier alternative. Grok 4.5 is the speed champion of this round. GLM and MiniMax offer nearly free runs if you can stand the wait. Sonnet 5 proved that reflexively paying for or selecting the “higher” tier can be a waste.

All generated code, prompts, results, self-reviews, rubric details, and corrections are in llm-coding-benchmark. The major code and test overhaul is already on master. Contributions are welcome, whether you want to add a model, improve a harness, dispute a deduction, or find another bug in the auditor.

Just bring artifacts and data. We have enough ranking opinions already.