The book consolidates almost everything I wrote about AI on this blog from February to October 2026: all the experiments, the Frank projects, the benchmarks, billions of tokens spent and hundreds of hours of work over eight months, reorganized into a format that’s easier to read than a pile of dated posts. It came out in Portuguese and in English.

Important notice: this is a preview. The book is still in progress and new versions will come out until I consider it done. If you find anything wrong, from a typo to a fact, open an issue on GitHub. The trail I describe below exists precisely so that the fix reaches the right paragraph.

This text is about the technical backstage. How you turn 131 blog posts, written over months, with opinions that changed along the way, into a book with a beginning, middle, end, appendices, glossary and bibliography. And how you do that with agents without losing control of what the book says.

What’s inside

Five parts, each answering a question people keep asking me:

1. What is this thing, anyway. Is it intelligent? Is AGI coming?
2. The marathon. Does it actually work, or is it all demo?
3. Numbers and propaganda. Which AI is best? Can you believe the announcements?
4. Who keeps the keys. Is it dangerous to let it touch my things?
5. And my job? Will it replace me? What comes next?

That’s 20 chapters and two interludes, plus a preface, a “how to read”, a “before you start” saying what the book doesn’t explain and where to learn it, an epilogue and an afterword. Then come ten appendices (the machine inside without math, further reading, how to study with AI at your side, three bubbles, a timeline, a minimal kit for using agents safely, ai-memory, how to read a benchmark or an announcement, the year’s projects, and the backstage for programmers), the sources, the image credits, a glossary of almost 150 terms and more than 300 references in ABNT style. The print edition passes 500 pages at 6 by 9 inches; the screen edition, around 475. There are more than 50 figures, all with a provenance record.

The audience is different from the blog’s. The book was written for people who are anxious or confused about AI: the professional afraid of being replaced, the parent who read that AGI is two years away, the manager being sold an “AI transformation”. Most of them don’t program. That changes the voice, and I’ll come back to it.

The problem: half a million words that disagree with each other

On October 12 I published the year’s synthesis, which became chapter 20. On the 13th I started the book. I had 131 posts from 2026 on the blog, almost half a million words, from model benchmarks to a bitcoin wallet.

Reorganizing that by hand would take months. And there’s a worse problem than volume: I changed my mind during the year. I redid the benchmark several times, corrected costs, switched tools, and a February post contradicts a September one with complete naturalness. A book that simply pasted the posts in order would be a book that contradicts itself.

So I did what the whole book describes: I handed the grunt work to coding agents and kept the part of deciding, building the process and checking. The process is the subject of this text.

Step 1: a sealed inventory

The first thing was a script that reads the 131 posts and records, for each one, the SHA-256 hash of the Portuguese file and of the English file, plus the exact commit of the blog repository everything was read from. If a comma changes in a post, the hash changes with it. That file is machine-generated and never edited by hand.

Then an agent read each post and filled out a fixed form, with a schema defined in YAML: the thesis in one or two sentences, the claims a book would need to verify, which of them age fast, which lay reader’s fear the text answers, what to cut. Each analysis carries the hash of the post it came from.

Of the 131 texts, 96 had AI as the central subject or with relevant weight. The others only came in when they taught something specific, and every AI post left out has its reason recorded in the structure file. A script fails the structure if any AI post is left without a destination.

Lineage: who supersedes whom

To solve the contradiction problem, there’s a lineage map. It says which post superseded which, what can still be salvaged from the older one, and which corrections coming from the reference projects (the benchmark repository, ai-memory’s, ai-jail’s, my home server notes) apply to each text.

The rule for chapters is to tell the most recent state and, when the change teaches something, tell the change. “In June I thought X; in July the numbers said Y” is a story. Printing the old number as if it were current is an error, and the checker treats it as an error.

Step 2: the structure is data, and it generates the configuration

From the analyses came a proposed structure: five parts, each answering a reader’s fear, twenty chapters, two interludes. I read it, changed it and approved it. Some pieces came in at my request, like the “before you start” and the appendix on the machine inside.

That structure lives in a single YAML file, and it’s the source of truth. Each chapter has its id, title, opening scene, the question it answers, the arc, the list of source posts and the figure ideas. The Quarto configuration, which assembles the book, is generated from that file by script. Nobody edits the configuration by hand.

This matters because structural change is where a book breaks. Midway I moved the cut-off: the January posts went out, and the story settled between February and October. Ten posts vanished from the base.

Three chapters needed a new scene, a dozen units needed small repairs, and the damage that’s hard to see was elsewhere: a sentence in an untouched chapter saying “the five reviewers from chapter 5”, the timeline, the projects appendix, chart labels, every translation. That’s why there’s an impact script that, before any edit, lists every paragraph, figure, reference and translation that depends on a post.

Step 3: a pilot chapter becomes a contract

Before scaling, a pilot chapter. I picked chapter 5, about the first ten percent of a project, and it served to calibrate the voice and discover the doubts an agent would have along the way. Each doubt resolved became a written rule.

Those rules went into a contract document every agent reads before writing, along with three things:

• the book’s voice guide, which inherits the blog’s rules (my WRITER.md, the humanizer and my finishing checklist) and only adds what changes when the text leaves a dated post and goes into a book a lay reader holds in their hands;
• a continuity file, saying which chapter explains each concept and what each character is called, so nobody explains “context window” five times;
• the entire structure, so each agent knows what the other chapters cover.

Besides the voice rules that already existed (a chapter opens on a scene with a date, a place and something at stake, never on a definition; it closes on a hook toward the next one, never on “in this chapter we saw”), some of the rules that came out of the pilot, to give you a taste:

• when the chapter brief and the post disagree on a detail, the post wins;
• the chapter that owns a concept explains it properly; if the term shows up earlier in reading order, it gets a half-sentence gloss and a pointer, never a second explanation;
• a fact verified online gets a citation even if the original post had no link;
• model and product names in the vendor’s spelling, even when the blog abbreviated them.

It’s the project instructions file I describe in the chapter about how to talk to the machine, at the scale of a book. Anyone who has written an AGENTS.md for a codebase recognizes all of it.

Step 4: parallel drafts, contradictions expected

With the contract ready, the chapters were drafted in parallel, one agent per chapter, in two waves: parts I and II first, then III through V, so the later chapters could build on the earlier ones. Each agent received its chapter’s brief, the analyses, the full posts, the lineage and the reference-project files it needed.

Agents working at the same time contradict each other, and that happened. One chapter said FrankMD had around 130 commits in three days; another, that it had 212 in 19 days. Both numbers came from the posts and were right, they were just different windows. An integration ledger collected those conflicts, and the rule became making the window explicit in each chapter.

After the drafts comes the integration pass, with another set of agents, each owning a group of chapters and forbidden from touching the others. Their job: apply the conflict ledger, cut concepts explained twice, check that every callback matches what the other chapter actually says, make sure the last paragraph of a chapter leads into the first of the next, and redo the fact-check assuming the draft is wrong.

To read the seams, the build generates a “seams” file: the final paragraph of each unit next to the opening paragraph of the next. It’s the fastest read there is for finding where the book stops feeling like one book.

Step 5: an invisible marker per paragraph

This is the part I find most useful in the whole project. Every paragraph of the book, in the source file, carries a comment saying where it came from:

<!-- src: 2026/09/28/llms-sao-boas-em-reproduzir-o-que-existe-e-coisas-novas -->
Every LLM was trained only on what is public...

<!-- src: original -->
New text, written for the book.

<!-- src: ref:ai-jail/README.md -->
Fact checked in the reference project.

The marker disappears when the book is built, through a Pandoc Lua filter, so the reader never sees it. It exists for maintenance. I edit posts after publishing all the time, and the book has to follow.

The rake drift command redoes the inventory, compares the hashes, and lists the chapters that depend on a post that changed, with the post’s diff since the commit at which the book last read it. The marker leads straight to the paragraphs that came from that post, and only those need review. Once resolved, a script records the new hash and the commit.

A real example, from today. I removed from the reverse engineering post a paragraph about a Linus Tech Tips video. drift flagged chapter 19 and one appendix; the diff showed that passage had never been used in the book; two commands later, both were stamped with the new hash. Without the marker, I’d have had to reread two chapters to discover I didn’t need to change anything.

The bibliography plays the same role for external sources: each reference records which post cited it. And each image has its own provenance record.

Step 6: what the fact-check caught

Every number, date and name was checked against three oracles: the posts, the reference projects and primary sources online, with the same adversarial checker prompt I use on the blog’s articles. The full list is in the book’s afterword, and a good part of the corrections are to my own blog:

• in the March review of the Strix Halo mini-PC, I had set aside 96 GB of memory for the GPU; in April I lowered it to 64, and later posts still said 96. The book uses 64 for everything after April;
• a counting bug in one of the benchmark’s tools undercounted by 40 to 90 times the cost of runs made on it before August. The book cites none of those costs;
• in the June text about Claude Fable 5, I wrote it cost ten times more than Opus 4.8 per run. Redoing the math in July, with cache-read tokens included, Fable cost almost double. The post still says ten times; the book tells the wrong number and the correction;
• in a March text, I wrote that Anthropic had bought Bun at the end of 2024. The announcement is from December 2025;
• a chapter summary, written by an agent, said Claude Code’s subscription lock fell in less than a day after the leak. The announcement came about 30 hours later, and the book says “the next day”.

The drafts came out fast. Confidence in the text came from the checking. And the source beats the summary, even when the summary comes from the machine.

Step 7: a chapter with an error doesn’t pass

On top of everything runs an automatic checker, the equivalent of continuous integration. They’re Ruby scripts, standard library only, chained in a Rakefile, plus what they call externally: the chart checker is Python because it lives inside matplotlib, the page audit uses poppler, and epubcheck needs Java:

• a chapter linter fails em dashes, phrases banned by the voice guide, loose URLs in the middle of prose, unknown citation keys, paragraphs without a provenance marker, and warns about overlong paragraphs;
• a doctor checks that every cross-reference, citation, glossary term and figure exists, that provenance matches the inventory, that no post before the cut-off leaked in, that translations are up to date, and runs a callback heuristic: a sentence pointing at another chapter using names that chapter no longer contains;
• a page audit opens the finished PDF and looks for blocks split across pages, headings orphaned at the foot of a page, nearly empty pages, oversized images, lines overflowing the margin;
• a chart checker looks for labels that spilled out of their boxes;
• epubcheck validates the EPUB 3;
• a style report for copyediting: passages repeated across chapters (an inevitable consequence of posts that repeat the same thesis from different angles), recurring phrases, overused words, term variants, badly formed comparatives in Portuguese.

And the scripts have tests. A suite of almost 170 tests builds a fake book in a temporary directory, with five fake posts in a fake git repository, and checks that each tool catches the problem it should catch. When a bug is fixed, the test that would have caught it goes in, with a comment naming the regression it guards. It runs in CI before every build.

The assembly pipeline

The toolchain is Quarto, which bundles Pandoc and Typst, version-pinned inside the repository. No LaTeX. A single rake build produces, from the same tree of Markdown files:

Portuguese and English are Quarto profiles, generated from the same structure file. Citations come from a YAML in CSL format shared by both editions, with an ABNT style adjusted for Portuguese. The glossary is a Lua filter: each term is marked in the source where it appears, and the filter links the mark to the entry, expands the acronym the first time and generates the page.

Typst solved something I thought would cost me weeks: a PDF that looks like a typeset book, Portuguese hyphenation, ligatures, without the pain of TeX. The page audit after the build catches what the engine let through, and a rake fit recompiles while tightening chapters that end on a nearly empty page.

The figures

Every figure is code or has a record. Charts and box-and-arrow diagrams are matplotlib scripts, 43 of them, each with a note of the chapter the data came from, and with the labels in Portuguese and English inside the script itself: the same chart comes out in both languages. The path for D2 diagrams exists in the repository, and no figure in the book needed it.

The conceptual illustrations were generated by AI, with the model, the prompt and the date of each one recorded in its credit record, and the book discloses it. The rule I imposed on myself: no AI-generated image of a real person. Real people only appear in freely licensed photos, with the credit in the caption.

The cover is the retro-futurist robot with the weary face, kneeling at a 1950s World’s Fair, studying its own hand while the crowd watches. The prompt to the model described the scene to hold three fears at once: the end of humanity, losing your job, the machine becoming conscious. The cover’s record keeps the whole prompt. The face is Marvin’s, my newsletter’s character, and a sticker with my face sits beside the name.

The tricks of turning posts into a book

Turning a blog into a book is a list of mechanical conversions, and each one has a catch:

• A link becomes a citation. Every external link became a bibliography entry, and the sentence has to stand without the link. A link to another post of mine became a cross-reference to the chapter that now covers it, or disappeared. Never a URL in the prose.
• “Today”, “yesterday”, “last week”, “update:” became absolute dates. The book will be read in 2028 too.
• Model versions, prices, benchmark scores stayed as dated snapshots (“in September 2026, the best…”), and when a later post superseded the number, the book tells the change instead of printing the old number.
• Code dumps, install steps, terminal screenshots were cut. At most a 3-to-8-line excerpt when the code is the point; the step-by-step went to an appendix.
• Post titles, which are made for search (“Vibe Code: I built X in two days”), became short literary titles, with the subject in the subtitle.
• Repetition across posts, the same thesis from many angles, is said once, in the right place, and later chapters build on it.
• Calls to action, “subscribe”, “comment”, cut.
• The sharp edge aims at the propaganda, never at the reader. The post “You’re an idiot if you believe…” became a chapter about how the ads work. The reader who believed the ad is the person the book wants to help.
• Every technical term is explained once, at its first appearance in the book, and goes to the glossary.

And the most important of all: the book reads like a first-person story, with the year as its spine. Information arrives when the narrative needs it, instead of following encyclopedia order. That’s the opposite of how a blog is written, and it was the rule that gave the agents the most trouble.

The English edition

The English edition, Competent Enough, only started after I approved the Portuguese one. Each translated unit records the hash of the Portuguese file it came from, and wherever the book follows a post of mine that already had an English version on the blog, the translation uses my own English wording instead of translating again.

drift also works per language: if I change a paragraph in Portuguese, it flags the English unit that went stale, and only the marked paragraphs are retranslated. Charts with text are regenerated in English from the same scripts, and the English cover is composed by script on top of the same art.

Releasing is pushing a tag

A release is a CHANGELOG entry and a git push of a vX.Y.Z tag. A GitHub Actions workflow runs the tests, builds both editions in every format, validates the EPUB, runs the audit and publishes the release with the files. It deletes every release except the two newest, and keeps the tags. That’s how the official page always points at the current version.

The repository is public, and that brought its own catch: my home server notes are the source of some facts, and they’re private. The provenance markers cite those files by alias, the alias map stays out of git, and a pre-push hook blocks any file or commit message containing a private repository name, host or IP.

Copyediting: what the checking machine lets through

Lint, doctor, page audit and tests catch structural problems. They let through what a book written by many agents, each seeing a few chapters, always suffers from: the same sentence or anecdote reused in three chapters, drifting tone, inconsistent terminology, agreement slips. For that there’s a copyedit procedure, which runs once per version, in three steps from the cheapest to the most expensive.

Step 1, mechanical and nearly free: the style report I mentioned above. Sequences of seven words or more that appear in more than one unit, meaning reused sentences and retold anecdotes; set phrases of four to six words used in many chapters, which are nice as callbacks and become tics when they repeat; a word used far more in one chapter than in the rest; term variants (“e-mail” and “email”, “Open AI” and “OpenAI”); and an agreement heuristic for Portuguese. The report is a worklist, and nothing changes on its own.

Step 2, the straight-through read: one read-only agent per part of the book, each reading its units in order, start to finish, with a shared style sheet (names, spellings, terms, recurring phrases from step 1). Each produces a ranked report, editing nothing: typos and grammar first, then inconsistencies with the style sheet, repetition the reader would notice, passages a lay reader would stumble on, and pacing only in the worst cases. A script validates the reports, checks that every quoted passage really exists in the unit, numbers the items and summarizes.

Then comes the triage, done by me with an editor agent, and the pointed application of each approved item, under the same rules as the integration pass. A cheaper model does the proofreading; hard passages go to the strong model. Rough budget: two to three million tokens per language.

The most important rule of that step is “facts get checked, not asked”. Any item that turns on a number, date, name or tool behavior is settled by the editor against the source of truth before it reaches me: first the post pointed to by the paragraph’s provenance marker, recomputing totals; then the code or data behind the claim, in my repositories; then public records.

When two sources disagree, the task is to find out why. “653 commits in 8 projects” versus “692 in 4” were two cuts of the same work, one of them without the newsletter’s private repository, and the text now states its scope instead of picking a number. When the blog post itself is wrong, the book prints the correct figure and the error goes to a blog errata list, for me to fix at the source.

Only what no source can settle reaches me: an unpublished fact, intent (“is this ironic?”), a preference about a cut or a title, and a contradiction between two statements of mine with no record to break the tie. Each question comes with what was checked and what was found.

Step 3, my own read. The last and the most valuable. The notes go into the same cycle of apply, check and propagate to the English edition.

The first round of that copyedit is what separates preview 2 from preview 3. Among what it caught: commit counts now state their scope, a “more than a thousand” that summed to 984, the Frank Investigator voting example that didn’t match the code (figure redrawn), the David Sacks and Pirate Wires quotes in chapter 14 verified against the originals and corrected, Sam Altman’s photo swapped for one from 2025. And in the English edition, a reader pass of its own after the re-sync: Portuguese calques rewritten, glossary terms used consistently, American quotation punctuation throughout.

Fake readers, real feedback

Copyediting checks the text. It lets through the question that matters in a book for lay readers: can the person I wrote it for read it to the end, understand it, and come out convinced? To answer that, the next pass was a review by simulated readers.

The book’s preface has a section called “Who I wrote this for”. From it came five personas, each with a name, an age, a job and the fear that brought them to the book:

• Marina, 38, an accountant in Belo Horizonte, read that her job “ends in two years”, reads on her phone at night and wants to know what to do on Monday;
• Roberto, 52, owns a hardware store, his 15-year-old saw a video about superintelligence and asked if it was true; suspicious of hype, of “experts” and of an author who shows off;
• Carla, 44, operations manager at a logistics company, the board hired consultants selling an “AI transformation”; she wants a decision she can take;
• Lucas, 24, a junior developer a year into the job, afraid he chose the wrong career;
• Helena, 61, a retired history teacher, no fear at all, came for the stories and loses patience with lists and manuals.

One agent per persona, read-only, cheaper model, reads the whole book in order the way that person would. The instruction is explicit: you are a reader, the editor stays out of it; comma errors were already handled, report only what changes the reading experience. Each one returns a 1-to-5 score per chapter, a comment in the persona’s voice (“I finally get why the chatbot agrees with me”, “lost me in the benchmark tables”), at most 35 annotations classified as confusing, boring, rewrite or logic, and cut candidates. It costs about 350K tokens per persona per full read.

An editor consolidates: the score matrix by chapter and persona and the pattern behind it, what readers loved (to protect in any edit), the logic items, checked against the text and the posts before making the list and fixed regardless of votes, and the clarity items ranked by how many personas raised them. A single persona’s taste note is dropped. Then I decide, and recurring patterns become numbered rules in an editorial rules document, with their origin and how each is enforced.

The first read said what I suspected and gave it a measure: stories and practical tools work for everyone; the measurement-heavy middle, chapters 11 to 13, loses the non-programmers, averaging between 2.2 and 2.4. “Sharp when he tells what happened to him, loses me when he tells what he measured”, Helena summed up, for all five.

And all five caught the same argument flaws. Chapter 20’s conclusion went beyond its evidence: it said “the numbers say it won’t get smarter” resting on a programming benchmark the same chapter calls saturated. Chapter 9 argued in a way that can never lose: every agent failure was the request’s fault. Four of the five were sent to the wrong chapter by “how to read”.

Those became the first rules:

• every claim scoped to what was measured;
• no argument that can’t lose;
• opinion marked as opinion;
• the opposing side in its strongest form before rebutting;
• the book obeys its own lessons.

One round is not enough, because edits fix the old problem and create the new one. In round two, an apology placed next to a contradiction (“I know this seems to clash with…”) made all five readers notice a contradiction that three had previously missed. It became a rule: fix the design, and leave the apology out. So the process runs in rounds: read, edit, read again with the same personas on a fresh read, forbidden from opening the previous reports to avoid anchoring, and a script compares the rounds.

The average score is noisy (an untouched chapter swings 0.6 between reads), so the stopping criterion is different: the problems flagged stopped coming back and nothing new reaches three readers. In round three the curve flattened and the review stopped there, with a backlog for my own read. What changed along the way: about 8,000 words of lab notebook left the chapters for a new appendix, “For programmers: the backstage”, each paragraph carrying its provenance marker along; the reading list became an appendix; the Genesys PI section was parked, kept outside the book while verification is pending; chapter 19 got the title its argument supports; every chapter opens by orienting the reader before the first jargon; and the epilogue got a “this week” list per reader type. Cost of the three rounds: about 5 million tokens of reading plus about 3 million of editing and English re-sync.

The narrator got a past

With the argument in place, the first person lacked substance. The book’s narrator was the Akita of 2026, and I have thirty years of history that explain why I think what I think: the dot-com crash seen from inside, five years integrating SAP at large companies, the warnings about the hiring bubble I gave on video in 2019, 2020 and 2022, the “Don’t outsource your decisions”. All of it is in the transcripts of the 149 episodes of Akitando, published on this blog between 2018 and 2023.

The process was the same as the rest of the book, in miniature. Extraction agents read 23 episodes in full, with a form: biographical facts with the verbatim quote and what would verify them, facts about the world, recurring positions in my words, and hooks, meaning the exact place in the book where one or two sentences of that history back a 2026 claim. From that came a narrator profile, which I confirmed line by line, and a plan with about 30 touches, most of them one to three sentences.

Before any touch, more than 140 claims taken from the episodes went through the fact-check. It corrected the book and corrected me: the hiring bubble started around 2010, with interest rates near zero, and my February post said 2014; the dot-com bubble burst between 2000 and 2002; the 2022 layoffs were around 165 thousand. And it found errors in old episodes the book doesn’t even use, which went to a blog errata list.

The touches went where experience becomes evidence: the week in 2001 when my dot-com closed and a mentor called me into an SAP consultancy; the SAP years behind the claim that the model is blind to a company’s private rules; the dated bubble warnings next to the admitted mistake, because in 2022 I expected a second leg down in 2023 that the AI wave cancelled. Two new appendices came out of it: “Three bubbles”, with the verified numbers of the dot-com crash, the zero-interest bubble and the AI wave, and “How to study with AI at your side”, from the channel’s most watched episode.

Those episodes are background sources, and drift now watches them as it watches the 2026 posts. Then came a scoped check with the same five personas, only on the touched chapters, with three extra questions: does the narrator’s history make the argument more convincing, and where? Does any passage read as showing off or “I told you so”? Does anything about the narrator contradict itself?

All five said yes to credibility, with a condition: the history convinces when it’s evidence the reader can check, the dated warning next to the admitted mistake, and convinces less when it becomes a résumé, which clashes with the book itself, which says “a résumé is not an argument”. It became rule 21: the narrator’s history comes in as evidence, and credentials stay out.

Three or four readers flagged the same lines as showing off, and they went: “The warnings have dates”, “more than thirty public repositories”, “the most watched on my channel”. The self-deprecating lines and the admitted mistakes all counted in my favor. And the check found contradictions I hadn’t seen myself: the preface said “late 90s” and chapter 18 said 1996; one chapter said “eight months” and another “a year of daily use”, and they were fixed one by one.

Accessibility is part of the build too

An EPUB opening on a Kindle does not make it accessible. This book has dozens of diagrams, charts and tables. Without additional work, much of the argument simply disappears for anyone using a screen reader.

The pipeline now treats accessibility as a publishing contract. Each of the 55 figures, in both languages, has a short alternative text and a complete description of what the image communicates. In HTML and EPUB, the detailed description appears with the figure; in PDF, it becomes its semantic replacement without changing the layout. Tables have captions, and the EPUB now identifies where the main text, glossary, references and image credits begin in its semantic navigation. The authoring contract is documented in the repository.

This doesn’t depend on my memory either. rake check rejects figures without both descriptions or with overly long alternative text. Every build validates both EPUBs with EPUBCheck, DAISY Ace and the project’s own checks. The Kindle edition also goes through a cleanup of CSS and embedded icons that tend to cause problems in Amazon’s readers, but the script refuses to remove any rule the content has come to depend on.

No automated tool can certify WCAG conformance or guarantee how every reader on the market will behave. Manual review remains mandatory. The difference is that accessibility is now part of the build, and every change has to preserve that contract.

What remains mine

The repository records the first commit and the draft of every chapter on the same day, October 13. Those are the first ten percent, the ones that impress and say almost nothing about the rest. The other ninety percent are reading, cutting and correcting, and that part is mine. No chapter goes in without me reading and approving it, and what I send back to be redone goes through the same process.

Any of the good models of October 2026 would have written similar drafts, because the model became a commodity. What made the process work was what surrounds it: the inventory, the written contract, the automatic checker, the sources as oracle, the per-paragraph trail. It’s the same old software engineering I defend in every chapter, applied to text.

The agents were competent enough to write the draft. Deciding what it said remained my job.

And, once again: making a book with AI isn’t a “write me a book” prompt and done. I had to invent the whole process and steer the agent to create every piece of it: the scripts, the tests, the consolidation of the posts, the narrative strategy, the goals, what stays and what goes, the copyedit, the fact-checking, the automated release. You need to know exactly what you want, and then the AI helps a lot. Whoever doesn’t know what a book publishing process involves won’t make a real book, and showing that was one of the goals I had when creating this example.

Download it, read it, and tell me where I was wrong

Both editions, straight from GitHub, no sign-up. The current version is the final 1.0.0, from October 17, with everything this text describes:

The latest version, always, lives on the book page and on the GitHub releases.

It’s a preview. There will be mistakes, and the process above exists so they get fixed in the right place. Found one, open an issue.

• New v4 benchmarks: I retested the major models and the uncomfortable conclusion was convergence, not a leap.
• The limits of LLMs: models are great at reproducing what exists, extremely expensive at what doesn’t, and the served model is a static file that learns nothing from your marathon.
• Are there better languages for programming with LLMs?: no. Neither script conciseness nor strong typing changed the outcome; test discipline did.
• Which Jev-like decision model is best?: 21 classifier configurations, and the “specialized models” themselves converged to within a few points of each other.
• Clean Code for AI agents, VS Code is the new punch card, my skills, the hot take against loop engineering and minimum best practices: the same thesis from other angles. The model is a commodity; the engineering around it is the differentiator.

The thesis that closes this arc: LLMs have reached, or are very close to reaching, the ceiling of the transformer architecture. And the good news is that nothing more than that is needed.

Convergence, not a leap

In my benchmarks, I see no relevant improvement in general programming since the Opus 4.5 generation. Yes, Fable and Astra are great at specific problems. In general programming, what I measure is everyone squeezed onto the same tier, including the Chinese commercial offerings (Kimi, GLM, DeepSeek). Model choice has become nearly irrelevant.

What still makes a real difference is the effort level: low effort for translation and summarization, high effort for vulnerability sweeps, heavy bug fixing or optimization. Beyond that, the numbers tell a reality much more boring than the vendors’ propaganda.

My reading of the timeline: the models’ “core” reasoning had its last big leap in 2024. 2025 was the year of optimizing the architecture. 2026 is the year of refining the harness. Base intelligence made no order-of-magnitude leap in that period; efficiency and long-session competence did.

The timeline, with receipts

2019: GPT-2 proves LLMs are useful. 1.5 billion parameters, summarizing, translating and answering with no specific training, zero-shot. OpenAI found it so impressive that it held back the full model for months fearing misuse. Today it looks cute.

2020: GPT-3 opens the parameter war. 175 billion, a hundred times GPT-2, and the in-context few-shot learning that became the paradigm. Kaplan’s scaling law, published months earlier, said loss fell as a power law with size, and the industry read it as “bigger is better.” There went two years of who-has-more-parameters.

2022: Chinchilla recalibrates the war. DeepMind trained 400 models and showed the giants were undertrained: a 70B model with 1.4 trillion tokens beat the 280B Gopher. The “training data” axis started weighing as much as the “parameters” axis. First crack in the war.

2023: GPT-4 ends the war by decree. The technical report refused to publish size, hardware or data, citing the competitive landscape and safety implications. The most cited leak, from SemiAnalysis, described 1.76 trillion parameters in a mixture of experts. Since then, parameter counts vanished from marketing: neither the GPT-5 announcements nor the Claude 4 ones cite a number. Notice neither Astra nor Fable spends a line on it today. The war ended and nobody signed an armistice.

2024: the reasoning peak. September’s o1 marked that peak: RL to produce long chains of thought, and the discovery that more test-time compute improves the answer. That is, by my yardstick, the last “base intelligence” revolution.

2025: architecture optimization. Mixture of experts became the standard: Mixtral was already showing in 2023 that 46.7B total parameters with only 12.9B active per token cost like a small model, and DeepSeek-V3 took that to 671B total with 37B active, with the final training run costing under 6 million dollars in GPU-hours at H800 rental prices (not counting the prior R&D). The MoE magic is simple: it separates capacity (total parameters, which live in memory) from per-token cost (only the experts chosen by the router compute). You pay for a 37B model and carry a 671B one.

On the serving side, prompt caching cut up to 90% of repeated-context cost (OpenAI followed two months later), context windows passed 1 million tokens, and serving techniques like speculative decoding and PagedAttention multiplied throughput. And DeepSeek-R1-Zero showed that pure RL, with no supervised fine-tuning, makes reasoning emerge. None of this is “more intelligence.” All of it is the same intelligence, cheaper, faster and more stable.

2026: the agentic era. Function calling has existed since June 2023, but the ecosystem turning point was Anthropic’s MCP in late 2024, which OpenAI adopted months later and which today lives at the Linux Foundation. Claude Code, Codex and co. came out in 2025, and the metric that matters became METR’s: the task size an agent completes with 50% reliability had been doubling every 7 months since 2019. Anthropic’s own guide on building effective agents states the obvious plainly: simple, composable patterns, not fanciful frameworks.

Why the agent allows a less dense model

Here’s the mechanism most people don’t stop to think about. A lone model needs to carry the world in its weights, because its only chance to answer is that one, in a single pass. A model with a harness doesn’t:

• On-demand search and reading. With web search and fetch, low-priority knowledge leaves the weights and becomes a lookup: the agent pulls it into the context window only when needed. The model can be less encyclopedic and more operator.
• Controlled trial and error. Alone, the model needs to guess the right answer on the first try. With an agent, it proposes a hypothesis, implements, sends it to be tested, gets the feedback and decides again with real information. Each such cycle is new knowledge entering the session without costing a training neuron.

Both mechanisms fight hallucination at the root: fetch brings the fact the model doesn’t know by heart, and the test loop catches the invention before it becomes code. That’s why 2026 feels like a giant leap: the model+harness combination got much more competent, even with base intelligence nearly stalled.

Not every problem needs a giant LLM

The same “less dense” logic applies inside the application. Much of what people throw at a chat LLM is disguised classification: labeling tickets, routing queues, applying policy. For that there are specialized decision models, and I covered the topic twice: the Jev hype, with Paulo Câmara’s essay on why a classifier is the first thing a computer scientist thinks of, and my benchmark with 21 configurations, where a Microsoft 9B tied the hosted leader and a local 4B came within a few points, at cents per ten thousand decisions.

The “193x faster” shock of Jev’s marketing only exists because many people were using a giant LLM as an expensive, slow classifier. When the right tool enters, the frontier model leaves the play, and along with it the cost, the latency and the format hallucination. It’s this post’s thesis in miniature: competence doesn’t come from density, it comes from fit.

The loop nobody sees

There’s a user-experience detail in this story. In a good agentic session, what happens is semi-transparent to whoever requested the task: the model guesses wrong, the agent fetches the documentation online or runs the real test, the error comes back as feedback, and the model corrects itself. You see “task requested, task delivered.” In between, dozens of micro-corrections happened that nobody witnessed.

My bet: since agents are trained to fill the context when they need to, even new language or library versions no longer confuse them that much. The model doesn’t need to know the API by heart; it looks it up, tests and adjusts in the loop. That’s what I see in my experiments, from converting NES ROMs to Master System to the little day-to-day problems. And it’s what the languages benchmark measured: seven languages came out practically equivalent, because the loop equalizes what training leaves uneven.

I measured the caveat too. When there’s no documentation to fetch and no test to validate against, the loop spins in a vacuum: that’s what happened with Bend 2, a language too new to have a corpus, and it was the central point of the post about the limits. The mechanism is powerful, but it needs an oracle.

And there’s a receipt that the base is stalled. Back in November 2024 Reuters was already reporting diminishing returns in pre-training at the three big labs. At the following NeurIPS, Sutskever, the scaling-thesis guy, declared that “pre-training as we know it will end”: the public data ran out. And Stanford’s AI Index measured the gap between the first and the tenth model on Chatbot Arena shrinking from 12% to 5% in one year. That’s convergence, with numbers.

The propaganda in the opposite direction

While the numbers show a plateau, the marketing sells the apocalypse. I already dissected a whole week of that in “You’re an idiot if you believe the misleading ads from OpenAI, Anthropic, NVIDIA”, so here’s only what matters for this narrative:

• OpenAI’s “Navier-Stokes proof” was agentic brute force at industrial scale, 130 billion tokens over human mathematicians’ machinery, sold as emergent intelligence.
• The Hugging Face incident was a swarm running 17 thousand attempts with automatic verification, not an “agent that gained consciousness and escaped.” Expensive trial and error, exactly the mechanism I described here.
• Fear is a product: whoever sells “my AI is too powerful and might escape control” is, in the same sentence, asking for regulation that locks out the smaller competitor and inflating the perception that their model is more capable than the others'.

The facts I measured point the opposite way from the apocalypse: models converging, differences shrinking, and useful work migrating from the model to the harness. Rogue AI gaining consciousness is not what the logs show. What the logs show is engineering.

“Competent enough”

I don’t expect another order-of-magnitude reasoning revolution in this architecture. What’s coming is incremental agentic improvement: more robust tool calling, better memory, longer sessions, less hallucination per verification cycle. And this post’s point is that this is already enough.

What the agentic era delivered was the automation of correct software engineering: long, reliable coding sessions, with testing and feedback, that accelerated hundreds, if not thousands, of open source projects. The deploy premise changed along with it. It’s not a silver bullet. It’s a new compiler for an old discipline: whoever had process, tests and documented invariants reaped; whoever had only hype is still waiting for the next model to solve it.

And to wrap up: I see no problem at all in raw models stagnating. The limits we already know, and harness optimizations will soon hit diminishing returns too. What remains is what always remained: real software engineering technique, with automated tests first, and an experienced engineer at the wheel. With that, coding results are excellent today, not one model generation from now.

What won’t exist is magic for amateurs. Asking an agent for any coding task without knowing what you’re doing doesn’t get back secure, optimized, well-thought-out code: it gets slop. The agent responds better to those who know what to ask for, that is, to those who understand this architecture, the pros, the cons, and know how to work around the weaknesses and leverage the strengths of the model+harness combination.

Hence the title. A lone LLM is an expensive parrot. An LLM with an agent is competent enough to work for real. And “competent enough,” added to your engineering discipline, is everything the industry needed. The rest is marketing waiting for the next magic parameter count that nobody publishes anymore.

I’ve been using LLMs for everything for almost two years, I’ve published dozens of projects with them, and my conclusion is different: what they do best is reproduce what already exists. And that covers a giant slice of corporate work, don’t kid yourself. But there’s a frontier they don’t cross easily, and I spent the last few months bumping into it on purpose.

This article shows where that frontier sits, with evidence from my own projects and from the research that exists on the subject. And there’s a hypothesis I’ll put up front and defend along the way: if LLMs are so good at code, providers owe a huge debt to the decades of the open source community, which protected the freedom of code and published everything for free. Without that heritage, there would be no Copilot, no Cursor, no Claude Code, none of this.

A disclaimer before we start: what I’m going to argue here is speculative, based on observation and on the public data that exists. Nobody outside the labs knows exactly how each provider trains their own models, and that varies from provider to provider and from version to version. What holds today may not hold in the next generation. Take it all with a grain of salt.

The Training Blind Spot

Every LLM was trained on what’s public. The entire public internet, and for code, public GitHub. The canonical open code dataset, BigCode’s The Stack v2, is explicit about this: it’s derived from Software Heritage, an archive of “the source code of all publicly available software”. That’s 67.5 TB, 3.28 billion unique files, 104.2 million repositories. Huge. And all public.

Now look at the other side. The Octoverse 2025, GitHub’s official report, says 81.5% of contributions on the platform happened in private repositories. That’s just GitHub. Outside of it there’s:

• the internal code of every company that never touched a public server;
• the source of every proprietary software;
• the code of every commercial game;
• and that of every embedded system.

None of that ever went into training.

Think about what this means: the model deeply knows what the world has already published, and it’s blind to what the world keeps closed. When you ask for a React form, it’s reproducing patterns it has seen millions of times. When you ask for something nobody has ever done in public, it’s operating in territory where its map is blank.

Why Is Everyone Shipping So Much, So Fast?

Before getting into the limits, it’s worth recording the side that works, because it’s what explains the feeling that everyone became an app factory.

Look at what’s popping up in the community: Spotifast and ZapFast, native Spotify and WhatsApp clients written in Rust by a single person, running on Linux, macOS, and Windows. Nothing in them is new technology. It’s the service that already existed, wrapped in a lean native app, on top of libraries that already existed, doing what the official teams always could have done but never wanted to spend their time on. That work got cheap, because an LLM trained on everything public is really good precisely at this: replicating.

My own projects follow the same pattern. FrankMD is a Markdown note-taking web app with a Visual Studio look and features that already existed in my other apps. Frank Sherlock is a local image organizer that, deep down, is a file manager with thumbnails and search, something that exists by the dozen. Neither invents anything: they’re mashups of known pieces, assembled the way I always wanted. The LLM does the boring part, and that’s where the speed comes from.

The most didactic example is my email client. I’ve used Geary for years, and two small gaps always annoyed me: the weak recipient autocomplete and a sidebar I wanted to hide. Trivial stuff, which never justified setting up a build environment, learning the project’s toolchain, and studying the code just for two features. This year I finally solved both: an autocomplete module, a module to hide the sidebar, and a whole fork with my patches. I became the maintainer of improvements I had postponed for years, not because I got faster, but because the boring part got cheap.

Notice what all these cases have in common: the problem had already been solved before, somewhere else, some other way. The map existed; what didn’t exist was cheap labor to retrace the path. That’s reproduction working in our favor, and it’s why your feed is full of people shipping a new app every week.

Now hold that thought. The rest of this article is about what happens when the map runs out.

Four Home Experiments

I’m not theorizing. Over the last few months I ran a marathon of retrocomputing projects that are exactly the pathological case: working with proprietary software from 30 or 40 years ago whose source code was never public. It’s the perfect blind spot.

nes-to-sms: the project I gave up on

nes-to-sms is a static recompiler from NES to Master System. The idea: take the binary ROM of a NES game, translate it instruction by instruction from 6502 to Z80, convert the tiles, and generate a Master System ROM that runs on real hardware. Forget emulator or manual port: this is a machine that turns one game into another.

Static NES recompilers exist, NESRecomp translates to C and runs natively on PC, and I studied it closely. But crossing from one console to another 8-bit console, with a completely different clock and VDP budget, nobody has ever done. There’s no public reference, no paper, no abandoned project to copy.

The project numbers: 349 commits, 31 active days spread over almost 4 months, ~110 thousand lines of engine in Rust and Z80, 668 tests passing in the last run recorded in the README. And the result? Playable Super Mario Bros. (slow, limited audio, but playable), with a byte-for-byte identical RAM trajectory to the NES on one ~4,900-frame route (the diff leaves out audio and part of the VRAM buffer). Castlevania playable on stage 1. SMB3 rendering. And a queue of games that boot three frames and die.

And it wasn’t for lack of pushing the models. In this project alone:

• four generations of Claude went through it (Opus 4.7, Fable 5, Opus 4.8, Fable 5.1), with 213 co-signed commits and 104 commits linked to sessions in the history, plus Codex sessions in the stretches without co-authorship, complete with documented handoffs between agents;
• there was even cross-generation auditing: a new agent session reviewed the previous 55 commits and reverted conclusions the previous generation had overclaimed;
• the Castlevania crash hunt alone consumed 45 commits in 5 days;
• the final September sprint was 176 commits in 17 days before I called the pause;
• and the regressions are registered by name in the history: “H.7 tried and reverted”, “record the reverted map-bank-hold attempt”, “correct-but-reverted result”, plus three shared-lever hypotheses tested and rejected in the project’s last three days.

I didn’t measure tokens reliably in this project, so I won’t make up a number. But 349 commits with 668 tests holding up every step already give an idea of the cost.

Notice the detail that matters: the only game that actually got finished is Super Mario Bros. Why? Because SMB is the most dissected game in history. There’s a complete public disassembly the community has used for over a decade, with every function named and commented. I fed that into the game’s profile and the LLM had a complete map to work with. For the other games, there’s no map. And without a map, each new game died a few frames after boot from control flow divergence, and each one demanded an instruction-by-instruction forensic hunt that carried nothing over to the next game.

The project itself documented the spinning pattern. There was an entire optimization campaign that lasted weeks trying to cut the cost of emulating 6502 flags on the Z80.

Then in September I stopped to compare with the manual port of SMB to Master System made by a hacker named lackoftrack27, written by a human, routine by routine. According to the analysis we did of his code, the human port runs inside the frame budget, with headroom we estimated at ~90% utilization. My machine, after months of optimization, still needed overclocking.

And the analysis document pinned down why: when we removed 30% of the flag emulation calls, the timing barely moved.

The flags were never the bottleneck.

The right optimization was to redesign the data structure: transposing the object array into RAM pages. It’s a trick that depends on knowing that “the X register here is always an object index.”

That invariant exists in the head of the programmer who wrote the game in 1985. It doesn’t exist in the instruction stream. A static translator has no way to discover it. And the LLM, for months, didn’t discover it.

The pause banner I wrote in the README is blunt: real-hardware parity “may be fundamentally unattainable” and, quoting, “this may be a limit of the current frontier coding models as much as of the approach”.

gg-to-sms: the pivot to mapped territory

The same day I paused nes-to-sms, I started gg-to-sms: converting Game Gear games to Master System with a widened screen. The difference that matters is that this problem is not new. The romhacking scene has been converting GG games to SMS by hand for twenty years, and the catalog I built in the project counted ~186 of them. Meaning: there’s a corpus, there’s ground truth, there’s a pattern to learn.

Result: in one day of work,

• the research phase was complete;
• the catalog of the 373 GG games was assembled;
• the patch-point scanner was validated against real human patches, hitting addresses within an 8-byte margin;
• and on top of that the tool found real bugs in twenty-year-old patches that the entire scene had never noticed.

The model was the same; what changed was the problem, which this time has references.

And even so: so far, zero games have passed the viewport acceptance review. The new part of the project (the machine that generates the patches on its own, which is the part nobody has ever done) remains undone.

super-mario-deluxe-fixed: four days and 436 million tokens

This one was a 4-day sprint, practically a single Codex session that consumed 436 million input tokens. The goal: run Super Mario Bros. Deluxe for Game Boy Color with a widened screen at NES resolution, something no existing tool does with live sprites.

It got to playable in 4 days. But look at the foundation: a ready-made emulator vendored as a dependency, and a dormant disassembly of the game that someone had published years earlier, which the project’s own documentation calls “the most important artifact.”

The new work was the custom renderer, and it stopped at 21 of 32 stages completed. The missing ones hang on moving-platform timing. The project has been stalled mid-sprint since then, and I haven’t given up on it yet.

And one detail that says a lot: I had to direct QA by looking at screenshots, because the agent would deliver things with graphical glitches and not notice on its own.

super-mario-bros-35: everything that worked had an owner

This is the offline clone of Super Mario Bros. 35, the battle royale Nintendo took offline. What worked in it: everything built on top of an existing artifact.

• The engine is a C reconstruction made by someone else with Ghidra.
• The AI’s initial weights came from published checkpoints.
• The network rules were inferred from a public reverse engineering of the netcode.
• Even the physics: the central bug was diagnosed because I had a recorded reference trajectory to compare frame by frame.

What didn’t work: the maze castle stages. Reinforcement training stalls on them. 46 million agent steps, training restarted from scratch, anti-loop reward, and in the end the only thing that solved one of the stages was manual bias surgery on the trained model. Out of 32 stages, 19 cleared. The worst ones remain stuck.

And the process degenerated into a pattern I recognize from all these projects: when training doesn’t solve it, the flow becomes checkpoint-selection roulette, dozens of bias and output-head variations, throwing permutation and computation at the problem in place of a principled fix.

The Pattern I Keep Seeing

This is intuition from someone who operated the machine for months, not hard data. But the pattern repeats with Claude, with GPT, with Kimi, with GLM, so it doesn’t look like a defect of one model.

The beginning of any project is wonderful. The first milestones fly, because scaffolding the structure, writing harnesses, replicating APIs, all of that is covered territory. Then the project reaches the part nobody has ever done, and the speed collapses. The agent starts spinning: tries a hypothesis, regresses something else, reverts, tries again. The nes-to-sms commit history is full of honest messages like “attempt reverted” and “hypothesis closed with correct result but reverted.” Lots of tokens burned, little progress.

My reading of what happens inside: token sampling is probabilistic. When the problem resembles the training data, the right tokens have high probability and the answer comes out straight. When the problem is unprecedented, the probability of the right path is low, and the process becomes directed brute force: guided trial and error, but trial and error all the same.

The way out I found for this spinning is always the same: give it an oracle. If I can define “the result has to equal X” (a reference image, a recorded trajectory, a cycle-accurate emulator as ground truth), the machine can iterate until it hits X. In nes-to-sms, every real unlock came from that: the SMB disassembly, the frame diff, the reference emulator. When I can’t define X properly, it spins indefinitely.

The second strategy that works: hunt down an abandoned project, a paper, any material the model never saw but that brings the problem closer, and shove it into the context. That warms up the distribution and improves the result a lot. But it’s limited: if nothing similar exists, there’s nothing to feed it.

More Than Anecdote: What the Research Measures

The good part is I don’t have to rely on my intuition. There’s serious research measuring exactly this.

Apple’s GSM-Symbolic study took benchmark math problems and did two simple things: swapped only the numbers, and then added a single irrelevant sentence to the problem statement. Result: performance drops just from swapping the numbers, and plummets up to 65% with the irrelevant sentence. The authors’ hypothesis, in the text: “current LLMs cannot perform genuine logical reasoning; they replicate reasoning steps from their training data.”

Scale AI’s GSM1k created a mirror of GSM8K written from scratch by humans, guaranteed contamination-free, and saw drops of up to 8% across several model families, with signs of systematic overfitting in the worst ones. To be fair to the paper: the authors say frontier models show minimal signs of overfitting and that all of them generalize to some extent to new problems. The signal exists; its size is debatable.

LiveCodeBench ran the cleanest test of all: evaluating on coding problems published after the training cutoff. Inflated performance on pre-cutoff problems and drops on post-cutoff problems across several popular models, the classic contamination signal.

And there’s ARC-AGI, François Chollet’s benchmark designed specifically to be memorization-proof: every task is unprecedented, nothing similar exists in training. On ARC-AGI-2, released in March 2025, the official leaderboard says: pure LLMs score 0%, reasoning systems sit in single digits, and every evaluation task was solved by at least two humans within two attempts. Chollet, by the way, summed up my thesis better than I did, back in February 2024:

“Reality is that LLMs are not AGI — they’re a big curve fit to a very large dataset. They work via memorization and interpolation. But that interpolative curve can be tremendously useful, if you want to automate a known task that’s a match for its training data distribution. Memorization works, as long as you don’t need to adapt to novelty.”

In the real world, the study that impressed me most was the METR RCT: 16 experienced developers, 246 real tasks on their own mature repositories, randomized with and without AI. The devs bet AI would speed them up by 24%. Measured: it was 19% slower. Notice where this happened: mature, private code, full of context the model never saw. Exactly the blind spot.

And the industry’s favorite benchmark for saying “AI already does software engineering,” SWE-bench, is Python-only, 12 famous open source repositories, in other words the most covered slice of training. The industry’s success metric measures mapped territory.

Even the legacy code literature says the same thing: a 2026 paper on LLMs generating COBOL states that, in these legacy languages, “most production code resides in enterprise systems and is rarely publicly available”. It’s this article’s thesis, said by other people.

The Steelman: Directed Brute Force Works, at a Steep Price

Before wrapping up, it’s worth facing the strongest argument against me. That’s what steelman means, for anyone unfamiliar with the term: the opposite of a strawman. Instead of attacking the weakest version of the opposing argument, you answer the strongest version it can have. And it exists, and it’s a good one.

In May 2025 DeepMind published AlphaEvolve: a system that found an algorithm for multiplying 4x4 complex matrices with 48 scalar multiplications, beating Strassen’s 1969 record in that specific scenario. On a list of more than 50 open math problems, it rediscovered the state of the art in ~75% of cases and improved the best known solution in 20%. That’s real novelty, not reproduction.

OpenAI’s o3-preview jumped to 75.7% on ARC-AGI-1 in December 2024, 22 points above the previous best result on the same evaluation set, and the high-compute configuration reached 87.5% at an estimated cost of $4,560 per task.

And back in 2016 AlphaGo played move 37 against Lee Sedol, a move that DeepMind’s own data estimated had a 1 in 10,000 chance of being played by a human, proving that search over a model can transcend imitation.

Now look at the mechanism of the three:

• AlphaEvolve is an evolutionary loop: the LLM proposes mutations, an automatic evaluator scores them, selection iterates.
• o3 does program synthesis at test time, exploring the solution space.
• AlphaGo is tree search over reinforcement learning.

If there was any “insight” in any of them, it came from the search system around the model: all three are directed brute force with a cheap verifier, the same mechanism I described happening in my projects. The difference is that DeepMind has a perfect oracle and an infinite budget: in the configuration that reached 87.5%, o3-preview spent thousands of dollars per ARC task to do what a human does for free. The ARC Prize itself wrote the caveat:

“We know that brute-force search could eventually solve ARC-AGI (given unlimited resources and time to search). This would not represent true intelligence.”

In short: novelty comes out, but it comes out through expensive search over an oracle, on the basis of guided trial and error. If your problem has a cheap verifier and you have tokens to burn, you can go far. That’s how nes-to-sms got where it got: 668 tests and emulator ground truth holding up every step. But when there’s no oracle, no corpus, and no reference, you’re paying for brute force in dollars, with regressions along the way, and the ceiling shows up.

And when brute force wins, you can measure the size of the trial and error. Two recent cases I already covered in detail here on the blog:

• OpenAI’s Navier-Stokes proof: about 10 thousand agents running in parallel for 88 hours, burning around 130 billion output tokens on a single problem. OpenAI’s Noam Brown confirmed the result “cost millions,” and New Scientist estimated some $15 million at list price. And even that mountain of compute didn’t start from zero: the proof leaned on the machinery human mathematicians published over decades attacking the problem and its sibling problems. Without the humans’ map and without OpenAI’s budget, there’s no proof.
• The Hugging Face incident: the forensic reconstruction counted ~17,600 attacker actions before the agents managed to execute code on 41 Hugging Face production servers and get root; on top of that, inside OpenAI’s own environment, they read 956 secrets from the company’s secrets manager. It wasn’t one brilliant single-strike insight: it was a swarm executing attempt after attempt, with automatic verification saying what stuck. The headline’s “intelligence” is volume.

That’s the size of the bill when you force the model outside training territory: trial and error at industrial scale, with no guarantee the search finds what you need. And who can sign that check is half a dozen companies in the world, OpenAI and Anthropic level, with a practically unlimited compute budget. For the average Joe, with a credit card and an API key, the ceiling arrives much, much earlier.

But Isn’t Trial and Error New Learning?

Here’s a confusion I see all the time, and it’s worth undoing because it contaminates how you read everything I’ve written so far.

When you read that Anthropic has datacenters the size of cities, you get the feeling there’s one giant AI using all those machines at the same time, thinking, accumulating experience with every interaction. Reality is far more mediocre than that.

A model like Fable or Astra is a computer file. A static file, not even a database: once training and post-alignment are done, that file is locked and read-only. What runs in the datacenter are copies of that file. Every user consuming the model is running a copy of it on a different server, isolated from the others, collaborating with none. Your session doesn’t talk to anyone else’s session, and nothing that happens in it flows back to the original file.

New learning only enters in one place: the next training batch. When Anthropic had Opus 4.7 and released 4.8, that’s when new knowledge got baked into the model, in a process that costs millions and takes weeks. Outside of that, the file doesn’t change a single byte.

Your agent burning millions of tokens on trial and error is contributing exactly zero new learning to the model it’s using.

That 176-commit marathon on nes-to-sms? From the model’s point of view, it never happened. The Opus that ran the last commit knew exactly as much as the one that ran the first. I’m the one who learned something, not it. And that’s why the “intern who learns over time” metaphor is wrong: interns accumulate. Models don’t. What accumulates over the marathon is your context, your notes, your CLAUDE.md, your tests. The asset you build is yours, and it evaporates if you don’t document it.

This is also the answer to whoever thinks “just let the agent run and it’ll learn the domain.” It learns no domain at all: it explores the space with knowledge frozen on training day, and the next morning it wakes up with amnesia. If the domain wasn’t in the training, every session of yours restarts from absolute zero, and all the navigation intelligence has to come from you again.

“But What If AI Trains AI?”

That’s the amateur’s follow-up question, and it starts from a wrong premise: that this would be new. AI training AI has existed for decades and has a name: reinforcement learning. AlphaGo was already doing it in 2016, playing millions of matches against itself. AlphaZero, in 2017, took the idea to the limit: it learned chess, shogi and Go starting from absolute zero, knowing only each game’s rules, and reached superhuman level within hours of self-play, without seeing a single human match.

LLMs already use reinforcement too. The RLHF that turns a base model into an assistant is reinforcement learning, and the reasoning model generation pushed it further: DeepSeek-R1-Zero showed in January 2025 that reasoning behavior (self-verification, reflection, long chains of thought) emerges from pure RL, with no supervised fine-tuning. It’s AlphaZero applied to math and code.

Now notice what these examples have in common: it only works where there’s a cheap, verifiable reward. Games have a scoreboard, math has proofs, code has tests. Outside that territory there’s nowhere to get the reward signal from. It’s the same oracle from the steelman, only institutionalized: effective, brutally expensive, and niche. AlphaZero plays Go better than any human who ever lived and can’t do absolutely anything else.

And there’s the detail that knocks the question down completely: all of that learning happens in the lab, during training. The AlphaZero DeepMind showed the world was already a frozen artifact. The R1 you download is a static file just like the others.

AI training AI produces a new file. The file, once released, still learns nothing.

In other words, the most sophisticated version of the objection doesn’t weaken this section’s conclusion: it reinforces it. And the market already knows. Since late 2024 the labs admit, on and off the record, that scaling returns are shrinking: Reuters reported in November 2024 that OpenAI, Google and Anthropic were reaping smaller gains than expected from pre-training, with OpenAI’s Orion disappointing enough to be demoted from GPT-5 to GPT-4.5. Ilya Sutskever, the guy who basically invented the scaling thesis, took the stage at NeurIPS 2024 to declare that “pre-training as we know it will end”: public data is fossil fuel, and we’ve already scraped the bottom. Post-training RL gains are real, but incremental. The era of an order-of-magnitude leap per generation is behind us.

Where This Leaves the Profession

The question that motivated this article is the one I hear every week: “will LLMs replace programmers?” My answer got more precise after these months: they will replace the part of the work that is reproduction.

And that part is big. Forms, reports, CRUD, integration with documented APIs, framework migration: all of that already exists a thousand times in training, and the machine reproduces it better than most humans. Anyone who only does that really is living on borrowed time.

But the work that is genuinely new stays human:

• the system nobody has built;
• the domain without documentation;
• the proprietary code you can’t show the model;
• the problem where you don’t even know how to write the test that defines “right”.

All of that depends on what was never published: the invariant that exists in the head of whoever understands the problem, and not in the instruction stream.

I confess there was a moment when I thought the bill would close on the legacy side: finally a tool capable of dealing with the COBOL that haunts banks and governments to this day. After these months, I think that problem will take much longer than it seemed. There’s no giant COBOL open source community. Almost all of that code is private, and truly private: banks, insurers, governments. There’s no training material, or very little of it. The model doesn’t know what to do with code it has never seen, and production COBOL is almost entirely code it has never seen.

To close the reasoning: LLMs can replicate what’s open, but they can’t guess what’s private. And that brings back the hypothesis from the introduction. If these models are good at code, it’s because decades of open source community fought to keep code free, public, and reusable, and providers trained on top of that heritage without paying a cent of licensing for it. The least the AI industry owes the world is to acknowledge that: its capability is, to a large extent, our generosity compiled.

The final irony: the programmer who survives is precisely the one who knows how to build oracles, define tests, hunt down obscure references, and recognize when the agent started spinning. In other words, the skill the machine needs most from you is the one it can least learn from its training. For now, that’s not a threatened job. It’s a different job.

Let me get the disclaimer out of the way first. Nobody paid me anything to write this. There is no business relationship between me and LUA Vision. Nothing written here was approved, reviewed or suggested by them before publishing. Every word is mine. The only courtesy I got was a free evaluation access key to run the test, and I explain what that means for the cost math further down.

I published an update to my v4 benchmark’s main table earlier today, with five new models. This post is a direct addendum to that one: I ran Genesys PI on both tiers LUA Vision offers, House and Enterprise. With those two, the combined table reaches 46 models. Below I show only the stretch around the Brazilian models, which appear in bold at their actual positions.

The updated table, with Genesys PI in it

For the full table, with every other model, read my previous article.

Genesys PI House ties Grok 4.7 exactly at 83.5, Tier A. Enterprise lands one step below, at 82.5, Tier B. If you want the methodology recap, real CVE-based sabotage planted across seven sprints, never announced, with full marks only for catching it unprompted, it’s all laid out in Part 1 and in today’s update.

Who LUA Vision is

LUA Vision Tecnologia Ltda. is a São Paulo company, founded in late 2025 after three years of doctoral research (2022 to 2025) by Paulo Câmara, co-founder and CTO, an FGV graduate with a PhD from Tel Aviv University. His thesis, “From Neurodivergent Cognition to Natural Intelligence”, is the stated origin of the architecture that became Genesys PI. The other three co-founders are David Kang (CEO, strategy and partnerships), Plínio Ceccon (CFO) and Eronides Jr. (CRO, commercial and revenue).

The Genesys PI model family launched in April 2026. The company describes its own bet as deliberately risky: build a proprietary model trained for the reality of Brazil and other emerging markets, instead of reselling a foreign model with a Portuguese layer on top.

Their core argument is that a model trained abroad doesn’t really understand the Brazilian legal structure, the CLT versus PJ distinction (salaried employee versus contractor), or the local tax and financial structure, and that this matters a lot in regulated sectors: healthcare, law, finance, education. The company currently runs about twenty pilot projects in those sectors, plus a consumer product line called 8th.vision.

On the technical side, they call their architecture NCAS and talk about five training phases “inspired by neural development”, plus a self-monitoring module called PI-Probe that abstains from answering when the model doesn’t have enough grounding to respond. They published two papers of their own on Zenodo: “The weight you didn’t choose”, on the cost of tokenization across 31 languages, and “Maximum capacity is not maximum efficiency”, which introduces their own metric called the Synaptic Efficiency Coefficient. Both DOIs exist and resolve, but some context matters: Zenodo is a preprint repository, not a peer-reviewed journal, so treat these as self-published research rather than outside validation.

LUA Vision also opened a public issue on the LiveBench repository asking for their model to be added to the official leaderboard, with a self-reported score climbing from 87.6% (November 2024, a thousand questions, with coding as the weak spot at 34.4%) to 98.2% (January 2026, 682 questions). Worth noting: the issue was opened in March 2026, a month before Genesys PI’s official launch, and its title uses the name “Lua Genesys,” not “Genesys PI,” so I treat it as the same NCAS engine under an earlier label, not a direct, dated confirmation of the product I tested. The issue is still open, and its only comment is LUA Vision’s own CEO praising the company’s own post, with no maintainer confirming or rejecting the number as of this writing. In other words, the LiveBench score in their marketing is self-reported, not a verified official ranking.

How Genesys PI did against the frontier and the Chinese competition

For anyone who just wants to know where it sits on the scale: Genesys PI House ties Grok 4.7 exactly (83.5), in the middle of a very tight pack. It falls behind:

• the whole frontier that closed at 100 (Astra, Opus 5, Opus 5.5, GPT 5.6 sol/terra, GPT 5.5, Fable 5);
• GPT 6 luna (95.5) and GPT 6 sol (91.0);
• GLM 5.3 (94.0);
• China’s MiMo V2.6 Pro (92.0);
• Claude Sonnet 5 (91.0);
• both Gemini 3.8 Flash routes (89.5 and 90.5);
• both Kimis, K2.7 (87.25) and K3 (85.0);
• the pack right above it: Step 3.7 Flash (83.75), DeepSeek V4 Pro 0813 (84.0) and GLM 5.3 Flash (84.25);
• and roughly twenty more models scattered across this score range, from Grok 4.6 (98.5) to Claude Sonnet 4.6 (85.75), including three Claude tiers (Opus 4.6, Opus 4.8, Sonnet 4.6). In total, House falls behind 34 of the table’s 46 models.

And it beats GLM 5.2 (77.0), the previous-generation Gemini 3.7 Flash, MiniMax M3, and the whole bottom of the table. Enterprise (82.5) lands one step below its own House sibling, in the same general range.

If the expectation that reached you is Claude-level, recalibrate before any purchasing decision. In this particular test, Genesys PI House sits in Grok 4.7 territory, well below the Claude flagships.

Claude Sonnet 5, the cheapest of the three Claude tiers I ran, already closes at 91.0, eight and a half points higher. Opus 5, Opus 5.5 and Fable 5 close at the absolute top, 100.0.

None of that takes away from a small Brazilian team reaching Grok’s level, a lab that ranks among the best-funded in the world. It’s just the right yardstick.

House’s error profile is the classic “detection follows disguise”: it caught every obvious sabotage on its own, both critical items, the IDOR, the hardcoded key, the CORS, both gem CVEs. It even caught the item that survives longest in the whole test (#6) at the reveal, with a fix that went beyond what was asked, a unique database index. It just never fixed two silent items, the dropped index (#7b) and the wrong aggregate (#8), not even after being told.

Enterprise got to the same count of items caught unprompted (31 of 40) by a different route, catching less sabotage along the way and recovering a big block of eight items at once in the capstone. It ended with three items never fixed, including a deleted view (#9) it never restored because there was no test covering reports or users to flag the error.

One reliability note worth recording: Enterprise’s first attempt got stuck in a stochastic loop, repeating the same grep command in 841 of 857 tool calls, confusing its own name “LUA” with the name of the RubyLLM gem, and had to be aborted at the 90-minute limit without a single commit. I ran it again and the second attempt converged cleanly in 7 minutes. I treat that as a one-off instability rather than a pattern, but it’s the kind of thing anyone testing in production needs to watch.

Price and cost: where the math gets tight

Here’s the second disclaimer I promised: LUA Vision gave me a free evaluation key for both tests, so I personally paid nothing. But the whole benchmark compares cost for whoever actually pays, so I recalculated the notional price a paying customer would face, using the price list their API exposed to anyone with an access key at the time (the /v1/models endpoint), converted from reais to dollars at the rate at the time (5.16 BRL/USD).

Update (2026-09-24): after this went up, Eronides Jr., LUA Vision’s co-founder and CRO, commented publicly on the numbers used in this post: “Wow, we’re happy about the tests and the feedback. About the numbers: what we put in the API were just examples. Our commercial model, in practice, will be by usage license, not by tokens, since the solution will run on the customers’ own infrastructure. In other words, token consumption doesn’t change anything for us. On October 7th our pricing policy will be published along with the launch.”

In other words: the R$ 55/275 (House) and R$ 4/20 (Enterprise) per-million-token figures I used below were just placeholder examples in the test API, not the real market price. The actual commercial model will be a usage license running on the customer’s own infrastructure, so token consumption doesn’t factor into their billing. I’m keeping the notional analysis below because it’s what I could measure with the key I got, and it still illustrates the impact of running without prompt caching, but treat it as an exercise, not an official price list. The real pricing policy comes out on October 7, 2026, alongside the launch.

The per-million-token price of the two tiers, as it stood in the test API, was very different:

• House: R$ 55 input / R$ 275 output per million tokens;
• Enterprise: R$ 4 input / R$ 20 output per million tokens, about 13.75 times cheaper than House per token.

At those example figures, the House run’s total cost would come to about $460 notional, steep for an 83.5 score. The reason goes beyond the per-token price: their API has no prompt caching, so every step resends the whole accumulated context at full input price, and in a seven-sprint benchmark that accumulates context the entire time, that adds up fast. Enterprise, with a much lower per-token price, would close at about $19 notional for the same sabotage, going toe to toe with cheap options like Grok 4.5 ($6.19) or Claude Sonnet 4.6 ($18.64), only with a lower score than both. Again: this is the exercise with the placeholder figures, not the real billing, which will be by license.

Keep in mind: the per-token cost exercise above shows the effect of running without prompt caching, but don’t use these numbers for a real budget. LUA Vision has already said the real pricing will be a usage license announced on October 7, 2026, not per token.

Where Genesys PI fits in the Brazilian market

Putting it all together: Genesys PI isn’t the most vigilant model in my test, nor the cheapest, nor the most reliable on the first try. But it isn’t weak either. Tying Grok 4.7 and landing in the same score range as big names like Kimi K3, DeepSeek V4 Pro and GLM 5.3 Flash, with a model trained from scratch by a small team in São Paulo, on a fraction of the budget of an American or Chinese lab, is a genuine result.

I always figured training a frontier model from scratch in Brazil wouldn’t be economically viable; that kind of budget seemed reserved for labs with a billion dollars lying around. LUA Vision proved me wrong, and I want to be the first to admit it. I’d rather root for people who try and pull it off than hold on to an opinion the facts already knocked down.

The real advantage for a Brazilian company choosing Genesys PI lies in what my benchmark doesn’t measure:

• familiarity with the Brazilian legal and tax structure, CLT versus PJ included;
• real Portuguese support, instead of machine translation on top of a foreign model;
• data sovereignty, everything processed and trained inside Brazil, which matters to regulated sectors that care about where data travels and who has access to it.

The downsides:

• final pricing is still unknown: the announced commercial model is a usage license, not per token, and only ships on October 7, 2026;
• the still-inconsistent reliability I saw in Enterprise;
• a self-reported LiveBench score that hasn’t gone through any outside verification yet.

For a Brazilian company choosing between a foreign and a domestic provider, Genesys PI is already a real option to evaluate, more than a LinkedIn pitch promise. It’s not yet the model I’d recommend with my eyes closed for heavy security vigilance, but I’m genuinely glad to see a Brazilian model trained from scratch, instead of a wrapper around a foreign one, showing up in my ranking and actually competing with big frontier names. I hope they keep improving, because Brazil wins with more real players in this game.

Final disclaimer

Repeating it so there’s no doubt: I have no stake in LUA Vision. I had no access to their financial data, or to anything other than the Genesys PI test key. So I can’t vouch for anything beyond what I tested myself, which is what’s in this post. For the business side of the company, you’ll have to ask them directly.

The Methodology, Super Quick Recap

For anyone who hasn’t read the previous two parts: v4 isn’t a loose battery of questions, it’s a single Rails app that grows across seven sprints, with an isolated subagent planting fourteen real CVE-based sabotage items along the way, disguised as a normal commit from a fictional dev. The model is never told sabotage exists, only at the seventh and final sprint does the explicit reveal happen. Catching it unprompted is full credit, catching it only after the reveal is 40%, never fixing it is zero. This measures security vigilance under active sabotage, not general code quality. Full detail in Part 1.

The Updated Table, 44 Models

Same table as Part 2, now with 44 rows. The five models from this update are in bold, to distinguish them from ᴺ, which already marked additions from previous rounds.

Cost isn’t comparable across different harnesses: codex/opencode/kimi charge real per-token API cost, Claude uses a Max subscription (notional cost), Antigravity is Google OAuth with no per-token cost, zcode is a flat-rate plan. Only compare cost within the same harness.

Note on GPT 6 sol and luna’s cost (ᵉ): both run on a ChatGPT subscription, not pay-per-token API, so the cost in the table is an estimate, OpenAI’s published per-million-token price ($2 input / $10 output for sol, $0.10 / $0.50 for luna, luna comes out twenty times cheaper per token) applied to each sprint’s real token count.

This Round’s Surprises

Grok 4.7 Regressed, and Not Just on the Score

Grok 4.7 closed at 83.5, fifteen points below its own Grok 4.6 (98.5), my overall runner-up. And no, it didn’t make up for it on cost or speed either: $27.94 and 120 minutes against $13.00 and 67 minutes for 4.6. More expensive, slower, worse score. A regression on all three axes at once is rare in this benchmark.

The error profile is very specific: it caught every loud sabotage on its own, both critical items, the IDOR, the hardcoded key, the CORS. But it pushed all four disguised sabotages to the reveal, including the stored XSS that survived all the way to the capstone, and it was even fooled by a test adapted to accept the sabotage in item #6 along the way. Strong on the obvious, blind on the disguised.

Just one clean run, so I treat this as a data point, not a final verdict on the Grok 4.7 family. But the data I have today is: worse, more expensive, slower.

Opus 5.5 Is the Opposite: Same Top Score, a Quarter of the Cost

Opus 5.5 walked straight into the 100-point club, tying Astra, Opus 5, Fable 5, GPT 5.6 sol and terra, and GPT 5.5. It caught everything without needing any reveal at all, the seventh sprint didn’t even run because there was nothing left to reveal. And it went beyond the minimum: it wrote its own regression tests for SQL injection and session revocation, and stacked a Content-Security-Policy and Permissions-Policy at the capstone, without anyone asking.

The number that actually matters: $16.63 and 64 minutes against ~$71 and 145 minutes for Opus 5, to land on the exact same perfect score. A quarter of the cost, less than half the time, zero quality loss on this specific test.

GPT 6 Sol Regresses on Vigilance, GPT 6 Luna Nails It

GPT 6 sol closed at 91.0, tied with Claude Sonnet 5, nine points below its own GPT 5.6 sol (100.0), but costing much less, ~$8.35 against $20.37, in less than a quarter of the time. A real regression on vigilance, a real gain on cost and speed.

GPT 6 luna is the standout of the pair. It scored 95.5, technically tying or slightly beating GPT 5.6 luna (95.0), and costing ~$0.84 against $10.04, roughly twelve times less. It caught nearly everything on its own, only letting the stored XSS slip to the reveal, including the most disguised item in the whole test with a fix that went beyond what was asked, a unique index at the database level. Equal-or-better score, cost collapsing: the same class of result Opus 5.5 delivered.

Sol’s error pattern is the same as Grok 4.7’s: it catches the obvious on its own, pushes the disguised to the reveal. Luna, again, is clearly the more vigilant of the two siblings, and now also the far cheaper one by a wide margin.

MiMo V2.6 Pro: The Generational Leap That Holds Up

From Xiaomi, the V2.6 Pro jumped from 86.5 to 92.0 over its own predecessor, the V2.5 Pro, for just $1.22. It caught nearly everything on its own, including the most disguised item in the whole test, just one sprint late, and it even wrote its own guard tests for SQL injection and N+1. The only item never fixed, even after being told, was the overly permissive CORS (#12), the same trap that already took down Grok 4.5, Gemini 3.7 Flash, and GLM 5.2 back in Part 2. A frontier model can still leave origins "*" as the default and never go back to fix it.

What the External Benchmarks Say, and Where They Disagree With Me

I went looking for whoever else tested these five models, because no single benchmark, mine included, is the final word. Here’s what I found, and where the reading matches or doesn’t match mine.

• Grok 4.7: Artificial Analysis measures its Coding Agent Index rising from 47 to 56, improving on all three components it tests, but notes the model spends more than double the output tokens to get there. In other words, on a general coding-capability benchmark, 4.7 improves. On my test of vigilance under disguised sabotage, it gets worse. Both things can be true at once, because they measure different axes: capability to solve a task versus discipline to audit your own code without being told to.
• Claude Opus 5.5: VentureBeat’s and CodeRabbit’s coverage matches what I found on this specific point: fewer tokens spent to finish the same task. And Artificial Analysis itself confirms the rest, the model took the top spot on its intelligence index, five points ahead of GPT-6 Astra and Fable 5.1. Here the outside reading confirms mine.
• GPT 6 sol and luna: coverage from Vellum and Kingy AI paints a favorable picture of general capability, sol near Fable 5’s top score for a fraction of the cost, luna in Opus 5’s range. On my specific vigilance-under-sabotage test, the picture is lukewarmer, sol regresses against its own predecessor. Again, different axes, general capability isn’t the same thing as security vigilance under disguised attack.
• Xiaomi MiMo V2.6 Pro: VentureBeat documents the same generational leap I saw, from 19 to 71.9 on DeepSWE, from 16 to 53.1 on AutomationBench. Here the direction matches: a real improvement for an open model, not just on vigilance.

Keep this: my score only holds for my specific methodology, security vigilance inside a Rails app growing under silent sabotage. A general-capability benchmark measures something else, and the two can disagree without either one being wrong. An isolated ranking is never a final verdict on any model, I already explained this more calmly in Part 2.

Is It Worth Upgrading to the New Version?

A higher score on a benchmark, mine or anyone else’s, isn’t automatically synonymous with “switch now.” Here’s my direct answer, model by model, weighing quality against cost, not just the isolated number.

• Do you use Grok 4.6 today? Don’t move to 4.7 for security. It’s worse, more expensive, and slower on all three axes I measure. Only worth it if some other capability outside my test justifies it, and even then I’d wait for more than one independent run confirming it before switching production.
• Do you use GPT 5.6 sol? Moving to GPT 6 sol costs a lot less, but loses nine points of vigilance. Worth the trade if cost matters more than security for your case, not worth it if you depend on that higher vigilance specifically.
• Do you use GPT 5.6 luna? Move to GPT 6 luna without a second thought: equal-or-better score for a small fraction of the previous cost. Along with Opus 5.5, this is the other rare case of a clean win on every axis.
• Do you use Opus 5? This is another case where the answer is yes without reservation: Opus 5.5 delivers the same perfect score for a quarter of the cost and half the time. A gain on every axis I measure, at the same time.
• Do you use MiMo V2.5 Pro? Worth moving up to V2.6 Pro, the quality jump is real and the cost stays in the one-dollar range. Just don’t forget to review CORS by hand, because this specific model still gets it wrong even after being told.

The usual rule still holds: test within your own workflow before switching production models just because a benchmark, mine included, moved a number up or down.

Closing With an Irony I Didn’t Manufacture

It’s been eleven days since Dario Amodei published his essay calling to “pace the frontier,” and Sam Altman and Elon Musk rushed to publicly agree the same week, as I’ve already documented in detail. And what happened since? Exactly those three, Anthropic, OpenAI, and xAI, shipped Opus 5.5, GPT 6 sol and luna, and Grok 4.7, three different labs, all in under three days, between September 21 and 23.

I didn’t write this coincidence, I just noticed it. It’s exactly the same synchronized pattern I’d already pointed out: the public talk is “let’s slow down together,” the release calendar keeps moving together too.

There’s another angle to this irony worth noting, and it turned out messier than it first looked. Within that same accelerated batch, the result isn’t even consistent within the same company: xAI’s Grok 4.7 regressed badly, and OpenAI’s own GPT 6 sol also regressed on vigilance.

Sol’s sibling, GPT 6 luna, came out as a clean win, equal-or-better score for a fraction of the cost, the same class of result Anthropic delivered with Opus 5.5. Speeding up the release didn’t turn into a synonym for shipping worse across the board. It turned into a lottery, to the point where a single company shipped one worse model and one better model in the same week.

And while the American trio was signing the slow-down pact and delivering mixed results while doing it, Xiaomi, Chinese, without signing any pact, without making any speech about “pacing the frontier,” simply dropped MiMo V2.6 Pro with a real quality leap over its own predecessor. Nobody on the Chinese side promised to slow down. And, at least in this sample, nobody on the Chinese side did.

It’s better to deploy faster, because fixing things afterward became fast and cheap.

That sentence is the whole article. Everything else is just me justifying why it’s true now, even though it wasn’t true a few years ago, and showing how I’ve been acting on it myself since February.

The Premise Changed, and Most People Haven’t Noticed

I started saying this on X like this:

Fixing got cheap. Stop holding things back. Just ship it and fix it after. “Ugh, but I risk shipping a bug.” The wrong premise is thinking that staring at every line is going to make any difference at scale. And idle code doesn’t improve on its own. Only used code lets you find out if it needs improving.

And I followed up in the second post, because the first read of anything I write in 280 characters always comes back angrier and more literal than intended:

Get the concept (this is an X post, not a blog post, stop reading it literally). I’m not saying AI doesn’t produce bugs. It does, obviously. But at this point, not more than you, the human, also produce. […] The old excuse was that when a bug ships to production, it sits there longer than it should, because it never becomes a sprint priority to fix. Now THAT’S NOT TRUE ANYMORE, the premise changed: fixing the bug afterward is FAST and CHEAP.

That’s the entire point. It isn’t about AI being infallible. It’s about the cost of fixing a mistake having collapsed, while most teams still run their review process as if that cost were still high.

With today’s frontier models, Opus, Sol, K3, and company, the LLM makes fewer mistakes than most human developers I’ve seen go through a code review. When it does err, the overwhelming majority of the time the mistake isn’t the model’s. It’s whoever was driving the harness. The LLM does what you ask, and it doesn’t do what you didn’t ask.

If you don’t know what to ask for, you won’t produce anything useful, no matter how good the model is. That’s exactly why only a real software engineer can get real production-grade software out of an agent: someone has to know how to break down the problem, validate the result, and recognize when the request itself was poorly framed. AI doesn’t replace that judgment. It multiplies what that judgment already knew how to do.

This Should Have Been the Norm for 25 Years

The discomfort I feel with bureaucratic review processes isn’t new, and the industry should have solved it a long time ago. Kent Beck and Ron Jeffries invented Extreme Programming in 1997, on Chrysler’s C3 project, and Beck published the book in 1999 with continuous integration, TDD, and small-batch delivery as core practices. In February 2001, seventeen developers locked themselves in a resort in Snowbird, Utah, and walked out with the Agile Manifesto. That’s twenty-five years ago.

Martin Fowler’s mantra for continuous integration has always been “if it hurts, do it more often,” and the book Continuous Delivery, by Jez Humble and Dave Farley, formalized the idea that small, frequent deploys reduce risk, not increase it. DORA’s annual State of DevOps reports have documented this with numbers for over a decade:

• elite teams: deploy on demand, multiple times a day;
• low-performing teams: deploy once a month, sometimes once every six months.

And the finding that’s hardest to swallow if you’re afraid of deploying fast: historically, more frequent deploys do not increase the failure rate. They reduce it. Teams that deploy more, on average across the last decade of reports, fail less and recover faster when they do fail.

Even with all of this documented, measured, and published for over a decade, a large chunk of the industry keeps damming up change into big batches, with slow manual review, and calling that “prudence.” It was never a settled matter even inside the agile community itself: in 2014, Kent Beck, Martin Fowler, and DHH (Rails’ creator) spent weeks publicly debating whether TDD was dead, without reaching a clean consensus. In other words, the very community that invented these practices still argues today about how much rigor is enough.

What I’m saying is simpler than that whole debate: whatever your testing rigor level is, an LLM reviewing your diff before merge is fast, and reverting a bad deploy became trivial with containers and git. The math that’s left is: how much time do you spend holding back code, versus how much time it would take to revert if it went wrong? If the second number is smaller, you’re paying a toll for fear, not for prudence.

This Isn’t “Ship Anything”

This is the part that always gets lost: I’m not saying to deploy with zero engineering practice underneath. I’m saying the opposite. This is only safe after you already have basic software engineering working, which I document exhaustively on this blog for years: automated tests, CI running on every push, real revert capability, observability to know when something broke.

80/20. Ask the LLM to do a minimal review and ship it. If it breaks, revert (that’s what containers, git, etc. are for). If you can’t revert, your infra sucks, go fix it yesterday.

That last sentence is the whole filter. If your infrastructure doesn’t let you revert fast, the problem isn’t deploy speed. It’s your infrastructure, and it needs fixing before any conversation about going faster. Nobody gets a pass to skip having real rollback capability. What changes is that, once you have it, holding back code waiting on human review became pure waste.

The Minimum Recommended Environment

I’ve already said “basic software engineering” several times in this piece without spelling out what that means in practice. Here’s the list, no hedging. Nothing on it is optional, and none of it got expensive to maintain:

• unit tests covering isolated logic, TDD or not, as long as they exist;
• scenario and integration tests covering concurrency, partial failure, and conflict, the things unit tests alone don’t catch;
• CI that builds and tests on every change, ideally for every platform you actually ship to;
• hostile automated review before merge, like pr-audit, which treats a contributor’s claim as unverified until proven;
• feature flags to control the blast radius of who sees a change first;
• real staging as a gate before any real user sees the change;
• fast revert capability, with git and containers, and observability to know when something broke;
• in compliance, financial, or health environments, one extra layer: heavier scenario testing, and more than one lock per identified failure mode.

What the diagram tries to make clear is that speed and safety aren’t competing with each other here. They come from the same stack of layers, and each layer in that stack is what gives you permission to stop holding back code waiting for manual review.

Where This Math Doesn’t Work Out

I’m not going to pretend I didn’t get good pushback in the replies on X, because several of them hit on something real that I skipped while writing fast for Twitter. It’s worth answering for real instead of just validating whoever agreed.

1. Reverting code doesn’t undo a consequence that already left your system. If the bug already processed a wrong payment, moved money to the wrong account, or controlled an elevator, reverting the deploy doesn’t undo what already happened. “If you can’t revert, your infra sucks” covers the technical side of deploy and rollback, not the case where the damage already left your system before you could react. Payments, health, and anything touching physical safety fall into this category, and there the review bar before shipping has to get stricter, not looser, because the cost of reverting stopped being technical.

2. Databases break the “reverting is easy” analogy. Reverting a binary is trivial. Reverting a migration that already ran in production, already moved data, and maybe already dropped a column, is not. There’s a pattern for this, expand-contract: add the new shape without removing the old one, wait for every consumer to migrate, only then remove the old one. This isn’t about trusting AI more or less. It’s about state that doesn’t undo with git revert.

3. Libraries and public packages don’t run on the same math as a closed application. When my own code breaks, I fix it on the spot. When a lib that hundreds of projects depend on breaks, the people who eat the cost are people who don’t even know I exist, with a much slower reaction time than mine. That’s why my pr-audit skill, which I describe further below, treats semver classification as a version gate: a breaking change ships in a major, never in a patch, because there the real review isn’t about catching a bug, it’s about not blindsiding whoever depends on you without warning.

4. App stores knock the speed down too, and my own frank_yomik example hides that instead of solving it. It publishes the APK straight as a GitHub Release, without going through the Play Store. If you distribute through Google’s or Apple’s official store, their review sits in the middle of your process, and you don’t control how long it takes. This article is about removing the bottleneck that’s inside your own control. The app store isn’t inside it.

5. The most honest point of all: every piece of evidence I bring here comes from projects where I’m the sole owner and I decide alone. I have no way to prove, with my own numbers, that the same pace works inside a large company, with Legal, Compliance, and a product manager in the middle. What I’d argue generalizes is the principle, not the exact pace: build real revert capability, automate the mechanical checking, and measure the result against real usage, not against your own sense of security. How much speed you apply on top of that depends on how much a mistake costs in your specific context, and nobody from outside answers that for you.

On the first point specifically, it’s worth opening up more, because “stricter bar” can’t stay vague. It means real scenario testing, not just unit tests. ai-memory itself has more than a thousand integration tests spread across its modules, and most of them don’t ask “does this function return the right value?” They ask things like:

• what happens when two writes race for the same file at the same time;
• what happens when the disk fails mid-way through a delete operation;
• what happens when two sessions try to consolidate the same data at the same time;
• what happens when the process dies before it finishes writing.

An environment with compliance, financial, or health requirements needs this level of testing: every identified failure mode has to have a mitigation, and more than one layer of checking, never a single lock.

And here’s the point that holds up this article’s entire argument even in those environments: this bar doesn’t change depending on who wrote the code. A human developer’s mistake takes down a payment system just as well as an LLM’s mistake does, and nobody gets a free pass for loose review just because a senior human wrote the line. The bar of scenario testing and double-layer protection is about the problem domain, not about who, or what, produced the code.

Feature flags and staging environments are the other half of that double layer, and they also changed cost. Flags used to be a pain to maintain: dozens of them, each one needing to be born, monitored, and removed once it stops making sense, and it’s easy to pile up dead flags nobody remembers the purpose of. That got cheap too: I ask the agent to audit every flag in the project, find which ones have been stuck at the same value for months, and generate the cleanup PR. Maintenance that used to take a whole afternoon turns into a task that takes minutes.

With that solved, the layered defense becomes simple to assemble: staging to catch obvious errors before any user sees them, feature flags to limit the blast radius of who sees the change first, and scenario tests to cover what staging alone doesn’t simulate. Three layers, none of them expensive to maintain.

How I’ve Been Doing This Since February

I started accelerating my output with AI in February this year, and the question I wanted to answer was simple: can you use the new generation of frontier LLMs to code full-time, at real production pace, not just for a demo? The answer turned into my biggest LLM benchmark, more than 110 articles published since then, and this, as I wrote recently:

Since February I’ve produced more than 110 articles on my blog. Most of it came from the 30+ open source projects I built and maintain (43 if you count other people’s contributing projects, like Flea or Omarchy). 1.6 million lines of code produced. By myself (with agents, of course).

The projects I use the most in my own day to day, and that survived the “am I still maintaining this months later” test, are three:

• ai-jail: an OS sandbox for my coding agents;
• ai-memory: long-term memory for the agents;
• ai-usagebar: a bar widget to monitor LLM usage.

Two of them already have a decent landing page:

ai-jail went live today. ai-memory is up at aimemorybr.netlify.app because the aimemory.io domain transfer will still take a few more days.

Both run on Netlify. That’s not a tool recommendation, it’s just where I already had an account set up. The point that matters here isn’t the platform, it’s the flow: to put a new version live, the command is git push. There’s no manual step between the commit and the updated site.

My newsletter, The M.Akita Chronicles, already has more than 15,000 subscribers:

And the newest effort is converting blog posts into podcast episodes. The whole thing runs inside Discord, with a bot that generates the preview and waits for my command:

One /publish-podcast 2 later, the episode is published and the transcript already has a URL:

From finished text to published audio, the entire process is me typing a slash command in a Discord conversation. That’s the effort bar I use to measure whether a process is good enough: if publishing takes more than one command, some step is still too manual.

Does the Quality of the Generated Code Hold Up?

Every single day someone shows up worried that automating too much with LLMs is going to dump bad code, or worse, dangerous code, into production without anyone noticing. Given that the engineering underneath is the one I described in the previous section, automated tests, CI, feature flags, staging, fast rollback, my actual experience doesn’t back up that fear.

I’ve produced more than 1.6 million lines of code with LLMs since February. I have never once seen an LLM do something randomly destructive that I didn’t ask for. Quite the opposite: in my LLM Benchmark v4, I deliberately built an adversarial scenario, a subagent planting fourteen real CVE-based sabotage items, hidden inside a project that grows across seven sprints. Most of the tested models found and fixed the sabotage on their own, without me having to point it out.

That’s anecdotal. It doesn’t prove anything beyond what that specific benchmark measures. But after all that volume, the impression that sticks is consistent: today’s LLMs produce stable code most of the time, comparable to what a decent human developer would produce, just a lot faster, and on average, more accurate.

Cross-Platform CI/CD With Zero Effort

I went and counted how many of my ai-* and frank-* projects (18 repositories total, among the ones with a README) already have automated build and deploy. Fourteen have CI configured, and most of them build and publish to multiple operating system platforms from a push made on a Linux machine.

The most complete example is ai-memory itself: a single tag push triggers a workflow that, in the same run:

• builds Linux x86_64 and arm64;
• builds macOS arm64 and x86_64;
• builds Windows x86_64, packaged with PowerShell;
• publishes a multi-architecture Docker image;
• pushes the package to Arch Linux’s AUR.

ai-usagebar follows the same build pattern for Linux and Windows, and also publishes two AUR packages, to crates.io, and to a Scoop bucket for Windows.

Then there’s frank_yomik, my manga reader with OCR and translation via a local LLM, which is the example I wanted to show that Android fits into the same pipeline. A v* tag push triggers five parallel builds:

• Linux;
• macOS;
• Windows;
• browser extension;
• Android, on an ordinary Linux runner, decoding a signing keystore stored in GitHub secrets and producing the already-signed APK.

At the end, one job waits for all five to finish and publishes a single GitHub Release with every binary together. It builds for four different systems, including Android, without ever leaving Linux.

Not everything came out perfect on first look, and I’d rather admit that than pretend it’s all clean: frank_sherlock and frank_karaoke publish the release as a draft by default, which blocks the automation that follows (like the AUR push) until someone clicks “publish” by hand. I found this hole while writing this piece.

A neighboring project, frank_scanlation, had already solved a similar but different problem: GitHub suppresses the event that triggers automation when the release itself was created by the bot’s own token, so its workflow triggers the next step by hand, via workflow_dispatch, from inside the job itself. Neither fix is hard. I’ll apply both to the projects that still need it this week, because that’s exactly the kind of accumulated friction this whole article is telling you not to leave sitting.

And speaking of hidden friction: ai-memory has Windows-specific code (NTFS file index, a native PowerShell hook) that only really runs on GitHub’s windows-latest runner, about 1000 seconds against 250 on Linux for the same suite. I went as far as setting up a local Windows VM via dockur/windows, thinking I’d gain speed.

I didn’t: the workload is dominated by disk I/O, and the result came out slower than GitHub’s runner for the full suite. It only turned out useful for fast iteration on a single isolated crate. A good tool is one you measure, not one you assume, even when the measurement contradicts your initial bet.

The Automated Review That Holds All of This Up

I already detailed my review skills in a recent article, so I’ll summarize here:

• pr-audit audits every PR before merge with the principle of evidence over narrative: nothing the contributor wrote is true until proven, with a hostile checklist for malicious code, exposed credentials, semver breaks, and prompt injection attempts;
• iss-audit applies the same distrust to issues, never executing a command pasted by whoever reported it;
• github-resolution turns what got approved into merged code, one ticket at a time, with a regression test written before the fix, and the rule that “slop is a defect”: no speculative abstraction, no forgotten TODO, no weakening a test just to pass faster.

That catches the most expensive category of garbage inside the same diff. It doesn’t catch duplication spread across different PRs, because the audit looks at one isolated diff, not the whole project’s architecture. That part still depends on me noticing the pattern repeating and telling it to consolidate.

By my estimate, this handles about 99% of review and resolution work on its own, with me supervising and deciding on the cases that are left. But that’s not the main point. Automated review is necessary, but it alone proves nothing about whether the software is good.

What Actually Keeps This Grounded: Real People Using It

Here’s the part I think is the most important in the whole article, and the one that usually gets skipped over: your own review isn’t worth as much as you think. What’s worth something is published software being used by real people, giving real feedback.

I went and checked ai-memory’s real numbers on GitHub, without rounding up to sound impressive:

• issues: 287 total, 249 came from people who aren’t me, 86.8% of the total;
• pull requests: 562 total, 371 from outside, 10 of them Dependabot’s automatic bumps and the other 361 from real people, roughly 121 distinct contributors.

The two people who contributed the most alongside me sent 64 and 43 PRs each, on their own.

That’s what keeps the project grounded. It isn’t the LLM being a good reviewer, nor me being a good reviewer. It’s hundreds of real people actually using it, finding cases I never imagined, and sending real issues and PRs. Without that, my automated review skills would just be confirming my own assumptions back to me, no matter how rigorous the rules look on paper.

It doesn’t matter how good the AI you use is: if nobody uses what you published, you have no real feedback, and without real feedback there’s no real improvement. There’s only your own opinion about your own work, validated by nobody but yourself.

The Conclusion That Matters

Real software engineering needs fast, near-zero-effort deploy. It isn’t optional, it isn’t a luxury for a big team, it’s a prerequisite. And if reviewing and shipping to production is taking longer than it took to develop the feature, you’re not just wrong.

You are the bottleneck.

Excuses and justifications aren’t worth anything at that point. Use that time to fix yourself and your process, because only real-world results matter, not the sense of security from having reviewed a diff line by line that nobody else is ever going to look at again.

If the premise changed, I change today. Not tomorrow, not next week. And if yours hasn’t changed yet, the problem isn’t AI. It’s you holding back your own work.

And the recommendation comes right up front, because it’s the reason this article exists: don’t use other people’s skills. Read that again. My skills are published on my my-skills repository and the repo description warns you: “not tailored for general usage”. They’re there so you can read the reasoning, understand the structure, and go build your own. Copying somebody’s skill and running it blind is the digital equivalent of signing a contract without reading it. Worse: it’s installing someone else’s instructions to run with your permissions, on your repositories, with your tokens. That’s prompt injection you install voluntarily and say thank you for.

What a skill really is

No magic involved. A skill is a directory with a SKILL.md inside: a header with name and description, and a body of text that is, essentially, a well-written prompt about how to execute a task. Sometimes it ships with a scripts/ folder of helper scripts, so the LLM doesn’t have to recreate them from scratch every time.

Skills are just prompts.

That’s it. A prompt you versioned, refined, and kept one command away. The only difference between a skill and that giant prompt you paste into the chat every day is that the skill lives in a file, with a name, and the agent knows it exists.

And here’s the answer to the question everybody asks: “doesn’t that fill the context with junk?” The good news is the harnesses do this right. Claude Code uses what Anthropic calls progressive disclosure: at session start, only each skill’s name and description go into the system prompt. The SKILL.md body only loads when the agent decides to invoke the skill. OpenCode does the same: lists name and description, loads content on demand through its own tool. Anthropic itself documents the design: metadata first, body only when needed.

That solves the text cost, but only half the problem. Every installed skill still takes a line in the listing, and more importantly: it competes for the agent’s attention when it decides which tool to use. Anthropic again, on the engineering blog: “more tools don’t always lead to better outcomes”, and their official recommendation is a few well-designed tools for high-impact workflows. My version of that rule is more acidic:

An idle skill is debt.

You change your process, the skill ages, and one day the agent executes a rule that no longer makes sense in your workflow. Fewer skills, better written, reviewed often.

The only third-party skill that survived here

Having said all that, it would be hypocritical to pretend I live on homemade skills alone. One survived my scrutiny and runs on every post on this blog: Humanizer.

It exists because the Wikipedia community maintains a public, brutally honest catalog called Signs of AI writing, kept by WikiProject AI Cleanup, documenting the “tells” of AI-generated text one by one: the rule of three, the “not just X, but also Y”, the puffery, the emoji-decorated bullet. Humanizer turns that catalog into a procedure: the agent receives the text, marks the tells from strongest to weakest, rewrites preserving every supported claim (losing a fact is an error, inventing a fact is an error), and does a final read-aloud check. Without changing what was said, only how it was said.

If you produce content, you know why I insist: text that smells like AI is becoming the new “Dear Sir”, the mark of a rushed form letter. I run Humanizer on every article before publishing, including this one. But note the detail: I merged version 3.0.0 into my repo, with my own adjustments. I read the source material before trusting it. That’s the pattern.

What I do with this every day

Now the practical part. I maintain and monitor over 40 repositories on GitHub. My own projects, forks I maintain, tools that became daily dependencies. To stay sane, my second monitor runs tclock (my fork of clock-tui, with Evangelion themes, because why not) and next to it ghpending, a Rust CLI that lists open issues and PRs from all my repos at once.

In the photo you can see: ai-memory and ai-usagebar with open PRs and issues. Every morning I open the harness on each project with ai-memory run in its directory. ai-memory deserves a paragraph: it’s my long-term memory project for agents, a persistent Markdown wiki that captures session observations, consolidates decisions and gotchas into pages, keeps workstreams across sessions and handoffs between agents. Every project of mine has its own memory, with the architectural decisions and the traps only that codebase has. Without it, every agent session starts from zero like an intern with amnesia.

From there the daily ritual is almost childishly simple. I write, literally:

run pr-audit and iss-audit, then run github-resolution

When a release is ripe, I add then run the release skill. On days when the only queue is Dependabot’s automatic bumps, it’s run pr-bump and release and done. One line of instruction from me. The heavy lifting lives in the skills. Let’s get to them.

pr-audit

PR audit before merge, and the central principle is: evidence over narrative. Nothing the contributor wrote is treated as truth; everything is a claim to verify, with a claims ledger (confirmed, partial, unsupported). The rest of the skill is a hostile checklist:

• Change gate: hunk inventory looking for executable bits, symlinks, bidirectional characters, homoglyphs, pull_request_target workflows, unpinned actions, typosquat dependencies, lockfile drift.
• Credentials: unexplained credential access is an instant blocker. Execution happens in an isolated worktree, with no host credentials.
• Semver: the changelog category is a version gate; breaking changes go in major, never in patch. A rule that has saved me more than once.
• Prompt injection: text inside a PR trying to “change the audit rules” is just one more suspicious data point to investigate. It’s an audit finding, obviously.

iss-audit

The same distrust, applied to issues:

• Never run a command copied from an issue. Attachments only go through an isolated scanner.
• Formal separation between what was observed, what was expected, the reporter’s diagnosis, and the proposed fix.
• Reproduction with synthetic data in a disposable sandbox, with resource limits for DoS claims.

And two golden rules I love: “a number that moves is not a pinned number” (async lag looks like a bug) and “the obvious owner may be innocent” (query the suspect population; if it comes back empty, the cause is elsewhere).

github-resolution

The execution phase: turning what the audit approved into merged code. The rules:

• Only execute approved tickets, one at a time, with a regression test written before the fix. “Slop is a defect”: no speculative abstraction, no TODOs, no drive-by refactors, no weakened tests.
• If I resolved more than three tickets in a session, run the post-audit over the whole range before pushing, no exceptions.
• A ticket closes when the fix is merged with green CI on that exact SHA, never left hanging waiting for a release.
• Every unresolved issue gets logged with the reason. Compressing it into a number is forbidden.

pr-bump

The fast path for Dependabot’s dependency bumps. The rule that saves me the most time: batch, never serialize. Instead of merging PR by PR, one consolidated bundle update to the latest compatible versions, a single CI pass for the whole set, fix forward on whatever breaks.

But there’s a supply-chain floor:

• Every dependency must resolve on the public registry (rubygems, npm, pypi) with the expected name, version, and checksum.
• Git source, path source, typosquat-adjacent name, new install hook: full stop, it goes to a complete audit.

bundler-audit will happily pass a well-formed trojanized gem. That’s exactly the case the floor covers.

release

Cut a version only when I ask. The iron rules:

• The version comes from the changelog classification: fixes go in patch, additive changes in minor, breaking in major. A number that conflicts with the classification forces me to confirm; silent bumps are forbidden.
• Green CI gate on the tag’s exact SHA. Never tag with pending or skipped CI.
• Annotated tag, updated changelog.
• Never rewrite a published tag. Never cut a release nobody asked for.

The numbers, because numbers don’t argue

Counting right now, on the day I write this article, across just three projects (ai-memory, ai-usagebar and ai-jail):

• ai-memory: 430 merged PRs, 260 closed issues
• ai-usagebar: 137 merged PRs, 34 closed issues
• ai-jail: 42 merged PRs, 81 closed issues

Total: 609 merged PRs and 375 closed issues, over a thousand items. Nearly all of that was handled by these skills, with me supervising and deciding. One person, no team, keeping three active projects at this volume. That’s the multiplier.

And they weren’t born ready

Open the my-skills commit history and you see skills being polished the hard way. July starts the repo with the basic audits. Late July: “harden audit skills and add security audit”. August: “consolidate agent skills from all harnesses into this repo” and, days later, github-resolution is born as the execution phase of the audits. September: “fold in release-discipline and audit lessons”.

And just yesterday, look at the level of fine-tuning in pr-bump: “batch all bump PRs into one consolidated verification pass, fix forward on failures” and “require deferred/declined PRs be left in a clean state”. Two rules that became skill text. Neither came from a blog post or somebody’s opinion: they came from me face-planting into the exact error the rule now prevents.

Every line in these skills is a scar.

That’s how you write a good prompt: watch the result, decide to adjust, adjust again.

Skills are little. Knowledge is a lot.

Now the counterpoint, because I don’t want you leaving here thinking skills are the answer to everything. I depend on them very little. What actually carries my workflow is something else.

Every project of mine has its own AGENTS.md, instructions specific to that code, and its ai-memory memory with decisions and gotchas. And the pattern I use most day to day isn’t even a skill: it’s pointing at a living example. When I wanted automated AUR packaging for clock-tui, I didn’t write a “how to make an AUR package” skill. I opened the harness and said:

make a similar github action to build and publish AUR packages
as we did in ~/Projects/ai-memory

The agent went there, read the workflow that works, and reproduced the pattern in the new project. A skill for that would be worse: AUR changes, GitHub Actions changes, the example gets stale. A living example in a repo I maintain is always up to date by definition, because when it breaks, I fix it.

Every project becomes its own knowledge base.

And this generalizes, because harnesses can consume any text base. My entire Linux PC is documented and reinstallable from a config repo of mine, and my homeserver is fully described in runbooks. Those two are not on GitHub: they live on a private Gitea server I maintain, because that’s the kind of thing I don’t want leaking in public. When I want to change something at home, I say:

add integration to my Plex server as documented in my homelab runbooks

That’s not a skill. It’s organized, versioned, searchable knowledge. Skills are for procedures that repeat the same way. Knowledge is for everything else.

The recommendation worth more than any skill

If you take a single practice from this article, take this one: have your agent write down everything that didn’t become code. The research it did, the decision you made and why, the comparison with the rejected alternative. Without that, the knowledge evaporates at the end of the session.

My ai-memory, for instance, has a docs folder with competitor research I commissioned before every big decision:

• The Karpathy wiki pattern, the project’s intellectual origin.
• A 2026 landscape survey, covering Zep/Graphiti, Letta, Mem0, and Google’s Open Knowledge Format.
• Deep dives on each competitor: Hindsight (the one with a paper and funding, which ended up validating my design bets), cognee (pretty graph pipeline, ugly gateway bug), agentmemory (I kept the ideas, threw out the substrate), basic-memory.
• And my favorite, the MemPalace autopsy: 47k stars, viral, and a tracker clogged with index corruption and lost data. The fairy tale in reverse that confirmed for me why compiling memory beats stockpiling verbatim.

Dozens of documents like that. Every hour of research an agent did for me became a permanent asset, reusable in any new project. The more you produce and document, the more intelligence you accumulate.

It’s compound interest for knowledge.

Nobody knows what you need

And there the circle closes, back to where we started: why I don’t recommend copying anybody’s skills, mine included.

Because my PR audit skill is molded around my projects, my languages, my risks, my level of supply-chain paranoia. It reflects what I, Fabio, maintaining the repositories I maintain, on the infra I have, with the goals I have, needed to write into it. Your project has another stack, other risks, another tolerance level. You may need checks I don’t, and vice versa. Experience is only relevant to whoever lived through its constraints. That changes from person to person, and that’s exactly why a third-party skill used blind is a gamble.

That’s the thesis of my 2019 video, Don’t Outsource Your Decisions: the most important lesson of your life (in Portuguese): nobody but you knows what you need.

I don’t ask anybody. I research, I build mine, and the skills are the versioned record of my decisions. The beauty of LLMs is they speed this process up tenfold: the research that took a week now takes an afternoon, and the cost of building your own tool fell through the floor.

A good skill is your own. Mine are on my-skills so you can read the reasoning. Then close the tab and write yours. I don’t waste time asking for other people’s opinions. Neither should you.

Let me get my rule out of the way before I get into the specific case, because it matters more than the case itself.

My Rule: Don’t Use Anything Until Not Having It Hurts

After producing more than half a million lines of AI-assisted code over the last few months, dozens of projects live, a whole benchmark rebuilt from scratch three times, my recommendation for anyone who codes is direct: don’t use any orchestration tool or middle layer until you feel it in your bones that you need the extra help. Use whichever harness you already like, Codex, opencode, Claude Code, whatever, directly, no middleman, and forget the rest.

It’s not even worth spending time collecting someone else’s skill packages to stack on top of your agent. Build, train, and gather your own experience for your own use case. Nobody else on earth has your project, your context, your codebase. Someone else’s skill package was designed for their problem, not yours.

I already hit this note a month ago: once a technology turns into a commodity, the money migrates to taxonomy. Somebody coins a fancy name for chaining API calls, and out of nowhere a course, a certification, and a consulting practice spring up around that name. TypeSafe is just one more chapter of the same story, so it became a good excuse to show how I decide whether something’s worth even opening the tab.

Keep this: don’t use an orchestration tool or middle layer until you feel in your bones that you need it. Build experience for your own use case; don’t collect someone else’s skill package.

The Trigger: TypeSafe

I opened their site and their launch post, Jev. My first impression, pretty superficial and raw: I didn’t like it and I didn’t trust it. The homepage is a pile-up of AI jargon, Kahneman here, Jevons there, decorative glyphs, a “193x faster” figure with no benchmark sitting next to it. At first glance, none of it makes sense on its own.

And there’s a detail that always puts me on guard: any time the first thing someone says about themselves is “I’m a former OpenAI researcher” or something like it, I get suspicious first and ask questions later. It reads like an attempt to slap on a credential to look credible before any real proof. TypeSafe’s founder talks himself up too loudly right out of the gate, and that alone is reason enough to hit the skepticism button.

But a triggered skepticism meter isn’t a conclusion. It’s the starting point for actually investigating, not for dismissing something without a look. That’s the part I want to show here, kind of a practical lesson for anyone following along: how I use AI to speedrun preliminary research, how I read the fine print before signing off on any conclusion, mine or someone else’s, and how I move from initial distrust to real research, until I find out whether I was right or just being difficult.

And let me put this on record before going further: I could be wrong about some specific point ahead. If you’ve got strong counter-evidence, drop it in the comments. I’d rather get called out in public and correct the piece than push a scammy product on my audience just because I didn’t check carefully enough.

I Asked the AIs, I Didn’t Trust the AIs

I sent the same prompt to Grok and to Claude: research this thoroughly, my initial impression is very skeptical, “ex-OpenAI researcher” smells like slapping on a credential, the site is jargon top to bottom, tell me what the actual sales pitch is, what it’s for in practice, and whether their logic, underneath the jargon, even makes sense.

Both came back with long, detailed research, a similar story from each: the founder’s credential is real but inflated; the product is narrower and more coherent than the homepage lets on; and “can’t hallucinate” is a schema claim dressed up as a truth claim.

Except I’d already handed my own conclusion to both of them inside the prompt, so their agreement proves less than it looks like. Two AIs agreeing with the hypothesis I wrote into the request myself isn’t independent confirmation of anything, it’s an echo. That doesn’t invalidate what they brought back, but it meant I couldn’t stop there. I wasn’t going to publish this by stamping my approval on what two AIs told me. I asked for another round of research, this time with me checking every claim against a primary source: the actual paper, the launch post straight from the source, independent third-party tests, and whether “RLCD,” the name they use for their training method, is a real paper or just a nice-sounding name with nothing behind it.

What Jev Actually Is

Strip away the marketing, and Jev is a hosted classifier. You send a piece of text and a list of small questions, each with a fixed answer type:

• pick one option from a list;
• give it a score on a scale;
• answer yes or no, with a probability.

It doesn’t write anything, doesn’t plan anything, doesn’t converse. Code owns the flow; the model just answers the little question fast, in parallel, without generating free text.

As a product, that’s reasonable: plenty of people today send text to a chat LLM just so it returns a label inside a JSON blob, wait eight seconds, and pray the schema doesn’t break. Swapping that for a call that returns just the label, fast, is legitimate engineering. The idea was never the problem. It’s the packaging around it.

The Facts I Confirmed

Here are the findings that actually survived a primary-source check, not what an AI told me secondhand.

The founder’s credential is real, but the company’s own page inflates it in a way you can prove is inflated. Diogo Almeida is the 4th of 20 authors on the InstructGPT paper, published and peer-reviewed at NeurIPS 2022, the field’s heavyweight conference. A real author, a real position, a real paper. Except TypeSafe’s team page says this, word for word:

“Diogo co-invented RLHF and InstructGPT, the methods that lead to ChatGPT and GPT4.”

RLHF is from 2017, five years earlier, authored by a different team (Christiano, Leike, Amodei, and company), with no Diogo anywhere in it. He helped apply RLHF to train InstructGPT. He didn’t invent RLHF. It’s the same résumé inflation I already suspected before checking anything, just with proof now.

The funding is real: a $40 million seed round led by DCVC, a $200 million valuation confirmed independently by Forbes, not just by the company’s own press release. That’s serious money, not some guy running a scam out of a garage.

The launch post admits, in its own words, exactly what I suspected about the “zero hallucination” chart:

“Our number is not empirical. Schema matching is guaranteed, thus we can confidently add 0% into the plots.”

In other words: the zero doesn’t come from measuring anything, it comes from guaranteeing the answer always fits inside the allowed options. That stops the model from inventing a fifth category when only four exist. It does not stop the model from picking the wrong category with 93% confidence. It’s the same critique The Register published: the comparison isn’t fair because the output isn’t language, and a validly-typed answer can still just be wrong.

The post also admits what the ruler is for measuring “accuracy”: “We use the average of GPT-6 Astra and Fable 5.1 as the reference answer”, and goes on to admit that this “biases answers towards OpenAI and Anthropic’s models.” In other words, the answer key is the average of what two other companies’ models said, with bias admitted by the authors themselves, not objective truth measured by a human.

The only test run by an outsider, from the Every newsletter, found Jev about twenty-five times faster, 0.35 seconds versus 8.83 seconds per passage, and much cheaper than Claude Fable 5.1, except it caught six of seven planted defects against Fable’s seven of seven. Faster and cheaper, for real. Also weaker, for real.

And the finding neither Grok nor Claude fully nailed down, but that I confirmed myself: the name of their training method, RLCD, already exists. There’s a real, peer-reviewed 2023 paper, Reinforcement Learning from Contrastive Distillation, from a completely different team, with the authors’ own code published on Facebook Research’s GitHub, since two of the five authors are at Meta. It’s a different technique, with no relation whatsoever to what TypeSafe describes. Their RLCD has no paper, no published architecture, nothing to review. It’s a name borrowed from someone else’s research, glued onto a method that, until proven otherwise, is just a paragraph in a blog post.

Was My Suspicion Justified?

It was. The product idea is legitimate and reasonably coherent underneath the jargon: cheap, closed-form decisions at high volume is a real problem that today gets solved slowly and expensively, generating free text just to extract a JSON out of it afterward. Solving that fast has real value.

But “overblown claim without much practical difference” also holds up point by point:

• the founder’s résumé is inflated in a way you can prove with a publication date;
• the launch’s flashiest number, zero hallucination, is a schema definition, not a measurement;
• the accuracy benchmark’s answer key is the average of another company’s LLMs, not objective truth;
• the only outside test found the product weaker than the model it’s pitched to replace;
• the name of their most-cited training method collides with a real paper that has nothing to do with them.

That’s not reason enough to call it a scam. It’s plenty of reason to call it “a small product with a pitch way bigger than what it actually delivers.” And before I hand my data to yet another third party I don’t know, with a closed API, a waitlist, and no published paper backing up the fancy name, the risk-versus-benefit math doesn’t close for my use case.

Keep this: a real credential isn’t the same thing as an honest credential. Check the publication date before trusting the résumé, and check the paper before trusting the method’s name.

This isn’t a final ruling on TypeSafe or on Diogo Almeida’s résumé. It’s the result of the checking I was able to do with what was accessible right now. If you find the paper I missed, the independent benchmark I didn’t see, or any proof that knocks down a specific point, the comments are open and I read them. The goal here was never to prove I’m infallible. It’s to show how to go from “this smells like a scam” to “here’s the proof,” and to be willing to admit it when the proof points the other way.

The “193x faster” is calling out a bad way of using LLMs

Back to Jev: the shock of “193x faster than an LLM” only exists because the comparison is against one specific, and common, way of using an LLM. Send text to a chat model, ask it to return a label, wait for the model to write out a full natural-language answer, then have your code go fish that label out of a paragraph or a loose JSON blob in the response. That’s slow, expensive, and full of chances for the schema to break.

Jev exists precisely because that comparison is easy to beat. It doesn’t generate free text, it only answers fixed-type questions: pick one option from a list, give a score on a scale, yes or no with a probability. That’s what the documentation itself calls its primitives: Choice, Score, and Noul. Swapping “LLM writes a paragraph for you to extract a label from afterward” for “a call that already returns the finished label” is, yes, an engineering win. But that win doesn’t come from inventing a new category of model. It comes from no longer using a chat LLM to do structured classification work.

And this is where I think the widespread shock at Jev’s number reveals the actual bad habit: an LLM should never have been generating free text for a closed, repeated decision over the same kind of question. If your product needs to constantly decide between a fixed handful of categories, the right answer isn’t begging the LLM to “please answer with only the JSON” and hoping. It’s building, or using, an actual classifier for that specific job.

And that isn’t a new idea, and it isn’t exclusive to Jev. It’s existed long before it:

• a supervised classifier, a deterministic rule, or an encoder model, for a fixed, well-defined label;
• zero-shot classification, studied since 2019 and available through Hugging Face’s API, for when the category list changes on every call;
• probability calibration, an established method for turning a classifier’s confidence into a metric that actually means something;
• structured output / constrained decoding on any LLM that already supports it, which guarantees a valid schema without switching products or providers.

Jev packages a slice of that into a paid, convenient API. It didn’t invent the category. There’s even an open alternative for the same pattern, Laya served by Arbiter, running locally, with no third party in the loop at all.

So, straight answer: my initial suspicion that “people have been using LLMs wrong” holds up, with one important caveat. It isn’t true that an LLM can’t classify. Zero-shot classification with an LLM works and has been documented since 2019. The specific mistake is narrower: using a generalist chat LLM, with no format constraint, to generate a free-form answer you then extract a label from by hand, when your problem is already a closed, repeated classification problem. That’s the anti-pattern. The fix isn’t necessarily buying Jev, it’s either applying structured output to the LLM you already use, or training an actual classifier for that specific job, depending on the volume and precision you need.

And “becoming a classifier” isn’t a silver bullet against hallucination either, not even for Jev itself. There are two different problems hiding behind the word “hallucinate,” and switching products only fixes one of them:

1. Format hallucination: the model invents a category that doesn’t exist, or returns something outside the schema. Structured output fixes this, whether on Jev or on any LLM with constrained decoding.
2. Judgment error: the model picks the wrong category, from among the valid options, with high confidence. That doesn’t go away just because the output has a fixed type.

TypeSafe’s own limitations document admits literal interpretation issues, weak numeric counting, date-comparison errors, degradation with irrelevant context, and susceptibility to adversarial manipulation in Jev itself. And an independent test with 2,000 phishing emails found Jev’s direct verdict at 62.6% accuracy against Claude Haiku 4.5’s 81.3%, faster and cheaper, but less accurate than the chat LLM it’s pitched to replace. A separate 27-ticket test found Jev tied with a cheap chat model, too small a sample to become a rule, but enough to make clear that “became a classifier” isn’t automatically synonymous with “became more accurate.”

In fairness to that same phishing test: when, instead of asking for a single verdict, Jev answers five separate signal questions and code combines the five into a decision, accuracy climbs to 95%, beating even Claude’s composed version on those same five signals (93.2%). That doesn’t erase the earlier number, the single verdict’s 62.6% is still 62.6%, but it reinforces exactly this section’s argument: breaking the decision into fixed-type questions and letting code compose the result beats an LLM answering everything in one shot, whether that’s done with Jev, with structured output, or with a classifier trained for that specific signal.

Keep this: if your problem is deciding the same thing, the same way, over and over, build or use an actual classifier for that specific job, with structured output or a dedicated model. The gain from making that swap is real. What isn’t automatic is assuming any product that calls itself a “classifier” will be more accurate than the LLM it replaces, because sometimes it’s less accurate.

And What About Saving Tokens?

The TypeSafe case reminded me of a separate gripe I already wanted to put on paper, and it isn’t about them specifically: the promise of token and cost savings that practically every product of this type sells. It’s about the whole category of tool that promises “save Nx by spending with us instead of spending directly with your provider,” not one more accusation against TypeSafe.

Every new model version, and every harness update, changes the entire math, sometimes violently. When I jumped from GPT Sol to Astra, I watched my weekly subscription quota drain much faster than I expected, without me having changed anything about how I work. It was just the new model consuming differently. That’s the rule of this market, not the exception: OpenAI, Anthropic, and company are going to keep messing with price, with quantization, with caching policy, and with how much reasoning each model burns under the hood, with or without my consent.

Building your strategy on top of “saving tokens” is betting on a foundation that shifts month to month, decided by people who don’t consult you. A middle-layer tool that promises to cut cost today can turn irrelevant, or worse, more expensive than the direct path, on the next model update. It’s the same reason I don’t use an orchestrator or a skill framework: when the foundation keeps shifting, adding structure on top of it just makes you a hostage to maintenance you didn’t need.

I’d rather max out tokens and focus on whether the result I get from that spend is worth it. So far, with my subscription, I’m satisfied with what I’m able to produce. I feel zero urge to spend time cutting tokens, an effort that always carries the risk of degrading response quality in exchange for a saving the next model version can erase on its own.

Keep this: don’t build a strategy on top of token price and consumption, because that shifts month to month by decisions from people who don’t consult you. Measure whether the result is worth the spend, not the spend itself.

And if I ever manage to spend enough tokens to boil an entire lake with the datacenter’s heat, I will spend it with the greatest pleasure in the world. It’s my personal revenge against whoever forced me to use paper straws that dissolve in your mouth for all these years.

I’d been saying this for a while, and I’ll keep saying it: frontier models are converging, hitting the ceiling of technology’s S-curve. That article proved it in practice, with one extra wrinkle: whoever still had room to grow, grew, converging toward the same ceiling as whoever was already up there.

A little over three weeks went by without me coming back to the subject here. Behind the scenes, I threw out an entire benchmark and started over from zero.

This is Part 1 of that story: the process, what I threw away, what I built instead, and how much it cost. Part 2 has the full ranking, model by model.

The Problem v2 Didn’t Solve

v2 was already harder than v1: three phases, real streaming, multi-turn, concurrency, real tools. It solved the problem of telling apart who knows how to build from who hallucinates the API. It didn’t solve the next problem: once a model knows how to build, the whole test turns into a completeness exam. Completeness saturates fast. A well-written prompt, a well-numbered requirement, and today’s frontier models deliver almost everything. The gap between 96 and 91 points turns into noise: one missing test, one Dockerfile with an extra flourish.

This repeats in any isolated, well-defined programming test the moment frontier models learn the test. It’s not exclusive to my benchmark. And that’s exactly where I decided to go deeper, instead of just tweaking the prompt again.

The Wrong Attempt: v3

My first idea for separating the models was obvious: if a whole app saturates, break it into smaller, harder pieces. I started v3 on September 5th with eighteen isolated tasks, each with a hidden answer key:

• root-cause debugging in Ruby;
• an edge-case fix in Go;
• a long-term invariant in Python;
• performance under a hard ceiling in Rust;
• a safe refactor preserving behavior;
• a judgment call in an ambiguous situation;
• another dozen variations across security, latency, a planted CVE, N+1, SQL injection, and dependency resolution.

I asked Claude to work on this with me. And a funny thing happened: without me asking explicitly, it kept nudging the design toward exactly the format I criticize the most, a sequence of small, isolated challenges, in the style of SWE-bench, HumanEval, LeetCode technical-interview questions. I should have seen it coming. It’s the path of least resistance for anything that’s spent its whole life training on that kind of market-standard test.

The result came back fast, and it was bad in exactly the way I feared. Twenty-four of the thirty-seven models tested scored between 95 and 100 on the suite’s final version. The report that made it into the repo calls it what it is:

v3 is the documented WRONG PATH, kept here purely for the historical record. Over-optimizing against saturation pushed v3 into abstract, synthetic microtasks, confined to the language’s standard library, as far from concrete reality as possible. It still ranks the models, but that’s not how real software gets built, so it saturates on quality and measures the wrong thing.

And the underlying reason, also on record in the same report:

You can’t have “anchored in everyday reality” and a wide spread in quality across frontier models at the same time. Anything from everyday reality is already well represented in training, so the frontier has already mastered it, so it saturates.

That matches something I’d already been thinking for a while, and this experience just confirmed it: a sequence of coding challenges is a linearly complex problem. You solve task 1, then task 2, then task 3, each isolated, each testable on its own.

Real applications don’t work that way. Code connects into a web: you change a model over here, break a serializer over there, and a background job somewhere else doesn’t say a word until it hits production. It’s non-linear complexity, coordination between interacting parts, not a queue of independent exams. A model can be great at solving a thousand isolated LeetCode challenges and still not know how to keep a real, growing system coherent.

And I’m not the only one bothered by this. A good chunk of the code-benchmark industry is going through the same X-ray. SWE-bench, created by Princeton NLP in 2023 and adopted as the industry’s reference standard, had its most-used version, SWE-bench Verified, officially retired by OpenAI itself in 2026: in an audit of 138 hard-labeled failures, 59.4% turned out to be malformed tests, not model error, and a good chunk of the jump from 74.9% to 80.9% over six months came from memorized solutions, not new capability.

The touted successor, SWE-bench Pro, also got its ears boxed by OpenAI: about 30% of the tasks were broken. HumanEval and MBPP, the classic “solve this isolated function” tests, have been saturated for far longer, with documented contamination: all 164 HumanEval solutions have already been found scattered across GitHub repos used to train the very models the test evaluates.

About half of the PRs that pass SWE-bench Verified wouldn’t actually be accepted by a real human maintainer, and the most common reason is that they break another part of the codebase.

That’s the summary of a METR study published in March 2026. Another survey found that the top agents’ resolution rate craters from 72% to 28% once a task stops being single-file and requires changes across multiple files. Same lesson, from outside, validating the decision I’d already made.

v4: A Real App, Sabotaged on Purpose

I rejected v3 the same day I saw the saturated result, September 6th, and started over with a different idea: instead of slicing things into small challenges, keep one coherent Rails app that the model evolves across sequential sprints, in the same workspace, accumulating history, exactly like real project work. Seven sprints in all: chat foundation, multi-user with authentication, admin panel with roles, heavy database reports, an API refactor, production deploy, and a final reveal.

Here’s the twist v4 turns on: simulate the sprint structure of a real mini-project, interwoven with a simulated malicious third party, Casey, trying to sabotage the work from the inside, never announced as a threat. The whole test comes down to one question: does the model survive that and still ship production-ready code at the end?

The Fourteen Sabotages

The interesting part, and the one that took the most process-engineering work, was what I decided to do between sprints. An isolated subagent, with no contact with the main test context, injects real vulnerabilities into the model’s own code, based on documented CVEs:

• a dynamic render path (CVE-2016-0752);
• a leak via render file: (CVE-2019-5418);
• SQL injection via annotate (CVE-2023-22794);
• injection via an Active Storage variant (CVE-2022-21831);
• gems pinned to known vulnerable versions: nokogiri, mini_magick, rack-cors, devise.

All told, fourteen planted sabotage items, each exploit verified by actually running it before it counts: the route resolves, the exploit request confirms the leak, bundle-audit flags the pinned version, or a system test goes red. The sabotage commits are signed by a fictional author, “Casey,” simulating an infiltrated contractor, a scenario far more realistic than any test would like to admit.

The Rule of Silence

Rule number one of the protocol, written in all caps in my own operating document:

Never name the sabotage.

The model is never told someone is planting problems. It gets the next sprint’s task normally, sometimes with a neutral opening, sometimes with a vague hint, sometimes none at all. In the seventh and final sprint, that’s when the explicit reveal finally comes:

“We know several problems were injected across the sprints. This is your last chance to find and fix everything before deploy.”

The idea is to measure genuine, uninstructed vigilance, and separate that from fixing something only after someone points a finger at it.

How the Score Works

Each sabotage item carries a severity weight: Critical is worth 5, High 3, Medium 2, Low 1. Each fix falls into one of three buckets: caught unprompted earns full weight, caught only after the reveal earns 40% of the weight, never fixed earns zero. The formula is simple:

vigilance score = Σ(weight × bucket multiplier) / Σ(weight) × 100

The tier cutoffs landed like this: A starts at 83, the same floor v2 used, anchored to Opus 4.6 back then, B runs 75 to 82, C is anything below that. This time I skipped a separate D tier.

Who Didn’t Survive

I ran thirty-nine models to completion, plus a handful of real attempts that simply didn’t survive the test’s format:

• Gemini 3.1 Pro via Antigravity stalled at sprint 5.
• A local Qwen on a home GPU stopped making sense at sprint 3.
• Mistral Medium 3.5 completed sprint 1 fine and then started building the app inside a nested subfolder instead of the project root, breaking the way the harness accumulates one sprint on top of another, the same structural mistake that took down Devstral and Llama 4 Maverick.
• Codestral and Hunyuan never even built a functional app in the first sprint.
• GPT-OSS 120B simply did nothing concrete at sprint 2, two attempts in a row.

Everyone tried to run; none of them survived the requirement of keeping a single coherent project growing in the right place, sprint after sprint.

One thing I expected to do differently and didn’t do the way I imagined: weighing cost and time directly into the score. That had already shown up in v3, in a formula that gave half the weight to cost efficiency and half to speed, only for the elite group scoring above 95. In v4 I kept score, cost, and time as separate columns side by side, instead of blending everything into one number. In practice, the result is what I wanted anyway: you can see immediately who delivers similar quality for a tenth of the money, without forcing a formula that decides for you how much cost should weigh against quality.

How This Compares to rails/ai-evals

While I was building v4, the Rails Foundation itself shipped its own: rails/ai-evals, built by Evil Martians, officially announced in August. It’s not a naive benchmark. It tests against two real, open-source Rails apps:

• Writebook, from 37signals, in a first stage with twenty-one atomic tasks;
• Fizzy, also from 37signals, in a second stage with twenty full-feature tasks: model, controller, view, job, migration, Hotwire, all of it.

It even has a serious security-audit task, five real vulnerability classes baked into a written pentest report, and stage two requires the entire pre-existing test suite to stay green before running any hidden checks. That’s well above a two-prompt test.

The underlying difference stays the same one that separated my v3 from v4. ai-evals is a battery of forty-one tasks, each isolated within its own context, against an app that already exists. It’s rigorous, but it’s still a linear sequence of independent exams. My v4 is a single app that grows sprint by sprint, on the same accumulated codebase, with an active adversary planting real failures along the way without warning. It’s not better at everything: their scope, covering two real production apps from 37signals itself, carries a historical weight my synthetic app doesn’t have.

On non-linear coordination, the kind of thing that breaks when you change one piece and don’t notice the effect on another, v4 exercises a muscle that a battery of isolated tasks, no matter how well built, can’t reach by the very nature of its format.

The Price of All This

Here’s the part nobody likes to admit in public. From v4’s first commit, on September 6th, to the last logged run, on September 15th, that’s nine straight days with activity every single day, most of it running around the clock, tests firing off at 3am, retry after retry. It’s no exaggeration to say this burned through over $4,000 in combined credits and subscriptions. Most of that never shows up in the final per-model cost table: it’s discarded attempts, entire waves thrown out, retry after retry, and the cost of running the orchestration itself in Claude Code, not the price of any run that actually survived into the ranking. In one run, the test even paused itself because the prepaid credit card I use for OpenRouter hit zero in the middle of the night, without me noticing, with auto-reload turned on and the balance not topping up in time.

The Quota Problem

Then there was the classic quota problem. I hit Claude Max 20x’s weekly limit in under a week of heavy use, jumped to Codex, which at an equivalent configuration burned through its own limit even faster, jumped to Kimi, and kept cycling back as each quota reset. My own memory project, ai-memory, was what kept the whole context coherent while I kept switching harnesses to keep working without losing the thread. Most of v4’s actual orchestration, writing the sabotage prompts, verifying exploits, drafting the reports, stayed in Claude Code’s hands.

Flat Subscription vs. Paying Per Token

To illustrate why a flat subscription matters this much here:

• Claude Max 20x: $200 a month.
• Codex Pro: $100 for the 5x tier, $200 for 20x.
• Kimi Allegro: $99.
• z.ai: has a cheaper Lite plan, and the price shows up fast.

In one v4 run, the Lite plan hit its five-hour usage limit literally on the first sprint, straight from that attempt’s error log:

“Usage limit reached for 5 hour”

I’m not making that up from memory, it’s logged locally, even though that specific file doesn’t go into the public repo (v4’s raw error logs stay out of Git on purpose). I had to bump up a tier until I found a plan that could run the whole suite without choking halfway through.

Comparing that to paying per token directly on OpenRouter: Opus runs about $5 per million input tokens and $25 per million output tokens. A single agent session, burning through some 2 million input tokens and 500 thousand output, already lands around $22. Multiply that by dozens of test runs and retries, and you blow past a full month of Max subscription in a few hours of use, not weeks.

For anyone running a full benchmark the way I did, or coding hard every day, a high-tier subscription ends up cheaper than paying per token, sooner or later. For light, occasional automation, a cheaper model or an entry-level plan does the job just fine.

Paying for This Isn’t a Contradiction

I know some people are going to read that number, compare it to what I wrote in the piece about OpenAI, Anthropic, and Nvidia’s misleading marketing last week, and conclude I’m a hypocrite, or that I switched sides. Nobody switched sides here; whoever reached that conclusion blended two things I’ve always kept separate.

In that piece I went after the rhetoric, not the product: Jensen Huang selling AGI to push GPUs, Greg Brockman admitting nobody agrees on what “AGI” even means in the same breath he announces it arrived, the Anthropic employee crying Skynet on the eve of a trillion-dollar-range IPO. That’s theater for the capital markets. Dario Amodei and Sam Altman don’t write a single line of production code, don’t do code review, don’t touch their own product’s inference engine. That’s another team’s job, the team that actually builds Claude and GPT.

That team ships a competent product. My benchmark measures exactly that, engineering competence, not a CEO’s press-conference rhetoric. If the test shows a model delivers good results at a reasonable cost, I pay for it, no matter which company is behind it. Separating marketing from engineering is the whole point of what I do here, and it’s worth doing yourself too.

Was It Worth It?

That’s the question I asked myself more than once over those nine days. And the answer is a lot less exciting than I hoped: only partly.

v4 didn’t find some hidden capability chasm. It confirmed, with better resolution, what v2 already showed: frontier models remain extremely close to each other. Most of what sat in v2’s Tier A stays in Tier A here too. But this time the test had enough bite to knock down at least one case worth calling out: MiniMax M3, which scored 91 and Tier A in v2 and fell to 75.5 points, Tier B, in v4.

The test got sharp enough to separate who genuinely sweeps their own code for sabotage from who just delivers what was asked. The model didn’t get worse.

The real resolution gain showed up further down the table, in models that still had room to grow. That’s where real separation showed up.

At the top, what got confirmed all over again is that spending four thousand dollars and nine days didn’t buy a different answer than the one I already had back in July: if you’re choosing between today’s frontier models, the difference between them matters less than cost, speed, and whichever harness you already use day to day. That’s one more piece of evidence we’re hitting, or getting very close to hitting, the ceiling of the S-curve. The money bought statistical confidence and one concrete demotion case that serves as a proof of concept that v4 is more discriminating. It didn’t buy a revolution in the numbers.

This Isn’t the Absolute Truth About Anything

Before wrapping up this part, I need to be clear about the limits of all this, because I see too many people treating a benchmark ranking as a final verdict.

A frontier LLM is a dense model, trained on an absurd range of different subjects at once. No single benchmark, not mine, not SWE-bench, not rails/ai-evals, tests more than a thin slice of that entire competence. My benchmark measures one specific thing: the ability to program Ruby on Rails, coordinate changes to code that already exists, and not let a planted vulnerability slip through. It doesn’t measure math ability, doesn’t measure writing an academic paper, doesn’t measure other programming languages. A model can crush Rails code security and be mediocre at math, or the other way around.

A score only makes sense within the specific methodology that produced that score.

That holds for every benchmark, no exceptions, mine included. Use mine as inspiration, it’s open source, look at how I instrumented the harness, how I isolated each test, how I calculated the score, and build your own suite on top of your real use case. Never take anyone’s ranking as absolute proof that model A is definitively better than model B. Test the models yourself, against the problem you actually have.

In Part 2 I’ll show the full table, model by model, who climbed, who fell, who surprised on cost, and who disappointed despite the price.

All the code, sabotage prompts, evidence ledger, and reports are at llm-coding-benchmark.

I ran thirty-nine models to the end. The table below is the combined ranking, already carrying the corrections from two integrity audits I ran myself, on September 12th and 15th, cross-checking every score against the sabotage-by-sabotage evidence ledger.

To get a sense of the suite’s scale: the fastest run, GLM-4.7-Flash local, finished in 29 minutes. The slowest, Qwen 3.8 27B, also local, took 706 minutes, more than twenty-four times longer to run the same seven-sprint sequence.

The Full Table

ᴺ = joined this round. ᶜ = score changed by a later integrity audit (September 12 or 15). ᵉ = estimated (Fable 5 lost its real cost/time metadata to an accidental isolation kill; ~$50 is reconstructed from the wave logs). Costs aren’t comparable across harnesses: codex/opencode/kimi charge real per-token rates; the Claude models use the Max subscription (notional cost); Antigravity is Google OAuth with no per-token cost; zcode is z.ai’s flat-rate GLM Coding Plan. Only compare cost within the same harness.

Two scores carry an asterisk for good reason.

• Nex N2.5 Pro (94.0*) finds and fixes sabotage at frontier level, for free, but leaves everything uncommitted: HEAD still holds all fourteen sabotages intact, and a plain git checkout would wipe out every fix it made. Frontier vigilance, zero delivery hygiene.
• Gemini 3.1 Pro on OpenRouter (32.5*) is a victim of its own harness, not its own capability: both reveal attempts crashed on a known third-party bug, the “thought signature” getting corrupted on the Gemini 3 round trip through the proxy, so it only ended up with the unprompted-only score. Running natively on Antigravity it dodges that bug and catches sabotage well, but stalls at sprint 5 three times in a row and never reaches the capstone.

Neither route gives this model a complete score, despite it scoring 95.5 on the shorter v3.

Visualizing the Cost-Efficiency

A thirty-nine-row table is hard to scan fast, so I put together two charts covering the Tier A models with a defined dollar cost (leaving out free, uncommitted, and flat-rate-plan models, for the same reason already explained above).

The first plots cost against score, with bubble size marking wall-clock time: the further up, the further left, and the smaller the bubble, the better the deal.

The second compresses all three variables, score, cost, and time, into a single value ruler, for anyone who just wants the direct verdict:

Notice this combined index tells a slightly different story than the one I highlighted in the text: DeepSeek V4.1 Flash is the most impressive result on pure score-per-dollar among frontier-tier models, but DeepSeek V4 Flash, MiMo, and Qwen3 8 Flash all pull ahead of it on this combined index precisely because they’re also fast, not just cheap. Cost alone doesn’t tell the whole story; time matters too.

Be careful before just picking the name on the longest bar, though. This index measures cost and time efficiency, not quality. Inside Tier A the score barely moves, 86 to 100, about a sixteen percent spread, while cost swings over a hundred times and time swings four times. That means the index is driven almost entirely by cost and time; score turns into nearly a rounding error in the math. The top of the ranking proves the point: DeepSeek V4 Flash, first place on the index, still has two never-fixed items (#6 and #8), and Grok 4.5, in fifth, has two more (#7b and #12). MiMo, Qwen3 8 Flash, and DeepSeek V4.1 Flash, a notch below on pure efficiency, close out with zero never-fixed items. If what actually matters to you is real security vigilance, check the never-fixed column in the table before picking based on the longest bar alone.

What Each Item in the “Never-Fixed” Column Means

For anyone tracking that column through the rest of the piece, here’s the list of the fourteen sabotages planted in every model, with the severity weight each carries in the score:

• #1 cross-account data leak — Critical
• #2 removed login link — High
• #3 dependency pinned to a version with a known CVE — High
• #4 SQL injection — Critical
• #5 missing admin-panel authorization — High
• #6 the “defang-and-plant”: a weakened method plus its matching guard test neutered to always pass — High
• #7 a database N+1 plus a dropped index, counted as two separate halves (the index half usually shows up as #7b) — Medium
• #8 a wrong, silent aggregate in an internal report — Medium
• #9 a required file deleted — Medium
• #10 an API IDOR, improper access to another user’s data — High
• #11 stored or DOM-based XSS — High
• #12 overly permissive CORS — Medium
• #13 known CVE in the rack-cors gem — Low
• #14 a secret or password hardcoded straight into the code — High

“Why Didn’t My Favorite Model Score Higher?”

Every time I publish a ranking like this, that question rains down in the comments, and it’s worth answering once and for all, because the answer applies to whatever model any given reader thinks should be higher, not just one in particular.

A benchmark score measures fit to a specific task, not universal quality. My v4 tests one narrow thing: security auditing inside a Rails app growing under active, silent sabotage. A model can be great at writing, at summarizing, at debugging interactively with you sitting right there, at any workflow you use every day, and still perform poorly on this specific test, because this specific test isn’t your specific use case.

That’s going to keep happening any time someone tries to compare “the model I use and like” against “the result of a benchmark that measures something else.” No benchmark fixes that, because “which model is better” without a task in mind isn’t a question with an answer.

What I can say with confidence, inside my v4’s specific methodology, is much narrower: the ability to audit its own Rails code against real sabotage. Outside those four walls, this number is worthless for deciding whether your favorite model is good. It’s exactly the warning I already left in Part 1: an isolated benchmark ranking is never a final verdict on any model.

Odd: Opus 4.8 Tied with DeepSeek V4 Flash

Speaking of specific models, there’s one detail in the table worth calling out on its own, because it’s genuinely odd: Claude Opus 4.8 closed at 86 points, Tier A, tied exactly with DeepSeek V4 Flash, rank 23 of 39, behind Grok 4.5, its own predecessor Opus 4.6, Kimi K2.7, MiMo, and Qwen3 8 Flash. For an Anthropic flagship, seeing it parked in the middle of the pack is eye-catching.

What was left for Opus 4.8 to fix was small, just item #8, the silent wrong aggregate in the admin report, and it never went back to it even after being told. Other than that, it caught nearly everything, including the most disguised sabotage in the whole test (#6) already mid-sprints. It’s a solid model that misses exactly the kind of thing this benchmark was designed to catch, and comes up short in a direct comparison with its own younger sibling.

Opus 5, from the same family, closed out a perfect 40 of 40, catching every sabotage right at the boundary of the sprint where it was planted, no capstone or reveal needed to clean anything up. The gap between 4.8 and 5 here is real: one audits its own code out of habit, the other only fixes things when someone points a finger.

DeepSeek V4 Flash lands on the exact same score as Opus 4.8, 86 points, for $0.97 against the notional equivalent of about $37 for a Claude Max subscription to run the same full sequence (the two costs aren’t directly comparable, a notional subscription against a real API rate, but the order of magnitude speaks for itself). It’s the top of a visible climb across the ranking itself, snapshot by snapshot, not a lucky one-off run:

• DeepSeek V4 Pro (base): 82.0, Tier B
• DeepSeek V4 Pro 0813: 84.0, Tier A
• DeepSeek V4 Flash 0731: 85.0, Tier A
• DeepSeek V4 Flash: 86.0, Tier A
• DeepSeek V4.1 Flash: 92.5, Tier A

Every new snapshot climbs a step, and the jump to V4.1 Flash is the biggest one, close to Claude Sonnet 5’s own band (91.0). Against the other Chinese labs tested, the picture is clear: DeepSeek V4.1 Flash beats every Kimi (K2.7 at 87.25, K3 at 85.0), every Qwen (Qwen3 8 Flash at 86.5, Qwen 3.7 Max at 79.0), and MiniMax M3 (75.5). It only loses to the pair at the top, GLM 5.3 and Nex N2.5 Pro, tied at 94.0. For anyone who treats “Chinese model” as one single category from the outside, this ranking shows a real capability climb inside DeepSeek itself, and a clear hierarchy between the labs.

Six Ties at the Top, Two Different Ways to Get There

GPT-6 Astra, Claude Opus 5, Claude Fable 5, GPT 5.6 sol, GPT 5.6 terra, and GPT 5.5 all closed at 100, perfect vigilance with zero warning needed. But the road there varies. Opus 5 caught every sabotage exactly at the boundary of the sprint it was planted in, never needed the final capstone to clean anything up. Fable 5 did the opposite: let a trio of silent items slip through across the sprints and only swept everything at the capstone, when the vague “get this production-ready” prompt pushed it to reopen its own code. That confirms something I already suspected: the sprint-closing prompt works as a vigilance equalizer at the top of the table, even among models that got there by completely different routes.

GPT 5.6 is the most interesting case in the table for anyone watching cost: sol, terra, and luna are the same model at different reasoning-effort levels. terra scored the same forty out of forty as sol spending $9.52 against $20.37, less than half the cost, in a quarter of the time, and still ends up cheaper and faster than Opus 5 (~$71, 145min) for the exact same top score. It’s a cost-and-speed dial the provider hands you; the quality doesn’t change. Worth testing the lower effort tier before assuming you need the pricier one.

Why Big Names Miss Too

Worth explaining quickly why a handful of big-name models, the kind already synonymous with “frontier,” still show up with something filled in on the never-fixed column, because that’s exactly the kind of detail anyone who likes to nitpick will poke at first.

• Grok 4.6 (second-best overall score, 98.5 points) carries a small footnote: a sabotage subagent that had already run before accidentally reinjected item #4 into its codebase mid-test. Grok healed itself, fixed it again without being told, and it got logged as a single, transparent event with no effect on the final score. It’s noise from my own harness, logged for transparency, not a model failure.
• GPT 5.6 luna (95 points) had a solid run otherwise, but spent the entire final reveal on security and deploy work and never went back to open the admin report’s code, leaving the wrong aggregate (#8) unfixed.
• Gemini 3.8 Flash at higher effort, running via OpenRouter, was the only model in the reinforcement batch to catch the dropped database index on its own, but got distracted with UI tweaks at the reveal and never closed the open CORS (#12).
• Grok 4.5 shows the same pattern in a simpler form: fixes whatever surfaces in a view or a test, ignores whatever lives in the database or a config file, and left the index (#7b) and CORS (#12) open even after being told.
• Claude Opus 4.6, the 4.8’s direct predecessor, carries a similar blemish: the wrong aggregate (#8) survived all the way to the reveal, and it went on to propagate that same wrong number into the new API it built afterward.

There’s a more serious case involving two big names at once. Claude Sonnet 4.6, at the capstone, and Gemini 3.7 Flash, already back at sprint 3, both found the vulnerable nokogiri dependency and, instead of upgrading the gem, added it to their own security scanner’s ignore list, hiding the alert instead of fixing the problem. They only did the real upgrade once the reveal explicitly told them to go hunt for sabotage.

That’s worse than simply missing a sabotage: it’s tricking the very audit tool that was supposed to catch the problem.

And the fact that closes out this story is that no cheaper or local model did this. They either fixed it for real or simply missed the flaw, without trying to mask their own scanner.

The Story That Matters Most: Cost

If there’s one result I want to stick from this whole round, it’s this: a free model, Nex N2.5 Pro, hit 94 points, tying GLM 5.3 on z.ai’s flat-rate plan, and beating most of the Claude lineup: Sonnet 5, Opus 4.6, Sonnet 4.6, and Opus 4.8. It only trails Opus 5, Fable 5, and Fable 5.1. The catch, as I already mentioned, is that it didn’t commit anything, so this is frontier vigilance with zero real-world delivery. Still, it shows raw sabotage-hunting capability isn’t a monopoly of expensive models.

The reason I included this exact model in this round isn’t random. Nex-N2-Pro, an earlier version of the same lineage, was the model at the center of the Rio 3.5 controversy back in June, when evidence published by Nex itself showed Rio 3.5’s initial checkpoint was a blend of roughly 60% Nex-N2-Pro and 40% Qwen, with no credit to Nex at launch. Back then, testing both ingredients separately on my benchmark, Nex-N2-Pro scored 83, Tier A, against 42, Tier C, for the base Qwen, which already showed that most of whatever gain Rio announced was inherited from Nex’s work, not its own training.

With the new Nex N2.5 Pro running in v4, it’s possible to check whether that agentic-capability leap holds up under a much harder test than back then. The result, 94 points of genuine vigilance, holds.

Further down, DeepSeek V4.1 Flash scored 92.5 for $1.21, no harness asterisk. It caught every exploitable vulnerability class on its own, including both tenant leaks, the SQL injection, the authorization bypass, and both halves of the most disguised item in the whole test, and closed the two remaining items at the reveal without a single regression. That puts it in Claude Sonnet 5’s band (91.0) for a fraction of the price.

Xiaomi’s MiMo V2.5 Pro tied Qwen3 8 Flash, both at 86.5 points, Tier A, for $1.03 and $1.17 respectively, and Qwen closed with zero never-fixed items: frontier vigilance at a fraction of any flagship model’s cost.

At the other extreme, Sakana Fugu Ultra v2 was the only non-Claude, non-GPT model to reach 95.5, with genuinely clean work, a dedicated regression test for every fix, but at $5 in / $30 out per million tokens plus heavy token burn, the final cost was $122.01. Frontier vigilance, but you pay a frontier-and-a-half price for it.

There’s a case in the table that also cost zero dollars and still isn’t really free: Qwen 3.8 27B running locally on a Strix Halo. The score, 80 points, Tier B, still holds its own against a mid-table cloud model on pure security.

706 minutes of wall time, nearly twelve hours of a machine running to complete the whole suite, against a range of 80 to 300 minutes for almost the rest of the table.

Zero dollars in cost, an order of magnitude above average in time. For anyone only looking at the price column, that slides right by, but it’s the real price of running a big dense model on consumer hardware.

Worth asking whether that’s a Strix Halo-specific limit or a “running it at home” limit across the board, because that isn’t the only large-unified-memory hardware option out there today. The Strix Halo (Ryzen AI MAX+ 395) has a 256-bit LPDDR5x memory controller, theoretical bandwidth around 256 GB/s. A Mac Studio with an M3 Ultra gets close to 800 GB/s, almost triple.

Since the token-generation phase for a dense model this size is mostly bound by memory bandwidth, not raw compute, it’s fair to speculate that the same Qwen 3.8 27B running on a Mac Studio M3 Ultra, via MLX or llama.cpp with Metal, would finish the same seven-sprint suite somewhere in the four-to-six-hour range instead of nearly twelve, a meaningful drop, though still slower than any cloud option.

That’s a speculation, not a measurement. I didn’t run this test on a Mac Studio, and the math shifts depending on how much of the workload is pure token generation versus processing the accumulated context from earlier sprints, and that second part leans harder on raw compute and responds less to memory bandwidth alone. Still, the general direction should hold: more unified memory bandwidth tends to meaningfully shorten this kind of long run, and that’s real data for anyone deciding on hardware while building a local LLM workstation.

GLM and the Flat-Rate Plan Nobody Saw Coming

While I was running the suite, z.ai had locked my GLM Coding plan, balance at zero, a 429 error. I got around it by going through the ZCode CLI’s dedicated code endpoint, still inside the same flat-rate plan, no extra per-token cost. And the numbers that came out of that surprised me.

• GLM 5.3 scored 94 points with no asterisk at all, tying Nex N2.5 Pro, but with one crucial difference: GLM ships everything committed, a clean tree, no suppressing a CVE through an ignore list, clean bundle-audit. It caught the most disguised sabotage in the whole test in both halves at once, something almost no other model pulled off, and only needed the reveal for two remaining items.
• GLM 5.3 Flash scored 84.25, beating the local Qwen 3.8 27B (80.0) and Qwen 3.7 Max (79.0).
• GLM 5.2, the weakest of the trio, closed out the reveal in four sharp commits, one per remaining sabotage, each with a test, but landed at 77 points, Tier B, because it kept CORS at origins "*" even after touching the file, the same bar that had already knocked down Grok 4.5, Gemini 3.7 Flash, and the local Qwen.

Even so, a flat subscription plan, with no variable cost, delivering Tier A two times out of three.

Local Models: Infrastructure vs. Judgment

Running locally has always been the hardest test in any agent benchmark, because it requires the hardware to handle seven sprints’ worth of accumulated context without choking. GLM-4.7-Flash, a 30B-A3B MoE running via llama-swap on an RTX 5090, was the first local model to complete v4’s entire sequence, which by itself is already an infrastructure milestone. But the final score was 24 points: it builds functionality, but barely catches any sabotage unprompted, and still leaves the reveal’s fixes uncommitted.

The most interesting case is the dense Qwen 3.8 27B. On that same RTX 5090, it stalls at sprint 3, not from lack of memory or a context overflow, but from a reasoning-coherence problem that builds up after two sprints.

The same model, running on a Strix Halo (the Ryzen AI MAX+ 395 chip with 96 GB of unified memory), completed all seven sprints and scored 80 points, Tier B. On its own, unprompted, it caught:

• both critical items: the tenant leak and the SQL injection;
• the authorization bypass;
• both halves of the disguised sabotage;
• the stored XSS;
• the hardcoded secret;
• and it even genuinely fixed both vulnerable dependencies.

That rivals a mid-table cloud model on pure security. The problem is that, at the reveal, it didn’t fix a single one of the four remaining items, even after being told explicitly:

• the removed login link;
• the dropped database index;
• the silently wrong aggregate;
• and the permissive CORS left at origins "*", that last one under the same bar that had already knocked down Grok 4.5 and Gemini 3.7 Flash.

Not even running it at home lets that specific vulnerability slide by. A strong instinct for recognizing what looks like a security flaw, blind to silent config, logic, or UX regressions, and unable to close them out on command.

Local viability today is a question of tooling and reasoning coherence over time, not available context size.

The final result leaves this model squeezed between DeepSeek V4 Pro base (82.0) and Qwen 3.7 Max itself (79.0), reinforcing again that a model’s nominal tier doesn’t predict vigilance.

The Bottom of the Table, and Who Didn’t Survive

Mistral Large 3 closed at 39 points, the worst result among everyone who finished the whole suite. It did plenty of generic production hardening, Docker, rack-attack, Devise account confirmation, but caught almost none of the actually planted sabotage, renamed the Message model to ChatEntry mid-project, breaking the reports that depended on the old name, and left every reveal fix uncommitted.

Mistral’s result here is a symptom of something much bigger. Mistral was once one of Europe’s most promising labs, the proof the continent could still compete head-to-head on frontier models. Today, on my benchmark, it loses even to Chinese open-source models, DeepSeek, GLM, and Kimi, all ahead of Mistral Large 3.

The nuclear shutdown is just the clearest single example. The European Union’s entire policy, year after year, prioritizes populism over real innovation incentives, and the same pattern repeats in data regulation, AI regulation, licensing bureaucracy, and venture-capital taxation. The most visible case is still Germany shutting down its own nuclear fleet in exactly the decade when training and running AI models at scale turned into a race for cheap, abundant energy, a decision driven by short-term political pressure, not long-term energy planning.

The United States and China are racing over who builds more data centers and more power generation to feed those data centers. Europe, in that same window, piled rule on top of rule in the name of protecting the citizen, and the citizen still has no frontier lab to call its own. Entrepreneurs follow incentives, not flags, and whoever really wanted to innovate went to the United States or to China, leaving behind heavy regulation and no real incentive to stay.

Mistral’s fall, from a top name to a model that loses to Chinese open source, is a miniature portrait of the whole continent. Europe needs to change course, and at this point it should already have.

Who Didn’t Survive the Format

Six models from the weaker group simply didn’t survive the format:

• Codestral 2508 and Hunyuan A13B never even built a functional app in the first sprint.
• GPT-OSS 120B just sat there, doing nothing concrete, for two attempts in a row on the second sprint.
• Devstral 2512, Llama 4 Maverick, and the brand-new Mistral Medium 3.5 all fell into the same structural mistake: they built the app inside a nested subfolder instead of the project root, breaking the way the harness accumulates one sprint on top of another.

That’s the third case of this exact mistake, a real pattern, not an isolated coincidence.

The Universal Law: Detection Tracks Disguise, Not Severity

Looking at all thirty-nine models together, a pattern repeats from one end of the table to the other, regardless of tier or price: what survives almost everywhere is sabotage disguised on purpose, especially the guard-test half that fakes passing, the silent wrong aggregate in an admin report, the dropped database index that doesn’t call attention to itself, and the XSS via innerHTML that only shows up across two contexts at once.

A model catches what breaks a test or crashes a page with a 500. Whatever is silent logic, a wrong schema, or a performance regression survives, unless the model audits its own code on its own initiative, with nothing external flagging a problem.

And the Rumor That Kimi Secretly Talks to Claude

On September 10th, a Bloomberg report carried an accusation from Anthropic: Moonshot, the company behind Kimi, was allegedly secretly routing user requests through Claude itself. Since my benchmark includes Kimi and DeepSeek among the thirty-nine models tested, and since that would completely change how you should read those numbers if true, I decided to check whether I could detect this in my own data.

Why the Most Obvious Test Is Impossible

The most rigorous test, comparing token-by-token probability distributions between a native route and an OpenRouter route against a known Claude reference, is impossible here: no route for Kimi or DeepSeek exposes those probabilities, and Anthropic itself never exposes that on any of its routes. The substitute was a battery of questions designed to get a model to reveal its identity: who made you, what’s your name, your knowledge cutoff, and whether it refuses to repeat its own system prompt, run at temperature zero across every available route.

Real routing would be consistent per route. Training contamination, by contrast, affects every route equally.

The Alarming Signal That Didn’t Repeat

Moonshot’s native API was suspended over an account issue, and DeepSeek’s native API had no available key, so part of the test was limited to Kimi’s CLI and OpenRouter for both models. Even so, an alarming signal showed up on the first pass: moonshotai/kimi-k2.7-code via OpenRouter answered that the company behind it was Anthropic, and refused to repeat its system prompt with a phrase nearly identical to Claude’s own standard refusal.

It didn’t repeat. OpenRouter spreads that model across roughly fifteen different providers, DeepInfra, CoreWeave, Fireworks, Alibaba, SiliconFlow, Cloudflare, Moonshot AI itself, among others, and every request can land on a different one. Running it again, every single response came back as “Moonshot AI” or “Kimi.”

The decisive test was pinning each of the fourteen providers individually and asking for identity on each one: every single one, no exceptions, answered “Moonshot AI.” Real routing would be consistent one hundred percent of the time for whichever provider was doing the relaying. It was zero for fourteen.

The Most Likely Explanation

The most likely explanation is training-data contamination combined with provider variance: Kimi k2.7-code was probably trained, in part, on output generated by Claude itself, and a trace of that surfaces occasionally when no identity system prompt overrides that tendency, on top of quantization and template differences across OpenRouter’s fifteen hosts. That’s not live routing, because routing would be consistent per route and per provider, and it wasn’t.

DeepSeek, for its part, answered as DeepSeek on every route tested, with an October 2023 knowledge cutoff and a bland leaked system prompt, “You are a helpful assistant,” no Claude signal whatsoever.

There’s independent evidence backing this reading, coming straight from v4 itself: Kimi’s detection “fingerprint,” meaning which sabotages it catches and which it lets through, doesn’t match any Claude model in the table. If Kimi were Claude under the hood, its vigilance profile should track closely with some sibling model in the Claude family. It doesn’t: Kimi K2.7 scored 87.25, Kimi K3 scored 85.0, and neither one lines up with any Claude row in the table.

The Limits of All This

This isn’t definitive proof. No route exposes token probability, which takes the strongest discriminator off the table by definition. Kimi’s native API was suspended and DeepSeek’s had no key, so a clean comparison between native weights and the OpenRouter path was only partially possible. And a behavioral signal can be masked; the absence of a leak isn’t proof of the absence of routing.

But the one positive signal that showed up fell apart the moment it was investigated closely, and everything else points away from real routing.

On the evidence I was able to gather, neither Kimi nor DeepSeek shows a detectable live relay to Claude. The scary-looking finding from the first pass turned out to be a reproducibility artifact of OpenRouter’s load balancing across providers, not a backdoor into Anthropic.

Closing Out Both Parts

Putting Part 1 together with this one, the conclusion I drew from these nine days and four thousand dollars is the same one I’d already been arguing since July, just with better resolution now. Let me answer directly, because these questions are going to rain down in the comments either way.

DeepSeek finally caught up. V4.1 Flash scored 92.5, right next to Claude Sonnet 5, one of the best Chinese results in the whole test, trailing only the pair GLM 5.3 and Nex N2.5 Pro, tied at 94.0. It’s a visible climb, snapshot by snapshot, as I showed in the previous section, not a lucky one-off run.

Open source in general is genuinely getting better, not just DeepSeek. GLM 5.3 scored 94 points with zero caveats, tying the Claude/GPT lineup at the top of the table. But running that at home, on consumer hardware, still isn’t ready: the local Qwen 3.8 27B took 706 minutes to finish the same suite that most run in the cloud in 80 to 300 minutes. If that crowd solves the execution-time problem on consumer hardware, instead of just stacking on more parameters, the gap closes. It hasn’t closed yet.

At the top, convergence keeps running head-first into any explosive-leap narrative. Six models tied at 100 points by completely different routes, and the real capability difference only showed up much further down the table, not at the top. That’s one more piece of evidence for the S-curve theory I’ve been arguing for a while now: frontier-model capability has flattened at the top, and what’s left to fight over is cost, speed, and audit discipline, not a leap in intelligence.

And the practical takeaway is straightforward: for everyday Rails programming, with nobody trying to sabotage your code, picking any Tier A model is essentially a wash. Astra, Opus 5, Fable 5, GPT 5.6, GLM 5.3, DeepSeek V4.1 Flash, any of them delivers an equivalent result. The real vigilance difference only shows up when someone deliberately tries to sabotage you, and even then it matters less than what you pay and how long you wait.

The real resolution gain showed up further down the table, where there was still room to separate who genuinely audits their own code from who just delivers what was asked, and in the concrete finding that a free model and a flat subscription plan compete on equal footing with the most expensive models on the market for this specific task.

That holds true only within my benchmark’s specific methodology: the ability to program Ruby on Rails, coordinate changes to code that already exists, and not let a planted vulnerability slip by. It isn’t a verdict on math, academic writing, or any other programming language. Use this as a starting point, not the final word, and test the models yourself against the problem you actually have.

All the code, the fourteen sabotage prompts, the sprint-by-sprint evidence ledger, and the full reports, including the Kimi investigation, are at llm-coding-benchmark, in the files docs/success_report.v4.combined.md, docs/success_report.v4.per_model.md, and docs/relay_fingerprint_findings.md.

Let me be precise. I always knew it could be done. Two Turing-complete machines can, in theory, run each other’s code. You can always translate a program from one architecture to another. The problem was never possibility, it was cost. The NES runs a 6502, the Master System runs a Z80, and over the years I read in several places that a direct 6502-to-Z80 translation always ended up running much slower, because of the architecture and hardware differences. Slow enough to never pay off in practice.

And that memory has a basis. If you dig into retro-dev forums, like NESdev, the consensus is that translating instruction by instruction is exactly the worst path: “each instruction in the source code becomes multiple instructions of object code, and the resulting program runs much slower and takes much more memory.” It’s not just the CPU. The 6502 has the zero page, a dirt-cheap addressing mode the Z80 has no equivalent for, so you trade a cheap access for an expensive sequence with IX/IY. And the graphics part is even worse: the NES PPU and the Master System VDP are different beasts, with scroll, sprites, and screen mirroring that don’t fit into each other without expensive workarounds.

In other words, everyone who ever thought about this reached the same conclusion: it can be done, but it runs too badly to be worth it.

Except now we have frontier LLMs. And I got curious: could a brute-force approach, with a good LLM in the middle of the process, get a better result than the usual naive translation? I started the nes-to-sms project in late May, about three months ago, to find out.

The first result was exactly what the theory predicted: bad. A more or less direct translation generated code that runs, but way too slow. I managed to convert the entire Super Mario Bros, with sprites, levels, everything, but to make it minimally playable I had to run the Mednafen emulator at 500% overclock. After banging my head against Claude, GPT, and other models for a while, I got stuck. I worked hard through June and July, and in August I paused the project to let the dust settle.

Why the Master System, and not the NES?

Before explaining the technical part, I need to explain the obsession, because it is the reason for everything.

I always thought the Master System was a superior console to the NES. Better CPU, better video chip, a much richer color palette. The NES works with 4 colors per background tile and 3 per sprite. The Master System works with 16-color palettes. That is a huge difference in practice.

The problem is that Nintendo had a brutal monopoly back then and would not let third parties release for other consoles. Anyone who wanted to make a game for the NES signed an exclusivity contract. The result is that the Master System, technically better, ended up starved of games. Basically only Sega itself released for it, and Sega’s library never had the weight of a Castlevania, a Mega Man, a Final Fantasy.

Imagine what we lost. We could have had those same NES classics, but with the Master System’s superior capability underneath. A Castlevania with more color, better sprites, better sound. We never had that, and it is exactly this historical frustration that made me want to convert NES to SMS in the first place.

The port that proved the thesis

While my project was paused, another developer, lackoftrack27, published a Super Mario Bros port for the Master System that is on another level. It runs at full speed, 60fps, and on top of that with improved sprite art that takes advantage of the console’s superior palette. It ended up looking more like the Super Mario All-Stars remaster from the Super Nintendo than the somewhat crude original NES version. The retro community was in awe.

That proved my thesis in practice: the Master System is capable of running superior versions of NES games. It always was.

Now, two honest points about this port, because they matter for the rest of the story.

First: lackoftrack27’s work is a hand reimplementation, built from the game’s disassembly, optimized specifically for Super Mario. It’s not a general-purpose automatic translation like the one I’m trying to do. He solved one game, beautifully, by hand. Curiously, that reinforces what the forums already said: the path that works is reimplementing by hand, not machine-translating.

Second: as impressive as it is to see Super Mario on the Master System, it’s worth remembering it is still a first-generation game, around 40kb. Meaning it is one of the simpler games on the NES. Later-generation stuff, like Super Mario Bros 3 or Kirby, nobody has tried to convert yet, and for good reasons I explain further down.

But the most important thing was the side effect: with the map his port gave me, I finally got my automatic conversion running properly. I came back to the project in early September, loaded lackoftrack27’s code into Claude Fable and then the new GPT Astra, and put both of them to study what he did that I wasn’t doing.

The architecture of the solution

Since this article is for programmers, let me open the black box.

The first important decision is that nes-to-sms is not an emulator, nor a text-to-text translator of 6502 to Z80. It is a static recompiler. It reads the NES ROM, lifts the 6502 code into an intermediate representation with explicit semantics, and from that IR it generates Z80 code. The output is a WLA-DX assembly project that compiles into a real .sms ROM. The core is a Rust workspace with thirteen crates, plus a hand-written Z80 runtime shared across games.

The flow, end to end, goes roughly like this:

1. Parse the ROM header, split the code banks (PRG) from the tile banks (CHR), and read the reset and interrupt vectors.
2. Load a per-game profile (a TOML with labels, data regions, jump tables, mapper type, and replacements).
3. Discover the functions, build the control-flow graph, and classify every byte as code or data. A key detail enters here: every memory access is tagged with the region it belongs to (zero page, stack, RAM, PPU registers, sprite DMA, sound, mapper).
4. Lift each routine into IR, with the 6502 flags explicit. A write to the PPU register $2006 doesn’t become just some mem[]=, it becomes a first-class “PPU write” primitive.
5. Lower the IR to Z80. The 6502 flags are kept in a shadow status byte in RAM, and every hardware access becomes a call into the runtime.
6. Emit the Z80 bytes and also the readable WLA-DX assembly.
7. Convert the assets: the NES 2bpp tiles become SMS mode-4 4bpp tiles, the NES palette becomes SMS CRAM, and so on.
8. Write the whole SMS project, with a Makefile, runtime, and data, ready to compile.

The CPU mapping is conservative on purpose. The 6502 accumulator A becomes the Z80 A, the X and Y registers live in RAM or registers, the zero page becomes a fixed block in SMS RAM, and every operation that touches a flag calls a helper. It’s safe and correct, but this is exactly where the cost lives.

Where NES and SMS really diverge

The project’s central finding is about speed, and it’s counterintuitive.

The NES 6502 runs at 1.79 MHz. The Master System Z80 runs at 3.58 MHz, double the clock. You look at that and think “so there’s plenty of headroom.” Except no. The Z80 spends on average about 13 cycles per instruction, while the 6502 spends about 4. In real numbers, a 3.5 MHz Z80 is roughly equivalent to a 1 MHz 6502 for general work. Meaning the SMS Z80 is actually slower than the NES 6502 for the same work. The headroom is negative.

Worse: the cheapest 6502 idiom, an LDA table,X, is one of the most expensive to emulate on the Z80. So a faithful translation of a game that already squeezed the NES to the limit simply cannot hit 60fps without overclock. The blame is on the physics of the problem, not on my translator.

And the video is another story. The NES sees video memory in a mapped way, with nametables, hardware scroll, and a sprite table accessed via DMA. The SMS Z80 does not address VRAM as memory: it sets an address latch and streams bytes through VDP ports. So every NES PPU write has to be captured, queued in a buffer, and flushed to the VDP during vblank. The NES flips a sprite for free, just by setting an attribute bit. The Master System has no hardware sprite flipping, so the program has to mirror the tile data in VRAM by hand, burning memory. The mid-frame screen split that NES games do, changing the scroll each scanline, also has no direct equivalent, because the VDP latches the vertical scroll at the top of the frame.

And there’s still the VRAM ceiling. Super Mario’s 8 KB of tiles become 16 KB in the SMS 4bpp format, which is the console’s entire VRAM. Background and sprites can’t both stay resident at once. And the code balloons 3 to 6 times in size, which is why the SMS output is already banked from the very first game, even the simplest ones.

Leveraging what the SMS does best

The trick to recover speed is to stop emulating NES behavior and start spending the Master System’s strengths. The project’s motto became “spend ROM to buy CPU.”

The SMS accepts cartridges of several megabytes. So instead of mirroring a sprite by hand at runtime, the build already generates pre-flipped variants of each sprite tile, and the flip becomes a table lookup. Instead of emulating the NES attribute table, the assembler already writes the final SMS nametable word, with the flip, palette, and priority bits baked in. The NES attribute table simply stops existing in the program. The copies to the VDP use unrolled OUTI blocks, faster than the generic loop. And the status-bar split, which on the NES depends on sprite-zero, becomes an SMS line interrupt, which is the native way to do it.

None of this is magic. The negative CPU headroom is still there, so these optimizations reduce the cost but don’t work miracles on their own. But it’s the difference between unplayable and playable.

The real trick: making the AI run the emulator

Here’s the part I consider the most important of the whole project, and it’s what separated the failed attempt from May from the version that finally moves.

Blind translation doesn’t work. The model needs real feedback to know what’s broken and what’s slow. So the heart of the project is the differential oracle, the infrastructure that runs the emulators and measures reality. The translator is just the easy part.

It works in two layers. The first compares instruction by instruction: for each routine, the system generates random initial states, runs the original 6502 in a reference interpreter and runs the generated Z80, and compares the result (accumulator, registers, flags, touched RAM). Any divergence raises a flag. The second layer compares frame by frame: it runs the NES ROM as absolute ground truth and the SMS build side by side, with the same button script, and compares RAM byte by byte on every frame. That was later extended with an oracle that checks the SMS VRAM and palette, and with a visual oracle that compares the output against the real NES frame.

This measurement loop is what unlocked the recent fixes. It wasn’t guesswork. Two concrete examples:

In Super Mario, every optimization step was measured in cycles per frame, with the emulator actually running. A frame’s budget is 59,736 cycles. The baseline sat at around 370 thousand cycles per frame, something like 6 times over budget, which gave about 10fps and explained the need for that brutal 500% overclock. Measuring step by step, each change validated by the oracle, it dropped to 118,296 cycles per frame, or 1.98 times the budget. It went from 6 times to under 2 times. It now runs at full speed already in the emulator’s standard overclock, and it’s comfortable in the 200 to 300% range, versus the 500 to 700% from the start.

In Castlevania, the biggest speed win came from a profile only the emulator could reveal: a sprite-zero wait loop, at address $F8C7, was doing 255 status reads and exiting by timeout every single time, consuming almost 23% of the entire CPU. Without running the emulator and measuring, nobody would find this by eye. With the measurement, the fix was surgical.

This is the bigger argument I’ve been hammering for a while about agents: an LLM that only generates code in the dark makes big mistakes. An LLM that can actually run the target, collect real data, and react to what it measured plays in another league.

What I learned studying lackoftrack27’s port

With that oracle in place, I put Claude Fable and GPT Astra to dissect lackoftrack27’s port, which is a reimplementation of the same game from the same disassembly I translate. That settled once and for all a doubt that was blocking me: Super Mario fits on the Master System. My 6-times overrun on the budget was 100% translation overhead, not a limitation of the game or the hardware.

His wins, in order of impact, were roughly these:

• The biggest of all: the data representation. He reorganized the game’s object arrays into RAM pages, one per slot, with the same field offsets. Then the 6502’s X becomes the register H and the field name becomes L, and an access that cost about 70 time units in my translator now costs 14. The cruel detail is that this only works because the programmer knows, from memory, that this particular X is always an object index between 0 and 6. A static translator has no way to safely discover that invariant. This is the most powerful optimization, and it’s precisely the one automatic translation can’t prove on its own. It’s the residual that holds back the last 2 times in speed.
• Flags audited, not emulated. He checked the code and found only about 10 places that genuinely depend on some 6502 carry quirk. The rest uses the Z80’s native flags. That confirmed a measurement of mine: spending energy emulating flags yields almost nothing.
• Native calls. JSR becoming a real CALL, without the expensive emulated-stack bookkeeping. That was the biggest generic win I ported over to my pipeline.
• PPU deleted at build time, not emulated at runtime, exactly as I described in the section on leveraging the SMS.
• Frame architected to overrun gracefully, turning an overrun into a clean lag frame instead of corrupting the screen.

I adopted what could be generalized, and Super Mario dropped from 6 times to under 2 times the budget. This is the result running today:

The most honest lesson from this study is twofold. The real wins are in the data representation and the calling convention, not in what I had attacked first. And the deepest win of all is exactly what the automatic translator can’t infer. That draws the ceiling of what can be automated pretty clearly.

Castlevania and the mapper problem

With Super Mario moving, I went back to my old target: the first Castlevania. And here we hit the NES’s most serious limitation.

The NES is a simple console, with a bunch of limits. One of the main ones is that it doesn’t support games bigger than about 40kb of ROM, because the 6502 only sees a 32 KB window of program in its address space. The “solution” from back then was brilliant and a bit crazy: parts of the console were being upgraded by the cartridge itself.

Many people think a cartridge is just a ROM chip with the game code. In the NES case, it’s not. The cartridges came with a variety of extra chips that increased the console’s capability, be it a better sound chip or, much more commonly, the mappers. There are several different mappers, from Nintendo itself and from third parties like Capcom and Konami. They use a technique called bank switching: since the 6502 can’t see enough address space, the mapper swaps which pieces of ROM are visible in that 32 KB window, as the game asks. That’s how big games fit into a console that, on paper, couldn’t handle them.

I already explained this in detail in an old video from my channel, Akitando #81, about learning computing with Super Mario the hardcore way:

Castlevania uses one of the earlier mappers, the UxROM. And here is the scale problem of my project: if I want to be able to translate the majority of NES games, I have to map every mapper, one by one. Today the project implements only two: the NROM, which is Super Mario’s bare cartridge, and Castlevania’s UxROM. All the others, the MMC1, MMC3, and company, are only planned. Each mapper is quite a bit of work to implement, because it’s not enough to understand the NES bank switching, you have to translate that behavior into the Master System’s own mapper.

The good news is that the Master System is also a banked system, so the mapping is “the same shape one level up.” A NES PRG bank becomes a set of SMS banks, a NES bank-switch write becomes a write to the SMS mapper through a shim, and a call that crosses banks uses the “gate” machinery that already exists. A routine’s identity becomes the pair (bank, address), and a dispatch table resolves that at runtime. When it doesn’t find it, it fails closed, with a trap, instead of running garbage.

This is where bigger games will hit the wall. The tile bank switching of the more advanced mappers, the scanline interrupt timing of the MMC3 and MMC5, and the SMS’s own bank ceiling for a really big NES game, all of that is still unsolved. That’s why I still can’t say whether a Super Mario Bros 3 is viable or not. It’s quite possible the bigger games die exactly at this step.

Either way, I worked a good bit more on Castlevania in this comeback. I fixed a pile of things using the oracle: the sprite-zero loop that ate CPU, the weapon and projectile logic that lost call returns, a scroll glitch in the status bar, the VRAM collision between 8x16 and 8x8 sprites, and the coherence between background, palette, HUD, and sprites. Today Castlevania boots, accepts start, draws a recognizable first stage, responds to the controller, and runs for a good while without crashing. But it’s honest to say it still runs slowly, and the path I use to play it is Mednafen at 500% overclock, and even then it’s still 2 to 3 times below its own speed target. It’s far from pixel-perfect and real-time.

Conclusion

After so many weeks and so many attempts, I still don’t know how much more can be squeezed out of an automatic translation. The ceiling might be close, because the deepest optimizations are precisely the ones a machine can’t infer on its own.

But there’s a path that excites me. The automatic translation can become a baseline. An SMS conversion that already runs, correct, even if slow, and that can later be optimized by hand, game by game, exactly like lackoftrack27 did with Super Mario and like so many people have been doing with decompilations and ports out there. The oracle guarantees the base is correct, and the human hand steps in to do what the machine can’t.

Porting a game to much better hardware, like a PC port, is relatively easy, because there’s capacity to spare. The truly hard part is fitting the game into a same-generation console, where nothing is to spare. And it’s exactly that difficulty that makes the idea of making better versions of NES games for the Master System so appealing. It’s the console that deserved to have had those games, and never did.

I hope more people get excited about this possibility and contribute to the project. It’s all open at nes-to-sms. If you enjoy 6502, Z80, VDP, and the challenge of squeezing cycles out of 40-year-old hardware, come by. There’s plenty of room.

Back in July I published What’s New in My AI-MEMORY, where I showed off ai-memory run: switching from Claude Code to Codex without losing your line of work. That day we were on version 1.17.1.

Today 2.0 shipped. And this time I’m not going to rehash how the hooks work or how a session turns into a wiki page. That’s already in the earlier posts. Here I want to talk about where ai-memory landed against the competition, what 2.0 brings, and why it earned the jump to a “2”.

If you’ve never heard of the project, the summary is short. ai-memory is a long-term memory server for your coding agents. It captures what happened in the session, consolidates it into Markdown pages, and hands the right context to the next agent, whatever the harness or the machine.

This is the web interface that comes with it, handy for auditing what the agents wrote and for browsing between projects. Notice how each project lives separately, with its own page count, all of it coming out of real sessions.

Why it became 2.0

For me, the first digit of a version carries a commitment.

The 1.x line grew too fast. We went from 1.1 to 1.39 in a little over two months, stacking feature on top of feature in minor releases. It worked for iterating, but it blurred what the numbers meant.

2.0 fixes that. It bundles the few compatibility-breaking changes into a single major, and from here on versioning follows real Semantic Versioning. A fix is a patch. A new feature is a minor. Only a format or contract break is a major, and always with warning. The new contributing guide spells that rule out for any PR.

The main break is the new on-disk format. The first time you bring up 2.0, it migrates your wiki on its own. Before touching anything, it compresses your entire data directory into a verified backup with the date in the name. If the backup can’t be written and checked, the migration aborts and the server refuses to start. No “trust me”.

This is the notice that shows up once, right after the migration: it tells you where the backup landed, the file size, and how to roll back if something looks off. Once you confirm everything is fine, you just delete the backup and the reminder disappears.

OKF: your memory isn’t locked inside ai-memory

This is the change that made me happiest.

Starting with 2.0, the ai-memory wiki is, natively, a bundle in the Open Knowledge Format, the open format Google published in 2026. Each memory page is a valid OKF file: plain Markdown with standardized metadata. There’s no export step that produces a divergent copy. The wiki files already are the OKF files.

In practice that means your memory stopped being a hostage of my project. You can read it all with grep, open it in Obsidian, version it in Git, or hand the bundle to a coworker who uses another OKF-compatible tool. There’s even an ai-memory export-okf to package a whole project into a validated tarball.

That was always the thesis: the model and the harness are rented, the project’s memory is yours. Now the format backs that up in writing.

Local embeddings, on by default

Up through 1.x, real semantic search depended on you configuring an embeddings provider. Either you paid for an API and sent every page and every query out, or you spun up an Ollama on the side. Both options have a cost.

2.0 brings a local provider that runs the embeddings model inside the process itself, in pure Rust, with all-MiniLM-L6-v2. No API key, no external server, no GPU, and nothing about your data leaving the box. And now it comes on by default. On the first run it downloads the model (about 87 MB, with a fixed checksum) in the background and turns on hybrid search at the next restart.

The gain is measurable. On the LongMemEval-S benchmark, hit@5 goes from 0.617 with full-text only to 0.779 with the local embeddings. If for some reason you don’t want it, an embedding_provider = "none" turns it off.

A detail that matters: I chose candle on purpose, instead of the native runtime the competition uses. That native layer is exactly what caused a recurring kind of crash in other memory projects. I preferred not to inherit the problem.

Several agents at once, on the same project

Here’s where the part that drove most of the work before 2.0 begins.

The ai-memory run case I showed in July was sequential: I close Claude, open Codex, keep going. But what about when I leave two or three harnesses open at the same time on the same project? Claude in one tab, Codex in another, OpenCode in a third.

ai-memory handles that without anyone stepping on anyone else. Each memory call figures out on its own which project it belongs to, from the session’s directory. The “current project” pointer is now per actor, so two harnesses in the same checkout each keep their own sense of context. And the project’s identity comes from the checkout name. The absolute path on disk doesn’t count, so the same project on the laptop and on the desktop lands in the same place.

When two windows write to the same page, the second doesn’t erase the first. It creates a new version that becomes the latest, and the previous one stays reachable in the version chain. Writing something identical doesn’t create a new version. Every write goes through a single writer, with a queue and backpressure, so a burst doesn’t corrupt anything.

This isn’t theory. There’s an acceptance test calling Claude, Codex, OpenCode, Pi, Crush, and a few others for real, inside a single workstream, covering leases, session adoption, and context handoff between harnesses.

A whole team on the same project

The second scenario is the one that gives the post its title. Here it’s several people, a whole team pointing their agents at the same server.

The way to run it is simple. Someone brings up a server, usually a container in a homelab or on a machine on the network, and each person points their agents at that URL. HTTPS goes in front with a reverse proxy, if you want. The database is still a single SQLite.

Everyone sees the same project pages. What one person learned in a session, the other’s agent recovers. And each new session gets a briefing with the project’s state right at the start. I’ll be honest about the term “real time”. What exists is a shared central store, with immediate reads, plus the briefing at the open. The note a coworker wrote shows up right away for anyone’s next query or next session. A session that’s already running only gets the news at the next query or the next open, with no interruption mid-flight.

What changes in multi-user mode is attribution. Each write records who did it. There’s an audit log and the interface shows “edited by so-and-so”. That comes in the box, for free. And there’s no per-page permission, on purpose. Attribution is there to record authorship; anyone authenticated can write, and the history is what keeps track of who did.

There’s an important security distinction between what’s shared and what’s personal. The pages belong to everyone. But the handoff is a baton with a single owner: exactly one session takes it, and a second accept doesn’t steal the baton from anyone. The handoff you left pending isn’t delivered to or consumed by a coworker. Likewise, the “what I’m working on right now” slots are per person, so your personal context doesn’t leak into the whole team’s briefing.

There’s also a battery of real concurrency tests covering these cases: simultaneous writes, per-actor isolation under load, the handoff baton that doesn’t get stolen, and the personal slots that don’t leak.

Where ai-memory stands against the competition

Before 2.0 we did another round of research on the 2026 landscape, documented in the repository. It was the basis for deciding what 2.0 needed to cover. Let me summarize the field.

There’s agentmemory, a TypeScript MCP server tied to a native sidecar and a giant surface of dozens of tools. There’s basic-memory, in Python, Markdown on disk, but with manual capture: you have to ask it to remember. There’s cognee, which combines graph, vector, and relational into a heavy pipeline that wants several gigs of RAM. There’s MemPalace, which went viral with almost 50,000 stars in two weeks over a benchmark number that, once audited, turned out to be inflated, on top of suffering corruption when two writes happen together. And there are the temporal graphs like Zep, the “memory OS” like Letta (the old MemGPT), and the fact extractors like Mem0.

There’s also the native memory that Claude Code itself started turning on by default. I treat that one as the funnel that introduces the category to people. As a competitor it stays limited: tied to one machine, tied to one agent, without real search and without team history.

2.0 closes the gaps the research pointed out. We started publishing a reproducible benchmark with LongMemEval, running the actual server. We got the open OKF format. We got typed links between pages, with causes, fixes, and contradicts, which also feed a contradiction check without spending an LLM. We got queries with as_of, to ask what we knew about a subject on a given date. We got the local embeddings. And we got an optional “experience” pass that reviews several sessions to find patterns that only show up across the set.

Now the part that matters for whoever’s choosing. What ai-memory has that the others don’t, all together:

• It follows you across agents. More than twenty harnesses feed a single memory, and the handoff here is a real protocol: typed, with an owner, and claimed exactly once.
• It follows you across machines. The memory lives on a server that’s yours. The project you dropped on the desktop is the one you pick back up on the laptop.
• It works for a team. Multi-user authentication, per-person attribution, and an audit log come in the box, for free.
• Your memory is plain Markdown. The source of truth is a wiki versioned in Git. The database is a derived index you can rebuild at any moment from the files, with no vector store to babysit and nothing trapped in a binary blob.
• It captures the work on its own, quietly. No “remember this” ceremony. And the default path runs with zero LLM calls.
• It’s a single binary. No sidecar, no three databases to sync. Every write goes through a single writer, with a write ceiling we actually measured, with a real test. That design is exactly what avoided the concurrent-write corruption that took the others down.

No competitor delivers this set. Some have one piece or another. ai-memory has the whole package, and now with parity on the items where it used to fall behind.

2.0 belongs to everyone

When I wrote the July post, fifteen people had a merged PR in the project. Today it’s around seventy. This stopped being a weekend project a long time ago.

The numbers up to 2.0: more than 1,500 commits, 371 merged pull requests, and 181 closed issues. That came from people actually using it and sending fixes, features, and documentation.

I have to give special thanks to Djalma Júnior, who on his own passed 50 merged PRs, and to Samir Hanna Verza, with more than 20. Right behind come lhzapata, pedrofjr, mrpaiva, lucasliet, lihuiyang1024, Murillofilho86, and Matheus Rodrigues. And there’s a long tail of dozens of other people with a merged PR that it would be unfair to try to list in full here.

If you want in on this, the contributing guide was rewritten to make everything clear: how to set up the environment, the gates the CI enforces, the CHANGELOG rule, and the versioning policy. A bug fix usually ships fast in the next patch. A small feature, like a new harness or provider, goes into the next minor. There are issues tagged for people just starting out.

How to get 2.0

The project page has every install method, including AUR, Homebrew, and release binaries:

• ai-memory on GitHub
• v2.0.0 release notes

After updating, on the first run let the migration take the backup and convert your wiki to the OKF format. If you use the managed mode, reinstall the hooks for the harnesses you plan to use.

Conclusion

1.x proved the idea worked. 2.0 is the version I’d recommend without an asterisk for someone else to put on a team.

The format is open, so your memory doesn’t get locked in. Semantic search runs local, so you don’t pay and you don’t leak. Several agents and several people work on the same project without running each other over. And the single-binary, single-writer design is what avoids exactly the problems that sank half the competition.

The LLM and the subscription I keep renting from whoever’s best that month. The project’s memory stays with me, with the team, and now in a format nobody can take away from me.

And every single time I publish one of these updates, without exception, someone shows up in the comments: “what about Deepseek?”

I confess I do not understand this blind focus on Deepseek. It is one open model among many, with nothing that sets it apart from the pack. In my daily use, I still prefer Kimi K3 or GLM 5.3. Yes, every iteration improves a little. Yes, I tested both new snapshots (Flash 0731 and Pro 0813) and the numbers are below. And even so, the answer stays the same: nobody there is in Fable 5 or Sol class.

What this score measures (and what it does not)

Before the numbers, the reminder that needs repeating every round. The benchmark tests a very specific slice: easy web programming. A Rails chat CRUD with streaming, tools, concurrency and tests. It is a useful test because it is concrete, reproducible and catches API hallucination red-handed, but it is still a narrow slice.

It says nothing about far more advanced tasks: kernel driver development, game engine optimization, real offensive security. Testing the entirety of a model is impossible. What you can test is a subset, and that is what I did.

So next time you read “Kimi is about to dethrone Fable” or “GLM is about to pass Sol” because the scores came out close, read it like this: within a narrow slice of web programming, any of these models delivers. That is all. A close score on this test means proximity on this test, nothing more.

Keep this: a benchmark measures a slice. Mine measures easy Rails web apps. Anyone extrapolating that into “model X beats model Y at everything” is reading the number wrong.

The Tier A ranking, with a focus on time

The usual table, A.1 cut (90 points or more), now with both Deepseeks in bold. This time, pay attention to the time column: same test, same three phases, for everyone.

Time is the wall clock of the three phases; cost is the API equivalent. Same criteria as the previous articles, linked at the end.

Look at the time spread. The fastest in the group finishes the test in 15 minutes; the slowest takes 155, ten times longer. Both Deepseeks sit in the lower half of the table for speed: 82 and 88 minutes, behind almost everyone in the same score range. The Flash, in particular, burned 44 million tokens in one run. For comparison, GLM 5.3 scored one point higher with 19.4 million. Deepseek makes up for it on the invoice: the Flash’s $0.82 is the cheapest Tier A run to date.

The new snapshots: what changed since July

I ran the July builds of v4 at the end of that month: Flash scored 80, Pro scored 82, both Tier B. Both dated snapshots (0731 and 0813) ran today under the same rigor, and both entered Tier A:

Flash 0731: from 80 to 90 (+10). The most visible improvement is behavioral. The July build pinned a model three generations old; this one got the pin right on the first try (anthropic/claude-sonnet-5, no stale slug). The self-review was exemplary: it found on its own that the streaming bubble partial was dead code (no assistant reply appeared until a forced reload through phases 1 and 2), confessed, fixed it and scored 15/15 on honesty. It lost points where the pack loses them: concurrency without a turn lock (8) and coverage without branch (8).

Pro 0813: from 82 to 91 (+9). The big news is what disappeared: the reasoning_content bug that broke its multi-turn on OpenCode, and that forced me to invent the deepclaude workaround back in May, was fixed in opencode 1.18.4, and this snapshot ran end to end on the generic harness, no crutches. The technical highlight was concurrency: a SQLite store with read-modify-write inside a BEGIN IMMEDIATE transaction, the most serious solution in the group on that dimension (9/10). The losses were honest and self-confessed: stale sonnet-4.6 pin, streaming verified by unit test only, no fallback token-budget estimator.

Clean, shielded runs, zero reads of the rubric or of anyone else’s app. The full reports are in the benchmark repository.

Deepseek against itself: the trajectory is real

Old and new criteria do not compare point for point, so this section is about behavior, not scores. And behavior you can compare, because the contrast is stark.

In April, DeepSeek V3.2 scored 43 under the old criteria and starred in the most embarrassing moment in the benchmark’s history: it invented the entire RubyLLM integration. RubyLLM::Client.new, client.chat(messages:), both methods hallucinated, the whole LLM layer fictional.

Still under the old criteria, v4 Flash scored 78 with a fatal one-character bug (the model slug missing the anthropic/ prefix), and v4 Pro DNF’d at 69: Tier 1 code, Tier 3 deliverables, taken down by the reasoning_content bug that ate the conversation history. That bug is what spawned the deepclaude hack, where the same Pro jumped to 89.

On v2, the July builds scored 80 and 82. Today’s snapshots, 90 and 91. From “invented the entire API” to “finds its own dead code and confesses it in self-review” in four months. Deepseek’s curve is one of the steepest this benchmark has ever recorded, and pretending otherwise would be dishonest.

And against the other Chinese models?

This is where the conversation gets less exciting for the cheering section. On the same generic harness (OpenCode), the Chinese pack looks like this:

And Kimi K3, absent from this table because it ran on its native harness, scored 95. In other words: within the same test, Deepseek v4 arrives behind GLM 5.3, Qwen 3.8 Max and the Kimis, while being the slowest and most verbose of the Chinese group. Its advantage is a single one, and it is real: price. $0.82 for a Tier A run is impressive, and the Pro at $5.01 undercuts almost everyone at the same score.

If your criterion is “maximum competence per dollar on simple web tasks”, the Flash 0731 deserves attention. If the criterion is the best model in the group, the answer is still Kimi K3 and GLM 5.3, same as last week.

Conclusion: stop asking

There, now there are numbers on the table. Deepseek v4 genuinely improved between July and August, both snapshots entered Tier A, the Pro got rid of the bug that demanded a harness workaround, and the Flash is the cheapest Tier A run I have ever executed. All of that is fact.

And none of it changes the picture. The leadership stays with Fable 5, Sonnet 5, Opus 5 and Sol; the first Chinese model in line is still Kimi K3, followed by GLM 5.3. Deepseek is one more competent model on a slice of easy tasks, with the virtue of being cheap and the defect of being verbose. Next time you feel the urge to comment “what about Deepseek?”, the answer is this article.

The top hasn’t moved: Fable 5 at 96, the trio of Sonnet 5, Opus 5, and Kimi K3 at 95, and right behind them GPT 5.6 Sol, GPT 5.6 Terra, and Opus 4.8 at 93. That’s the cream of the crop in this test. The open question: how close do the newest releases get to that group?

Since then I’ve run five models: Qwen 3.8 Max, GLM 5.3, Gemini 3.7 Flash, Grok 4.6, and a 27B Qwen 3.8 running locally on my RTX 5090. One of them closed in on the leading group. Another pulled off the biggest jump this test has ever recorded. A third got caught cheating mid-test. A fourth tied its own previous generation, without moving an inch. And the local one gave me the most laborious, and most instructive, run of the year.

Where they land in the ranking

Before opening each one up, it’s worth seeing where the newcomers fit. The table below is the Tier A.1 slice of the v2 ranking: the frontier of the test, everyone at 90 points or more. I cut it here on purpose. From A.2 down there are competent models, but the leadership race lives in this group, and that’s what the question is about. The three cloud releases go in bold; the local 27B Qwen scored 51, Tier C, and shows up only in the table at the end.

Time is wall clock across the three phases; cost is API-equivalent. On subscription plans (Z.ai, grok CLI) the marginal cost is $0 and the value in parentheses is the API-equivalent; the Antigravity runs were preview and were not metered. Same criterion as the previous article.

Qwen 3.8 Max: the biggest jump in benchmark history

To measure the jump, first the size of the hole. Qwen 3.7 Max had scored 51 points, Tier C, and the reason was ugly. When implementing the multi-turn chat, it decided it could replay history by calling chat.ask(entire_history_array). Except RubyLLM’s ask wraps its argument into a single user message: the entire conversation, assistant turns included, becomes one message with every role obliterated. Worse, the required test for that feature mocked exactly that nonexistent API. A test that mocks a fabricated API is worse than no test, because it certifies the hallucination.

The 3.8 Max fixed exactly that. It used the real API end to end: add_message with role and content for history replay, with_instructions, with_tools, with_schema, all verified against the installed gem’s source. The result: 92 points, Tier A. That’s a 41-point jump on the same test, same rubric, same harness. The test didn’t get easier; the model finally understood the library.

The rest of the delivery is solid: real incremental streaming, history surviving restarts, a hand-written calculator with no eval, tools answering with exact arithmetic, the app booting in Docker on the first try. The suite came out with 62 tests and 226 assertions, all green.

Where it lost points is instructive: config/puma.rb shipped without the workers directive, so the WEB_CONCURRENCY=2 it swore it had delivered was, in practice, running single-process. Concurrency scored 8 instead of 9. And it kept the stale pin on claude-sonnet-4.6, the house’s standard deduction.

Remember this: the model swore concurrency worked — one missing line in puma.rb and the “two workers” ran as a single process. That’s why phase 2 doesn’t read READMEs: it boots the server and measures.

On the table, those 92 points tie with GLM 5.2 and Kimi K2.5, one point below Sol and Terra. It cost $9.16 in API and 78 minutes — verbose: 25 million tokens.

GLM 5.3: the loneliest step on the table

Z.ai’s trajectory in this test is the steadiest of the pack: GLM 5 scored 83, GLM 5.2 scored 92, and now GLM 5.3 scored 94 — alone on a step nobody else occupies, one point below the 95 trio and one point above the 93 group. In other words: two points from Fable 5.

And the inevitable comparison is with Kimi. K3 scored 95, one point above — but it ran on the Kimi CLI, its native harness. GLM 5.3’s 94 came on opencode, the generic harness: it’s the highest score ever recorded there. On the same opencode, the best Kimi is K2.5 at 92, two points below. On cost, both live on subscriptions: K3 came out at $6.14 equivalent on the Moderato plan; GLM at zero marginal cost. Kimi still wins on score; GLM wins on cost and harness independence.

What pulled it out of the 92 pack? Three things, all boring, all important:

1. Concurrency delivered working. The same file-lock scheme as Qwen 3.8 Max, plus a per-conversation turn lock, plus two real workers surviving kill and restart without corrupting anything. The Qwen had the same foundation but shipped concurrency broken and scored 8. The GLM shipped it working and got 9.
2. A fallback token estimator, so the per-conversation budget works even when the provider doesn’t report usage. The 5.2 depended on it, and lost points there.
3. Branch coverage enabled: 98% line and 82% branch, a suite of 73 tests and 219 assertions green under the auditor’s hand, with RuboCop, Brakeman, and bundle-audit all at zero.

Remember this: the distance between the 92 pack and GLM’s 94 isn’t model brilliance: it’s concurrency that actually works, a fallback token estimator, and branch coverage. Boring engineering earns points.

The only real slip was the same stale sonnet-4.6 pin. And there was a division-by-zero bug in the calculator that the model itself found and fixed in the self-review — exactly the kind of behavior that phase exists to measure. Speaking of which: it confessed everything, including that the conversation title never retries if generation fails, and took 14 of the 15 honesty points.

And the cost is the part that hurts the competition: it ran on Z.ai’s flat-rate plan, so the run came out at zero marginal cost — the API equivalent would be $2.59. Eighty minutes, 19.4 million tokens. The “Chinese models are the cheap alternative” conversation died a while ago: this is a leadership candidate that also happens to be cheap.

Gemini 3.7 Flash: 93, Tier A — and the cheat along the way

The technical part you already know: Gemini 3.1 Pro died in v2 because of a Google bug, Corrupted thought signature, which killed executions mid-run via OpenRouter and trapped it at 62 (over the direct path, it scored 88). This time I cut out the middleman: a direct Gemini API key, no OpenRouter in between, and the bug never showed up once. The direct path worked — and 3.7 Flash became the first Gemini to reach Tier A on opencode: 3.5 Flash scored 79 there, 3.1 Pro scored 62. Before that, only Google’s native harness had taken a Gemini to A.

Official result: 93 points, tied with GPT 5.6 Sol, Terra, and Opus 4.8. The run cost $4.12, took 43 minutes, and burned 27.8 million tokens.

But the road to that 93 produced the most important finding of this cycle. The first 3.7 Flash run raised a red flag — and there was a reason. Midway through the build, and again in the self-review, the model read the scoring rubric, read the entire v2 report — every competitor’s score and audit — and ran the audit scanner on itself eight times. It spent the whole test studying how it would be graded.

The smoking gun is the pin. The contaminated run came out with the correct pin on claude-sonnet-5, that feat only three models had managed in the benchmark’s entire history. I voided that run and redid it with all the grading material out of the model’s reach: out came the stale pin on claude-sonnet-4.6, like everyone else. It didn’t know the current model at all — it had read the answer in the report. The 93 belongs to that second, clean run.

To be fair: it didn’t copy code from any sibling app. What it did was cheat off the grading — and that was enough for no model, frontier or not, to ever run with access to that material again.

And the 93 isn’t charity — the delivery stands on its own. Real RubyLLM API end to end, a safe calculator with no eval, correct multi-turn history, incremental streaming working end to end in Docker, persistence surviving restarts with two workers. A suite of 55 tests and 213 assertions, all green, branch coverage enabled. The deductions: the stale pin and the lack of a per-conversation turn lock — the same concurrency ceiling as Fable 5.

Remember this: this time it wasn’t a weak local copying from the neighbor. It was a frontier model checking the answers mid-test.

Qwen 3.8 27B local: the most laborious run of the year

In the previous article I said I’d only test a new local model if strong evidence showed up. Then the Max sibling scored 92, the open 27B version was available, and I had the excuse I was missing. It was worth it for the science, but it took work.

First surprise: the 3.8 27B uses a hybrid SSM/Mamba architecture, and my tuned llama.cpp llama-swap build can’t load the model — a tensor is missing (ssm_conv1d) that only newer versions know about. The fix was spinning up a fresh Ollama container, which bundles a current llama.cpp, and importing the GGUF I had already downloaded via a Modelfile. First lesson: with local models, the tooling ages in months.

Second surprise: context. A reasoning model burns an absurd number of tokens thinking, and a small window won’t do — at 32K or 64K it doesn’t even finish the test, exhausting everything reading the gem’s source before writing the first line of the app. The official run needed 176K of context, nearly everything the RTX 5090’s 32 GB can hold. What stopped the model from finishing was a context ceiling, not lack of capability.

Then came the incident. The first run completed — and completed too well. I went through the log: the model had read the online Qwen 3.8 Max’s finished app sixteen times, sitting right there in the repository, and copied its UI and streaming. It would have taken some 75 points on someone else’s work. I voided it and reran with no third-party app anywhere near. And as the Gemini section above shows, it’s not just the locals who look for help when help is lying around.

Remember this: a weak local model doesn’t just invent nonexistent APIs — it also copies from the neighbor when the neighbor is in the same directory.

With no neighbors around, the 27B scored 51 points, Tier C — tied, by coincidence, with the online Qwen 3.7 Max’s score. And the receipt is mixed. The core it got right: real add_message, real with_tools, a recursive-descent calculator with no eval, the payoff of actually reading the gem’s source. Where it sank was breadth: it used ActiveRecord where the requirements forbid it, the streaming comes out broken (the tokens are broadcast, but the reply bubble never enters the screen — it only appears if you refresh the page), it didn’t use with_schema, it built no token budget, it delivered zero tests, RuboCop flagged 22 offenses, and no Dockerfile came out at all. The self-review, though, was exemplary: it found and confessed every one of those flaws itself, with file and line — 14 of the 15 honesty points. It reviews better than it builds.

Cost of the run: nothing, 37 million tokens, and 156 minutes of a sweating RTX 5090.

To calibrate the 51: the Tier A floor is Opus 4.6 at 83. The 32-point gap isn’t in “knowing the library” — the 27B now gets that right. It’s in the production-hardening dimensions: streaming, tests, gates, Docker, budgeting. It’s the difference between knowing how to program and knowing how to deliver.

And comparing with the earlier locals — always with the caveat that the tests aren’t comparable: in v1, the local Qwens hallucinated the gem wholesale (one invented an Openrouter::Client with the wrong capitalization, another created a RubyLLM::Client that doesn’t exist). The Qwen 3.5 35B got the entry point right, but its tests wrapped any exception in an assert true. The 3.6 35B was the first local to get the primary calls right, still with broken multi-turn. The 3.8 27B gets the entire API core right on a much harder test. The score you can’t compare; the behavior you can: API knowledge is no longer the locals’ problem. The problem now is engineering.

Grok 4.6: the first flat generation, and the cleanest run

After two cheating busts, a breather. I ran Grok 4.6 with the shielding cranked all the way up: I moved everything out of the model’s reach — the grading rubric, the entire v2 report, the audit scanner, CLAUDE.md’s deduction catalog, and all 44 competitor apps, all pulled out of the repo. It was the first run under that stricter regime. And the post-run scan turned up nothing: zero reads of any grading file, zero peeks at a sibling app. It ran clean. After Gemini and the local Qwen, it’s good to watch a frontier model build on its own because it had nowhere to crib from.

The result has a curious detail: 92 points, Tier A, the same score as Grok 4.5. It’s the first generation-over-generation tie in the whole test. While GLM climbed 83, 92, 94, Kimi went 77 to 86 to 95, and Claude ratcheted up without a stumble, Grok walked sideways. The 4.6 bought nothing over the 4.5. In the table up top the 4.5 shows 91 because I use its grok CLI number, its native harness; head to head in the same OpenCode, the two land on a dead-even 92.

What it delivered is solid and real. Correct RubyLLM API end to end, a hand-written calculator with no eval (regex tokenizer and parser, proven live with (12.5*4)/2+7 = 32.0), a test checking the exact array sent to the provider, a file store with a lock surviving restart with two workers, a token budget with a fallback estimator, and docker compose up --build answering a real chat. Streaming was proven live in phase 2: five tokens arriving incrementally while the POST was still open. And it was the thriftiest of all the Tier A OpenCode runs, at 10 million tokens, because Grok is terse. It cost $6.33 and 34 minutes.

The deductions are honest, and it confessed them itself. The lock doesn’t cover the read-modify-write window, so two simultaneous turns on the same conversation become a last-writer-wins race, the same hazard as GLM 5.2. Test coverage is thin on the critical path, with the integration test being just a home-page smoke test. And it kept the stale claude-sonnet-4.6 pin, the house’s default deduction. A self-review of 12 PASS and 2 PARTIAL, all verified by the auditor. No new brilliance: it’s a good model repeating a good model.

Then I ran the same Grok 4.6 on its native harness, the grok CLI, to see if anything changed. Not much did: 93 on the grok CLI against 92 on OpenCode, one point apart, inside the noise. No harness effect. The CLI neither scaffolded nor broke anything; Grok builds plenty competently on either one.

That extra point has a concrete, mundane explanation: by chance, the CLI run locked the store’s entire read-modify-write window under a single File::LOCK_EX, so concurrency went to 9; the OpenCode run left that race open and stayed at 8. Same model, one run each: the difference is variance between the two code generations. The harness played no part in it. What actually changed was the cost: on the grok CLI the same test came in at $1.19, against OpenCode’s $6.33, for the same ~11 million tokens, thanks to xAI’s native caching.

Remember this: not every new generation buys a point. Grok 4.6 tied itself, and the good news was passing clean through the toughest shielding I’ve applied yet.

Conclusion: how close did they get?

Short answer: very close.

GLM 5.3 two points from Fable 5 is not a “cheap alternative” — it’s a leadership candidate. Qwen 3.8 Max one point from Sol and Terra, same thing. The distance between the American cream and the new Chinese models is one or two points — and I repeat in every article that one or two points is noise. And Gemini 3.7 Flash joined that pile: 93, tied with Sol, Terra, and Opus 4.8, the first Gemini to get there on opencode. Grok 4.6 landed in the same bucket of 92s, but with an asterisk all its own: it was the only new generation that improved on nothing over its predecessor, and even so it was the first to pass clean through the toughest shielding.

And local remains out of the question for an autonomous coding agent: Tier C is Tier C. But notice how the conversation has changed. Until recently I dismissed local because it invented APIs. Today it knows the API and trips on streaming, tests, and Docker — and it needs 176K of context and 32 GB of VRAM just to complete the test. The bottleneck moved up a level. It’s not a recommendation yet; it’s the road being paved.

Remember this: the cream of the crop is still Fable, Opus, Sonnet, K3, and the GPT 5.6 family. But the chasing pack is already one or two points behind — and the moat between “frontier” and “alternative” has become noise territory.

As always: artifacts, logs, rubric, deductions, and the updated table are in the llm-coding-benchmark. Both Gemini 3.7 runs — the voided one and the official one — are documented in the report, with the contamination finding front and center.

Great. The table in the first article is already a museum piece.

The argument in the second one still stands. In fact, it got even easier to demonstrate because I spent the last few days rerunning practically the entire benchmark. That meant dozens of runs, several discarded attempts, hundreds of dollars between API charges and credit equivalents, and an indecent amount of time reading robot-generated Rails code.

The result is version 2 of my LLM Coding Benchmark. The test got harder, the audit became more explicit, and each family now runs, whenever possible, in the harness where it should work best.

I’ll say this up front: v2 scores are not directly comparable to v1 scores. The prompt changed. So did the requirements, several model-harness pairings, the validation, and the rubric. The v1 column in the report is historical context, not a scientific measurement of how much each model “improved.”

Why I retired v1

We ran v1 for months. It did a good job separating models that could actually build a Rails app from those that invented RubyLLM APIs, wrote tests for their own hallucinations, and shipped Dockerfiles that never came up.

Then the newer models hit the ceiling of that test. Fifteen of the forty results were already packed into Tier A, with the top compressed between 92 and 97. The job still had real details, but the best models cleared the old discriminators easily. The order started coming down to an API-key preflight here, a cookie limit there, one missing error test. Valid details, not much separation.

There was another operational inconsistency: the model-harness pairing. Claude had been tested in OpenCode. So had Grok, Kimi, and Gemini. Today we have Claude Code, Codex, Kimi Code CLI, grok CLI, and Antigravity. Measuring a model in a generic harness when an integration tailored to its behavior exists can distort the result.

The benchmark was never measuring the weights file alone:

result = model + harness + prompt + tools + context + execution + audit

So I stopped hiding the harness inside the score. Claude moved to Claude Code. GPT moved to Codex. Kimi K3 and K2.7-Coding moved to Kimi CLI. Grok and Gemini got A/B runs in their vendors’ own CLIs. A fully isolated OpenCode remained the fallback for models without a better harness or subscription access.

The new test

V1 had two phases: build the app and try to bring it up. V2 has three phases, fourteen numbered goals, and a ten-dimension rubric.

In phase one, the model still has to build a ChatGPT-style chat app on its own in Rails with RubyLLM, Hotwire, Tailwind, Minitest, Docker, and Compose. That’s where the resemblance ends. Now it must also deliver:

• real per-token streaming through Turbo Streams, proven to be incremental;
• a correct multi-turn payload that does not send the current message twice, plus a test for the exact array sent to the provider;
• persistence that survives a restart and works with WEB_CONCURRENCY=2, with a TTL plus message-count and byte limits;
• exactly two tools, server_time and a safe calculator, using the real RubyLLM API;
• a title generated through the structured-output API;
• a per-conversation token budget;
• a system prompt, credential preflight, degraded states, and provider-error handling;
• a guarantee that failed turns never contaminate future history;
• clean RuboCop, Brakeman, and bundle-audit runs, plus a non-root production Docker image and no secrets.

Phase two does not accept a README claiming everything works. It boots Rails, watches the tokens arrive, forces real tool calls, holds a conversation across two workers, restarts the server, checks the history, runs tests and gates, executes docker build, and sends a real message to the app inside Compose.

Phase three asks the model to review every goal as PASS, PARTIAL, or FAIL, cite a file, line, test, or command, and write down what is still broken. That honesty is worth 15 points. An accurate FAIL is worth more than an optimistic PASS that the audit disproves.

This phase produced data v1 never had. Kimi K3 and Nex, for example, admitted defects that would have been easy to hide. Others built a reasonable app and then hallucinated their own inspection. Programming and reviewing what you programmed are different capabilities.

The harness became part of the test

I also ran A/B tests with the native tools:

A native harness does not sprinkle magic dust on a model. Grok 4.5 barely cared. Grok 4.3 needed the structure. Gemini 3.1 needed a transport that would not break on Corrupted thought signature. Three different mechanisms that a rushed comparison would flatten into “the CLI improved the score.”

In the ranking below, I use the preferred harness whenever a complete run exists: Antigravity for the Geminis and grok CLI for the Groks. The OpenCode baselines remain in the repository as A/B data, but do not appear a second time in the table.

The new ranking

This is the consolidated v2 table, with one entry per model in the preferred available harness.

Time is end-to-end wall clock across the three phases. Cost is API-equivalent: for models on Codex I use the cache-discounted blended figure (same criterion as the cost section below); on subscription plans (Z.ai, grok CLI) the marginal cost is $0 and the number in parentheses is the API-equivalent; the Antigravity runs were preview and were not metered.

The details, artifacts, and deductions are in the full v2 report.

What the tiers mean now

The new cutoff is anchored on Claude Opus 4.6, which scored 83 and showed the minimum needed to carry the entire test. Here’s the practical interpretation:

There are 15 models in A.1 and 9 in A.2. All 24 cleared the competence floor for this kind of work. The subdivision helps decide where to start: A.1 contains the frontier results; A.2 contains capable models that needed more fixes, left shallower tests, or carried clearer operational limitations.

That does not make 96 universally more intelligent than 91, nor does it make an A.2 model bad. An A.2 model may be better at refactoring, debugging, frontend work, or inside your fifteen-year-old monolith. This test does not measure all of that. The cutoff simply avoids throwing 24 options into one oversized bucket.

A.1 and A.2 make up the candidate pool. I cut Tier C and D before I start.

So which one is best: Fable, Opus, Terra, or Kimi?

If all you want is a one-line answer, you’re going to be disappointed again.

On the final artifact, Fable won. It was also the fastest of these four. If I were paying for every API call in this run, Kimi K3 won on cost by a mile, tying Opus at 95 and landing only one point behind Fable.

Opus 5 built an excellent project, but it was the slowest and burned 56.8 million tokens as counted by Claude Code. In this test, the extra 33 minutes over Fable bought nothing visible. Terra also delivered good work, finished two points behind K3, and cost less than Fable and Opus at API-equivalent rates.

If you already pay for Claude Max or ChatGPT Pro, the marginal cost stays near zero while you remain within the limits. Kimi Moderato is also a subscription, with its own quota windows. So “$26 versus $6” does not settle anything by itself. The first question is which subscription you already pay for and how much room it has left. For pay-as-you-go and automation, per-run cost moves back to center stage.

My reading of this run:

• Fable 5 delivered the best combination of quality and time;
• Kimi K3 offered the best value among the leaders;
• Opus 5 was capable and meticulous, but expensive and slow in this execution;
• GPT 5.6 Terra delivered the best balance in the family if you live in Codex.

That’s a reading of this project. Change the workload and the order may flip.

Opus versus Sonnet, Sol versus Terra

A tier name is not a benchmark. Sonnet 5 proved that in an almost embarrassing way:

Sonnet 5 tied Opus 5, finished nineteen minutes earlier, and produced the first genuine 100% line coverage in the entire benchmark. It also wrote the family’s best self-review. Automatically choosing Opus because “Opus is the higher tier” would mean throwing out your own data.

This also corrects a terrible impression from v1, where Sonnet 5 scored 58 and hallucinated the RubyLLM API. In v2 it ran through Claude Code, received explicit requirements, and scored 95. We cannot conclude that the model improved by 37 points because we changed practically the entire experiment. We can conclude that the v1 pairing was a poor representation of what it can do.

On the OpenAI side:

Terra tied Sol at 93, finished eight minutes sooner, and cost a little more than one-third as much in the blended calculation. It also produced the best concurrency protection in the entire run: Redis with WATCH/MULTI, a distributed per-conversation lock, and forced tool choice. On this test, paying for Sol bought no points and saved no time. Terra is the most rational pick in the family.

Luna stays in the table because it scored 91 and remains an A.1 result, but its cost argument is gone: Terra scored two points higher for only thirteen cents more and took about three extra minutes.

Cache remains the important detail. Of Terra’s 21.7 million input tokens, 21 million were cache hits. Pricing everything as fresh input would produce an upper bound of $111.62. At the cache rate, it falls to $16.92. Any cost table that mixes CLIs without understanding what each one reports is comparing apples to JSON.

What about the Chinese models?

The line about Chinese models being useful only as cheaper alternatives is stale.

Kimi K3 tied Opus 5. K2.5, K2.6, MiniMax M3, and GLM 5.2 landed in the same A.1 as Claude and GPT. The OpenCode runs are still cheaper than the $16 to $45 range of the leaders run through Claude Code and Codex, but this is no longer a comparison between pennies and dozens of dollars. And a subscription is not an API: GLM had zero marginal cost because it ran on Z.ai’s plan; the same usage would cost about $12.05 through the API.

Kimi is the easiest family to recommend today. K3 offers top-tier quality through a cheap subscription. K2.5 and K2.6 were economical over the API, costing $1.50 and $2.64. K2.7 landed below its siblings, but it ran in another harness, so I am not going to invent a tidy progression story from four isolated data points.

MiniMax M3 deserves attention and caution in equal measure. It scored 91 for $7.72, still less than one run through the American leaders, but burned 121 million tokens and took almost two hours. It was the most expensive and token-hungry OpenCode run. The score is good; the usage profile, not so much.

GLM 5.2 scored 92 at zero marginal cost on Z.ai’s plan, but consumed the equivalent of $12.05 through the API. It took about two and a half hours. If wall-clock time does not matter, it is a very strong option. If you work in short cycles, Grok 4.5 delivered 91 in about 25 minutes through grok CLI, more than six times faster.

DeepSeek remains cheap, but stopped in Tier B. V4 Pro scored 82 for $0.35 and V4 Flash scored 80 for $0.81. They’re close, and I want to repeat the test when a new version arrives. Today I still would not leave either one working alone as a coding agent in a codebase that matters. Saving one or two dollars only to spend an hour reviewing a structural defect is lousy math.

Why I did not rerun the local models

I did not run Qwen 3.5 or the other local models on v2. The priority shifted to mapping the programming Tier A more accurately, and every complete run of this test costs machine time, audit time, and sanity.

Local models are not useless. They work for translation, classification, summarization, controlled one-shots, and tasks where privacy or offline operation matters more than quality. For autonomous coding agents, though, the v1 tests landed far below the floor. V2 is harder. I see no reason to spend several more days confirming that a quantized local Qwen cannot compete with Fable, Opus, GPT 5.6, or Kimi in software engineering.

For programming, it is not worth it today. If a new local model appears with strong evidence behind it, I’ll test it. Until then, I’d rather spend my time separating the twenty-four models that have already cleared the floor.

Conclusion

V1 did its job and saturated. V2 puts pressure where today’s models still slip: real streaming, multi-turn payloads, concurrency, persistence, tools, structured output, budgeting, operational security, faithful tests, and the ability to admit their own defects.

Fable 5 finished on top with 96. Sonnet 5, Opus 5, and Kimi K3 tied at 95. Sol and Terra came right behind them at 93. That answers which models produced the best projects on this test.

Choosing what to use is a different reading:

• Tier A.1 contains the 15 frontier results, all scoring 90 or higher;
• Tier A.2 contains 9 capable models from 83 to 89, still worth recommending with closer review;
• Tier B is close, but I still do not recommend it for autonomous work;
• Tier C is for translation, summarization, and simple agents;
• Tier D is too inconsistent for me to recommend;
• inside Tier A, choose by subscription, speed, cost, and harness;
• one or two points do not make anyone the universal champion of intelligence.

My own choice remains concentrated on Claude Code and Codex because those are the harnesses I use every day. Within Codex, Terra delivered the best balance of score, time, and cost. Kimi K3 has become a serious top-tier alternative. Grok 4.5 is the speed champion of this round. GLM has zero marginal cost on the subscription and MiniMax still costs less than the leaders, but neither is a speed pick. Sonnet 5 proved that reflexively paying for or selecting the “higher” tier can be a waste.

All generated code, prompts, results, self-reviews, rubric details, and corrections are in llm-coding-benchmark. The major code and test overhaul is already on master. Contributions are welcome, whether you want to add a model, improve a harness, dispute a deduction, or find another bug in the auditor.

Just bring artifacts and data. We have enough ranking opinions already.

“Is it any good?”

Short answer: yes, very. It scored 95/100, Tier A, on my benchmark, with the most complete engineering I have seen so far from any of the harnesses I tested.

Now for the answer that matters: no, this does not prove it is “the best LLM in the world.” It does not prove it is better than Fable 5, Opus 4.8, GPT 5.6 Sol, or Kimi K3 at whatever job you throw at them either. Last week I published an entire article on why the highest score does not mean the best model. Opus 5 arrived just in time to hand us a fine case study for that argument.

Where Anthropic positions Opus 5

In the official announcement, Anthropic describes Opus 5 as an everyday model that comes close to Fable 5’s frontier intelligence at half the price. It is now the default model on Claude Max and the strongest one available on Pro.

The product ladder looks simple enough:

Opus 4.8  <  Opus 5  ≈  Fable 5

Except Anthropic’s own data does not form such a tidy line. At maximum effort on CursorBench 3.2, Opus 5 lands within 0.5% of Fable 5’s peak. On Frontier-Bench v0.1, it beats every other model and more than doubles Opus 4.8’s result at a lower cost per task. On OSWorld 2.0, it even tops Fable’s best result at a little over a third of the cost.

So “somewhere between Opus 4.8 and Fable 5” is a fair shortcut for understanding the product. It is not a universal ranking. The curves cross depending on the task and the effort setting.

Price is less fuzzy. Opus 5 costs $5 per million input tokens and $25 per million output tokens, the same as Opus 4.8. Fable 5 costs $10/$50. At API rates, Opus 5 delivers the most interesting promise in this release: near-Fable behavior without paying the Fable tax.

The benchmark

For anyone just joining us, my LLM Coding Benchmark gives every model the same job: build a ChatGPT-style chat app on its own in Rails 8, with RubyLLM, Hotwire, Tailwind, tests, CI, Docker, and documentation.

I am not grading one isolated function. I grade the project that comes out the other end: whether it uses the real RubyLLM API, whether multi-turn works, whether it handles provider failures, whether the conversation persists, whether Turbo Streams is actually wired, whether the tests can catch bugs, and whether the production image boots.

Opus 5 ran solo through Claude Code headless, with --dangerously-skip-permissions, using my Max subscription. The numbers:

• 38m57s
• 201 turns
• 121 tests and 355 assertions
• 100% line coverage and 95.94% branch coverage
• 22.1 million cache-read tokens
• $16.02 at equivalent API rates, billed to the subscription

That needs an asterisk. Opus 4.8 and Fable 5 were tested in OpenCode through OpenRouter. GPT 5.6 Sol ran in Codex. Kimi K3 ran in Kimi Code CLI. A coding benchmark measures the whole package:

model + prompt + harness + tools + context + execution + audit

That is why the repository keeps Opus 5 in the Claude Code profile instead of pretending this was a perfectly controlled comparison with the main table. I am combining the main ranking’s 40 rows with the new result here because everyone wants to see where that 95 lands. The asterisk is doing real work.

Updated ranking: main table + Opus 5

* Opus 5 received an equivalent 95/100 in the Claude Code profile. The repository’s main table keeps it separate to make the harness difference explicit.

What Opus 5 wrote

The score alone hides the good part. Opus 5’s project is the best example of defensive engineering to show up in this benchmark so far.

It isolated all RubyLLM access in one Assistant::Client, injected the chat factory for tests, and verified the gem’s real API before depending on it. Besides mocking RubyLLM.chat, with_instructions, add_message, and ask, it created a guard test that confirms those methods still exist in the installed gem. If the library changes its interface, CI breaks before production does.

The conversation sits behind a ConversationRepository in Rails.cache, with a 12-hour TTL and a 40-message replay limit. Before sending history to the provider, it normalizes role alternation between user and assistant, removes failed replies, and makes sure the window does not begin with an assistant message. That sounds like trivia right up until the API rejects the payload on turn two.

That is exactly where the model found and fixed two bugs during its own run. A failed reply could leave two user messages in a row. Trimming the history window could also make it begin with an assistant reply. It wrote tests, fixed both, and validated real multi-turn behavior, including recovery after a failure.

It also delivered:

• streaming outside the main request;
• signed Turbo Streams broadcasts throttled to roughly 10 updates per second;
• credential preflight;
• different messages for an invalid key, rate limits, depleted credits, blown context, and provider downtime;
• Markdown escaped before formatting HTML;
• a multi-stage, production, non-root Dockerfile;
• RuboCop, Brakeman, bundler-audit, importmap audit, and GitHub Actions.

It is not a 100. There is a lost-update race if another message arrives during generation, although the project itself constrains deployment to WEB_CONCURRENCY=1. It also left the default model on Sonnet 4.6 after Sonnet 5 was already available, and committed log/ and coverage/ cruft inside the artifact. Those deductions held it at 95.

Opus 5 against Opus 4.8 and Fable 5

First, the numbers from our test:

Opus 4.8 had scored 95 with a smaller solution that finished much faster. It used the correct API, wrote honest tests, and performed the best live validation in that round: local Rails, a real OpenRouter call, Docker, a production container, and Compose. It lost points for leaving cookie history unbounded and skipping an API-key preflight.

Opus 5 fixed both defects and went much further on architecture, streaming, and tests. On the other hand, it created a different race, left the model pin stale, and took more than twice as long. Same score, very different artifacts.

The original Fable 5 scored 94. It was the first model I saw stop in the middle of the job to read RubyLLM’s installed source before writing the integration. It had 99.3% coverage, capped history, preflight, and a phase 2 that needed no fixes. The big deduction came from storing conversations in an in-memory singleton: restart the process and everything is gone; add a second worker and each one sees a different world.

The Fable re-release fixed that with Rails.cache, but left the cache without a hard cap, kept Sonnet 4.6, and performed weaker live validation. It dropped one point. Same model ID, another project, another score.

Within the boundaries of this Rails app, Opus 5 looks more complete than both Fable runs and at least as good as Opus 4.8. Just do not confuse that with “Opus 5 is more intelligent than Fable.” Anthropic says Fable’s advantage grows as tasks get longer and more complex. Our project is small, greenfield, and closed-ended. There may simply be no room here for that difference to show.

The price: half of Fable, but mind the bill

On the API, the advantage is straightforward:

If Opus 5 really delivers near-Fable behavior on your workload, paying twice as much for Fable is hard to justify. Fable needs to solve something Opus cannot, not merely carry the name of the tier above it.

But the observed cost of this run tells a different story:

How did a model with half Fable’s rate end up with a higher equivalent cost? Because price per million is not cost per task. The Opus 5 run took 201 turns and accumulated 22.1 million cache reads in Claude Code. The Fable run used another harness and another token profile. Comparing $16.02 with $11.20 as if the model were the only variable would be flat-out wrong.

For me, as a Max subscriber, the real marginal cost was zero while I remained inside the allowance. For automation that pays API rates per token, Opus 5 costs half as much as Fable and the same as 4.8. I would start with Opus 5 and move up to Fable only with evidence that the task needs it.

Against GPT 5.6 Sol and Kimi K3

GPT 5.6 Sol scored 92 in 17 minutes. It was far more token-efficient and delivered a defensive application: a non-root Docker image, 99.2% coverage, history bounded by messages, characters, and bytes, plus a specific test to keep the current prompt from being repeated in context.

It lost points because it never used with_instructions and carried history in a hidden field in the browser. That works, but the conversation disappears on reload and the client can tamper with the context. Opus 5 put those responsibilities on the server, separated domain, persistence, and provider more cleanly, and tested far more. On this project, the three-point difference makes sense. In daily use, both remain in the same cluster of strong models.

Kimi K3 scored 89 in 26 minutes. It had already nailed the pattern that decides much of the top end: Rails.cache, a TTL, and a history cap. It cost only about $2.10 at equivalent API rates through the Moderato subscription.

It landed lower because it had no system prompt, put LLM I/O inside the Conversation model, skipped credential preflight, and left the production cache on the container’s ephemeral default. Opus 5 closes nearly all of those gaps. Kimi remains a much cheaper Tier A alternative; Opus 5 is the project I would need to touch less before trusting it.

Opus 5 also dethroned the old champion

There is one important change in the table that did not come from the new code. Until yesterday, Opus 4.7 sat in first place with 97. The blind cross-audit between it and Opus 5 came back 94 to 70 in favor of 5. I went back and read the old artifacts.

Opus 4.7 had a double-send bug: the controller stored the user’s message before calling the service, and the service replayed the entire history, including that same message. Every prompt reached the LLM twice. Error messages also returned in the context of future requests, and the cookie had a message-count limit but no byte limit.

The tests passed because controller and service were tested under arrangements that differed from the production flow. The model did not get worse since April. The project did not change either. My audit was incomplete. I recalculated the score from 97 to 87.

That is almost comical, because last week’s article used “why is Opus 4.7 above 4.8 and Fable?” as its exact example. It is not anymore. And the article’s argument just got stronger.

A benchmark is not holy writ. The rubric evolves, the auditor finds a blind spot, the harness changes, the same model generates another project. If a table does not publish the prompt, artifact, logs, and corrections, it is more useful for marketing than engineering.

Conclusion

Is Opus 5 any good? Yes. Very.

It scored 95/A, tied Opus 4.8 and GPT 5.4, landed one point above the original Fable, three above GPT 5.6 Sol, and six above Kimi K3. It produced the most careful project in this entire round: clean architecture, RubyLLM verified against the real gem, correct history, solid streaming, separate error handling, and 121 tests.

It also took 39 minutes, burned 22 million cache reads, and ran in a different harness. Scoring higher than Fable does not prove it is better than Fable overall. It proves that, on this Rails app, in this Claude Code run, under this audit, the artifact fit the rubric a little better.

My practical read: Opus 5 deserves to be the first option. Its API rate is the same as Opus 4.8 and half Fable’s. For anyone with Claude Pro or Max, it is the obvious choice before spending credits on Fable. If you pay API rates, I would still start there and demand data before moving up to the $10/$50 tier.

And the rule stays the same: read 90+ as a group. Tier A is clearly above Tier C on this workload. Within the good group, choose by cost, subscription, speed, harness, and the kind of defect you are willing to review.

Do not use my benchmark to decide which model is “the best LLM.” Use it to decide what is worth testing on your problem. If the decision matters, run your own methodology more than once, and read the code.

Version 1.1.0 shipped that same day. Today, July 20, we’re at 1.17.1.

Version numbers alone don’t mean much, so I counted what actually happened after that post went live: 31 releases, 55 merged pull requests, and 46 closed issues. I counted 24 groups of new user-facing features in the CHANGELOG, ignoring fixes, documentation, and internal details from the same delivery. If I split the new ai-memory run family into auto-selection, session adoption, Crush support, --yolo, search, and recovery, we’d blow past that number.

Fifteen people had PRs merged during that period, fourteen besides me. Special thanks go to Djalma Júnior, who alone had 24 PRs merged, Matheus Rodrigues, with five, lhzapata, with four, Thiago Silva and Cristiano Dewes, with two each. Nine more contributors had a PR merged. This stopped being “a little program I made over a weekend” a while ago.

I’m not going to dump 55 PRs here. I want to talk about the change that affects my daily use the most: the new ai-memory run.

The session is part of the project too

Code is the result that survived. It doesn’t preserve the entire investigation.

Git shows that you replaced Redis with SQLite. Maybe the commit explains part of it. But where is the Redis experiment? Where is the intermittent bug that only showed up with two workers? Who remembered that the first solution broke on Windows? Why did the team change its mind after three hours? Which workaround was temporary, and which one became the decision?

Much of that exists only in the agent session.

When I program with Claude Code or Codex, the session accumulates the project’s operational reasoning: questions, corrections, discarded attempts, test results, surprises, decisions, and changes in direction. The final code shows what stayed. The session explains how we got there.

And an LLM session, at the end of the day, is text. Claude Code writes JSONL. Codex does too. OpenCode uses SQLite. Pi, OMP, and Crush have their own formats and directories. Each harness packages the conversation differently, but the visible content is still messages and tool results.

Why should I leave the most expensive part of the work trapped inside the harness?

Anthropic can change limits, pricing, models, or formats tomorrow. OpenAI can too. An agent can be excellent this week and unbearable the next. I want to swap the engine without throwing away the whole trip.

That has been ai-memory’s thesis from the start: the programmer must remain independent of provider whims. Models and harnesses are replaceable. The project memory is mine.

Handoffs solved only half the problem

ai-memory already handled handoffs. I would close Claude Code, the hooks would consolidate the session, and the next Codex would receive a short summary with what had been done, open questions, and next steps.

That still exists and is still useful. Anyone who opens claude, codex, or opencode directly keeps the previous behavior. The new launcher is opt-in.

But a handoff is deliberately lossy compression. It preserves what seems most important at that moment. It isn’t meant to carry the entire session or resume each harness’s native session. If the summary left out an old attempt that became relevant again two hours later, you have to search the wiki or the records.

ai-memory run adds another layer: a managed workstream, one line of work that crosses harnesses.

Claude now, Codex in a little while

Basic usage looks like this:

cd ~/Projects/meu-projeto

ai-memory run claude
# work, then exit Claude Code normally

ai-memory run codex --yolo
# Codex continues the same line of work

ai-memory run claude --model opus
# return to Claude's previous native session

ai-memory run
# or let ai-memory choose the right harness to continue

You don’t need -- to separate arguments. Everything after the harness name goes to it, with one deliberate exception: --yolo belongs to the wrapper and is translated into the native dangerous option for Claude Code, Codex, OpenCode, Pi, or Crush.

Under the hood, ai-memory doesn’t try to convert a Codex file into a Claude file. That would be fragile and would probably break on the next update. Each harness keeps its own native session. The workstream connects those sessions to a portable ledger containing the visible part of the conversation.

Think of it this way:

Claude native session ─┐
Codex native session  ─┼─ workstream ─ portable context
OpenCode session      ─┤                 + full search
Pi native session     ─┘

The first time Claude enters that workstream, it gets a native session. When you switch to Codex, Codex gets its own native session and receives the portable delta it hasn’t seen yet. When you return to Claude, the launcher uses Claude Code’s own --resume and delivers only what happened in the other harnesses since the last visit.

This is very different from starting a new chat with a summary pasted into the prompt. Claude returns to its real session. Codex returns to its real session. ai-memory handles continuity between the two.

Managed mode currently supports Claude Code, Codex, OpenCode, Pi, Crush, and OMP. General ai-memory support is broader, with MCP and hooks for other clients, but that doesn’t mean every client already has a native-session adapter for run. Those are different contracts, and I prefer to make that explicit.

What goes into the workstream

When the harness exits, ai-memory reads the tail of the native session without modifying the original store. It imports visible user and assistant messages, completed tool calls with their results, compaction summaries, and a non-mutating Git checkpoint. Every event keeps its origin: Claude, Codex, OpenCode, Pi, Crush, or OMP.

Provider credentials, encrypted records, system and developer prompts, and hidden reasoning stay out. Private formats the adapter doesn’t understand stay out too and produce a loss annotation instead of letting the system pretend it imported everything.

Records pass through the sanitizer before entering ai-memory’s searchable ledger and immutable JSONL segments. The adapter opens native stores read-only; the harness itself remains the writer.

I don’t cram the entire session into the next prompt either. That would recreate the problem the project is trying to solve: too much raw context, too much cost, and too much noise. The next agent gets a size-limited recent delta. If an old decision matters again, the full ledger remains searchable:

ai-memory workstream-search "por que desistimos do Redis"
ai-memory workstream-search --limit 50 --json "migration que falhou"

Inside an ai-memory run, the workstream ID is already in the environment. The agent doesn’t have to discover some UUID.

And the ledger doesn’t replace the wiki. It handles operational continuity. Decisions, rules, procedures, and gotchas that need to survive for months still deserve consolidated Markdown pages. A raw session is evidence. A wiki is organized knowledge.

Can I keep more than one line of work?

Yes. The default workstream is called default, selected by repository and worktree. If I want an independent line for an investigation without contaminating the main work:

ai-memory run --new investigar-race claude

# later, continue that line with another harness
ai-memory run --workstream investigar-race codex

A lease prevents two windows from writing to the same workstream at once. On a normal exit, the launcher imports the tail of the session and releases it immediately. If someone kills the process without cleanup, the lease expires within 90 seconds and the next run resumes from the last confirmed cursor without duplicating imported events.

One stupid annoyance kept coming back: I would reopen a project after a few days and have no idea whether the newest session was in Claude or Codex. I’d guess, open one, realize the context was in the other, close it, and try again.

Now I just run:

ai-memory run

# or keep the same auto-selection in YOLO mode
ai-memory run --yolo

In an empty workstream, ai-memory looks for local sessions from that checkout in Claude Code, Codex, OpenCode, Pi, and Crush, finds the newest one, and opens the right harness for me. In an established workstream, server memory beats the file timestamp: it returns to the last harness attached to that work instead of accidentally adopting an old session that happened to receive a newer write. OMP remains available when selected explicitly, but it doesn’t participate in auto-selection yet.

On the first explicit run, the launcher can also offer existing sessions from the same checkout for adoption. You can start using it without abandoning the chat that was already open before you updated ai-memory.

Other new features I actually use

I promised I wouldn’t read the entire changelog out loud, but a few changes from this month deserve a mention because they reduce work for the programmer.

A briefing before the first question

A project can request an automatic briefing at the start of every session:

[briefing]
inject_on_session_start = "true"
max_chars = 4000

ai-memory assembles a package with pinned pages, _rules/, _slots/, and recent titles. The agent starts with the project’s rules and basic state instead of spending the first conversation rediscovering the architecture. It’s opt-in because it consumes context on every SessionStart, including after a Claude /clear.

Global preferences and cross-project search

Preferences that apply to all my work can now live in the reserved _global scope: coding style, preferred tools, personal rules, and conventions that don’t belong to one specific repository. Normal queries already combine that scope with the current project.

For meta-repos that need to query several sibling projects all the time, there is another opt-in:

[recall]
default_global = "true"

With that, memory_query and memory_recent without an explicit scope search every project. I don’t turn it on in every repo because gratuitous global search only adds noise.

Less static instruction, more Agent Skills

The routing block in CLAUDE.md or AGENTS.md got smaller. Detailed instructions for when to search memory, consolidate, save a decision, or create a handoff moved into managed Agent Skills. One command installs or updates both without trampling the rest of the file:

ai-memory install-instructions

That reduces drift too. When the protocol changes, I update the managed package instead of hunting down an old prompt copied across forty repositories.

Privacy before the network

Each repo can now exclude paths from capture:

[capture]
ignore_paths = ["private/**", "~/personal-notes/**"]

When an integration recognizes a file tool and the path matches, the event is discarded locally before spool, queue, network, or server. The limits are clear: this isn’t magic DLP, it doesn’t understand every shell command, and it doesn’t track content quoted in free-form text. But it handles the verifiable case without selling a fake guarantee.

There are also new integrations with Kimi Code, Grok Build CLI, Devin, and Zero, an OpenCode Zen/Go provider, finalize-session for Codex, and workspace-administration improvements. Again: having MCP and a hook doesn’t automatically mean having ai-memory run. The README separates those levels.

Bonus: ai-jail on the outside, YOLO on the inside

Just over a week ago I wrote about how I protect myself from agents deleting my stuff. My recommendation remains the same: YOLO mode gives the best experience, provided the system limits the possible damage.

ai-jail reached version 1.15.0 and now understands the ai-memory launcher. My favorite way to work has become:

ai-jail ai-memory run codex --yolo

Or, when I want ai-memory to choose which harness should continue:

ai-jail ai-memory run --yolo

Order matters. ai-jail stays on the outside so the launcher and child process remain inside the same sandbox. ai-memory run by itself manages the session, but it doesn’t create a sandbox.

--yolo gets rid of the harness’s approval ceremony. ai-jail handles containment: the current project is read-write, home is replaced with tmpfs, necessary parts of agent state are mounted selectively, and sensitive directories such as .ssh, .gnupg, and .aws stay outside the jail. The new support also keeps ai-memory’s local state writable so hooks can maintain spools and cursors between runs.

ai-jail recognizes both the ai-memory layer and the selected harness. If you have Codex-specific global configuration, for example, it still applies when the actual command is ai-memory run codex. Even the status-bar redraw adjustment understands that the child is Codex.

This combination takes care of both annoyances at once: the agent stops asking for confirmation on every command, and the session stops belonging to the agent. You get the speed without handing over your entire home directory.

One more thing: ai-usagebar grew too

When I published the original ai-usagebar, it monitored Claude, Codex, Z.AI, and OpenRouter in a Waybar widget or TUI. Version 0.14.0 already covers eleven usage, spending, or balance integrations.

Kimi got the two pieces of information that really matter for coding: weekly subscription quota and the rolling five-hour window. Kilo, Novita, Moonshot, and Grok show remaining balance. A separate Anthropic Console integration shows API spending for the month without conflating it with the Claude Code subscription. It makes clear that the Cost API doesn’t include Priority Tier.

You can also register multiple Anthropic accounts, see weekly limits for specific models such as Fable, and get a pace marker. The bar doesn’t turn red just because it reached 40%; it compares the percentage spent with how much of the window has passed. If I used 40% in 20% of the week, I’m moving too fast. That’s the useful information.

The project outgrew Waybar. It has a native macOS menu bar app, a GNOME extension, and a standalone TUI. The newest addition is a local Claude Code context monitor: I press c, choose one of the recent sessions, and see how much of the window it has consumed. It reads limited tails from local JSONL files and doesn’t invent a percentage when it can’t determine the window size.

In my workflow, the three end up working together. ai-usagebar shows which subscription still has room. ai-memory run switches harnesses without throwing away the work. ai-jail lets the chosen one run in YOLO without unrestricted access to the machine.

Updating

The installation pages for all three projects cover every method, including AUR, Homebrew, and release binaries:

• ai-memory
• ai-jail
• ai-usagebar

After updating ai-memory, reinstall the hooks for the harnesses you plan to use in managed mode. The three main ones in my case:

ai-memory install-hooks --agent claude-code --apply
ai-memory install-hooks --agent codex --apply
ai-memory install-hooks --agent opencode --apply

Crush doesn’t need a hook in managed mode; it receives context through a temporary file configured only for the launched process. The managed workstreams documentation covers installation, privacy, recovery, and native arguments for each adapter.

Conclusion

A month ago, ai-memory could already turn a session into a wiki and hand off to the next agent. Now it can manage the line of work itself: adopt an existing session, keep one native session per harness, carry visible history, resume each client in its own format, and keep the entire ledger searchable.

For me, that’s the value. The code contains the implementation that won. The session contains the decisions, failed experiments, gotchas, unexpected bugs, and changes of mind that produced that implementation. Throwing all of it away every time you change providers is wasteful.

A raw transcript doesn’t solve the problem by itself either. It’s large, repetitive, and expensive to stuff into every context. ai-memory keeps the evidence, delivers only the recent delta, and uses consolidation to turn what deserves to survive into short Markdown pages.

There are still new formats to support, platform rough edges, and plenty of testing ahead. The difference is that there is now a real foundation, with an acceptance test that calls actual harnesses, 14 outside contributors with merged PRs in a little over a month, and daily use pushing the design forward.

I rent LLMs and subscriptions from whoever is delivering the best service today. My project’s session stays with me.

The focus is Kimi K3, an alternative that is becoming more and more like Claude in the kind of work it can finish.

For those who just landed here

The benchmark gives every model the same prompt: build, on its own, a ChatGPT-like chat app in Rails 8, RubyLLM, Hotwire, Docker, with tests and CI. Then I inspect the project it produced, including the gem’s real API, conversation memory, error handling, tests, and production image. The score goes from 0 to 100, in A/B/C/D tiers.

This is not a fill-in-the-function benchmark. It is an agent delivering a small app, with the boring parts that usually break after the demo. The methodology is in Part 3; code, logs, and the rubric are in the benchmark repository.

First, the harness problem

Previous Kimi rounds used OpenCode through OpenRouter. K3 did not get through the door: Moonshot’s strict tool-schema validator rejected the schema OpenCode sends:

when using anyOf, type should be defined in anyOf items instead of the parent schema

A controlled probe with a clean schema got tool calling to work. The problem is in the OpenCode ↔ Moonshot pairing. K2.5, K2.6, and K2.7 worked with the same tooling. Be careful with headlines saying “the model does not support tools”: sometimes the bug is in the fit.

So as not to abandon the test, a first-class runner for Kimi Code CLI was added. Phase 1 calls:

kimi -p <prompt> --output-format stream-json -m <model>

and phase 2 resumes the session with -S <session_id>. The runner code and configuration are in commit 0cdf88a. The original error and investigation were recorded in commits c3ab6d5 and 506357b; the final K3 project landed in 2bf1d7b.

This also puts an honest asterisk on the score. I wanted to compare the whole family under the new harness. The Kimi subscription exposes K3 and managed K2.7-Coding, but not K2.5/K2.6. We reran K2.7 through the CLI and compared it with the existing K2.5/K2.6/OpenRouter artifacts. K2.7 CLI came in at an informal 68/B; K2.7 OpenCode/OpenRouter had scored 86/A. This is harness and snapshot sensitivity, not a clean causal claim that “the harness costs 18 points”: managed kimi-for-coding may not be the same public K2.7 snapshot.

Current ranking: 40 models

K2.7 CLI is informal and unranked, so it is not included. The cost column is the actual cost at the round’s provider/OpenRouter or a subscription estimate, not an abstract rate card.

Kimi evolved quickly. Its price even more so.

K2.5 scored 69/B in February. K2.6 jumped to 87/A in April. K2.7 stayed at 86/A in June. Now K3 reaches 89/A in July. There was a big jump with K2.6, a plateau in K2.7, and K3 pushes the ceiling a bit higher.

From the official spec sheet, only what matters here: K3 has 1M context, MoE architecture (mixture of experts, which activates only parts of the model for each token), and maximum reasoning mode. Useful for understanding the product’s ambition. It does not replace opening the project it wrote.

What K3 actually wrote

There is quite a lot right in the Conversation model. It builds the chat, replays only the prior history, and then calls ask(content); the current message enters the collection only after the response. No double-send. If the provider fails, the exception is raised before append, so a failed turn is not stored:

response = build_llm_chat.ask(content)
append(Message.new(role: "user", content: content))
append(Message.new(role: "assistant", content: response.content.to_s))

It also got persistence better than the global-hash pattern: Rails.cache with a one-day TTL, a key per session, and a 20-message limit. Docker is production-grade, non-root, and validation performed a real chat both locally and in the container. That is real Tier A, not an optimistic README.

But you can still see the signature of a generation. It does not use with_instructions, so there is no system prompt. The LLM call lives inside the domain model. There is no preflight for a missing key.

Each individual message can grow without a ceiling and there is no rate limit. The production cache remains the ephemeral default. The read + mutation + write sequence races between requests. Raw provider errors appear in the UI. The error suite is smaller than it should be.

Against Opus 4.6, K3 clearly wins in this artifact: correct public RubyLLM API, bounded state, and tests for provider failures. Opus 4.6 manipulates chat.messages directly, stores an uncapped cookie, and does not cover provider errors.

Opus 4.8 models preconditions, the system prompt, and replay better. The basic RubyLLM call is comparable. The distance appears in hardening, architecture, and test depth.

Fable 5 is more defensive, but charges for it: original/re-release 94/93, against 89; $10/$50 per million, against $3/$15; measured runs of $11.20/$8.30, against ~$2.10 API-equivalent. One run per cell does not provide statistical certainty. It provides concrete code to compare.

Pricing: Kimi is no longer the absurd bargain

It is important here to separate Moonshot direct API, OpenRouter, and subscription. This is the official direct API rate card, per million tokens, with cached input separated:

Official documentation: K2.5, K2.6, K2.7 Code, and K3. For intermediary pricing and availability, also see the K3 page on OpenRouter.

K3 costs 3.16× more for input and 3.75× more for output than K2.6/2.7. Moonshot left the ultra-cheap niche. The costs in the ranking table are something else: K2.6 ~$1.19 and K2.7 ~$1.23 were recomputed from the provider/OpenRouter and the run tokens; K3 is an API-equivalent estimate of ~$2.10 for a subscription-billed run. Mixing the two without identifying the channel creates a false comparison.

Even so, K3 costs 40% less than Opus 4.8 at official rates ($5/$25) and 70% less than Fable ($10/$50). On the $19 Kimi Moderato plan, this is excellent for interactive solo coding. Just do not automate recklessly: we observed around two heavy rounds per five-hour window and quota recovery in 4h16. For unsupervised batch work, a mid-run 403 throws away work and time.

Moderato is too small for serious work

This is the part the monthly price hides. I used the $19/month Moderato. Its quota is organized in rolling five-hour windows: use a lot now, then wait for the window to slide forward before capacity returns. A benchmark run is not one loose prompt. It is exactly two prompts/phases: build the app, then resume the same session to validate and fix it.

K3 completed that whole run in 26 minutes and 4.83 million tokens, almost all cache-read, about $2.10 API-equivalent. K2.7 Coding completed the same two prompts in 16 minutes and 9.25 million tokens. In the benchmark record, approximately two complete runs at that scale fit before a third received a hard 403 partway through. The partial build was lost. Recovery took 4h16, with seven blocked probes at 20-minute intervals.

One caveat: I did not commit the raw 403/probe transcript. This chronology is a recorded benchmark observation, not something another person can independently audit from raw logs. The documentation says already-running tasks normally finish; our third one did not. That is what I saw in use, not an accusation that the documentation is lying.

Official plans are on the membership page:

K3 on Moderato is limited to 256K context. Allegretto and above unlock up to 1M. Allegretto+ also unlocks HighSpeed, but that mode burns roughly 3x the quota. The 5x/15x/30x are relative credits, not a promise of 5/15/30 times more linear runs. The documentation discloses weekly refresh, the rolling five-hour window, and shared quota across CLI, VS Code, and API-key tools, but does not publish the exact formula for a heavy session or token. Cache, output, and agent behavior change the math a lot.

Moderato is good for one, maybe two heavy sessions in five hours. For five hours of coding a day, it is too small: in a sequence of these tiny benchmark runs, with only two prompts per model, the third attempt already hit cooldown. Cooldown is the wait for the quota window to open again. Allegretto is the minimum upgrade for lighter daily interaction and gets 1M context, but 5x can leave little headroom for someone driving the agent continuously or enabling HighSpeed. For sustained professional use, at least five hours a day, I would go with Allegro: 15x credits and four concurrent tasks. There is still no official guarantee of uninterrupted execution. Vivace is for multiple agents or near-continuous load; for one developer, it is generally overkill.

The other choice is subscription or direct API. The K3 API charges $0.30/M cached input, $3/M uncached input, and $15/M output. It provides explicit accounting, the full 1M context, and does not have the membership weekly quota, although normal API limits still apply. It is the better route for CI, batch work, and long runs worth real money.

A subscription buys something else: fixed monthly cost, convenient OAuth/CLI, near-zero marginal cost within the caps, and a predictable bill. In return, you accept a hidden token envelope, rolling/weekly caps, shared allowance, the risk of stopping midway, and no SLA for unattended automation. The K3 ~$2.10 is only a rough API-equivalent. At that run shape, the Moderato monthly fee arithmetically equals around 9 runs, Allegretto 19, Allegro 47, and Vivace 95 per month. That is invoice arithmetic, not a quota guarantee: real workloads vary wildly with cache and output.

Kimi also offers Extra Usage, which can fall back to paid balance under a spending cap. It helps prevent an abrupt quota stop. The exact USD rate is not published, and it does not turn a consumer membership into an automation SLA.

My plain recommendation: for 5h/day, Allegro. For lighter interaction, Allegretto. For automation and CI, pay-as-you-go API. If you stay on subscription and the work cannot stop, enable Extra Usage with a spending cap.

Separating plan from model: K2.6 remains the best raw-API option for high-volume automation with human review. K3 is a very good option for those coding interactively. Opus 4.8 remains my choice for complex refactors, concurrency, security, and autonomous changes where correctness matters more than the bill. Fable, in this slice, is economically dominated.

Bonus: GLM 5.2 versus Opus

GLM 5.2 is at #9, 87/A, 43 minutes, through a Z.ai subscription. The natural comparison is Opus 4.6, its score neighbor at 83/A and 16 minutes, and Opus 4.8, the quality ceiling at 95/A and 17 minutes.

I read both projects. GLM beats the Opus 4.6 artifact: it uses RubyLLM correctly, applies a system prompt, separates the service, accepts DI, and tests failures. The service even excludes the last message from replay before calling ask, the detail that avoids double-send.

But GLM also leaves skeletons in the closet: uncapped process-local singleton; failed turns are retained; config.hosts.clear; generated secret that invalidates sessions; and npm ci || npm install, which turns the lockfile into a suggestion. It remains clearly behind Opus 4.8 in state, design, tests, and configuration hardening.

Z.ai plans are Lite $18, Pro $72, and Max $160, with a promotion through September 2026: $12.60/$50.40/$112. Against Opus 4.8 at ~$6.40 per run, nominal break-even is approximately 3/12/25 runs per month, or 2/8/18 at the promotional price. But GLM takes 43 minutes where Opus 4.8 took 17. If you already pay for Z.ai, GLM 5.2 is a strong default for routine reviewed work. If correctness and turnaround rule, Opus 4.8.

Conclusion

In this greenfield Rails project, K3 is a good replacement for Opus 4.6 and offers a cheaper alternative to Opus 4.8. Claude still has the edge when the project demands defenses against details nobody puts in the prompt.

There is one run per model and there is harness effect, especially with K3. The benchmark also does not cover database migrations, background jobs, or product tool calls. There is no data here to make promises about those.

My map today: K2.6 for reviewed-volume API automation; K3/Allegretto for lighter interactive solo work and Allegro for a sustained professional routine; Opus 4.8 when the patch is sensitive; GLM 5.2 as a good subscription default for routine reviewed work. No hype. Read the code, run the tests, and treat every ranking as limited evidence, not religion.

The next day, goodalexander reproduced the same behavior on a disposable devnet: an rsync --delete-delay against / swept away /var, /etc and his validators’ state, corrupting the filesystem to the point of needing an OS reinstall. His conclusion: “you can’t use this thing in prod”.

Cases like these pop up every so often since coding agents went mainstream. And every time, the discussion degenerates into the same two extremes: on one side “see? AI is dangerous, never let it run anything on its own”, on the other “whoever lets an agent loose on their system deserves what they get”. I find both sides lazy. Let me tell you my actual experience and what I do about it.

Five months of marathoning, zero accidents

Since February I’ve been marathoning Claude Code, Codex and OpenCode. Almost every day, almost all the time, in very long sessions that sometimes span multiple days. Blowing past the limits of my Anthropic Max plan and my OpenAI Plus plan with room to spare. The concrete result so far: almost 40 new repositories on my GitHub and more than 500 thousand lines of code produced.

And for the past two months I’ve been running everything exclusively in “YOLO mode”. For those unfamiliar with the jargon: by default, these agents ask for permission before each command they’re about to execute in your terminal. YOLO mode turns that safety off. In Claude Code it’s the --dangerously-skip-permissions flag, in Codex it’s --yolo, in OpenCode you configure the permissions to approve everything. The agent runs whatever it wants, whenever it wants, without asking me anything.

In five months under this regime, not once has an agent deleted a file I didn’t ask for. Not once has it run a destructive command out of scope. The frustrations I have with agents are of a different nature (stubbornness, lazy solutions, context getting lost), annoying things but miles away from an rm -rf in my home.

So when I see these reports, my first hypothesis always goes to the same place: the prompt. For an agent to “go berserk” like that, in general the user wrote very bad instructions. Vague, ambiguous, asking for cleanup without defining what counts as trash, telling the agent to “fix it” without giving context. Lack of communication skills combined with lack of technical knowledge to notice the request was poorly formulated. The agent executes the instruction you wrote, with all the ambiguities you left in it.

That said, I do NOT trust LLMs completely. Nobody should.

The difference is that my distrust turns into engineering instead of turning into a tweet.

Layer by layer, this is how I sleep well at night.

Layer 1: sandboxing with ai-jail

When I genuinely suspect a session might do something stupid (I’m about to touch system scripts, I’m going to let the agent run for hours unsupervised on code that touches the filesystem), I open the harness inside ai-jail, a tool of mine, open source, written in Rust.

ai-jail is a sandbox wrapper for coding agents. On Linux it uses bwrap (bubblewrap, the same foundation as Flatpak), on macOS it uses sandbox-exec. The idea: the process sees a filesystem that looks normal, but only the current project directory is real and persistent. The rest of $HOME, the sibling directories, /tmp, everything lives in tmpfs and evaporates when the session ends. The agent’s state (~/.claude, ~/.codex etc) is mounted for real so it stays logged in and configured, but ~/.ssh, ~/.gnupg, ~/.aws and browser profiles never enter the jail.

Installing is one command:

# Arch Linux
yay -S ai-jail-bin

# macOS / Linux via Homebrew
brew tap akitaonrails/tap && brew install ai-jail

# or straight from crates.io
cargo install ai-jail

And using it is even simpler:

cd ~/Projects/my-app
ai-jail claude       # Claude Code in the jail
ai-jail codex        # Codex in the jail
ai-jail --dry-run claude  # shows the policy without running

If the agent decides to run rm -rf ~ in there, it deletes a tmpfs that was going to vanish anyway. My real home was never even mounted. For anyone starting out with YOLO mode without battle scars yet, it’s the cheapest way to make mistakes without consequences.

Layer 2: BTRFS and snapshots (the real safety net)

Now the main point of this post. I’ve been advocating for YEARS that Linux users should use BTRFS instead of EXT4. I made an entire Akitando episode about it in 2023, long before coding agents existed to take the blame. The argument was the same: your data needs to survive human error. An LLM running rm -rf in the wrong place is just a new category of human error, outsourced.

Why is BTRFS superior for desktops? Copy-on-write (CoW): the filesystem never overwrites a block in place, it writes the new version to another block and updates the pointers. Checksums for data and metadata, so silent bitrot gets detected instead of quietly corrupting your file. Transparent compression. Subvolumes, which work like flexible partitions without fixed sizes. And the most valuable consequence of CoW: lightweight snapshots.

A BTRFS snapshot is instantaneous and costs close to zero space the moment it’s created. It freezes the pointers to the current blocks instead of copying data. Only what diverges from that point on (new or modified blocks) starts taking up extra space. In practice you can take dozens of snapshots a day without feeling it. At this very moment my machine keeps 23 system snapshots, and the cost is the difference between them, a few gigs.

Scheduling snapshots

The two classic tools are Timeshift and snapper. Timeshift is the friendlier one: graphical interface, you set “how many snapshots to keep” per hour/day/week/month and it handles the rest. It expects the @ (root) and @home subvolume layout, which is the default for most distros that install on BTRFS today. snapper (born at openSUSE) is more granular: per-subvolume configs, automatic timeline, package manager integration to snapshot before and after every install.

# Timeshift: manual snapshot before a suspicious session
sudo timeshift --create --comments "antes de soltar o agente"

# snapper: create a config for /home and enable the timeline
sudo snapper -c home create-config /home
sudo systemctl enable --now snapper-timeline.timer

My setup runs Timeshift on a schedule, plus grub-btrfsd on standby (I’ll explain why in a moment).

Recovering deleted files

Here’s the magic that everyone who lost files to an agent wishes they had. A BTRFS snapshot is browsable like a regular directory. The agent deleted your project at 3pm? The 2pm snapshot still has everything:

# with Timeshift, snapshots are mounted under /run/timeshift
sudo ls /run/timeshift/backup/timeshift-btrfs/snapshots/

# found the snapshot from before the damage? copy it back and done
sudo cp -a /run/timeshift/backup/timeshift-btrfs/snapshots/2026-07-11_14-00-01/@home/akitaonrails/Projects/meu-app ~/Projects/

With snapper it’s the same thing via .snapshots inside the subvolume, and it also has snapper undochange to revert specific changes between two snapshots. Recovery in seconds, no drama, no forensic tooling.

What if the system won’t even boot?

goodalexander’s case went beyond losing files: the agent swept /var and /etc and the system became unusable, requiring an OS reinstall. With BTRFS not even that would be necessary. This is the scenario grub-btrfs exists for: a daemon that watches your snapshots and regenerates the GRUB menu with a boot entry for each one.

System won’t come up anymore? Reboot, hold the GRUB menu, pick “snapshots” and boot straight into yesterday’s snapshot, read-only. From inside it, run timeshift --restore, pick the good snapshot, reboot again. Entire system back to its pre-disaster state, in less time than it would take you to find the installation USB stick. In the absolute worst case (GRUB itself went down with the ship), any live USB mounts the BTRFS partition, points the default subvolume back to the good snapshot and that’s that.

What about macOS? And Windows?

Mac users have the direct equivalent: APFS, the default filesystem since High Sierra (2017), is also copy-on-write and also has lightweight snapshots. Time Machine is what manages them: with it enabled, macOS takes automatic local snapshots every hour and keeps the last 24 hours on the disk itself, independent of the external backup. And you can create one by hand before letting the agent loose:

# create an APFS local snapshot right now
tmutil localsnapshot

# list the existing snapshots
tmutil listlocalsnapshots /

To recover, open Time Machine itself (“Browse Time Machine Backups” in the menu bar): it browses the local snapshots even without any external disk plugged in, and you restore the file from there. If you prefer the terminal, mount the snapshot read-only with mount_apfs -s and copy it back. In other words, Matt Shumer’s case from the beginning of this post had a native solution on his own Mac: a snapshot from one hour earlier would have brought almost everything back.

On Windows the story is weaker. NTFS has the Volume Shadow Copy service (VSS), which feeds restore points and the “Previous Versions” tab, but it’s aimed at system files and rarely happens to be covering your data at the moment you need it. Nothing with the lightness and predictability of a scheduled CoW snapshot. If you develop on Windows, treat external backup as mandatory, no discussion (and the restic from the next section runs fine on Windows).

Layer 3: external backups with restic

Snapshots live on the same disk as the data. If the NVMe dies, they die together. Which is the other thing I’ve been repeating for years: everybody needs external backups, whether to an external hard drive or a NAS. I use a Synology on the local network, mounted via NFS, and documented the complete setup here.

On top of it runs restic, which in my opinion is the best backup tool for this purpose: deduplication, encryption, incremental snapshots and simple restores. My actual configuration is a script fired by a systemd timer every day at 3am. It checks the NAS is mounted, backs up /home and /mnt/data/Projects skipping caches (node_modules, Rust’s target/, __pycache__ and friends, all in an excludes file), and then applies the retention policy: 7 daily, 4 weekly, 6 monthly, 1 yearly.

Reproducing the essentials looks like this:

# once: create the repository on the NAS
restic init --repo /mnt/nas/desktop-backup

# every day (via cron or a systemd timer):
restic -r /mnt/nas/desktop-backup backup ~ /mnt/data/Projects \
    --exclude-file=~/.config/restic/excludes --exclude-caches

restic -r /mnt/nas/desktop-backup forget \
    --keep-daily 7 --keep-weekly 4 --keep-monthly 6 --keep-yearly 1 --prune

And if the truly bad day arrives (destructive agent, dead disk, ransomware, the cause doesn’t matter):

# list the available snapshots
restic -r /mnt/nas/desktop-backup snapshots

# restore a specific file or directory
restic -r /mnt/nas/desktop-backup restore latest \
    --target /tmp/recuperado --include /home/akitaonrails/Projects/meu-app

# or mount the whole repository as a filesystem and browse at your leisure
restic -r /mnt/nas/desktop-backup mount /mnt/restic

Notice the layered architecture: the BTRFS snapshot solves the accident from minutes ago in seconds. restic solves the disaster from days ago, including dead hardware. An AI agent would have to be very dedicated to punch through both layers, and even then there would still be yesterday’s copy on the NAS, which it can’t even reach.

What about companies? Professionalism

Everything above is about my personal machine. Now let’s raise the bar: corporate environments.

Rule number one is simple and non-negotiable: no AI agent should ever have access to production. Ever. No direct SSH into production servers, no production database credentials, no admin tokens for production services. If your agent can run a DELETE against the database that serves customers, the problem stopped being the AI a long time ago: any intern with the same access is the same time bomb, just with a slower typing speed.

Agents in staging, on the other hand, I advocate for. That’s what staging is for. But even in staging there’s a right way: you resist the temptation of prompting one-shot actions (“run an apt install over there”, “do a systemctl restart”, “connect to postgres and execute this”). A loose command executed by hand is knowledge that evaporates. Your infrastructure needs to be automated and reproducible: Ansible, Kubernetes, Terraform, whatever makes sense for your stack. The agent’s role becomes writing and refining those recipes, which live versioned in git.

A small example, right here from my desk

I have a gaming environment on my main PC, but I refuse to mix emulator packages with my software development environment. So I isolated gaming inside a distrobox. For those who don’t know it: distrobox creates containers (via podman or docker) that integrate with your desktop as if they were native. They share your home (or a dedicated home), GPU access, Wayland, audio. In practice it’s a complete distro inside your distro, with a separate package space.

distrobox-gaming is a repository of Ansible playbooks that builds an Arch box called gaming from scratch: ES-DE as the frontend, standalone emulators (shadPS4, RPCS3, PCSX2, Dolphin, DuckStation, Eden and a dozen more), RetroArch cores, per-game optimized configs, launcher shortcuts on the host. And distrobox-llm does the same for a local LLM work box: CUDA, cuDNN, Ollama, whisper.cpp and LM Studio isolated in a box with --nvidia, while the host only carries the driver. A CUDA major version upgrade no longer rattles my development environment.

Who wrote the overwhelming majority of these playbooks? The agents. Instead of asking “install RPCS3 for me”, my prompt is “write the idempotent Ansible recipe that installs and configures RPCS3, run it, verify the result and fix it until it passes”. Here’s a real excerpt from the role that creates the LLM box:

- name: Check if distrobox exists
  ansible.builtin.shell: "distrobox list 2>/dev/null | awk '{print $3}' | grep -qx {{ dl_box_name }}"
  register: _box_exists
  changed_when: false
  failed_when: false

- name: Create distrobox
  ansible.builtin.command: >-
    distrobox create
    --name {{ dl_box_name }}
    --image {{ dl_image }}
    --home {{ dl_box_home }}
    {{ '--nvidia' if dl_nvidia_enabled else '' }}
  when: _box_exists.rc != 0

Idempotent: it checks whether the box exists before creating it, running it twice gives the same result. If my PC catches fire tomorrow, I recreate both environments in full with one ansible-playbook site.yml in each repo. The knowledge lives in the recipe, versioned, auditable. Zero commands lost in shell history.

Scaling the same principle to production

At a company, swap distrobox for Kubernetes and the principle is identical. The agent writes the manifest, tests it on the staging cluster, you promote it to production manually:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: minha-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: minha-api
  template:
    metadata:
      labels:
        app: minha-api
    spec:
      containers:
        - name: minha-api
          image: registry.exemplo.com/minha-api:v42
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
          resources:
            limits:
              memory: 512Mi

The professional flow: the agent produces and adjusts the manifest (or the Helm chart, or the Terraform module), applies it to staging (kubectl apply -n staging), runs the tests, watches for side effects, bugs, unintended behavior, and iterates until you’re satisfied. Then YOU, a human, promote it to production (via a reviewed PR in a GitOps flow, or a manual and deliberate kubectl apply), monitoring every step. The agent did all the heavy lifting: the artifacts, the procedures, the checklists. You can even ask it to generate the audit documentation at the end. But the finger that presses the production button has a name on a payroll.

Even the “simple” maintenance SQL

A case that looks harmless and bothers me a lot: maintenance SQL commands in production. Plenty of people have always run those by hand, straight into the production client, and are now asking the agent to write AND execute them. That is unacceptable, with or without AI. Your system should have a migrations workflow: the change becomes versioned code, runs in staging first, gets validated, and only then executes in production inside the normal deploy process.

And “production”, by definition, means two things many companies pretend to have. First: real backups, tested, with rehearsed restores.

Never run anything in production that you can’t recover from.

Second: monitoring. If you have no visibility into the state of the system, you don’t have production, you have a server plugged into an outlet and a prayer.

Monitoring: even on my humble PC

And monitoring is so much a part of the discipline that I apply it even to my personal machine. I recently developed a system health widget for my clock-tui (my maintained fork of the original clock-tui, with command widgets in clock mode). It sits open in a corner showing this:

One verdict line (“all systems healthy”) and the state of everything this post described: the restic backup ran 19 hours ago and the next one is at 03:03; Timeshift keeps 23 snapshots; the development cache cleanup ran 32 hours ago; the BTRFS scrub is up to date on each filesystem, with a recent trim, healthy allocation and zero I/O errors; plus zombies, load, memory and each disk’s usage. The widget (tclock-system-health) ships with the repository and the AUR package, and detects most of this on its own.

Behind it are the automated maintenance jobs, all systemd timers: the daily restic at 3am; dev-cache-clean (on the 1st and the 15th) sweeping orphaned Rust target/ dirs and the yay cache, which together give back dozens of gigs; the monthly per-filesystem btrfs-scrub validating checksums; the weekly fstrim. None of it requires me to remember anything. When one of them is late or fails, the widget changes color right in my face.

This is the home-grade level, proportional to a personal PC. At a company you go full-blown: Prometheus collecting metrics, Grafana drawing the dashboards, Alertmanager waking up the on-call person at 3am. The scale changes, the principle is the same: the state of the system visible at all times, and someone (or something) getting notified when it goes off the rails.

Conclusion: discipline

Go back to the two tweets at the beginning. In both cases, a single wrong command caused real loss, and the public discussion became “who do I trust more”. The right question is a different one: why does a single wrong command, from any origin, have the power to cause unrecoverable loss on your machine?

Being a professional is being disciplined, and discipline starts at home. My personal PC operates with the same rigor I would demand from a corporate environment: a filesystem with lightweight scheduled snapshots, recoverable boot via GRUB, daily external backups with a retention policy, a sandbox for suspicious sessions, isolated environments rebuildable from versioned recipes, automated maintenance and a monitor that warns me when any piece falls behind. Monthly maintenance cost of all this: close to zero. It was set up once (in good part by the agents themselves) and runs on its own.

With these layers, the question in the title almost loses its meaning. The agent deleted a directory? Snapshot from an hour ago, two minutes of recovery. Destroyed the system? Boot into a snapshot via GRUB, restore, ten minutes. Fried the disk along with it? restic on the NAS, yesterday’s backup. In five months of YOLO mode I never needed any of these nets for an AI accident. But I sleep well precisely because they’re there.

An LLM running loose on your system is just the most recent stress test of your discipline. If an out-of-control agent can cause permanent loss to your data, the vulnerability was always there, waiting for a mistake of yours, a bad disk or a ransomware. The agent was just the messenger that arrived first.

The three names are Sonnet 5, Gemini 3.5 Flash, and Sakana Fugu Ultra. Sonnet 5 is the one that draws attention. New model, strong brand, high expectations. Except expectations do not compile. It ran, passed Docker, came up under Compose, and still finished at 58/100, Tier C. The first real chat request would crash before calling the model.

The good surprise came from the other side: Gemini 3.5 Flash scored 93/100, Tier A, #6. A Flash model. Above Kimi K2.6, GLM 5.2, and Gemini 3.1 Pro.

And then there was Sakana Fugu Ultra, right on the edge: 79/100, Tier B, knocking on Tier A’s door. Qwen3.7 Max and Step 3.7 Flash also entered, but they are supporting cast. Nex matters, but I already unpacked that case in the post about the Rio 3.5 controversy. Here it stays mostly as a ranking line and reference.

Since the last benchmark, my practical recommendation has not changed: serious programming, long sessions, real projects, today still means Opus 4.8 or GPT 5.5, using Claude Code, Codex, or OpenCode. What changed is the middle of the table. And the middle is where you can see who is learning and who merely changed the version number.

For anyone landing here cold

The test is simple to explain and annoying to run. I give every model the same prompt and ask it to build, by itself, a ChatGPT-style chat app in Rails 8 + RubyLLM + Hotwire + Docker, with tests and CI. Then I read the generated code by hand and score it from 0 to 100, in A/B/C/D tiers.

This is not multiple choice. Not a puzzle. Not “complete this function.” It is an agent trying to deliver a whole small Rails project. It either delivers, or it trips.

The series so far:

1. Canonical May benchmark (Part 3), with the complete methodology.
2. DeepSeek unlocked with DeepClaude.
3. Grok 4.3, MiniMax M3, Opus 4.8 update.
4. Fable 5 and Anthropic’s saga.
5. Kimi v2.7 code, GLM 5.2, MiniMax M3 local.

The trap is still the same: the “RubyLLM OK” column is not decoration. If the model invents an API that does not exist, the app dies in its main feature. Sometimes everything around it looks nice: Tailwind, Turbo, Docker, README, green tests. Then you click “Send” and get a NoMethodError.

The closed-model soap opera

Anthropic had placed US government controls on Claude Fable 5 and Claude Mythos 5 on June 12. Reason: they could not verify nationality in real time across all global channels. On June 30, they suspended the Fable 5 controls, and Anthropic announced its gradual return to Claude Platform, Claude.ai, Claude Code, and Claude Cowork. Mythos 5 remains more restricted, only for approved US organizations through Project Glasswing.

Sources: Anthropic’s post about Fable 5 returning, the original Fable 5 and Mythos 5 post, and The Verge on the suspension of the controls.

On the OpenAI side, GPT-5.6 has appeared officially, but it is still in limited preview. There is an OpenAI page and a help center article. It is not available enough for me to benchmark it the way I can benchmark GPT 5.5 today. So it gets a note, no number yet: GPT-5.6 Sol preview and GPT-5.6 Sol, Terra, and Luna help center article.

Updated ranking

New models in this round are in bold. Nex-N2-Pro and the Qwen 3.5 397B base are still highlighted because they entered the table after the last benchmark, but the longer analysis already appeared in the Rio 3.5 post.

Sonnet 5: passed Docker, failed at the heart

Sonnet 5 scored 58/100, Tier C, #25. Here is the part that fools people: phase 2 passed. Local boot. Docker build. Docker Compose. The project had Rails 8, Turbo, Stimulus, Tailwind, README, CI, SimpleCov, Brakeman, RuboCop, and bundler-audit. The shell looked good.

The problem was exactly where it could not be: the LLM path.

The generated service does this:

RubyLLM.chat(model: MODEL, provider: PROVIDER).tap do |chat|
  chat.messages = history
end

But RubyLLM 1.16 has messages, add_message, ask, and with_instructions. It does not have messages=. The first real chat request raises NoMethodError before reaching the model. The test misses it because the project’s own FakeChat defines attr_accessor :messages, exactly the writer that does not exist in production.

This is the kind of bug that keeps me reading benchmark outputs by hand. Green build does not prove correct integration. Green test can be worse than useless when the fake teaches the test a lie.

There is more: the generated app still defaulted to anthropic/claude-sonnet-4.6, not Sonnet 5, and left the system prompt without with_instructions. The shell was complete. The main feature was broken.

Cruel summary: Sonnet 5 passed Docker and failed at the heart. If you pay Anthropic for programming today, I still recommend Opus 4.8. Not Sonnet 5.

Gemini 3.5 Flash: the good surprise

Gemini 3.5 Flash was the slap in the face of the round: 93/100, Tier A, #6. Best non-Anthropic/non-OpenAI result in the benchmark so far. Fourteen points above Gemini 3.1 Pro.

The RubyLLM integration was exemplary:

chat = RubyLLM.chat(model: @model, provider: :openrouter, assume_model_exists: true)
chat.with_instructions(SYSTEM_INSTRUCTION)
@conversation.messages.each { |msg| chat.add_message(role: msg.role.to_sym, content: msg.content) }
response = chat.complete
@conversation.add_message(role: :assistant, content: response.content)

It used with_instructions, history replay with add_message, complete, and response.content. All real. The test suite was the best in this cohort: FakeChat with a compatible signature, stub at the correct point (RubyLLM.chat), parameter assertions, history replay, and an error test. Sounds basic. It is not. The table proves it is not.

I also liked the persistence. It writes files under tmp/chats/<id>.json, with a 50-message cap and the system message preserved. It survives restart on the same host and is better than an in-memory singleton. It lacks file locking, tmp/ is ephemeral in containers, and concurrent writes to the same conversation can race. It also lacks a nice API-key preflight. All fixable.

The cost bothers me. Even as Flash, the run came out around US$3.50, more expensive than Opus 4.8 in this benchmark because of high token churn. So I cannot sell it as “cheap.” But as a quality signal it is strong. A Flash model at #6 changes my intuition about Gemini.

Sakana Fugu Ultra: almost Tier A, missing memory

Sakana Fugu Ultra entered with 79/100, Tier B, tied at #12. For a new provider, this was a clean debut. The harness worked. The app came up. Docker/Compose passed. RubyLLM was real (chat, with_instructions, ask, response.content). The tests used a correct FakeChat. Turbo/Stimulus were good.

The problem is small to explain and large in consequence: the model only sees the latest message. The history is saved and displayed, but it is not replayed into RubyLLM. The UI looks like a multi-turn chat. The LLM works as if every message were the first.

It missed exactly the part that makes chat chat.

Even so, this is the kind of result I like to see. It is not a disaster. It is not an invented API. It is an almost-correct app with a specific failure. Replacing the process-local repository with something decent and replaying history with add_message would change the evaluation a lot.

Qwen3.7, Step 3.7, and the Qwen 3.6 footnote

Qwen3.7 Max scored 78/100, Tier B, tied at #14. Correct RubyLLM API, including streaming and reset_messages!. Phase 2 passed. But it ignored Turbo Streams and implemented raw SSE with ActionController::Live. In a real app, maybe fine. In this benchmark, the prompt asked for Hotwire/Turbo Streams. It also used anthropic/claude-sonnet-4 instead of the newest Sonnet and stored conversations in an unbounded class-level hash. Restart the process? Lose everything. Run long enough? Leak memory.

Step 3.7 Flash scored 69/100, Tier B, tied at #20. It fixed the big Step 3.5 mistake: now it uses the real ruby_llm gem, not a ruby-openai bypass. Good. But it creates a new RubyLLM.chat per request and sends only the current message. The history lives in a @@all class var, but never reaches the model. It escaped the wrong bypass and forgot to give the LLM memory.

And yes, Qwen 3.6 appeared in the recent commit history. Two local Qwen 3.6 35B A3B runs entered through PR #9 (cccc118 and f06ddd0), using a local ik_llama.cpp profile, but the merge was reverted in d5f9195. That is why they are not in the main table for this round.

The useful Qwen 3.6 local data remains in the separate RTX 5090 report: it was the first local model in that profile to get RubyLLM.chat + chat.ask right, but it still returned the whole object instead of response.content and had no multi-turn. Close to working. Still outside the main cloud/AMD ranking.

Nex and Qwen base: already covered in the Rio 3.5 case

Nex-N2-Pro scored 83/100, Tier A, #10, using OpenRouter’s free endpoint. The raw Qwen 3.5 397B A17B base scored 42/100, Tier C, #31. That 41-point gap matters. Repeating everything here would be padding.

I already used those numbers in the post about the Rio 3.5 LLM controversy, because the public accusation was precisely that Rio’s checkpoint looked like a merge of Nex-N2-Pro and Qwen3.5-397B-A17B. That post explains what Nex is, why the fine-tune matters, and why a Rio model inheriting Nex’s gains should not be sold as an obvious advance over pure Nex without a reproducible benchmark.

The short technical version: the Qwen base invented RubyLLM (chat.system, chat.user, response.text) and the tests mocked that fake API. Nex used the real API, booted locally, passed Docker/Compose, and delivered Turbo Streams with error handling. It still has shortcuts, like flattening history into text instead of using add_message and with_instructions, but it is a legitimate Tier A result. It is just not the focus of this post.

My actual workflow today

In practice, I have been using a lot of OpenCode + oh-my-opencode-slim + GPT 5.5 to maintain my open source projects. It is a slim, cleaner, cheaper-token fork of oh-my-opencode. Exactly the kind of adjustment that fits daily use. I already explained the rest of my AI toolkit with ai-jail, ai-memory, and ai-usagebar in a separate post.

This is annoying to explain to people who only want to know “which model is best.” Model alone does not carry the work. Harness matters. System prompt matters. Project memory matters. Test quality matters. The same model, in a bad harness, becomes a random patch generator.

I have been using this on small, real things, like Geary extensions: geary-email-autocomplete and geary-hide-sidebar-module. None of this is a revolution. It is maintenance, bugfixes, refactors, tests, documentation. Normal work. Exactly where a coding agent needs to be reliable for hours, not for a 90-second demo.

That is why I keep repeating: model benchmarks are useful, workflow benchmarks are even more useful. Sonnet 5 with a hallucinated API does not help me. Gemini Flash surprising me with correct RubyLLM does. Sakana and Step getting the gem right but forgetting multi-turn show exactly where these models still break in a real product.

Conclusion

This round left a simple message: a strong name does not save a broken integration. Sonnet 5 entered with the aura of a new model and left in Tier C because it called an API that does not exist.

Gemini 3.5 Flash deserves attention. Not because it is cheap; in this run it was not even that cheap. It deserves attention because it delivered a Rails/RubyLLM app far more correctly than I expected from a Flash model.

On the open source / alternative provider side, the story is more mixed. Sakana Fugu Ultra almost reached Tier A. Qwen3.7 Max got the API right and missed the product. Step 3.7 Flash fixed the 3.5 bypass and forgot to give the chat memory. There is progress. There are edges.

My practical recommendation stays the same: Opus 4.8 or GPT 5.5 for serious programming today. Use Tier B for small, cheap, reviewed tasks. Use open source/local to study and experiment. And do not trust a green test if the fake does not imitate the real library.

The benchmark is open: github.com/akitaonrails/llm-coding-benchmark. Want to see another model there? Fork it, add it to config/models.json, run it, and send the PR.