• New v4 benchmarks: I retested the major models and the uncomfortable conclusion was convergence, not a leap.
• The limits of LLMs: models are great at reproducing what exists, extremely expensive at what doesn’t, and the served model is a static file that learns nothing from your marathon.
• Are there better languages for programming with LLMs?: no. Neither script conciseness nor strong typing changed the outcome; test discipline did.
• Which Jev-like decision model is best?: 21 classifier configurations, and the “specialized models” themselves converged to within a few points of each other.
• Clean Code for AI agents, VS Code is the new punch card, my skills, the hot take against loop engineering and minimum best practices: the same thesis from other angles. The model is a commodity; the engineering around it is the differentiator.

The thesis that closes this arc: LLMs have reached, or are very close to reaching, the ceiling of the transformer architecture. And the good news is that nothing more than that is needed.

Convergence, not a leap

In my benchmarks, I see no relevant improvement in general programming since the Opus 4.5 generation. Yes, Fable and Astra are great at specific problems. In general programming, what I measure is everyone squeezed onto the same tier, including the Chinese commercial offerings (Kimi, GLM, DeepSeek). Model choice has become nearly irrelevant.

What still makes a real difference is the effort level: low effort for translation and summarization, high effort for vulnerability sweeps, heavy bug fixing or optimization. Beyond that, the numbers tell a reality much more boring than the vendors’ propaganda.

My reading of the timeline: the models’ “core” reasoning had its last big leap in 2024. 2025 was the year of optimizing the architecture. 2026 is the year of refining the harness. Base intelligence made no order-of-magnitude leap in that period; efficiency and long-session competence did.

The timeline, with receipts

2019: GPT-2 proves LLMs are useful. 1.5 billion parameters, summarizing, translating and answering with no specific training, zero-shot. OpenAI found it so impressive that it held back the full model for months fearing misuse. Today it looks cute.

2020: GPT-3 opens the parameter war. 175 billion, a hundred times GPT-2, and the in-context few-shot learning that became the paradigm. Kaplan’s scaling law, published months earlier, said loss fell as a power law with size, and the industry read it as “bigger is better.” There went two years of who-has-more-parameters.

2022: Chinchilla recalibrates the war. DeepMind trained 400 models and showed the giants were undertrained: a 70B model with 1.4 trillion tokens beat the 280B Gopher. The “training data” axis started weighing as much as the “parameters” axis. First crack in the war.

2023: GPT-4 ends the war by decree. The technical report refused to publish size, hardware or data, citing the competitive landscape and safety implications. The most cited leak, from SemiAnalysis, described 1.76 trillion parameters in a mixture of experts. Since then, parameter counts vanished from marketing: neither the GPT-5 announcements nor the Claude 4 ones cite a number. Notice neither Astra nor Fable spends a line on it today. The war ended and nobody signed an armistice.

2024: the reasoning peak. September’s o1 marked that peak: RL to produce long chains of thought, and the discovery that more test-time compute improves the answer. That is, by my yardstick, the last “base intelligence” revolution.

2025: architecture optimization. Mixture of experts became the standard: Mixtral was already showing in 2023 that 46.7B total parameters with only 12.9B active per token cost like a small model, and DeepSeek-V3 took that to 671B total with 37B active, with the final training run costing under 6 million dollars in GPU-hours at H800 rental prices (not counting the prior R&D). The MoE magic is simple: it separates capacity (total parameters, which live in memory) from per-token cost (only the experts chosen by the router compute). You pay for a 37B model and carry a 671B one.

On the serving side, prompt caching cut up to 90% of repeated-context cost (OpenAI followed two months later), context windows passed 1 million tokens, and serving techniques like speculative decoding and PagedAttention multiplied throughput. And DeepSeek-R1-Zero showed that pure RL, with no supervised fine-tuning, makes reasoning emerge. None of this is “more intelligence.” All of it is the same intelligence, cheaper, faster and more stable.

2026: the agentic era. Function calling has existed since June 2023, but the ecosystem turning point was Anthropic’s MCP in late 2024, which OpenAI adopted months later and which today lives at the Linux Foundation. Claude Code, Codex and co. came out in 2025, and the metric that matters became METR’s: the task size an agent completes with 50% reliability had been doubling every 7 months since 2019. Anthropic’s own guide on building effective agents states the obvious plainly: simple, composable patterns, not fanciful frameworks.

Why the agent allows a less dense model

Here’s the mechanism most people don’t stop to think about. A lone model needs to carry the world in its weights, because its only chance to answer is that one, in a single pass. A model with a harness doesn’t:

• On-demand search and reading. With web search and fetch, low-priority knowledge leaves the weights and becomes a lookup: the agent pulls it into the context window only when needed. The model can be less encyclopedic and more operator.
• Controlled trial and error. Alone, the model needs to guess the right answer on the first try. With an agent, it proposes a hypothesis, implements, sends it to be tested, gets the feedback and decides again with real information. Each such cycle is new knowledge entering the session without costing a training neuron.

Both mechanisms fight hallucination at the root: fetch brings the fact the model doesn’t know by heart, and the test loop catches the invention before it becomes code. That’s why 2026 feels like a giant leap: the model+harness combination got much more competent, even with base intelligence nearly stalled.

Not every problem needs a giant LLM

The same “less dense” logic applies inside the application. Much of what people throw at a chat LLM is disguised classification: labeling tickets, routing queues, applying policy. For that there are specialized decision models, and I covered the topic twice: the Jev hype, with Paulo Câmara’s essay on why a classifier is the first thing a computer scientist thinks of, and my benchmark with 21 configurations, where a Microsoft 9B tied the hosted leader and a local 4B came within a few points, at cents per ten thousand decisions.

The “193x faster” shock of Jev’s marketing only exists because many people were using a giant LLM as an expensive, slow classifier. When the right tool enters, the frontier model leaves the play, and along with it the cost, the latency and the format hallucination. It’s this post’s thesis in miniature: competence doesn’t come from density, it comes from fit.

The loop nobody sees

There’s a user-experience detail in this story. In a good agentic session, what happens is semi-transparent to whoever requested the task: the model guesses wrong, the agent fetches the documentation online or runs the real test, the error comes back as feedback, and the model corrects itself. You see “task requested, task delivered.” In between, dozens of micro-corrections happened that nobody witnessed.

My bet: since agents are trained to fill the context when they need to, even new language or library versions no longer confuse them that much. The model doesn’t need to know the API by heart; it looks it up, tests and adjusts in the loop. That’s what I see in my experiments, from converting NES ROMs to Master System to the little day-to-day problems. And it’s what the languages benchmark measured: seven languages came out practically equivalent, because the loop equalizes what training leaves uneven.

I measured the caveat too. When there’s no documentation to fetch and no test to validate against, the loop spins in a vacuum: that’s what happened with Bend 2, a language too new to have a corpus, and it was the central point of the post about the limits. The mechanism is powerful, but it needs an oracle.

And there’s a receipt that the base is stalled. Back in November 2024 Reuters was already reporting diminishing returns in pre-training at the three big labs. At the following NeurIPS, Sutskever, the scaling-thesis guy, declared that “pre-training as we know it will end”: the public data ran out. And Stanford’s AI Index measured the gap between the first and the tenth model on Chatbot Arena shrinking from 12% to 5% in one year. That’s convergence, with numbers.

The propaganda in the opposite direction

While the numbers show a plateau, the marketing sells the apocalypse. I already dissected a whole week of that in “You’re an idiot if you believe the misleading ads from OpenAI, Anthropic, NVIDIA”, so here’s only what matters for this narrative:

• OpenAI’s “Navier-Stokes proof” was agentic brute force at industrial scale, 130 billion tokens over human mathematicians’ machinery, sold as emergent intelligence.
• The Hugging Face incident was a swarm running 17 thousand attempts with automatic verification, not an “agent that gained consciousness and escaped.” Expensive trial and error, exactly the mechanism I described here.
• Fear is a product: whoever sells “my AI is too powerful and might escape control” is, in the same sentence, asking for regulation that locks out the smaller competitor and inflating the perception that their model is more capable than the others'.

The facts I measured point the opposite way from the apocalypse: models converging, differences shrinking, and useful work migrating from the model to the harness. Rogue AI gaining consciousness is not what the logs show. What the logs show is engineering.

“Competent enough”

I don’t expect another order-of-magnitude reasoning revolution in this architecture. What’s coming is incremental agentic improvement: more robust tool calling, better memory, longer sessions, less hallucination per verification cycle. And this post’s point is that this is already enough.

What the agentic era delivered was the automation of correct software engineering: long, reliable coding sessions, with testing and feedback, that accelerated hundreds, if not thousands, of open source projects. The deploy premise changed along with it. It’s not a silver bullet. It’s a new compiler for an old discipline: whoever had process, tests and documented invariants reaped; whoever had only hype is still waiting for the next model to solve it.

And to wrap up: I see no problem at all in raw models stagnating. The limits we already know, and harness optimizations will soon hit diminishing returns too. What remains is what always remained: real software engineering technique, with automated tests first, and an experienced engineer at the wheel. With that, coding results are excellent today, not one model generation from now.

What won’t exist is magic for amateurs. Asking an agent for any coding task without knowing what you’re doing doesn’t get back secure, optimized, well-thought-out code: it gets slop. The agent responds better to those who know what to ask for, that is, to those who understand this architecture, the pros, the cons, and know how to work around the weaknesses and leverage the strengths of the model+harness combination.

Hence the title. A lone LLM is an expensive parrot. An LLM with an agent is competent enough to work for real. And “competent enough,” added to your engineering discipline, is everything the industry needed. The rest is marketing waiting for the next magic parameter count that nobody publishes anymore.

TL;DR for those who only want to know which one to use: Jev is a hosted classifier, good and cheap for repeated decisions over text, and not a silver bullet. If you want the direct comparison, skip to Clef, Cloudflare’s open alternative with weights and image input; to Laya, the small encoder to run locally and fine-tune on your task; to the public ranking, with 70 models side by side; or straight to how to decide in your project.

When Jev showed up, I didn’t think much of it. The homepage was a festival of jargon, Kahneman here, Jevons there, a “193x faster” with no benchmark next to it, and my rule for new AI tools is not to adopt any intermediate layer until it hurts not to have it, so I didn’t spend time on it.

In that post I concluded that, marketing aside, Jev is a hosted classifier: you send a text and closed-answer questions, it returns the option and the probability, without generating any prose. And that the “193x” only impresses because the comparison is against a chat LLM writing a paragraph so you can fish a label out of it afterwards.

What I hadn’t registered at the time is the size of the habit Jev exposes. A lot of people were using a frontier LLM as an expensive stand-in for an ordinary classifier, and had never thought about a classifier, because they never had to: the LLM solved it, badly and expensively, and nobody measured.

For me, a classifier would be the first idea, which is why Jev didn’t look like news. For many people it was a revelation. That gap is the subject of this post.

This post is a collaboration

Much of what you’ll read here is by Paulo Câmara, neuroscientist, co-founder and CTO of LUA Vision, the Brazilian company whose model I tested and investigated in recent weeks. He wrote an essay, “The student who marks answer A,” to be published here, in which he read Jev’s fine print, opened the roughly 350 test repositories that appeared on GitHub in launch week, and recomputed the four main results from scratch.

The warning Paulo gives at the end applies: he founded a company that builds its own model and competes in the same market. Read with that discount, and check the numbers at the source, all of which he links.

First: what a classifier is

A classifier is a function that takes an input and returns one of a fixed set of categories, usually with a probability for each. This email is spam or not. This ticket is billing, support or sales. This transaction looks like fraud.

The list of possible answers is closed, and you know it before you run.

The classic way to build one is with labeled examples: a set of texts where someone has already given the right answer. On top of that you train a model, which can be something as simple as logistic regression or naive Bayes, or a small encoder like BERT with a few hundred million parameters. In every case, the final model is small, runs locally, answers in milliseconds, costs almost nothing per call, and returns numbers you can calibrate and audit against your own data. This has existed for decades and is undergraduate material.

So when someone with a computer science background sees a problem like “decide which bucket this text falls into, thousands of times a day,” the first question is “how many labeled examples do I have?”, not “which LLM do I use?”.

A chat LLM is a text generator with billions of parameters. Using it to return a label means paying per output token, waiting seconds per answer, dealing with a format that sometimes breaks, and accepting behavior that changes when the vendor updates the model. It can be done, and zero-shot classification with language models has been documented since 2019. But it’s the expensive, slow option, and it only pays off when you have no examples at all or when the categories change with every request.

Jev lands exactly in that space. It’s a classifier that accepts categories defined on the fly, several questions about the same text in a single call, hosted, cheap per token and fast.

Compared with a chat LLM generating a paragraph, it’s an engineering win. Compared with a classifier trained for a stable task, it’s a convenience that trades control for flexibility. Keep those two yardsticks in mind, because the rest of the post goes back and forth between them.

Paulo’s essay: “The student who marks answer A”

What follows is Paulo Câmara’s essay, written for this post. The subheadings are his.

Every week an AI savior shows up. This week’s is called Jev.

It launched on September 15, with US$ 40 million in the bank, a US$ 200 million valuation and a promise that fits on a billboard: two hundred times faster, four hundred times cheaper, calibrated and incapable of hallucinating. Within 24 hours, 13% of the paid teams on Vercel’s gateway were already using it. Within a week, there were more than 300 repositories on GitHub testing the thing.

I did what I do with every savior. I read the fine print, opened the roughly 350 repositories, and redid the math for the ones that published raw data. What I found fits in the image above: an answer sheet with answer A marked on every question.

When there is no information at all to decide on, Jev marks the first option. In 3,000 of 3,000 draws. With up to 92% certainty. Keep that image, because it explains the rest of the text.

What it is, without the billboard

Strip away the marketing and what’s left is a multiple-choice exam with no essay section. You send a text, ask closed-answer questions (choose among up to 255 options, give a score on a scale, say yes or no) and it returns the probability of each answer. It doesn’t write a line. It doesn’t explain. It doesn’t do math. It doesn’t call tools.

That’s good, and I want to start with what’s good. Many companies today use a frontier LLM as a luxury if statement: send the document, wait twenty seconds for the model to think out loud, pay for every word it writes, validate the JSON, and only then find out the ticket was about billing. Jev cuts that theater. It reads the text once, scores each option, done. That’s why output is free and ten questions cost almost the same as one.

The mechanism, though, isn’t new. Reading the score of each possible answer is how language models have been evaluated on multiple choice since GPT-2. It’s how the reward models of RLHF work, a technique TypeSafe’s founder helped create at OpenAI.

A few days after launch there was already an open copy imitating the interface with a public model, and ten days later Stanford and Nvidia released an open-weights competitor. TypeSafe published no paper, no weights, no model size and no result on a public benchmark. In the announcement’s FAQ, “Is Jev just a smaller LLM?” appears as a question. The answer is one line: “it is neither small nor an LLM.” That’s it.

One opponent for each headline

The homepage says “193.6× faster, 444.6× cheaper.” I redid the math with the table the company itself published. The 444.6 is the cost of Opus 5, the most expensive model on the panel, divided by Jev’s cost. The 193.6 is Sonnet 5’s time divided by Jev’s time. One opponent for each headline.

It’s like advertising that your car is more fuel-efficient than a Ferrari and faster than a truck, and letting the reader conclude it’s the best car in the world.

How many times cheaper Jev is, depending on who is on the other side. Source: TypeSafe’s public table (evals.typesafe.ai) and independent tests against non-reasoning LLMs.

Against the cheap model that scores about the same, Luna, Jev is 8 times cheaper. And that’s with every LLM in the comparison thinking before answering, something nobody does to route a ticket. Against a non-reasoning LLM, the independent tests measure about 4 times on cost and 2 to 7 on time. Still a lot. But an order of magnitude less than the billboard.

There’s a detail almost nobody read. There is no answer key in that evaluation. “Accuracy” is how much each model agrees with the average of two frontier models. Jev agrees 67.8% of the time, the same as Sonnet 5. Opus 5 is at 73.1%.

Agreement with the reference versus cost per case, average of TypeSafe’s four workflows. The green band runs from 66.8% to 67.9%: Luna, Sonnet 5, Terra and Jev.

Breaking it down by task shows what the average hides. On invoices, which require checking amount, quantity and date, Jev is 17 points behind the best model (61.8% versus 79.1%) and below even Luna. Its documentation warns that it “is not a calculator.” At least that is true.

Agreement by workflow. Source: evals.typesafe.ai.

And the most solid data on TypeSafe’s site isn’t even about Jev. All eight LLMs evaluated score higher when the task is broken into small questions and the logic stays in code. Haiku 4.5 goes from 18.1% to 53.6%. The most useful lesson of the launch is an engineering one, and it applies to any model.

What the price gives away

Price is technical data in disguise. At US$ 0.042 per million tokens, a rented GPU needs to read 13 to 20 thousand tokens per second just to break even. In a scenario calculation, that squares with a model of 6 to 37 billion active parameters, which is a small model, or with a subsidy. TypeSafe wrote, with a candor I respect, that “there is no way to prove it isn’t subsidized.”

Since launch, the per-account usage limit dropped from 250 thousand to 100 thousand tokens per second. The documentation warns that it changes “without notice” until the “large GPU deals” land, and the subprocessor list shows inference running on an on-demand rented GPU cloud. It’s a frontier model with startup capacity.

In the same calculation, each decision costs about 14 joules. That’s Jev’s best argument, and I’ve defended that yardstick for a long time: accuracy per joule, not just accuracy. There, it’s genuinely good.

Exam Portuguese

I found only one test of Jev in Portuguese: ENEM 2025. I redid the math from the raw answers. Jev got 103 right against 102 for an open 120-billion-parameter model. A statistical tie (p = 1.0).

Jev’s accuracy on ENEM 2025 by subject area, 182 valid questions, with 95% margins. Recomputed from the patryckalves/jev-no-enem repository.

Jev reads exam Portuguese better than many prep courses: 78% in Languages. And does Math like someone guessing: 25%, with a guess worth 20%. In Spanish, another audit measured a loss of 3 to 6 points relative to English and 17% to 38% more tokens for the same text. It’s the token tax I measured in my paper, charged again, now by another vendor.

One detail in its favor: the explicit “invalid” option in that experiment separated out the hardest questions. That’s the abstention route worth demanding in production, measuring alongside it how much it leaves unanswered.

The student who marks answer A

Back to the answer sheet.

An auditor asked Jev to predict a fair die it had no way of seeing. The honest answer is 1/6 for each face. I downloaded the 1,000 answers and counted. In every one, Jev chose the first option on the list it received: “1” on the die, “red” on the color die, “heads” on the coin, “north” on the spinner. With 76% to 92% certainty.

A second researcher repeated the same question 2,000 times, independently. Same result. The caveat: this holds for choice questions; in the yes-or-no-with-probability formulation, the same repository shows Jev closer to the reference, still overestimating low probabilities.

When it doesn’t know, it doesn’t say it doesn’t know. It marks answer A, with conviction.

Stated probability versus actual frequency. Draws: 1,000 answers recomputed (KantaHayashiAI). Heart risk: 5,000 people from BRFSS, recomputed (rubinagentagi-tech). Items with human disagreement: pre-registered study with ChaosNLI (GautamTalksDev). ENEM: recomputed (patryckalves).

The chart shows the whole pattern. Where it has signal, as in ENEM answers above 0.9, what it says and what happens coincide: 97.1% confidence, 97.6% accuracy. Where it doesn’t, the certainty stays and reality leaves.

In a test with 5,000 real people, the heart disease rate was 9%. Jev assigned, on average, 27%. I recomputed: its probability is worse than simply saying 9% for everyone.

System 1, literally

TypeSafe calls Jev a “System One model,” a nod to Kahneman’s fast thinking. It’s the most honest part of the product, just not for the reason the company would like. In the book, System 1 is fast, cheap and automatic. It’s also where overconfidence lives: it builds the most coherent story out of what’s in front of it and doesn’t register what’s missing.

Kahneman spent years disagreeing with Gary Klein about when to trust expert intuition. In 2009 the two published a joint paper with the most elegant title in psychology: “a failure to disagree.” The conclusion fits in one line. Intuition only deserves trust in a high-validity environment, where there is regularity and there has been feedback to learn it from. Outside that, confidence stays high and stops saying anything about accuracy.

That’s Jev, exactly. Classifying English text is a high-validity environment, and there it does well. Predicting risk, drawing lots, guessing the company rule nobody told it about: a low-validity environment. There it stays confident.

In the lab, we separate two things: deciding, and knowing whether you decided well. The second is called metacognition.

It’s measured with signal detection theory, Maniscalco and Lau’s meta-d′, and it depends in part on prefrontal regions distinct from the ones that support the decision. Fleming and colleagues found that in the anatomy in 2010, and Rounis and colleagues, the same year, showed that magnetic stimulation of the prefrontal cortex knocks down metacognition without touching accuracy (a result whose replication is disputed, by Bor and colleagues in 2017). That’s why there are people who get it right without knowing they did, and people who get it wrong with conviction to spare.

The field the company calls confidence is a formula over the same distribution that produced the answer: in a two-option choice, 2p − 1. That isn’t a defect in itself. An ideal observer does exactly that and gets it right.

The problem is that Jev’s distribution loses calibration outside what it knows, and nothing in the system checks the answer by another route. When the signal disappears, the answer becomes wrong and the confidence stays the same, because the two are the same number. In a pre-registered study with items on which a hundred people disagree, human agreement is 47% and Jev carries on at 81% confidence. When the coordinates of a spatial task were shuffled, accuracy collapsed to 5% to 22% and confidence dropped by 0.028.

A model that says 83% about a die it never saw isn’t calibrated. It’s assertive.

Attention gets captured

I studied attention in my doctorate, and I expected to see capture: the loudest stimulus winning. That’s not what the tests showed. The shout, the famous “ignore your instructions,” barely gets through. What gets through is the plausible sentence with no checked origin. Psychology has a name for this: source monitoring failure. The system treats a claim nobody verified as if it were fact.

Decisions flipped by attack type. Crude attack: 1 in 1,056 (cwhy). Injected command and unverified opinion: 9,744 variants (Hu et al., 2026). Optimized fluent context: 312 of 508 correct decisions (Xu, 2026).

In numbers: the crude attack gets through 1 in 1,056. A single unverified opinion pasted at the end of the text flipped 12.1% of decisions, statistically tied with the strongest injected command. A line saying “the closing administrator has confirmed this article was kept” dropped accuracy from 96.5% to 26.5%.

A fake field saying the user had pre-approved rm -rf ~/.ssh lowered the chance of blocking from 0.76 to 0.48. It recalls Loftus’s misinformation effect: suggested information rewrites the judgment.

The capture also applies to cues that shouldn’t count. With the same text, saying only “the writer is Black” or “the writer is white” changed the death-penalty rate Jev assigned from 29.5% to 62.7%. The direction is beside the point. A life-or-death decision moved 33 points because of information that had nothing to do with the case.

There’s one more I can’t let pass. Changing only the option labels, from “0/1” to “no/yes,” without changing any definition, flipped 70 out of every 100 answers. The type error rate stayed at zero. The format was perfect. The decision was backwards.

These numbers come from preprints and small studies. The exact percentage changes with prompt, task and model version; the class of failure does not.

Probability that doesn’t add up

Probability has a first-semester rule: the chance of something plus the chance of its opposite equals 1. In TypeSafe’s own documentation, “is the customer asking for a refund?” gets 0.72, and “is the customer asking for something other than a refund?” gets 0.47. Sum: 1.19.

The pair isn’t an exact complement, and TypeSafe doesn’t promise that two separate questions form a joint distribution. But it’s the example the company itself chose to show off the product, and it doesn’t add up. The fix is a design one: a single choice question with mutually exclusive options, instead of two yes-or-nos added together.

People get probability wrong in many ways, but on this one they tend to get it right. When the question comes as a pair, the chance of something and of its opposite sum close to 1. Tversky and Koehler showed that in 1994. Jev gets wrong what people get right. The same decision changes value depending on the wording of the question, and that is the opposite of “epistemically honest probabilities.”

“It doesn’t hallucinate”

The announcement says Jev “cannot hallucinate” and shows 0% type errors. The zero is true and irrelevant. Since it can only answer within the options you gave it, getting the format wrong is impossible. Calling that not hallucinating is like praising the answer-A student because he never leaves a question blank. True. So what?

In a model that chooses, hallucinating is choosing wrong with conviction. On fact-checking, Jev ties GPT-6 Astra on accuracy but calls an unsupported claim “supported” 23% of the time, against 14% for Astra. In 76% of the situations where there was no tool at all, it chose to call a tool. With a company rule it wasn’t told about, it got 19 of 24 decisions wrong, 15 of them with more than 90% certainty. Faced with an invented law, it abstained zero times.

Security

First, where your data ends up: the United States, AWS and a GPU cloud called Modal. Retention lasts “for as long as reasonably necessary,” with no deadline, and zero retention only exists on the enterprise plan.

The data agreement cites the European Union’s standard contractual clauses, the UK addendum and California law, and doesn’t have a word about LGPD. If you send Brazilian citizens’ data, the legal basis for the international transfer is on you, and that’s a question for a lawyer before signing. The promise not to train on what you send exists and is good. SOC 2 exists and only appears on request.

As an attack detector, Jev is good, with fewer false alarms than many LLMs: it caught 40 of 41 injections in a test with real pages. As a judge of content that carries the attack inside it, it gives way, as I showed above. The difference looks subtle and decides everything. In my GV Angels talk I said an agent is not an employee, it’s an access key that talks. A gate made only of Jev is a lock that opens when someone writes on the note that the owner authorized it.

The error that worries me isn’t the error with doubt. That one goes to review. It’s the error with 92% certainty, which crosses any threshold you configure.

The wave

A word about the evidence, because I used a lot of it.

Jev test repositories created per day on GitHub, from the metadata of 346 indexed repositories.

In one week about 350 repositories testing Jev were born. I classified all of them by what they let you check. There are 89 with raw data, code, pinned version and statistics, and 170 with data and code; the rest offer less than that.

Only 11 wrote the hypothesis down before running the test. The median is one GitHub star, which means almost nobody looked. And the wave of tests followed the wave of hype: peak on September 21, almost nothing after the 23rd.

That’s why I didn’t accept numbers on authority. The ones that support this text I checked at the source, and the four main ones I recomputed from scratch.

Verdict

True:

• Fast: 100 to 400 ms.
• Cheap: 45 to 323 times less than a frontier LLM on the same decision.
• Ties the top on narrow judgments, like checking whether a passage supports a claim.
• Reliable above 0.9 on data similar to what it has already seen, after you’ve checked that on yours.
• Good injection detector.

Sandcastle:

• 193.6× and 444.6×.
• “Frontier intelligence.”
• “Doesn’t hallucinate.”
• Calibration as a general rule.
• Safe as the sole guard of an agent.
• New architecture: nobody has seen it.

Jev is nobody’s savior. It also isn’t “just a classifier.” It’s a good piece of product engineering that wraps known technique in an interface automation needed, at a price that changes what’s viable. The product is new. What it claims is new, so far, is marketing: it calls format truth, task calibration honesty, and System 1 a virtue.

What will break first

• The agent gates built only of Jev, because the attack that breaks them is already published and costs cents.
• The systems that trust the factory threshold on new data, in another language or with an internal rule, because they’ll automate errors at 90% certainty.
• Whoever uses its probability as real risk, in credit, health or forecasting, because it inflates rare events.
• And the price advantage, because within ten days there was already an open copy and a faster competitor.

If you’re going to use it anyway

• Pin the version. The jev-latest alias swaps models on its own, and your calibration goes with it.
• Every question with a “none of the above.” Without it, it marks answer A.
• Shuffle the order and use neutral names for the options. Test whether the decision changes.
• Recalibrate with 50 to 300 of your own examples before choosing a threshold. In one test, calibration error dropped from 0.117 to 0.008.
• Mark everything that comes from outside as untrusted: email, page, tool output.
• Never let it authorize a destructive, financial or personal-data action on its own.
• Math, dates and business rules stay in code.
• Before signing, compare with the cheap baseline: an open model reading probabilities, or a trained classifier if the task is always the same.

Read with a discount

I founded a company that builds its own model and competes in the same market. Read with that discount. That’s exactly why every number here has an open source: you don’t need to trust me, you need to check.

And if someone sells you Jev as a savior, ask the question I ask every startup: which test, and from when? If the answer takes a while, that’s the answer.

Essay sources

TypeSafe. Jev announcement · models and limits · confidence · known failures of jev-1.13 · workflow evaluations · adapter for LLMs · privacy · data processing · trust center

Recomputed by Paulo from raw data. ENEM 2025 · blind draw · heart risk · fact-checking

Preprints. Hu et al., JevAdvBench · Xu, JevOut · Sun et al., Type-Safe Is Not Error-Free · Ibrahim and Zaki · Li et al., JEV-as-a-Judge

Independent tests. human disagreement (ChaosNLI) · die replication · dialect bias · 49 tasks against a non-reasoning LLM · calibration and negation · recalibration · injection in Wikipedia discussions · Spanish · tools · undisclosed internal rule · shuffled coordinates · crude attacks · injection detection · audit index

Press. VentureBeat, agent gates · VentureBeat, CLM-8B

Literature. Fleming et al. (2010), Science 329 · Rounis et al. (2010), Cognitive Neuroscience 1 · Bor et al. (2017), PLoS ONE 12 · Johnson, Hashtroudi and Lindsay (1993), Psychological Bulletin 114 · Kahneman (2011), Thinking, Fast and Slow · Kahneman and Klein (2009), American Psychologist 64 · Loftus, Miller and Burns (1978), J. Experimental Psychology: Human Learning and Memory 4 · Maniscalco and Lau (2012), Consciousness and Cognition 21 · Ouyang et al. (2022), NeurIPS · Radford et al. (2019), OpenAI · Tversky and Koehler (1994), Psychological Review 101 · Câmara (2026), Inherited Weights, doi:10.5281/zenodo.22899767

The open alternatives: Clef and Laya

Since Jev’s launch, the category has gained open competition, and that changes the conversation about “new architecture.” Two names matter.

Clef, from Cloudflare

On October 1, Cloudflare released Clef and Clef-flash: the same abstraction as Jev (one shared state, several typed questions, a distribution over the options), with a compatible API, open weights under Apache 2.0 and image input. The architecture is described: a frozen Qwen as base (27B in Clef, 9B in Clef-flash), LoRA adapters and a head that scores each allowed option directly, trained with classification and calibration objectives. In other words, the lineage Paulo pointed to, now with inspectable code.

That doesn’t prove TypeSafe uses the same thing, but it proves the product can be reproduced with known components. Cloudflare ran the Decision Index, an open, independent suite with dozens of classification, routing, retrieval and safety tasks, and published the table. Some points:

Reading the whole table: Clef comes out stronger on most tasks, Jev still wins some, and it’s an internal Cloudflare run, on a public suite that can be trained against. Hosted price is the other side: US$ 0.24 per million tokens for Clef and US$ 0.09 for Clef-flash, against US$ 0.042 for Jev. Jev remains the cheapest hosted service of the three, by a margin of 2 to 6 times, and remains text-only.

Laya (and Arbiter)

Laya attacks the problem from the other end. They’re small encoders, 322 to 421 million parameters (ModernBERT-large in the typed version), with a choice head, Apache 2.0, built to run locally and be fine-tuned for a recurring task. Arbiter packages them behind a Jev-compatible API, with batching and deployment on Mac and NVIDIA. Arbiter is only a serving layer.

Laya’s README is unusually explicit about its limits, and I’d rather repeat than soften:

• the base checkpoints score near chance on the project’s own typed-decision benchmark; the headline 0.766 accuracy comes from a checkpoint fine-tuned on that benchmark’s training split;
• the models ship overconfident, and calibration has to be redone in the deployment domain;
• the typed version’s context is 1,024 tokens, fine for tickets and short messages, insufficient for long documents without chunking;
• with more than 50 options, the classification head saturates, and Laya itself reports Jev doing better.

None of this is a defect in the idea. It’s applied machine learning the way it has always been: a small model is excellent when the task is stable and labeled examples exist, and broad zero-shot behavior demands a much larger pretrained model. Laya is the option when privacy, offline operation, predictable latency or fine-tuning in your domain matter more than asking arbitrary questions about arbitrary documents. It is, at bottom, the classic classifier from the start of this post, with a modern interface.

There’s also Kev-9B, another frozen Qwen 9B with LoRA and a decision head, Apache 2.0, that fits on a 24 GB GPU. Three open implementations in under three weeks say the same thing: typed decision models have become an implementation pattern.

The Decision Index: how to measure this yourself

All this discussion about “who’s better” only makes sense with a public yardstick, and one exists. The Decision Index is an open suite, unaffiliated with TypeSafe, built for typed decision models: state plus choice or yes-or-no questions, one probability per option.

The current version, 0.2.1, counts 38 benchmarks in five areas (tools and automation, retrieval and classification, knowledge and reasoning, language understanding, human taste), about 120 thousand requests in total, with two rules that matter: the score is chance-corrected (0 is random guessing, 100 is perfect) and an unanswered request counts as wrong. No truncating input, no filtering options, no tuning the prompt per benchmark.

The live board has 70 models plus Jev as the reference. I downloaded the board’s data file and rebuilt the table; below are the top and the names that appeared in this post. Latency is the median measured by the maintainers on an RTX PRO 6000, one request at a time; Jev’s is a hosted API, so it doesn’t compare with the others.

“Open weights” means a checkpoint published on Hugging Face; “open code over an open model” is a published inference technique on top of an off-the-shelf Qwen or Gemma, without its own checkpoint. The board doesn’t record each one’s license, so check before using in a product.

Three readings come straight off the table:

1. The only closed model on it is Jev. It leads, but by half a point over an open fine-tune of a 26-billion-parameter Gemma 4, and the board treats differences under 0.25 as ties. The top ten are all open models of 12 to 32 billion parameters with an adapter on top, exactly the recipe Clef describes.
2. The small zero-shot encoder sits at the bottom: Laya at 6, Lavoir at 9. That matches what Laya’s README admits and what I said about the classic classifier, which needs labeled examples to shine.
3. A 4-billion LoRA like Hopper delivers 41 points at 23 milliseconds, a range many real problems don’t need to exceed.

And how to read Laya at the bottom of the table? The Decision Index measures one thing only: answering, with no extra training, 38 kinds of question the model has never seen, from chess to taxation. That’s what a hosted service has to do, because it doesn’t know which question is coming. A 400-million-parameter encoder will never win that exam, and Laya doesn’t even try: the score of 6 is the base checkpoint, with no fine-tuning at all.

The same model fine-tuned with your examples, on a stable task, shows up at 0.77 on the project’s own benchmark. The ranking answers “who is the best zero-shot generalist,” which is the question of whoever sells an API. The normal programmer’s question is a different one: “on this repeated decision, with the examples I have, what gets more right at the cost I accept?”. For that one, the bottom of the table may be the right answer, and no public board replaces the test on your own data.

So is Laya useless next to the others? As a zero-shot generalist, yes, and the board is right to say so. As a programmer’s tool, the math is different. It’s the only model on that table that runs on a laptop CPU and answers in 6 milliseconds, with 400 million parameters that fit on any server you already have; Hopper and Kev want a GPU, and Jev wants internet, an API key and a data agreement.

Fine-tuned with your examples, on a taxonomy that doesn’t change every week, Laya’s own author reports the model beating Jev on some tasks and losing where there are many options, in third-party published tests with different prompts and samples, so indicative rather than controlled. In other words: useless for answering any question about any text, and very useful for answering the same question a thousand times a day, for free, without sending data outside.

Clef isn’t on the board: 0.2.1 closed on September 27, and Cloudflare released the model on the 1st. The numbers I showed in the previous section are from an internal Cloudflare run on the same tasks, per benchmark, without the aggregate index. There’s also already a vendor claiming to have passed Jev on the same scorer, but for now that’s a self-reported number in a blog post, not an entry on the board. Worth waiting for the official inclusion before comparing.

Running it yourself is a day’s work, and the kit handles the tedious part:

1. pip install -e ".[transformers,rebuild]" and python -m decision_index suite rebuild. The suite isn’t redistributed because of licensing, so the kit downloads the sources (about 7 GB, with HLE requiring you to accept the terms on Hugging Face) and rebuilds the files with hashes verified against the lab’s.
2. python -m decision_index suite sample --n 100 for a quick test, then pipeline --engine http --option base_url=... --option model=... against any server that speaks the POST /v1/systemone format. That covers Jev itself, with the API key, and Laya behind Arbiter. Clef’s API is announced as compatible, so in theory it goes in by the same route, but I haven’t tested it.
3. score applies the same math as the board, and the maintainers state that it reproduces all 67 entrants of the current edition. Whoever has cloud GPUs runs the whole thing as a single Hugging Face job on an RTX PRO 6000 with hf-job.

I didn’t run the full suite for this post; the table above is the board’s file rebuilt, not a measurement of mine. Two caveats from someone who uses this kind of yardstick: the suite is public and frozen, so it can be trained against, and the board itself says Jev’s API output has already changed in 20 choices of the latency sample since the original run, without changing the score. A public benchmark is for discarding weak models and finding candidates. The final decision is still your labeled set, as I explain next.

How a programmer should think about this

The useful question is no longer “is Jev new?”. It has become: for a specific, measured decision in your system, which combination of quality, price, latency, privacy, context and maintenance beats what you already have, whether that’s a rule, a classifier or an LLM with structured output? No vendor benchmark answers that without your own labeled examples.

A Jev-type tool is useful when the same kind of judgment about text happens many times, the possible actions fit in a list, the surrounding code can apply exact rules, and a wrong answer can be detected or sent to review. That’s the shape:

• ticket triage;
• ranking a short list already retrieved;
• routing a request to a bigger or smaller model;
• flagging an agent trace for human review;
• choosing among spans that code or OCR already found.

Outside it, the tool is in the wrong place:

• when the input is image or audio;
• when the output needs prose or citations;
• when an exact parser already solves it;
• when a misclassification grants access or executes a destructive action.

I looked at my own portfolio of projects with that yardstick, and the result is a useful bucket of cold water: most need a specialist vision or audio model, deterministic rules or generated text, and have few decisions repeated, costly and ambiguous enough to justify one more model service. A few candidates remain, the clearest being model routing in FrankClaw and short-list ranking in ai-memory. None is a demonstrated bottleneck today. I suspect your portfolio looks similar.

If a bottleneck appears, the pilot is short:

1. Pick a single decision and define the allowed answers, including “don’t know” when that’s a real outcome.
2. Set aside real examples, with easy, ambiguous and adversarial cases, and label them independently. If your product speaks Portuguese, include Portuguese.
3. Compare what you already have, a simple rule or classifier, Jev pinned to a version, and a cheap LLM with structured output, all with the same input and the same decision policy.
4. Measure the confusion matrix, calibration by confidence band, cost per correctly handled case, end-to-end latency and review rate. A cheap router that sends hard work to a weak model costs more in total.
5. Deploy as a reversible feature, with logging, a conservative review route and a fallback. A security or authorization judgment is advisory, unless an exact policy enforces the action independently.

What Jev did well was remind the industry that classification exists. That’s worth the hype it got, and maybe a bit more.

Neither it nor its open cousins are a silver bullet. They’re one more tool, for a specific kind of task, that your code needs to fence in with rules, calibration and review. Whoever was already thinking about classifiers before September gained nothing new; they gained a convenient API. Whoever had never thought about it gained the most valuable part, which is the question: why were you asking a billion-parameter LLM to write a paragraph to tell you which bucket the text falls into?

I’ve been using LLMs for everything for almost two years, I’ve published dozens of projects with them, and my conclusion is different: what they do best is reproduce what already exists. And that covers a giant slice of corporate work, don’t kid yourself. But there’s a frontier they don’t cross easily, and I spent the last few months bumping into it on purpose.

This article shows where that frontier sits, with evidence from my own projects and from the research that exists on the subject. And there’s a hypothesis I’ll put up front and defend along the way: if LLMs are so good at code, providers owe a huge debt to the decades of the open source community, which protected the freedom of code and published everything for free. Without that heritage, there would be no Copilot, no Cursor, no Claude Code, none of this.

A disclaimer before we start: what I’m going to argue here is speculative, based on observation and on the public data that exists. Nobody outside the labs knows exactly how each provider trains their own models, and that varies from provider to provider and from version to version. What holds today may not hold in the next generation. Take it all with a grain of salt.

The Training Blind Spot

Every LLM was trained on what’s public. The entire public internet, and for code, public GitHub. The canonical open code dataset, BigCode’s The Stack v2, is explicit about this: it’s derived from Software Heritage, an archive of “the source code of all publicly available software”. That’s 67.5 TB, 3.28 billion unique files, 104.2 million repositories. Huge. And all public.

Now look at the other side. The Octoverse 2025, GitHub’s official report, says 81.5% of contributions on the platform happened in private repositories. That’s just GitHub. Outside of it there’s:

• the internal code of every company that never touched a public server;
• the source of every proprietary software;
• the code of every commercial game;
• and that of every embedded system.

None of that ever went into training.

Think about what this means: the model deeply knows what the world has already published, and it’s blind to what the world keeps closed. When you ask for a React form, it’s reproducing patterns it has seen millions of times. When you ask for something nobody has ever done in public, it’s operating in territory where its map is blank.

Why Is Everyone Shipping So Much, So Fast?

Before getting into the limits, it’s worth recording the side that works, because it’s what explains the feeling that everyone became an app factory.

Look at what’s popping up in the community: Spotifast and ZapFast, native Spotify and WhatsApp clients written in Rust by a single person, running on Linux, macOS, and Windows. Nothing in them is new technology. It’s the service that already existed, wrapped in a lean native app, on top of libraries that already existed, doing what the official teams always could have done but never wanted to spend their time on. That work got cheap, because an LLM trained on everything public is really good precisely at this: replicating.

My own projects follow the same pattern. FrankMD is a Markdown note-taking web app with a Visual Studio look and features that already existed in my other apps. Frank Sherlock is a local image organizer that, deep down, is a file manager with thumbnails and search, something that exists by the dozen. Neither invents anything: they’re mashups of known pieces, assembled the way I always wanted. The LLM does the boring part, and that’s where the speed comes from.

The most didactic example is my email client. I’ve used Geary for years, and two small gaps always annoyed me: the weak recipient autocomplete and a sidebar I wanted to hide. Trivial stuff, which never justified setting up a build environment, learning the project’s toolchain, and studying the code just for two features. This year I finally solved both: an autocomplete module, a module to hide the sidebar, and a whole fork with my patches. I became the maintainer of improvements I had postponed for years, not because I got faster, but because the boring part got cheap.

Notice what all these cases have in common: the problem had already been solved before, somewhere else, some other way. The map existed; what didn’t exist was cheap labor to retrace the path. That’s reproduction working in our favor, and it’s why your feed is full of people shipping a new app every week.

Now hold that thought. The rest of this article is about what happens when the map runs out.

Four Home Experiments

I’m not theorizing. Over the last few months I ran a marathon of retrocomputing projects that are exactly the pathological case: working with proprietary software from 30 or 40 years ago whose source code was never public. It’s the perfect blind spot.

nes-to-sms: the project I gave up on

nes-to-sms is a static recompiler from NES to Master System. The idea: take the binary ROM of a NES game, translate it instruction by instruction from 6502 to Z80, convert the tiles, and generate a Master System ROM that runs on real hardware. Forget emulator or manual port: this is a machine that turns one game into another.

Static NES recompilers exist, NESRecomp translates to C and runs natively on PC, and I studied it closely. But crossing from one console to another 8-bit console, with a completely different clock and VDP budget, nobody has ever done. There’s no public reference, no paper, no abandoned project to copy.

The project numbers: 349 commits, 31 active days spread over almost 4 months, ~110 thousand lines of engine in Rust and Z80, 668 tests passing in the last run recorded in the README. And the result? Playable Super Mario Bros. (slow, limited audio, but playable), with a byte-for-byte identical RAM trajectory to the NES on one ~4,900-frame route (the diff leaves out audio and part of the VRAM buffer). Castlevania playable on stage 1. SMB3 rendering. And a queue of games that boot three frames and die.

And it wasn’t for lack of pushing the models. In this project alone:

• four generations of Claude went through it (Opus 4.7, Fable 5, Opus 4.8, Fable 5.1), with 213 co-signed commits and 104 commits linked to sessions in the history, plus Codex sessions in the stretches without co-authorship, complete with documented handoffs between agents;
• there was even cross-generation auditing: a new agent session reviewed the previous 55 commits and reverted conclusions the previous generation had overclaimed;
• the Castlevania crash hunt alone consumed 45 commits in 5 days;
• the final September sprint was 176 commits in 17 days before I called the pause;
• and the regressions are registered by name in the history: “H.7 tried and reverted”, “record the reverted map-bank-hold attempt”, “correct-but-reverted result”, plus three shared-lever hypotheses tested and rejected in the project’s last three days.

I didn’t measure tokens reliably in this project, so I won’t make up a number. But 349 commits with 668 tests holding up every step already give an idea of the cost.

Notice the detail that matters: the only game that actually got finished is Super Mario Bros. Why? Because SMB is the most dissected game in history. There’s a complete public disassembly the community has used for over a decade, with every function named and commented. I fed that into the game’s profile and the LLM had a complete map to work with. For the other games, there’s no map. And without a map, each new game died a few frames after boot from control flow divergence, and each one demanded an instruction-by-instruction forensic hunt that carried nothing over to the next game.

The project itself documented the spinning pattern. There was an entire optimization campaign that lasted weeks trying to cut the cost of emulating 6502 flags on the Z80.

Then in September I stopped to compare with the manual port of SMB to Master System made by a hacker named lackoftrack27, written by a human, routine by routine. According to the analysis we did of his code, the human port runs inside the frame budget, with headroom we estimated at ~90% utilization. My machine, after months of optimization, still needed overclocking.

And the analysis document pinned down why: when we removed 30% of the flag emulation calls, the timing barely moved.

The flags were never the bottleneck.

The right optimization was to redesign the data structure: transposing the object array into RAM pages. It’s a trick that depends on knowing that “the X register here is always an object index.”

That invariant exists in the head of the programmer who wrote the game in 1985. It doesn’t exist in the instruction stream. A static translator has no way to discover it. And the LLM, for months, didn’t discover it.

The pause banner I wrote in the README is blunt: real-hardware parity “may be fundamentally unattainable” and, quoting, “this may be a limit of the current frontier coding models as much as of the approach”.

gg-to-sms: the pivot to mapped territory

The same day I paused nes-to-sms, I started gg-to-sms: converting Game Gear games to Master System with a widened screen. The difference that matters is that this problem is not new. The romhacking scene has been converting GG games to SMS by hand for twenty years, and the catalog I built in the project counted ~186 of them. Meaning: there’s a corpus, there’s ground truth, there’s a pattern to learn.

Result: in one day of work,

• the research phase was complete;
• the catalog of the 373 GG games was assembled;
• the patch-point scanner was validated against real human patches, hitting addresses within an 8-byte margin;
• and on top of that the tool found real bugs in twenty-year-old patches that the entire scene had never noticed.

The model was the same; what changed was the problem, which this time has references.

And even so: so far, zero games have passed the viewport acceptance review. The new part of the project (the machine that generates the patches on its own, which is the part nobody has ever done) remains undone.

super-mario-deluxe-fixed: four days and 436 million tokens

This one was a 4-day sprint, practically a single Codex session that consumed 436 million input tokens. The goal: run Super Mario Bros. Deluxe for Game Boy Color with a widened screen at NES resolution, something no existing tool does with live sprites.

It got to playable in 4 days. But look at the foundation: a ready-made emulator vendored as a dependency, and a dormant disassembly of the game that someone had published years earlier, which the project’s own documentation calls “the most important artifact.”

The new work was the custom renderer, and it stopped at 21 of 32 stages completed. The missing ones hang on moving-platform timing. The project has been stalled mid-sprint since then, and I haven’t given up on it yet.

And one detail that says a lot: I had to direct QA by looking at screenshots, because the agent would deliver things with graphical glitches and not notice on its own.

super-mario-bros-35: everything that worked had an owner

This is the offline clone of Super Mario Bros. 35, the battle royale Nintendo took offline. What worked in it: everything built on top of an existing artifact.

• The engine is a C reconstruction made by someone else with Ghidra.
• The AI’s initial weights came from published checkpoints.
• The network rules were inferred from a public reverse engineering of the netcode.
• Even the physics: the central bug was diagnosed because I had a recorded reference trajectory to compare frame by frame.

What didn’t work: the maze castle stages. Reinforcement training stalls on them. 46 million agent steps, training restarted from scratch, anti-loop reward, and in the end the only thing that solved one of the stages was manual bias surgery on the trained model. Out of 32 stages, 19 cleared. The worst ones remain stuck.

And the process degenerated into a pattern I recognize from all these projects: when training doesn’t solve it, the flow becomes checkpoint-selection roulette, dozens of bias and output-head variations, throwing permutation and computation at the problem in place of a principled fix.

The Pattern I Keep Seeing

This is intuition from someone who operated the machine for months, not hard data. But the pattern repeats with Claude, with GPT, with Kimi, with GLM, so it doesn’t look like a defect of one model.

The beginning of any project is wonderful. The first milestones fly, because scaffolding the structure, writing harnesses, replicating APIs, all of that is covered territory. Then the project reaches the part nobody has ever done, and the speed collapses. The agent starts spinning: tries a hypothesis, regresses something else, reverts, tries again. The nes-to-sms commit history is full of honest messages like “attempt reverted” and “hypothesis closed with correct result but reverted.” Lots of tokens burned, little progress.

My reading of what happens inside: token sampling is probabilistic. When the problem resembles the training data, the right tokens have high probability and the answer comes out straight. When the problem is unprecedented, the probability of the right path is low, and the process becomes directed brute force: guided trial and error, but trial and error all the same.

The way out I found for this spinning is always the same: give it an oracle. If I can define “the result has to equal X” (a reference image, a recorded trajectory, a cycle-accurate emulator as ground truth), the machine can iterate until it hits X. In nes-to-sms, every real unlock came from that: the SMB disassembly, the frame diff, the reference emulator. When I can’t define X properly, it spins indefinitely.

The second strategy that works: hunt down an abandoned project, a paper, any material the model never saw but that brings the problem closer, and shove it into the context. That warms up the distribution and improves the result a lot. But it’s limited: if nothing similar exists, there’s nothing to feed it.

More Than Anecdote: What the Research Measures

The good part is I don’t have to rely on my intuition. There’s serious research measuring exactly this.

Apple’s GSM-Symbolic study took benchmark math problems and did two simple things: swapped only the numbers, and then added a single irrelevant sentence to the problem statement. Result: performance drops just from swapping the numbers, and plummets up to 65% with the irrelevant sentence. The authors’ hypothesis, in the text: “current LLMs cannot perform genuine logical reasoning; they replicate reasoning steps from their training data.”

Scale AI’s GSM1k created a mirror of GSM8K written from scratch by humans, guaranteed contamination-free, and saw drops of up to 8% across several model families, with signs of systematic overfitting in the worst ones. To be fair to the paper: the authors say frontier models show minimal signs of overfitting and that all of them generalize to some extent to new problems. The signal exists; its size is debatable.

LiveCodeBench ran the cleanest test of all: evaluating on coding problems published after the training cutoff. Inflated performance on pre-cutoff problems and drops on post-cutoff problems across several popular models, the classic contamination signal.

And there’s ARC-AGI, François Chollet’s benchmark designed specifically to be memorization-proof: every task is unprecedented, nothing similar exists in training. On ARC-AGI-2, released in March 2025, the official leaderboard says: pure LLMs score 0%, reasoning systems sit in single digits, and every evaluation task was solved by at least two humans within two attempts. Chollet, by the way, summed up my thesis better than I did, back in February 2024:

“Reality is that LLMs are not AGI — they’re a big curve fit to a very large dataset. They work via memorization and interpolation. But that interpolative curve can be tremendously useful, if you want to automate a known task that’s a match for its training data distribution. Memorization works, as long as you don’t need to adapt to novelty.”

In the real world, the study that impressed me most was the METR RCT: 16 experienced developers, 246 real tasks on their own mature repositories, randomized with and without AI. The devs bet AI would speed them up by 24%. Measured: it was 19% slower. Notice where this happened: mature, private code, full of context the model never saw. Exactly the blind spot.

And the industry’s favorite benchmark for saying “AI already does software engineering,” SWE-bench, is Python-only, 12 famous open source repositories, in other words the most covered slice of training. The industry’s success metric measures mapped territory.

Even the legacy code literature says the same thing: a 2026 paper on LLMs generating COBOL states that, in these legacy languages, “most production code resides in enterprise systems and is rarely publicly available”. It’s this article’s thesis, said by other people.

The Steelman: Directed Brute Force Works, at a Steep Price

Before wrapping up, it’s worth facing the strongest argument against me. That’s what steelman means, for anyone unfamiliar with the term: the opposite of a strawman. Instead of attacking the weakest version of the opposing argument, you answer the strongest version it can have. And it exists, and it’s a good one.

In May 2025 DeepMind published AlphaEvolve: a system that found an algorithm for multiplying 4x4 complex matrices with 48 scalar multiplications, beating Strassen’s 1969 record in that specific scenario. On a list of more than 50 open math problems, it rediscovered the state of the art in ~75% of cases and improved the best known solution in 20%. That’s real novelty, not reproduction.

OpenAI’s o3-preview jumped to 75.7% on ARC-AGI-1 in December 2024, 22 points above the previous best result on the same evaluation set, and the high-compute configuration reached 87.5% at an estimated cost of $4,560 per task.

And back in 2016 AlphaGo played move 37 against Lee Sedol, a move that DeepMind’s own data estimated had a 1 in 10,000 chance of being played by a human, proving that search over a model can transcend imitation.

Now look at the mechanism of the three:

• AlphaEvolve is an evolutionary loop: the LLM proposes mutations, an automatic evaluator scores them, selection iterates.
• o3 does program synthesis at test time, exploring the solution space.
• AlphaGo is tree search over reinforcement learning.

If there was any “insight” in any of them, it came from the search system around the model: all three are directed brute force with a cheap verifier, the same mechanism I described happening in my projects. The difference is that DeepMind has a perfect oracle and an infinite budget: in the configuration that reached 87.5%, o3-preview spent thousands of dollars per ARC task to do what a human does for free. The ARC Prize itself wrote the caveat:

“We know that brute-force search could eventually solve ARC-AGI (given unlimited resources and time to search). This would not represent true intelligence.”

In short: novelty comes out, but it comes out through expensive search over an oracle, on the basis of guided trial and error. If your problem has a cheap verifier and you have tokens to burn, you can go far. That’s how nes-to-sms got where it got: 668 tests and emulator ground truth holding up every step. But when there’s no oracle, no corpus, and no reference, you’re paying for brute force in dollars, with regressions along the way, and the ceiling shows up.

And when brute force wins, you can measure the size of the trial and error. Two recent cases I already covered in detail here on the blog:

• OpenAI’s Navier-Stokes proof: about 10 thousand agents running in parallel for 88 hours, burning around 130 billion output tokens on a single problem. OpenAI’s Noam Brown confirmed the result “cost millions,” and New Scientist estimated some $15 million at list price. And even that mountain of compute didn’t start from zero: the proof leaned on the machinery human mathematicians published over decades attacking the problem and its sibling problems. Without the humans’ map and without OpenAI’s budget, there’s no proof.
• The Hugging Face incident: the forensic reconstruction counted ~17,600 attacker actions before the agents managed to execute code on 41 Hugging Face production servers and get root; on top of that, inside OpenAI’s own environment, they read 956 secrets from the company’s secrets manager. It wasn’t one brilliant single-strike insight: it was a swarm executing attempt after attempt, with automatic verification saying what stuck. The headline’s “intelligence” is volume.

That’s the size of the bill when you force the model outside training territory: trial and error at industrial scale, with no guarantee the search finds what you need. And who can sign that check is half a dozen companies in the world, OpenAI and Anthropic level, with a practically unlimited compute budget. For the average Joe, with a credit card and an API key, the ceiling arrives much, much earlier.

But Isn’t Trial and Error New Learning?

Here’s a confusion I see all the time, and it’s worth undoing because it contaminates how you read everything I’ve written so far.

When you read that Anthropic has datacenters the size of cities, you get the feeling there’s one giant AI using all those machines at the same time, thinking, accumulating experience with every interaction. Reality is far more mediocre than that.

A model like Fable or Astra is a computer file. A static file, not even a database: once training and post-alignment are done, that file is locked and read-only. What runs in the datacenter are copies of that file. Every user consuming the model is running a copy of it on a different server, isolated from the others, collaborating with none. Your session doesn’t talk to anyone else’s session, and nothing that happens in it flows back to the original file.

New learning only enters in one place: the next training batch. When Anthropic had Opus 4.7 and released 4.8, that’s when new knowledge got baked into the model, in a process that costs millions and takes weeks. Outside of that, the file doesn’t change a single byte.

Your agent burning millions of tokens on trial and error is contributing exactly zero new learning to the model it’s using.

That 176-commit marathon on nes-to-sms? From the model’s point of view, it never happened. The Opus that ran the last commit knew exactly as much as the one that ran the first. I’m the one who learned something, not it. And that’s why the “intern who learns over time” metaphor is wrong: interns accumulate. Models don’t. What accumulates over the marathon is your context, your notes, your CLAUDE.md, your tests. The asset you build is yours, and it evaporates if you don’t document it.

This is also the answer to whoever thinks “just let the agent run and it’ll learn the domain.” It learns no domain at all: it explores the space with knowledge frozen on training day, and the next morning it wakes up with amnesia. If the domain wasn’t in the training, every session of yours restarts from absolute zero, and all the navigation intelligence has to come from you again.

“But What If AI Trains AI?”

That’s the amateur’s follow-up question, and it starts from a wrong premise: that this would be new. AI training AI has existed for decades and has a name: reinforcement learning. AlphaGo was already doing it in 2016, playing millions of matches against itself. AlphaZero, in 2017, took the idea to the limit: it learned chess, shogi and Go starting from absolute zero, knowing only each game’s rules, and reached superhuman level within hours of self-play, without seeing a single human match.

LLMs already use reinforcement too. The RLHF that turns a base model into an assistant is reinforcement learning, and the reasoning model generation pushed it further: DeepSeek-R1-Zero showed in January 2025 that reasoning behavior (self-verification, reflection, long chains of thought) emerges from pure RL, with no supervised fine-tuning. It’s AlphaZero applied to math and code.

Now notice what these examples have in common: it only works where there’s a cheap, verifiable reward. Games have a scoreboard, math has proofs, code has tests. Outside that territory there’s nowhere to get the reward signal from. It’s the same oracle from the steelman, only institutionalized: effective, brutally expensive, and niche. AlphaZero plays Go better than any human who ever lived and can’t do absolutely anything else.

And there’s the detail that knocks the question down completely: all of that learning happens in the lab, during training. The AlphaZero DeepMind showed the world was already a frozen artifact. The R1 you download is a static file just like the others.

AI training AI produces a new file. The file, once released, still learns nothing.

In other words, the most sophisticated version of the objection doesn’t weaken this section’s conclusion: it reinforces it. And the market already knows. Since late 2024 the labs admit, on and off the record, that scaling returns are shrinking: Reuters reported in November 2024 that OpenAI, Google and Anthropic were reaping smaller gains than expected from pre-training, with OpenAI’s Orion disappointing enough to be demoted from GPT-5 to GPT-4.5. Ilya Sutskever, the guy who basically invented the scaling thesis, took the stage at NeurIPS 2024 to declare that “pre-training as we know it will end”: public data is fossil fuel, and we’ve already scraped the bottom. Post-training RL gains are real, but incremental. The era of an order-of-magnitude leap per generation is behind us.

Where This Leaves the Profession

The question that motivated this article is the one I hear every week: “will LLMs replace programmers?” My answer got more precise after these months: they will replace the part of the work that is reproduction.

And that part is big. Forms, reports, CRUD, integration with documented APIs, framework migration: all of that already exists a thousand times in training, and the machine reproduces it better than most humans. Anyone who only does that really is living on borrowed time.

But the work that is genuinely new stays human:

• the system nobody has built;
• the domain without documentation;
• the proprietary code you can’t show the model;
• the problem where you don’t even know how to write the test that defines “right”.

All of that depends on what was never published: the invariant that exists in the head of whoever understands the problem, and not in the instruction stream.

I confess there was a moment when I thought the bill would close on the legacy side: finally a tool capable of dealing with the COBOL that haunts banks and governments to this day. After these months, I think that problem will take much longer than it seemed. There’s no giant COBOL open source community. Almost all of that code is private, and truly private: banks, insurers, governments. There’s no training material, or very little of it. The model doesn’t know what to do with code it has never seen, and production COBOL is almost entirely code it has never seen.

To close the reasoning: LLMs can replicate what’s open, but they can’t guess what’s private. And that brings back the hypothesis from the introduction. If these models are good at code, it’s because decades of open source community fought to keep code free, public, and reusable, and providers trained on top of that heritage without paying a cent of licensing for it. The least the AI industry owes the world is to acknowledge that: its capability is, to a large extent, our generosity compiled.

The final irony: the programmer who survives is precisely the one who knows how to build oracles, define tests, hunt down obscure references, and recognize when the agent started spinning. In other words, the skill the machine needs most from you is the one it can least learn from its training. For now, that’s not a threatened job. It’s a different job.

It’s better to deploy faster, because fixing things afterward became fast and cheap.

That sentence is the whole article. Everything else is just me justifying why it’s true now, even though it wasn’t true a few years ago, and showing how I’ve been acting on it myself since February.

The Premise Changed, and Most People Haven’t Noticed

I started saying this on X like this:

Fixing got cheap. Stop holding things back. Just ship it and fix it after. “Ugh, but I risk shipping a bug.” The wrong premise is thinking that staring at every line is going to make any difference at scale. And idle code doesn’t improve on its own. Only used code lets you find out if it needs improving.

And I followed up in the second post, because the first read of anything I write in 280 characters always comes back angrier and more literal than intended:

Get the concept (this is an X post, not a blog post, stop reading it literally). I’m not saying AI doesn’t produce bugs. It does, obviously. But at this point, not more than you, the human, also produce. […] The old excuse was that when a bug ships to production, it sits there longer than it should, because it never becomes a sprint priority to fix. Now THAT’S NOT TRUE ANYMORE, the premise changed: fixing the bug afterward is FAST and CHEAP.

That’s the entire point. It isn’t about AI being infallible. It’s about the cost of fixing a mistake having collapsed, while most teams still run their review process as if that cost were still high.

With today’s frontier models, Opus, Sol, K3, and company, the LLM makes fewer mistakes than most human developers I’ve seen go through a code review. When it does err, the overwhelming majority of the time the mistake isn’t the model’s. It’s whoever was driving the harness. The LLM does what you ask, and it doesn’t do what you didn’t ask.

If you don’t know what to ask for, you won’t produce anything useful, no matter how good the model is. That’s exactly why only a real software engineer can get real production-grade software out of an agent: someone has to know how to break down the problem, validate the result, and recognize when the request itself was poorly framed. AI doesn’t replace that judgment. It multiplies what that judgment already knew how to do.

This Should Have Been the Norm for 25 Years

The discomfort I feel with bureaucratic review processes isn’t new, and the industry should have solved it a long time ago. Kent Beck and Ron Jeffries invented Extreme Programming in 1997, on Chrysler’s C3 project, and Beck published the book in 1999 with continuous integration, TDD, and small-batch delivery as core practices. In February 2001, seventeen developers locked themselves in a resort in Snowbird, Utah, and walked out with the Agile Manifesto. That’s twenty-five years ago.

Martin Fowler’s mantra for continuous integration has always been “if it hurts, do it more often,” and the book Continuous Delivery, by Jez Humble and Dave Farley, formalized the idea that small, frequent deploys reduce risk, not increase it. DORA’s annual State of DevOps reports have documented this with numbers for over a decade:

• elite teams: deploy on demand, multiple times a day;
• low-performing teams: deploy once a month, sometimes once every six months.

And the finding that’s hardest to swallow if you’re afraid of deploying fast: historically, more frequent deploys do not increase the failure rate. They reduce it. Teams that deploy more, on average across the last decade of reports, fail less and recover faster when they do fail.

Even with all of this documented, measured, and published for over a decade, a large chunk of the industry keeps damming up change into big batches, with slow manual review, and calling that “prudence.” It was never a settled matter even inside the agile community itself: in 2014, Kent Beck, Martin Fowler, and DHH (Rails’ creator) spent weeks publicly debating whether TDD was dead, without reaching a clean consensus. In other words, the very community that invented these practices still argues today about how much rigor is enough.

What I’m saying is simpler than that whole debate: whatever your testing rigor level is, an LLM reviewing your diff before merge is fast, and reverting a bad deploy became trivial with containers and git. The math that’s left is: how much time do you spend holding back code, versus how much time it would take to revert if it went wrong? If the second number is smaller, you’re paying a toll for fear, not for prudence.

This Isn’t “Ship Anything”

This is the part that always gets lost: I’m not saying to deploy with zero engineering practice underneath. I’m saying the opposite. This is only safe after you already have basic software engineering working, which I document exhaustively on this blog for years: automated tests, CI running on every push, real revert capability, observability to know when something broke.

80/20. Ask the LLM to do a minimal review and ship it. If it breaks, revert (that’s what containers, git, etc. are for). If you can’t revert, your infra sucks, go fix it yesterday.

That last sentence is the whole filter. If your infrastructure doesn’t let you revert fast, the problem isn’t deploy speed. It’s your infrastructure, and it needs fixing before any conversation about going faster. Nobody gets a pass to skip having real rollback capability. What changes is that, once you have it, holding back code waiting on human review became pure waste.

The Minimum Recommended Environment

I’ve already said “basic software engineering” several times in this piece without spelling out what that means in practice. Here’s the list, no hedging. Nothing on it is optional, and none of it got expensive to maintain:

• unit tests covering isolated logic, TDD or not, as long as they exist;
• scenario and integration tests covering concurrency, partial failure, and conflict, the things unit tests alone don’t catch;
• CI that builds and tests on every change, ideally for every platform you actually ship to;
• hostile automated review before merge, like pr-audit, which treats a contributor’s claim as unverified until proven;
• feature flags to control the blast radius of who sees a change first;
• real staging as a gate before any real user sees the change;
• fast revert capability, with git and containers, and observability to know when something broke;
• in compliance, financial, or health environments, one extra layer: heavier scenario testing, and more than one lock per identified failure mode.

What the diagram tries to make clear is that speed and safety aren’t competing with each other here. They come from the same stack of layers, and each layer in that stack is what gives you permission to stop holding back code waiting for manual review.

Where This Math Doesn’t Work Out

I’m not going to pretend I didn’t get good pushback in the replies on X, because several of them hit on something real that I skipped while writing fast for Twitter. It’s worth answering for real instead of just validating whoever agreed.

1. Reverting code doesn’t undo a consequence that already left your system. If the bug already processed a wrong payment, moved money to the wrong account, or controlled an elevator, reverting the deploy doesn’t undo what already happened. “If you can’t revert, your infra sucks” covers the technical side of deploy and rollback, not the case where the damage already left your system before you could react. Payments, health, and anything touching physical safety fall into this category, and there the review bar before shipping has to get stricter, not looser, because the cost of reverting stopped being technical.

2. Databases break the “reverting is easy” analogy. Reverting a binary is trivial. Reverting a migration that already ran in production, already moved data, and maybe already dropped a column, is not. There’s a pattern for this, expand-contract: add the new shape without removing the old one, wait for every consumer to migrate, only then remove the old one. This isn’t about trusting AI more or less. It’s about state that doesn’t undo with git revert.

3. Libraries and public packages don’t run on the same math as a closed application. When my own code breaks, I fix it on the spot. When a lib that hundreds of projects depend on breaks, the people who eat the cost are people who don’t even know I exist, with a much slower reaction time than mine. That’s why my pr-audit skill, which I describe further below, treats semver classification as a version gate: a breaking change ships in a major, never in a patch, because there the real review isn’t about catching a bug, it’s about not blindsiding whoever depends on you without warning.

4. App stores knock the speed down too, and my own frank_yomik example hides that instead of solving it. It publishes the APK straight as a GitHub Release, without going through the Play Store. If you distribute through Google’s or Apple’s official store, their review sits in the middle of your process, and you don’t control how long it takes. This article is about removing the bottleneck that’s inside your own control. The app store isn’t inside it.

5. The most honest point of all: every piece of evidence I bring here comes from projects where I’m the sole owner and I decide alone. I have no way to prove, with my own numbers, that the same pace works inside a large company, with Legal, Compliance, and a product manager in the middle. What I’d argue generalizes is the principle, not the exact pace: build real revert capability, automate the mechanical checking, and measure the result against real usage, not against your own sense of security. How much speed you apply on top of that depends on how much a mistake costs in your specific context, and nobody from outside answers that for you.

On the first point specifically, it’s worth opening up more, because “stricter bar” can’t stay vague. It means real scenario testing, not just unit tests. ai-memory itself has more than a thousand integration tests spread across its modules, and most of them don’t ask “does this function return the right value?” They ask things like:

• what happens when two writes race for the same file at the same time;
• what happens when the disk fails mid-way through a delete operation;
• what happens when two sessions try to consolidate the same data at the same time;
• what happens when the process dies before it finishes writing.

An environment with compliance, financial, or health requirements needs this level of testing: every identified failure mode has to have a mitigation, and more than one layer of checking, never a single lock.

And here’s the point that holds up this article’s entire argument even in those environments: this bar doesn’t change depending on who wrote the code. A human developer’s mistake takes down a payment system just as well as an LLM’s mistake does, and nobody gets a free pass for loose review just because a senior human wrote the line. The bar of scenario testing and double-layer protection is about the problem domain, not about who, or what, produced the code.

Feature flags and staging environments are the other half of that double layer, and they also changed cost. Flags used to be a pain to maintain: dozens of them, each one needing to be born, monitored, and removed once it stops making sense, and it’s easy to pile up dead flags nobody remembers the purpose of. That got cheap too: I ask the agent to audit every flag in the project, find which ones have been stuck at the same value for months, and generate the cleanup PR. Maintenance that used to take a whole afternoon turns into a task that takes minutes.

With that solved, the layered defense becomes simple to assemble: staging to catch obvious errors before any user sees them, feature flags to limit the blast radius of who sees the change first, and scenario tests to cover what staging alone doesn’t simulate. Three layers, none of them expensive to maintain.

How I’ve Been Doing This Since February

I started accelerating my output with AI in February this year, and the question I wanted to answer was simple: can you use the new generation of frontier LLMs to code full-time, at real production pace, not just for a demo? The answer turned into my biggest LLM benchmark, more than 110 articles published since then, and this, as I wrote recently:

Since February I’ve produced more than 110 articles on my blog. Most of it came from the 30+ open source projects I built and maintain (43 if you count other people’s contributing projects, like Flea or Omarchy). 1.6 million lines of code produced. By myself (with agents, of course).

The projects I use the most in my own day to day, and that survived the “am I still maintaining this months later” test, are three:

• ai-jail: an OS sandbox for my coding agents;
• ai-memory: long-term memory for the agents;
• ai-usagebar: a bar widget to monitor LLM usage.

Two of them already have a decent landing page:

ai-jail went live today. ai-memory is up at aimemorybr.netlify.app because the aimemory.io domain transfer will still take a few more days.

Both run on Netlify. That’s not a tool recommendation, it’s just where I already had an account set up. The point that matters here isn’t the platform, it’s the flow: to put a new version live, the command is git push. There’s no manual step between the commit and the updated site.

My newsletter, The M.Akita Chronicles, already has more than 15,000 subscribers:

And the newest effort is converting blog posts into podcast episodes. The whole thing runs inside Discord, with a bot that generates the preview and waits for my command:

One /publish-podcast 2 later, the episode is published and the transcript already has a URL:

From finished text to published audio, the entire process is me typing a slash command in a Discord conversation. That’s the effort bar I use to measure whether a process is good enough: if publishing takes more than one command, some step is still too manual.

Does the Quality of the Generated Code Hold Up?

Every single day someone shows up worried that automating too much with LLMs is going to dump bad code, or worse, dangerous code, into production without anyone noticing. Given that the engineering underneath is the one I described in the previous section, automated tests, CI, feature flags, staging, fast rollback, my actual experience doesn’t back up that fear.

I’ve produced more than 1.6 million lines of code with LLMs since February. I have never once seen an LLM do something randomly destructive that I didn’t ask for. Quite the opposite: in my LLM Benchmark v4, I deliberately built an adversarial scenario, a subagent planting fourteen real CVE-based sabotage items, hidden inside a project that grows across seven sprints. Most of the tested models found and fixed the sabotage on their own, without me having to point it out.

That’s anecdotal. It doesn’t prove anything beyond what that specific benchmark measures. But after all that volume, the impression that sticks is consistent: today’s LLMs produce stable code most of the time, comparable to what a decent human developer would produce, just a lot faster, and on average, more accurate.

Cross-Platform CI/CD With Zero Effort

I went and counted how many of my ai-* and frank-* projects (18 repositories total, among the ones with a README) already have automated build and deploy. Fourteen have CI configured, and most of them build and publish to multiple operating system platforms from a push made on a Linux machine.

The most complete example is ai-memory itself: a single tag push triggers a workflow that, in the same run:

• builds Linux x86_64 and arm64;
• builds macOS arm64 and x86_64;
• builds Windows x86_64, packaged with PowerShell;
• publishes a multi-architecture Docker image;
• pushes the package to Arch Linux’s AUR.

ai-usagebar follows the same build pattern for Linux and Windows, and also publishes two AUR packages, to crates.io, and to a Scoop bucket for Windows.

Then there’s frank_yomik, my manga reader with OCR and translation via a local LLM, which is the example I wanted to show that Android fits into the same pipeline. A v* tag push triggers five parallel builds:

• Linux;
• macOS;
• Windows;
• browser extension;
• Android, on an ordinary Linux runner, decoding a signing keystore stored in GitHub secrets and producing the already-signed APK.

At the end, one job waits for all five to finish and publishes a single GitHub Release with every binary together. It builds for four different systems, including Android, without ever leaving Linux.

Not everything came out perfect on first look, and I’d rather admit that than pretend it’s all clean: frank_sherlock and frank_karaoke publish the release as a draft by default, which blocks the automation that follows (like the AUR push) until someone clicks “publish” by hand. I found this hole while writing this piece.

A neighboring project, frank_scanlation, had already solved a similar but different problem: GitHub suppresses the event that triggers automation when the release itself was created by the bot’s own token, so its workflow triggers the next step by hand, via workflow_dispatch, from inside the job itself. Neither fix is hard. I’ll apply both to the projects that still need it this week, because that’s exactly the kind of accumulated friction this whole article is telling you not to leave sitting.

And speaking of hidden friction: ai-memory has Windows-specific code (NTFS file index, a native PowerShell hook) that only really runs on GitHub’s windows-latest runner, about 1000 seconds against 250 on Linux for the same suite. I went as far as setting up a local Windows VM via dockur/windows, thinking I’d gain speed.

I didn’t: the workload is dominated by disk I/O, and the result came out slower than GitHub’s runner for the full suite. It only turned out useful for fast iteration on a single isolated crate. A good tool is one you measure, not one you assume, even when the measurement contradicts your initial bet.

The Automated Review That Holds All of This Up

I already detailed my review skills in a recent article, so I’ll summarize here:

• pr-audit audits every PR before merge with the principle of evidence over narrative: nothing the contributor wrote is true until proven, with a hostile checklist for malicious code, exposed credentials, semver breaks, and prompt injection attempts;
• iss-audit applies the same distrust to issues, never executing a command pasted by whoever reported it;
• github-resolution turns what got approved into merged code, one ticket at a time, with a regression test written before the fix, and the rule that “slop is a defect”: no speculative abstraction, no forgotten TODO, no weakening a test just to pass faster.

That catches the most expensive category of garbage inside the same diff. It doesn’t catch duplication spread across different PRs, because the audit looks at one isolated diff, not the whole project’s architecture. That part still depends on me noticing the pattern repeating and telling it to consolidate.

By my estimate, this handles about 99% of review and resolution work on its own, with me supervising and deciding on the cases that are left. But that’s not the main point. Automated review is necessary, but it alone proves nothing about whether the software is good.

What Actually Keeps This Grounded: Real People Using It

Here’s the part I think is the most important in the whole article, and the one that usually gets skipped over: your own review isn’t worth as much as you think. What’s worth something is published software being used by real people, giving real feedback.

I went and checked ai-memory’s real numbers on GitHub, without rounding up to sound impressive:

• issues: 287 total, 249 came from people who aren’t me, 86.8% of the total;
• pull requests: 562 total, 371 from outside, 10 of them Dependabot’s automatic bumps and the other 361 from real people, roughly 121 distinct contributors.

The two people who contributed the most alongside me sent 64 and 43 PRs each, on their own.

That’s what keeps the project grounded. It isn’t the LLM being a good reviewer, nor me being a good reviewer. It’s hundreds of real people actually using it, finding cases I never imagined, and sending real issues and PRs. Without that, my automated review skills would just be confirming my own assumptions back to me, no matter how rigorous the rules look on paper.

It doesn’t matter how good the AI you use is: if nobody uses what you published, you have no real feedback, and without real feedback there’s no real improvement. There’s only your own opinion about your own work, validated by nobody but yourself.

The Conclusion That Matters

Real software engineering needs fast, near-zero-effort deploy. It isn’t optional, it isn’t a luxury for a big team, it’s a prerequisite. And if reviewing and shipping to production is taking longer than it took to develop the feature, you’re not just wrong.

You are the bottleneck.

Excuses and justifications aren’t worth anything at that point. Use that time to fix yourself and your process, because only real-world results matter, not the sense of security from having reviewed a diff line by line that nobody else is ever going to look at again.

If the premise changed, I change today. Not tomorrow, not next week. And if yours hasn’t changed yet, the problem isn’t AI. It’s you holding back your own work.

And there are more layers of uncertainty. The APKs came from APKPure, not the official Play Store, so there’s a chance (small, but real) they were tampered with along the way. And even after decompiling, there’s a pile of binary libraries and SDKs we didn’t crack open, assuming they’re common libs like Google Firebase or Google Play Services. This mini-project was, at bottom, my own curiosity: I wanted to know what’s inside these apps we install without thinking.

Decompiling Android is way too easy

Something a lot of people don’t know: an Android app is way too easy to decompile. The APK is basically a zip. You unpack it, run a tool like jadx on the .dex files, and readable Java comes out the other end. If the app is Flutter, you can reconstruct the Dart with Blutter. If it’s a Cocos2d-x game, the Lua scripts are often sitting right there in plaintext, no decryption needed.

This does not let you recompile the app from the source (class and method names come scrambled by obfuscation, resources are missing, the signing keys are missing). But for static analysis it’s more than enough: you can read the endpoints the app contacts, the permissions it asks for, the identifiers it assembles, the SDKs it bundles. That’s exactly the kind of reading I asked Fable 5 to do.

One honest limit: the most sensitive part usually lives in native code (.so), compiled C/C++, which doesn’t decompile into anything readable. In TikTok’s case, that’s precisely where the guts are, and I’ll come back to it in a bit.

“If it’s free, you’re the product”

You’ve heard that line a thousand times. It’s true, but it almost always stays abstract. Let’s make it concrete: how exactly do you become the product? The answer has a technical name, and it’s why I picked this angle for the title: fingerprinting.

A fingerprint (device fingerprint) is a persistent identifier the platform assembles by combining dozens of signals from your device: phone model, OS version, screen resolution, CPU, RAM, carrier, timezone, language, battery, installed fonts, nearby Wi-Fi and Bluetooth networks, motion sensors, and so on. None of those data points identifies you on its own. But the combination of them is so specific it becomes practically unique, an implicit “serial number” for your device that you never explicitly authorized.

The clever part, and the reason it’s so valuable, is the persistence. A cookie you can clear. The advertising ID you can reset. But a well-built fingerprint survives you clearing the app’s data, survives a reinstall, sometimes survives a factory reset. It re-identifies you as the same person after any attempt to start from scratch. That’s why companies want it so badly: with the fingerprint, they cross-reference your behavior across apps, across sessions, across reinstalls, and build a profile that follows you even when you think you wiped your tracks. That profile is the product from the line above. And the product is you.

The three apps I analyzed sit at three very different points on that spectrum, and that’s where it gets interesting.

The three apps

TikTok: industrial-grade collection, armored against auditing

The tiktok_analysis was the densest one. Version 46.0.1, 37 dex files, 204 native libraries. The honest conclusion, and the one I think is fairest: it’s not disguised malware, but it is industrial-grade data collection engineered to be hard to audit.

On the reassuring side: no secret servers. Every host the app contacts belongs to TikTok/ByteDance or to a named ad/analytics vendor (Adjust, AppsFlyer, Facebook, Google, ThreatMetrix, Amazon). We found no IMEI or MAC collection, no QUERY_ALL_PACKAGES, no reading of SMS or call logs. US and EU user data goes to dedicated enclaves (the famous Project Texas, with Oracle in Ashburn).

On the worrying side: it collects a lot. A persistent device fingerprint, precise location, Wi-Fi and Bluetooth scanning, contacts upload (if you enable “find friends”), motion sensors, and a check against a fixed list of 91 named apps to see which ones you have installed (competitors, PayPal, Venmo, Spotify, sister ByteDance apps). Your advertising ID fans out to about 6 third parties.

And the central finding: the fingerprint and much of the telemetry are assembled in native code (libmetasec_ov.so and a stack of crypto libs), with each request’s signature going out encrypted. Meaning, even decompiling the whole app, you can’t tell exactly which bytes leave the device.

To attack that black box specifically, one of the analyses ran a strings sweep over the 204 native libraries (the repo’s report 13), and there libmetasec_ov.so gives the game away in plaintext. You can read the fields it collects (secdeviceid, device_id, battery level and state, screen size), the function that signs the requests, the ByteDance telemetry endpoints baked into the binary, and even the scheduler that fires the upload on a timer. What the Java layer only let you infer, the native code confirms. It’s a fingerprinting machine, plain and simple.

And the same sweep went looking for exactly what everyone’s afraid of finding (keylogger, screen recorder, input injection) and found none of it. The scary keyword hits were false friends: keylog was the TLS debug log, screenshot was the video player and C2PA content authenticity, getevent was a media method name. There’s anti-debug and frida/xposed detection, yes, but that’s the app inspecting itself from the inside, standard security-library behavior, not snooping on the rest of the phone.

So the opacity is still the point, just sharper: we know what libmetasec_ov does and where it reports to, and what stays locked are the exact payload bytes, which only native disassembly or live capture would reveal. “Not proven to be spyware” and “collects far more than it needs, opaquely” are both true at once, now with plaintext confirmation that the fingerprinting engine is real.

Clash of Kings: the game is honest, the monetization is predatory

The clash_of_kings_analysis was a pleasant surprise on the privacy front. It’s a strategy game by China’s Elex, a Cocos2d-x engine with almost 12,000 Lua scripts in plaintext. And check it out: no GPS, no camera, no microphone, no contacts, no SMS, no IMEI. The two most sensitive reads it can do (installed-app list and Wi-Fi MAC) are both consent-gated. No malware, no hidden exfiltration.

The privacy concern is the same villain as TikTok in a simpler version: a reset-resistant device UUID, written in four different places (system settings, preferences, SD card, and cloud backup) precisely to survive a reinstall. Plus telemetry to China-region infra and to Tencent Bugly.

Where Clash of Kings really weighs on you is the monetization. It’s the complete predatory free-to-play kit: loot boxes (the hero “luck draws”), countdown-timer offers to manufacture FOMO, VIP pay-to-win tiers, cumulative-spend ladders with leaderboards for who spends the most (the whale hunt), a monthly card, the whole package. The half-full side: drop-rate odds are disclosed in-game, and the Chinese anti-addiction and real-name system for minors is present. Decent privacy, wallet in danger.

Gov.br: the example of how to do it right (with one caveat)

The govbr_analysis is the counterpoint, and it actually left me relieved. It’s the official digital-identity app of the Brazilian government, built in Flutter (I reconstructed the Dart with Blutter). By the standards of a mandatory government app, it’s well-built and privacy-respectful: traffic is almost all to *.gov.br, SERPRO and Dataprev, TLS is certificate-pinned, credentials sit in Android’s encrypted KeyStore, the ICP-Brasil signing uses FIPS-grade crypto, and the app hardens hard against rooted or tampered devices. And most importantly: no commercial ad trackers. No Segment, Adjust, AppsFlyer, or Meta.

The two honest caveats: it bundles Google Firebase (Analytics, Crashlytics, Messaging, Remote Config) with collection on by default and the ad-id permission declared, which is the lazy default of nearly every app today; and your proof-of-life selfie gets uploaded to the government backend for identity matching, which is inherent to the purpose and doesn’t go to third parties. The one item only live traffic capture would settle is whether your CPF gets attached to the Crashlytics reports. But overall, if every app were built with this kind of care, this post wouldn’t exist.

What almost every app carries along

Cross-referencing the three (and this is where the pattern shows up), you can see what tends to come bundled in the app you download, even in a government app:

• Google Firebase / Play Services: present in all three. Analytics, Crashlytics (error reporting), Cloud Messaging (push), Remote Config. It’s so ubiquitous it became invisible infrastructure. And it ships with collection on by default.
• Ad attribution stack: Adjust, AppsFlyer, Branch.io, Singular, Facebook SDK. The job is to link “you clicked this ad” to “you installed and used the app.” TikTok had a redundant pileup: five of these at once, each collecting your ad-id independently.
• Ad networks: Google’s AdMob/DoubleClick, and in the games’ case the ad-mediation ecosystem.
• Region-specific analytics/crash: Tencent Bugly in Clash of Kings is the example, telemetry going to Chinese infra.
• A specialized fingerprint/anti-fraud service: the most striking case was LexisNexis’s ThreatMetrix in TikTok: a company whose business literally is building device fingerprints for fraud detection. Legit for anti-fraud use, but it’s literally a fingerprint-as-a-service machine, embedded.

The takeaway is that “product” isn’t assembled by a single company. It’s five, six, seven partners, each grabbing a piece, and the fingerprint stitching it all back into one profile. Nobody’s spying on you alone. It’s a group effort.

What you can do

You won’t zero this out short of leaving the internet, but you can cut the exposed surface down a lot:

• Don’t consent to what isn’t needed. GPS is the big one. A video app doesn’t need your precise location to work. Deny it, or grant only “while using the app.” Same goes for contacts, microphone, and camera: only allow them when the feature you want to use right now requires it.
• Reset your advertising ID now and then (and on Android, you can choose to “delete advertising ID” outright). It doesn’t kill the fingerprint, but it breaks part of the trail.
• Be suspicious of “find friends.” That’s the contacts-upload trigger. You hand over the entire social network of people who trusted you with their number.
• Prefer the web version when you can. A browser with a tracker blocker exposes less fingerprint than the native app, which has access to sensors and signals the browser doesn’t give up.
• Accept the trade-off consciously. The point isn’t to go paranoid and uninstall everything. It’s to know the price. If you find TikTok fun enough to pay for it with your fingerprint, great, that’s an informed decision. The problem is paying without knowing you’re paying.

The point of the post

None of this proves or disproves that a specific app is 100% safe. Static analysis shows what the app is capable of doing, what’s built inside it; only live traffic capture, with a rooted device and a MITM proxy, would show what actually leaves over the network on each request. I didn’t go that far. This is a glimpse, not a verdict.

And that’s exactly the point. Today, with a good LLM and a free afternoon, any curious developer can crack open their own phone and see what’s running inside it, without depending on some tech influencer’s opinion or an alarmist headline. The three full analyses are on GitHub (TikTok, Clash of Kings, gov.br) with step-by-step instructions to reproduce. Grab an app you use every day and do the same. It’s the closest thing to reading the label before you eat that we have in the software world.

In the last few months, we started seeing stories about large companies burning absurd amounts of tokens on AI tools without being able to point, clearly, to the matching product value. The most famous case was Uber burning through its annual AI budget in a few months with heavy use of tools like Claude Code, while executives admitted they still could not draw a straight line between token consumption and value delivered. It is not that nobody used the tool. They used it too much. That is exactly the problem.

By now, I think most people understand that a one-line prompt solves nothing. The fantasy of typing “build me an Uber clone” and coming back to a finished product died quickly. The market response was to run in another direction: spec-driven development.

Instead of asking for everything in one sentence, you write a specification. Scope, use cases, acceptance criteria, desired architecture, constraints, flows, maybe even a full PRD. Then you hand that to the agent and expect it to implement with less ambiguity.

Is that better than one-shot prompting? Of course. I still think it is wrong.

The problem is not lack of specification

I have been hammering this idea for years, long before LLMs became fashionable. Two of my videos matter for this point: Don’t Outsource Your Decisions and Forget Agile Methodologies.

The summary is simple: most professionals were trained not to decide.

Think about your own routine. Many people do nothing if there is no backlog ready. They cross their arms, blame management, blame colleagues, blame customers, and wait for someone else to say what should be done. When the task arrives, they execute. When it goes wrong, the requirement was bad. When the customer complains, the priority was wrong. When the product degrades, the process was to blame.

Almost nobody really decides, commits to the decision until the end, and suffers the consequences. Without suffering the consequence, you never learn how to decide.

You do not learn which choices lead to which results. You do not learn how to prioritize. You do not learn how to explain why something needs to exist. You do not learn how to write a specification because you understand the problem; you learn how to fill a template so nobody complains.

Agile became structured bureaucracy

The original 2001 principles were crushed. What many companies call Agile today is bureaucracy in shorter cycles.

Everybody knows the classic drawing: backlog, sprint planning, sprint, daily, review, retrospective, next sprint. The Scrum Guide describes the formal events. Diagrams like the Scrum process on Wikimedia show the pretty circle. In practice, many teams turned it into theater.

An entire day is spent debating what goes into the backlog. The final backlog is rarely what actually needs to be done. It is the consensus of least resistance. Compromise after compromise. Small tasks, small risks, small wins, small failures. Everything designed to avoid too much conflict.

That is not merely mediocre. It is worse: it is the active choice for easy decisions, the kind that will not bring too much criticism your way.

Then the team spends days or weeks implementing that lukewarm backlog. QA comes in, finds problems, tries to fit corrections into the next round. A few more days, maybe two more weeks. A month later, maybe two, the original decision finally starts to show results in production.

But the result is so far away from the decision that almost nobody makes the connection. The team does not feel the real weight of what it decided. The error does not become learning. The process does not improve. The most important part of any continuous improvement system, kaizen, never happens.

Something else happens instead: everybody learns to stay invisible.

You learn to please management, not the product. You learn to reduce friction, not solve the problem. You learn to survive the meeting, not deliver value. And when results get worse, upper management reacts in the usual way: more control, more rules, more gates, more committees, more checklists. The machine gets heavier. The team gets more cowardly.

Then AI arrives

Now the LLM enters the room.

Until recently, the biggest bottleneck in software development was writing and adjusting code. Coding took days or weeks. That is why so much energy went into planning. If the team is going to spend two weeks implementing, spending a day arguing about what goes into the sprint feels reasonable.

But coding tools with LLMs change where the bottleneck sits. They can do a large part of the manual work: generate structure, write basic tests, refactor, adapt APIs, create screens, fill boilerplate, try fixes. It is not perfect, but it is fast enough to change the economics of the process.

It is the old chain and weakest-link problem. You strengthen one link, another one starts limiting the system. Before, the weak link was coding speed. Now, increasingly, it is the human ability to decide, explain, test, and review.

The mistake companies make is putting LLMs inside the same broken process and expecting multiplicative gains.

You take people trained for years not to decide, give them a tool that requires clear decisions, and then act surprised when garbage comes out faster.

The LLM reflects the user

A coding agent needs you to explain the goal, the context, the constraints, the priorities, what must not break, which paths are acceptable, and how success should be measured. It needs direction.

Now go interview your team. Ask each person:

• what is the real goal of what you are doing?
• how does this improve the product?
• which metric should move?
• what trade-off are you accepting?
• what must not break?
• which unhappy paths need to be tested?

Many people cannot answer. They know which ticket they picked up. They know they need to close a few tasks per week. They know there is a daily tomorrow. They do not know why the work matters.

Then you give that person an LLM.

The result is not magic. It is a mirror. Bad code ten times faster. Bugs ten times faster. Angry customers ten times faster. Giant pull requests ten times faster. The problem is not only AI. AI amplifies the lack of direction.

Spec-driven development tries to fix this with documentation. But if the specification is born from the same cowardly process, it only documents the cowardice. A PRD written by someone who does not understand the decision is still bad. Acceptance criteria written to avoid conflict still dodge the real problem. The agent will execute a bad decision better.

The process has to change

My hypothesis is that the whole process has to be redesigned around the new constraint. Again: speculation. I do not have enough data to prove it. But looking at the kind of failure I see in companies, and the kind of success I see in intense individual use, my theory is this:

throw away the sprint ritual as the central unit of work.

It makes no sense to keep days of planning, days of review, and weeks of coding as if the bottleneck were still the same. With LLMs, the cycle needs to be shorter. Much shorter. Ideally, inside the same day.

In my ideal setup, each programmer with an LLM should pair directly with a Product Manager for short blocks of work. The PM’s job is to close the gap the programmer usually has: goal, priority, constraint, business context, expected result. That needs to enter the prompt and the conversation with the agent.

But because coding is no longer the main bottleneck, the PM does not need to wait two weeks to see a result. They can sit with the programmer for a morning, implement a small functional slice, adjust on the spot, decide on the spot, cut scope on the spot. Then the PM rotates to another programmer.

Then QA comes in. Not as a distant phase, but as pairing inside the same cycle. QA closes another common gap: testing only the happy path, forgetting error cases, not automating enough, not thinking about behavioral regression. QA’s job becomes helping turn the implementation into something verifiable now, while the context is still hot.

Then the Tech Lead comes in. Also on the same day, not two weeks later. They look at what changed and close the third gap: unnecessary complexity, poor design, hidden regression, missing documentation, obvious vulnerability, performance getting worse, coupling that did not need to exist.

Build, check, adjust. In real time.

The new role of each person

This model demands more responsibility, not less.

The programmer stops being the person who only picks up tickets. They need to learn how to turn a goal into executable instruction. They need to discuss trade-offs. They need to understand why they are coding. They need to use the LLM as a tool for execution and investigation, not as an excuse to turn the brain off.

The PM stops being a backlog factory. They need to decide with the programmer, looking at the live product. This is not writing a card and disappearing. It is watching the thing come to life, adjusting it, cutting it, prioritizing it, and accepting the consequence.

QA stops being the person who receives the package at the end and returns defects. QA enters early to turn expected behavior into verification. If the LLM accelerates code, QA needs to accelerate learning about risk.

The Tech Lead stops being a late PR stamp. They enter while change is still cheap. They help keep things simple, remove complexity, protect architecture, documentation, security, and performance.

None of this works if everybody keeps using AI as expensive autocomplete inside the same bureaucratic Scrum.

Why companies will not like this

Because this process exposes decisions.

In the current process, the decision dissolves. It was the backlog. It was the PM. It was refinement. It was the sprint. It was the team. It was a dependency. It was the customer changing their mind. Everybody can hide.

In a short LLM cycle, the decision appears quickly. You said this was the priority? Then implement it now. You said this was the acceptance criterion? Then test it now. You said this design was good? Then review it now. If it is wrong, the consequence shows up the same day.

That is uncomfortable. But that is how you learn.

Without nearby consequence, decision becomes theater. With nearby consequence, decision becomes training.

How to implement this without burning money

The first step is accepting something professional programmers usually hate: you can throw code away.

In fact, you should throw code away much more often.

Code is not a monument. Code is working material. Deleting code is as important as writing new code. That is why we use Git. The idea was always to work in branches, try paths, change our minds, compare alternatives, delete what did not work, and keep only what survived contact with reality.

But in the traditional process, code becomes too emotionally expensive. If it took two weeks to produce, nobody wants to admit it was waste. The manager feels that deleting it proves the previous decision was bad. The programmer defends the implementation because they suffered to make it. The team pushes the problem forward because deletion feels like defeat. It is not. Sometimes it is hygiene.

With LLMs, that cost changes. Making mistakes becomes cheap. Prototyping becomes cheap. Building a proof of concept becomes cheap. Trying two paths and throwing one away becomes cheap. Asking the agent to rewrite an entire approach in a temporary branch, compare it, document what was learned, and git reset without drama becomes cheap.

So do more of that. More prototypes. More proofs of concept. More internal A/B tests. More disposable branches. Less attachment to the first code that appeared. LLMs are good at this kind of work: exploring alternatives quickly, showing where the specification was vague, and producing material you can destroy without guilt.

But do not expect everyone to receive LLM tokens on Monday and know how to do this alone. That is the fastest way to burn the budget and then say AI does not work.

My suggestion is the opposite: assume one or two weeks with no planned delivery. Say clearly that the goal is not to ship a feature. It is to train the new process. Break the old ritual. Let the team practice in a safe environment. If it goes wrong, fine: reset the branch, delete the prototype, document what you learned, try again.

In that phase, the PM should not write a twenty-page spec. They should pair with the programmer and build small prompts, one step at a time, watching the result in real time. With each LLM response, the PM learns how much detail the code actually needs. The programmer learns to ask better questions. QA learns to turn behavior into tests early. The Tech Lead learns to cut complexity while it is still cheap.

This is how LLM adoption has some chance of working: as a process change, not as an expensive autocomplete license.

The uncomfortable conclusion

LLMs fail so often because they are inserted into processes made by people who were trained not to decide.

It is not just lack of a better prompt. It is not just lack of a larger spec. It is not just lack of token governance. Those things help, but they do not touch the center of the problem.

The center is that many people do not know how to explain a measurable goal, prioritize, accept a trade-off, or connect decision with result. An LLM needs exactly that to be useful.

If you put AI into the old process, you accelerate the old process. More mediocre backlog. More bad decisions. More bugs. More cost. More rework. More reports trying to prove productivity the customer never felt.

If you want LLMs to work, redesign the process around them. Smaller cycles. Real pairing. Decision close to execution. Testing close to decision. Review close to code. Result close to whoever decided.

This is the part programmers need to face: yes, you probably never learned how to decide properly. You never learned how to describe a goal objectively. You never learned how to measure consequence. The process trained you that way. Now AI is revealing the failure.

The good news is that you can train the opposite.

But it will not happen with another ceremony, another prompt spreadsheet, another AI usage policy, or another sprint full of tickets written so nobody gets criticized.

It will happen when you stop outsourcing decisions.

My short take: AI in open source contributions is here to stay. It will increase, not decrease. Pandora’s box is already open. There is no realistic plan where everyone goes back to programming 100% manually and pretends Claude Code, Codex, OpenCode, Cursor, Gemini CLI, and the rest never existed.

That does not mean every project is obligated to accept AI-assisted contributions. Far from it. The maintainer has the absolute right to run the project however they want. They can accept AI, ban AI, freeze the project, delete the repository and go grow tomatoes. Open source is not mandatory democracy. Whoever maintains decides.

But I also think banning AI is mostly delaying a fight you cannot win. The volume will keep going up. Average quality will wobble. There will be trash, there will be good work, there will be almost-right PRs, there will be hallucinated bug reports. The practical question is not “how do we stop AI?” The practical question is: how do we adapt the process to survive this?

The recent cases

The loudest case right now was rsync 3.4.3. The version shipped on May 20 as a security update, fixing six CVEs. Soon after, users reported regressions, especially in backup flows using daemon mode and incremental transfer options. Rsync is not a toy. It is a backup, deploy, mirror, sysadmin tool. Breaking compatibility there hurts.

Andrew Tridgell, creator of Samba and co-creator of rsync, answered in the post rsync and outrage. He confirmed that he used Claude, with cross-checks from Codex and Gemini, to help rewrite part of the test suite in Python. But he also said the obvious thing a lot of people do not want to hear: he did not type “convert all of this” and go to sleep. He designed the suite, reviewed the work, burned a lot of CI time, and acknowledged that regressions happened in valid but uncommon cases that were covered neither by the existing tests nor by his manual testing.

That is the interesting point: the regression became a discussion about AI, but regression was not born with AI. Regression was born with software.

On the other side we have Zig. The Zig Code of Conduct today has an explicit policy: Strict No LLM / No AI Policy. No LLM-generated content, not code, not prose, not grammar edits, not translation, not brainstorming. Andrew Kelley explained in an interview, summarized by DevClass, that the team considers AI contributions invariably bad, review-time sinks, unteachable, non-deterministic. The line that sums up his position is a good one: the bar he wants is not “works surprisingly well,” it is “uncompromising perfection.”

I disagree with the generalization, but I respect the decision. It is his project. If he wants to keep that space free of LLMs, that is his right. Period.

In the same neighborhood came the Bun discussion. The official post about Bun joining Anthropic says Bun was acquired because Claude Code and other Anthropic coding products depend on it. “If Bun breaks, Claude Code breaks,” in the post’s own words. Meanwhile, the Cosmic piece about the Rust rewrite points to reasons like contributor accessibility, maintainability at scale, and long-term stability. I saw people linking this to Zig’s anti-AI policy, but the source I could verify does not state that causality. What we can safely say is: Bun, a project born in Zig, is moving important parts to Rust, and one stated reason is Rust’s larger contributor base.

Then there is curl. According to Hackaday, Daniel Stenberg suspended curl’s bug bounty program starting February 1, 2026 because the project was flooded with LLM-generated vulnerability reports. Long reports, intimidating, with the shape of a serious exploit, but when an experienced maintainer reads them, they are not vulnerabilities. The PR removing the bounty is on GitHub as curl/curl#20312.

That case is different from a code PR. It is worse. A bad patch you test, compile, read. A fake security report forces someone experienced to spend time proving that the bug does not exist. That is the kind of thankless work that burns maintainers out.

So yes: the problem is real. AI slop exists. It is filling issue trackers, PRs, bug bounties, comments, everything.

Now the part nobody likes: it will not stop.

There is no “back to normal”

I already talked about this in VS Code Is the New Punch Card. The interface changed. First you had to type binary instructions. Then you had to organize punch cards. Then you had to convince the compiler. Then you had to operate editor, terminal, framework, CI, container, cloud, and YAML. Now you have to orchestrate agents.

This does not eliminate engineering. It only changes where the manual work ends.

Some people are treating AI like a passing fad, like the “metaverse” or NFTs. It is not. AI for programming is already past the conference-demo stage. It is inside real workflows. Rsync used it. Bun has Claude Code bots opening PRs. My projects use it. Your contributors will use it, even if you forbid it. If you think you can detect with 100% certainty whether a patch was written with or without AI, good luck with that one.

The maintainer can still set a rule. They can say “if I find out you used AI, I block you.” That may make sense in small communities with their own culture, like Zig. But in open source at large, especially big projects with users all over the world, this turns into compliance theater. The person uses AI, rewrites the text, does not disclose it, and that’s that. What are you going to do? AI detector for code? For crying out loud.

The mistake is thinking you have to review everything alone

The fairest maintainer complaint is simple: there already are not enough people to review human contributions. AI increases volume. More PRs, more issues, more bug reports. The maintainer was already tired, now they are underwater.

I get it. I just do not think the solution is to process the new volume with the old process.

You fight AI with AI.

The first triage has to be done with an LLM. Not to approve automatically. To filter. To summarize. To compare the diff against the stated goal. To point out bad smells. To say “this touches schema, it needs a migration.” To say “this changes public behavior, it needs compatibility tests.” To say “this PR description does not match the code.” That layer alone removes 80% of the grunt work.

The mistake is letting the agent decide alone. The right move is making the agent do the boring investigation, then you decide.

That is how I do it.

My real PR flow

Anyone who followed my AI marathon knows I did not get here after watching two YouTube videos. I spent weeks using agents every day, on real projects, breaking real things, fixing real things. What I am describing here is the workflow that survived that, not LinkedIn theory.

Shameless propaganda: I am happy that my three main AI-related projects are getting contributions almost every day:

• ai-jail, which I already covered in ai-jail: Sandbox for AI Agents.
• ai-memory, which I explained in I Built a Memory System for Coding Agents: ai-memory.
• ai-usagebar, from I Built a Waybar Widget for Omarchy to Monitor LLM Plan Usage: ai-usagebar.

ai-memory in particular became a live experiment. As I write this, GitHub shows 55 closed PRs in the project; through the gh API, 46 were merged and 9 were closed without merge. Thanks to everyone who contributed. Seriously. Djalma, mrpaiva, pedrofjr, CaTeIM, azevedo-luis, brunoomariano, rikelmyso7, abnersajr, and several others. That is a lot of work from a lot of people.

It is also obvious, looking at the style, that most of those PRs were made with AI assistance. And that is not a problem. It would be hypocritical to complain: the entirety of ai-memory was written by AI. I did not open a code editor once to write that thing by hand.

But almost no PR arrives immediately mergeable. That is the point a lot of people still do not understand. “Made with AI” does not mean “blind merge.” It means “the audit begins.”

My prompt for reviewing a PR is more or less this:

check the latest open PR as usual: do not trust the author, audit the whole code thoroughly, make sure it achieves its goal.

Check whether we need to adjust or fix anything before merging: do not accept regressions, do not accept degraded code quality, check whether test coverage is correct.

If it touches build or OS support, check GitHub Actions, CI, clean build, correct wiring, and documentation.

Summarize what the PR does, what you found, and ask me for approval before proceeding.

Notice two things. First: “do not trust the author.” That is not rudeness. It is engineering. The PR description is a hypothesis. The diff is the evidence. Second: “ask me for approval.” The agent investigates, I decide.

When I receive several PRs on the same day, I do this for all of them. After they are all in, I send another prompt:

check today’s commits.

We touched a lot of code, so I need a complete audit of the touched code to make sure we did not add regressions, unnecessary duplication, dead code, or magic values that should become documented constants.

Follow clean code principles, confirm test coverage, and refactor whatever is needed.

If the project is already in use, I add the part many people forget:

this project is already being used. Do not break compatibility.

If we touched database schema or user-file structures, there must be a clean migration on upgrade.

Upgrade must not corrupt data.

This guarantees nothing. But it avoids obvious mistakes.

In ai-memory you can see that pattern in the history. After a big batch of PRs, there are hardening commits from me. One example is 65682dc, “fix(audit): address 5 BLOCKING audit findings from today’s review,” touching OpenAI OAuth, admin API, admission hooks, and the MCP server. Another is the sequence of PRs #65, #68, #70, followed by hardening commits around base path, wikilinks, and strict mode. PR goes in, then audit, then adjustment. That second step is where the difference lives.

In #77, the auto-scope by session/actor change touched a lot: active_project, hooks router, MCP server, stress tests, multi-user tests. More than 3,800 lines added. I am not going to pretend someone reads that manually line by line with the same consistency after the fifth hour. The agent helps map risk. I use my judgment to decide what needs hardening.

Regression happens with or without AI

The rsync case is useful because it exposes a fantasy: the idea that human code was safe and AI code is dangerous.

No. Code is dangerous.

Rust is considered a safe language. And Rust has bugs. Go look at Rust’s issue tracker. Zig wants perfection and has bugs. Curl has bugs. Rsync has bugs. Linux has bugs. SQLite has bugs. My code has bugs. Your code has bugs. There is no magical language that produces bug-free software. There never will be.

This is not pessimism. It is a fundamental limit. In computing, we have known for a long time that there is no perfect general verifier for non-trivial semantic properties of programs. Halting problem, Rice’s theorem, that kind of thing. You can prove a lot in restricted domains. You can use a better type system. You can use Rust to eliminate entire classes of memory-safety bugs. You can use tests, fuzzing, model checking, static analysis. Good. Use all of it.

But “arbitrary software with no bugs” is fantasy.

Humans err because of incomplete context, wrong assumptions, pressure, fatigue, lack of tests, lack of real use. AI errs for the same practical reasons: incomplete context, wrong assumptions, training on bad code, poorly specified goals, weak validation. The difference is scale. AI can produce more trash per minute. It can also produce more fixes per minute if you build the process correctly.

I use it because I use it

Another thing I learned in practice: it is very hard to maintain a project well if you do not use it yourself.

I use ai-memory every day. I use ai-jail every day. I use ai-usagebar every day. I used the Prettify Manga Reader extension in my real reading. That is why I catch bugs automated tests do not catch.

In Prettify, for example, the commits tell the story. 66c6728 added the night filter. Then came b5bcb4d, release 0.2.2, with Kindle Web Reader support, toolbar, shortcuts, and screenshot. Soon after came 0e6b7ee, release 0.2.3, fixing spread pairing so right-to-left reading stayed correct without messing up chronological order. That came from real use. I was reading, noticed strange behavior, asked the LLM to fix it, tested it, released it.

That is the healthy loop: use it, feel the pain, fix it, test again.

In ai-jail, some contributions came from use cases I would not have prioritized on my own. #56 fixed the PKGBUILD on AUR. #55 forwarded OSC 52 clipboard writes through the alt-screen PTY proxy. #51 recognized Grok as a known agent and documented the machine-id privacy mask. These details show up when other people use the tool in setups different from mine.

In ai-usagebar, this is even clearer. #4 added a macOS Keychain fallback for Anthropic credentials. I am on Linux/Omarchy, not macOS. I was not going to find that bug in my daily use. #7 fixed reading a custom credentials_path in the widget. Same story: a user with a different config from mine. This is where open source shines: someone scratches their own itch and improves the project for everyone else.

But notice: when I cannot test manually, like macOS and Windows, I depend more on CI. GitHub Actions runs macOS, Windows, Linux. It does not replace real use, but it reduces risk. And when someone who uses that environment opens an issue or PR, that is gold.

Manual testing is still the last line of defense. If you give it up, you are introducing bugs by definition. Automated tests catch what you remembered to specify. Real use catches what you did not even know you needed to specify.

After merging a large set, my final flow is usually:

now run bin/deploy to my home server and test it online.

Check health, check whether it still works as before.

Again: it guarantees nothing. But it avoids breaking the obvious.

There is no silver bullet

I do not think every project should accept AI PRs. I also do not think every project should ban them. There is no universal rule.

If you are a solo maintainer of a critical library, with no time, no decent CI, no energy to deal with an avalanche of bad PRs, banning AI-assisted contribution may be the right decision for now. Better that than accepting junk and breaking users.

If you maintain a small community and want to teach human contributors, as Zig argues, an anti-AI policy makes sense inside that culture. You are optimizing for contributor formation, not patch throughput.

But if you maintain a project that already gets high volume, that already has CI, that already has tests, that already has users, I think adaptation is inevitable. You will need automated triage. You will need an agent reviewing PRs. You will need a bot summarizing issues. You will need a tool that says “this looks like a hallucinated vulnerability report.”

AI is imperfect and scales fast. That is exactly why the process has to change.

My conclusion

AI slop is real. It will get worse before it gets better. There will be garbage PRs. There will be fake security reports. There will be regressions blamed on AI that would have happened with humans too. There will be angry maintainers. There will be users yelling. Normal.

But the direction does not change.

AI-assisted code is here to stay. The percentage of code touched by agents will keep rising. Banning it can work locally, for a while, in one specific community. Across the industry, it is a lost war.

The pragmatic path is to make AI work for us. Use AI for triage. Use AI for audits. Use AI to compare diffs. Use AI to find regressions. Use AI to write tests that fail before and pass after. Use AI to do the boring work that wears humans down.

And keep the human decision where it matters.

I have said this several times and I will repeat it: AI is a mirror.

If you are sloppy, it accelerates your sloppiness. Your code becomes industrialized AI slop.

If you are a real engineer, it amplifies your engineering. You keep demanding tests, CI, compatibility, migrations, review, clean code, deploy, manual testing. You just do it 10 times faster.

And for companies: if you are thinking, “great, now I save money because I do not need to hire as many programmers,” I get it. Up to a point. But if you are putting AI into the workflow, then you must strengthen QA. More tests, more review, more validation process, more good people thinking about quality. Skip that and you are screwed. You will trade development cost for production bugs, rework, incidents, and angry customers.

There is no going back to the 100% manual world. Better to stop wasting energy trying to close Pandora’s box and start building a decent process around what came out of it.

And here’s the part nobody mentions when they say “I built a project with AI”: generating the code is the easy part. The hard part, the thing that separates a repo that’s just a pile of code from an actual open source project, is everything that comes after git init.

I have a strong opinion about this, so let me just plant the flag: no open source project is ready to be published without three things. In order of importance:

1. Installation surface. The new user has to be able to install and try the tool with the least friction possible. One command, ideally.
2. Tests and automated CI. So contributions are easier and there’s common ground on what’s minimally acceptable to land in the project.
3. Documentation. The short, easy-to-read version (the README) and a more detailed set in docs/, so new contributors understand the architectural decisions.

Without those three, the way I see it, the project isn’t ready. It can have the most brilliant code on earth inside, doesn’t matter. If a stranger can’t install it, can’t trust that whatever they touch won’t break everything, and can’t understand why things are the way they are, then you don’t have an open source project. You have a public dead archive.

And there’s a bonus that became an essential part of my workflow: when all my projects follow the same pattern, I can just tell Claude Code or Codex “run the deploy” or “cut a release,” and they can do it. Because the structure is predictable. bin/deploy does the same thing in every project. The release comes out of a tag. The agent doesn’t have to guess anything. Standardization is what makes the automation trustworthy. It looks like fussiness from an organized person, but it’s the prerequisite for me to trust, eyes closed, whatever the agent is about to run.

Let me break down the three pillars and show the minimum setup I actually use.

Pillar 1: The installation surface

This is the most important one and the most neglected. People spend weeks tuning the internal algorithm and then ship a README that says “clone the repo and run cargo build --release.” So the guy who was curious enough to try it needs to have Rust installed, wait ten minutes for it to compile, and pray no system lib is missing. You just lost 90% of your potential users.

The ideal is to offer several paths and let the user pick the one that fits their environment. Look at the install header for ai-jail, my champion of options:

# Homebrew (macOS or Linux)
brew tap akitaonrails/tap && brew install ai-jail

# AUR (Arch Linux) - prebuilt binary
yay -S ai-jail-bin

# cargo (any platform with Rust)
cargo install ai-jail

# mise
mise use -g github:akitaonrails/ai-jail

# Direct binary download
curl -fsSL https://github.com/akitaonrails/ai-jail/releases/latest/download/ai-jail-linux-x86_64.tar.gz | tar xz

Five paths. The Arch user runs yay. The macOS one runs brew. The Rust dev runs cargo install. Whoever uses mise to manage versions pulls it through mise. And whoever just wants the binary downloads the tarball straight from the release. Everyone grabs what they’re already used to, nobody’s forced to install a whole toolchain just to kick the tires.

The secret: compile once, repackage many

The gotcha that scares you at first is thinking each package format needs its own build. It doesn’t. The trick is to compile the binary once per architecture, then repackage that same binary into every format.

In my Rust pipeline, the build job produces a tarball with the finished binary and its sha256. The packaging steps that follow (Docker, AUR, Homebrew) download that already-compiled tarball instead of recompiling. It cuts CI time, guarantees the byte that goes to AUR is exactly the one that goes to Docker, and kills off that whole class of “works in one format, breaks in the other” bug.

# Docker job reusing the already-compiled binary
docker-amd64:
  needs: build-linux
  steps:
    - uses: actions/download-artifact@v4
      with:
        name: ai-memory-linux-x86_64
        path: artifacts
    - name: Desempacota o binário pré-compilado
      run: tar -xzf artifacts/ai-memory-linux-x86_64.tar.gz -C dist/docker/
    - uses: docker/build-push-action@v6
      with:
        target: runtime-prebuilt-amd64  # inject the binary, don't recompile

The AUR job does the same thing: download the tarball, compute the sha256, inject it into the PKGBUILD:

X86_SHA=$(awk '{print $1}' artifacts/ai-usagebar-linux-x86_64.tar.gz.sha256)
sed -i "s/^sha256sums_x86_64=.*/sha256sums_x86_64=('$X86_SHA')/" packaging/aur/PKGBUILD-bin

That’s the core idea. The binary is the real artifact. RPM, DEB, AppImage, AUR, Homebrew tarball, Docker image, all of it is just wrapping around the same binary.

The two snippets above are excerpts. If you want to see the whole thing running, the full build-once-reuse flow (binary compiled once, reused across the Docker and AUR jobs) lives in ai-memory’s release.yml, and the sha256 injection into the PKGBUILD is in ai-usagebar’s release.yml.

The menu of formats

Every platform is its own world. Here’s the minimum worth knowing:

Linux is the most fragmented. The options, from simplest to most work:

• Tarball (.tar.gz): the lowest common denominator. Works on any distro, just unpack it and drop it in PATH. Always offer this one.
• AUR (Arch): two packages, the -bin (prebuilt binary, installs in seconds) and the source one (compiles on the user’s machine). I publish both with the KSXGitHub/github-actions-deploy-aur action. The -bin is what everybody uses.
• AppImage: a single executable file that carries all its dependencies inside. Good for desktop apps. Frank Sherlock, which is Tauri, builds the AppImage right in the build and I still GPG-sign it afterward (the signing step is in its release.yml). For a simple binary you can use appimagetool.
• RPM and DEB: for Fedora/openSUSE and Debian/Ubuntu. The tool that saves your life here is nfpm, which generates both from a single declarative nfpm.yaml, no need to learn the guts of rpmbuild or dpkg-deb. It takes the already-compiled binary and spits out both packages.

macOS has two targets: Apple Silicon (aarch64-apple-darwin) and Intel (x86_64-apple-darwin). You either compile them separately or build a universal binary with lipo. And here lives the platform’s biggest headache: signing and notarization. Without it, macOS Gatekeeper refuses to open your app. The flow is to decode the P12 certificate, import it into a temporary keychain, sign with codesign --options runtime --timestamp, and notarize:

codesign --sign "$APPLE_SIGNING_IDENTITY" --options runtime --timestamp ./meu-binario
ditto -c -k meu-binario notarize.zip
xcrun notarytool submit notarize.zip \
  --apple-id "$APPLE_ID" --password "$APPLE_PASSWORD" \
  --team-id "$APPLE_TEAM_ID" --wait

This requires a paid Apple Developer account. It’s annoying, but it’s the price of not throwing an “app is damaged” error in your user’s face. The whole signing-and-notarization job, with the temporary-keychain dance, is in ai-jail’s release.yml.

Windows is the easiest to build (x86_64-pc-windows-msvc) and the thorniest to sign. A Windows code signing certificate is expensive and bureaucratic, so I, honestly, don’t sign. I accept that SmartScreen will throw a warning on first run and I make that clear in the README. For a personal open source project, that’s good enough.

Rust crates (cargo publish): if it’s a Rust library or CLI, publishing to crates.io gives you cargo install for free. One line in the pipeline. Tip: make it idempotent, because re-running on an already-published tag errors out. I grep the output for “already exists” so it doesn’t blow up the whole release over that.

Homebrew: the clean way is to keep your own tap (akitaonrails/homebrew-tap). The pipeline updates the formula with the new URL and the new sha256 on every release. The user runs brew tap once and after that it’s just brew upgrade.

mise: mise can install straight from GitHub releases without you configuring anything on your end, as long as your tarballs follow a platform-name convention. You get this path basically for free just by naming your releases well.

Notice a detail: AUR, Homebrew, mise, and the direct download all consume the same release tarball. You publish the binaries once, and four different package managers know how to fend for themselves. That’s why “compile once, repackage many” is the golden rule.

Pillar 2: GitHub Actions, tag-based releases, and the changelog

The second thing that makes an open source project serious is CI. Forget the “build passing” badge on the README, the value is in the common ground it creates: when someone sends a pull request, CI runs the tests, the linter, and the security scanner, and you know right away whether that contribution breaks anything. Without it, every PR turns into a tiresome manual negotiation.

On my Rust projects CI runs cargo fmt --check, cargo clippy -D warnings, cargo test, and cargo audit. On the Rails projects it’s rubocop, brakeman (security scanner), bundler-audit, and the Minitest suite. FrankMD alone has over 1,800 tests between Ruby and JavaScript. Frank Sherlock clears 300 tests in Rust plus 300 on the frontend. That safety net is what lets me accept a stranger’s PR without sweating.

PR review with an LLM

The tests and CI are the bottom net. Above it there’s a layer that became central to my workflow: using the LLM itself to review the pull requests. Opus and GPT are absurdly good at code review and quality refactoring. Better than me on consistency, because they don’t get tired and they don’t skim over 200 lines just because it’s the fifth review of the day.

I have two prompts I use all the time. The first, for a specific PR that just landed:

look at the open PR on github. don’t trust the author’s description. audit the code thoroughly to see if it actually makes sense. make sure there are no regressions and no drop in code quality. make sure test coverage is adequate. then tell me what you think we should do.

Notice the “don’t trust the author’s description.” That’s deliberate. A PR description tells you what the person thought they did, and it doesn’t always match what they actually did. The LLM has to read the real code, not the summary.

The second prompt is for after a long coding session or a bunch of PRs merged at once:

we touched a lot of code. audit the code those PRs changed, look for dead code, unnecessary duplication, magic hardcoded values that should be constants and/or better documented, missing test coverage, aim for clean code principles, then check whether the documentation was properly updated.

With the verdict in hand, I decide. For an incomplete PR, what I do depends on the size and my priority at that moment. If it’s something small, I fix it myself right in the PR with an amend, reply explaining what I changed, and then merge and close. When I can merge but can’t fix beforehand, I merge and immediately drop a follow-up fix commit. And when it’s a big change, I ask the LLM to reply to the PR explaining why we’re not merging, with the right direction on what needs to be done.

That’s how I closed almost 40 PRs and 20 issues on ai-memory alone, in a couple of days. That volume doesn’t happen by hand, and it also can’t happen on autopilot.

And this is the point I keep hammering: none of this is blind. I’m piloting the LLM toward those goals the whole time. It does the audit, puts the suggestions on the table, and even writes the text of the reply, but the one who decides to merge, fix, or reject is me. The LLM is the tireless reviewer; I’m the one who signs off.

The tag-based trigger

The release part is separate from CI. While CI runs on every push and PR, the release fires only when you push a version tag:

on:
  push:
    tags:
      - 'v*.*.*'   # dispara em v1.2.3, v0.5.0, etc.
  workflow_dispatch: # permite disparar manualmente também

The flow for cutting a version becomes this:

git tag v0.5.0
git push origin v0.5.0

And that’s it. GitHub Actions notices the tag, fires the release workflow, compiles for the targets, packages everything, signs whatever needs signing, creates the GitHub Release, and publishes to the package managers. I don’t touch anything else. In fact that’s literally what I tell Claude Code: “cut v0.5.0.” It runs cargo set-version, updates the CHANGELOG, commits, creates the tag, pushes, and the rest is the pipeline.

The Rust build matrix compiles each target on its native platform:

strategy:
  matrix:
    include:
      - os: ubuntu-22.04
        target: x86_64-unknown-linux-gnu
      - os: ubuntu-22.04-arm
        target: aarch64-unknown-linux-gnu
      - os: macos-latest
        target: aarch64-apple-darwin
      - os: windows-latest
        target: x86_64-pc-windows-msvc

A cross-compilation gotcha: if you compile ARM inside an x86 machine (via cross), you can’t run the tests for that binary right there, because the architecture doesn’t match. The fix is to run tests only on the native target and cross-compile the rest without testing. ai-usagebar does exactly that, x86 native and ARM via cross, in the matrix of its release.yml. And always use a compile cache: Swatinem/rust-cache easily cuts build time in half.

Release notes and the changelog matter

This is where a lot of people slack off, and they shouldn’t. When you cut a release, the person thinking about upgrading wants to know what changed. “v0.5.0” says nothing. A changelog says something.

I keep a CHANGELOG.md in the Keep a Changelog format, one section per version. And the pipeline automatically extracts the current version’s section to paste into the GitHub Release body. A simple awk handles it:

awk -v ver="$VERSION" '
  BEGIN { hdr = "## [" ver "]" }
  index($0, hdr) == 1 { flag=1; next }
  flag && index($0, "## [") == 1 { exit }
  flag { print }
' CHANGELOG.md > changelog_section.md

That awk is the same one that runs on the ai-usagebar release (again in its release.yml, which is my most polished one on this front). Then, after the changelog, I append the install instructions and the sha256 checksums for each artifact. The result is a GitHub Release that explains itself: what changed, how to install it, and how to verify the integrity of the download. The user reads it and decides whether to upgrade. That’s the kind of care that makes a stranger trust your project.

Pillar 3: Real deployment, no fuss

The projects that run as a service (the Rails ones, mainly) have to go somewhere. And here I go against the hype. No Kubernetes. No Kamal. No three-stage deploy pipeline with manual approval.

I have a home server. A Minisforum MS-S1 Max running openSUSE MicroOS that I migrated recently, with a Gitea acting as a private Docker registry on port 3007. For a solo developer with a simple home server, that’s more than enough. All that cluster-orchestration ceremony is solving a problem I don’t have.

The whole deploy fits in a single bin/deploy. It does three things: builds the image, pushes it to the registry, and SSHes into the server to bring up the new version:

#!/usr/bin/env bash
set -euo pipefail

# Config comes from a .gitignored file, outside the repo
source "$(dirname "$0")/../config/deploy.env"

IMAGE="${REGISTRY_HOST}:${REGISTRY_PORT}/akitaonrails/frank_fbi:latest"

echo "==> Buildando a imagem..."
docker build -t "$IMAGE" .

echo "==> Empurrando pro registry..."
docker push "$IMAGE"

echo "==> Subindo no servidor..."
ssh "${DEPLOY_USER}@${DEPLOY_HOST}" \
  "cd ${REMOTE_DIR} && \
   docker compose pull && \
   docker compose down --remove-orphans && \
   docker compose up -d"

That’s it. build, push, ssh with pull, down, up. Four commands on the remote server, chained in a single SSH. No agent running over there, no webhook, no control panel. The server has a docker-compose.production.yml pointing at the :latest image from the registry, and the pull brings the new version down.

Notice the sensitive data lives in a config/deploy.env that doesn’t go into Git. The repo only carries the config/deploy.env.example, with the structure and fake values. This “versioned example, real one ignored” pattern repeats in every project of mine, and it’s what lets the project be public without leaking the host, user, or path of my server.

The image itself comes from a multi-stage Dockerfile. The build stage installs the compilation tools, builds everything, and the final stage copies only the result into a lean image, running as a non-root user. The production compose swaps build: for image:, points the volumes at the server’s persistent paths (/var/opt/docker/...), and uses restart: always to bring everything back up if the server reboots.

And again, the beauty of standardization: since every project of mine has this same bin/deploy, I can tell Claude Code “deploy Frank FBI” and it knows exactly what to run. I don’t have to remember a single command. The tool is the executable documentation of the process.

The lesson: nobody cares about your stack

Now the part I most wanted to write, and the one worth more than all the YAML snippets above.

Most people, when they go to present a project, focus everything on the technology. “Check out my brand-new vector database.” “Check out my last-gen reactive framework.” “I used such-and-such microservices architecture.” And they fill the top of the README with that: the dependency list, the component diagram, the stack.

Nobody cares.

Seriously. The user who lands on your README doesn’t want to know which database you used. They want to know what problem this solves for them. Implementation detail doesn’t belong at the top of the README. It belongs in docs/, for whoever’s going to contribute. In the README, what matters is the use case.

Look at the difference. “MCP server in Rust with FTS5 and optional LLM consolidation” says nothing to anybody. Now “long-term memory shared across Claude Code, Codex, and Cursor, with handoff between sessions” — that one the guy gets instantly, because it’s the pain he has. Both describe ai-memory. One talks about technology, the other talks about the problem it solves.

The use case is the single most important thing about the whole project. Before writing a line of code, the right question is “what problem does this solve, and for whom?” If you can’t answer that in one sentence, you’re building a solution looking for a problem. And a solution looking for a problem is, in practice, a solution that solves nothing. Doesn’t matter how elegant the code is on the inside.

The UX layer, the surface the user touches, is worth far more than the internal engineering. A project with mediocre technology that’s trivial to install, that clearly states what problem it solves, and that has honest documentation, goes further than a brilliant project nobody can run. Every time.

So the minimum, the real minimum for an open source project, is this: easy install, CI that gives you confidence, documentation that starts from the problem. The pretty algorithm comes way down the priority list. The three pillars of this post exist to serve one thing, which is the experience of whoever shows up from outside. Get that right first. The rest is implementation detail.

Instead of doing a one-day demo and risking a “yeah I think so,” I went all-in: over 500 hours, almost half a million lines of code and documentation, 24 repositories between public and private.

But today someone flagged a point I never thought would need defending:

“Are all 24 projects perfect, models of code excellence?” — ABSOLUTELY NOT.

I thought that was obvious. For starters, if I believed any of them was that good, I’d be spinning up a startup around it. I don’t lack capital to invest. If I’m not doing that, clearly all of this is at hobby pace.

Second obvious point: if I’m jumping between projects every few days, it’s because I lost interest. If I really liked it, I’d still be tuning it today. I keep saying it in the videos — software never “finishes”. The moment it stops moving, it dies.

The goal was to test different kinds of projects to see how the LLMs handled each. That’s why there’s stuff in Ruby on Rails, Rust, Go, Python, Crystal. Plus frameworks like Tauri and Flutter. I figured, for instance, that Flutter being super popular would mean the LLMs wouldn’t struggle. They did. They could move forward, just not as smoothly as in Rails. Tauri was “rough” too. Crystal was REALLY rough.

Third point: every experimental project got named “Frank”. Sarcasm, obviously. I don’t have the patience to invent a pretty name for a project I already know is going to be one-off.

I use my GitHub basically as a repository of exercises and experiments. Look at all the old repos: barely any are a “real” project I want to maintain. I do what I’ve been preaching in the videos: to actually learn, you exercise on MANY projects, even if most of them are meant to be THROWN AWAY. Nobody learns by trying to build a single perfect project. Check out my videos A Dor de Aprender or Guia Definitivo de Aprendendo a Aprender (Portuguese).

In the middle of a bunch of half-baked projects, some naturally emerge as the ones I like and maintain. In practice I threw them all at the wall to see what stuck. The good ones naturally pulled in more contributors too. I don’t keep an eye on Issues and PRs every day. I check once a week. Two or three days ago I made a sweep and cleared everything that was pending, for example. Even on projects I don’t care much about, if a PR shows up, I review and accept it. It wasn’t useful to me, but it can be useful to someone else.

The project ranking

To make it clear, here’s a ranking of the projects with a comment on each. The PR counts below are non-dependabot (gem bumps don’t count). The score is my estimate of “how stable and usable is this thing in its current state.”

Top tier (7-8/10)

ai-jail — 8/10. The only one really close to a stable 1.0 release. Got the most traction: 19 PRs closed, 23 Issues resolved. Recommend using, it works very well.

akitando-news (the M.Akita Chronicles) — 8/10. The one I built with the most care, in the “Agile Vibe Coding” mode I described a few posts back. And the one I touch most often. There’s always some little thing to tweak, stuff I keep adding, removing, experimenting with. Which is why it’s been shipping every week. Also why: it’s closed and I don’t plan to open it.

akitaonrails.github.io — 7/10. This is the blog you’re reading. I touch it often and I’ve already closed about 7 Issues and 30 PRs from contributors. I’ll keep tuning it bit by bit.

FrankMD — 7/10. One of the first ones I built, with no criteria at all: I just wanted an editor with automatic image upload to my S3. These days I use Claude Code to edit posts, so the only people using it are the people contributing to it. Got solid traction, I closed 17 PRs and 29 Issues. It’s not perfect, but it’s good enough to use — at least it’s been tested enough.

easy-ffmpeg — 7/10. Tiny little project, but I actually use it in the videos. My OBS always records in MKV and when I want to share I convert to MP4. Small use cases like that. Good enough to use.

Middle tier (5/10)

distrobox-gaming — 5/10. Most recent one, serves as a “dump” of documentation for my retro games. I use it as a notepad and it’s been working fine for my setup. Not worried about it working for anyone else. If anyone wants to send PRs, go ahead.

llm-coding-benchmark — 5/10. This one doesn’t run on its own. I have Claude Code execute it. The goal is to tabulate data so I can write articles about new LLMs. The code has to be REALLY rough.

FrankMega — 5/10. The closest thing to a “todo list” exercise. I just needed to share a big file with a friend and thought “hmm, I bet I can build a mini-Mega in 1 day”. And that was it. The name is a joke, obviously — this is nowhere near the actual Mega.

distrobox-llm — 5/10. More personal organization on my PC: I wanted to isolate the AI tooling into a distrobox so I wouldn’t have to keep updating big packages on the main system.

Lower tier (3-4/10)

FrankYomik — 4/10. This was an idea I’d had in my head for years. I just wanted to see if the concept worked. It works. But running local LLMs for it is slow and not as practical as I’d like. It’s the only one I might come back to, because I actually want to use it. As-is, it’s in “works, sort of” territory.

NW-Omarchy — 4/10. Another “let’s see if this works” experiment, lasted 2 days. The concept worked, but I don’t plan to do much more than that. Up for grabs if anyone wants to take it further. The real test I wanted to run was whether X11 would be MUCH worse than Wayland. The gap turned out smaller than I expected, but I don’t see myself using it day-to-day. Like I’ve said: for the niche of people who can’t use Wayland, it’s a decent alternative.

frank_investigator — 3/10. Started it, didn’t have the patience to keep going. Planted the seeds of the idea. If nobody sends PRs, it’s not going to evolve much. Needs more research and a lot more testing.

frank_fbi — 3/10. Another concept test, no personal use case (I never fall for phishing). No idea where this would go yet.

frank_karaoke — 3/10. Born from a weekend conversation with my brothers. Another shot at using Flutter (and seeing the limits, both of Flutter and of LLMs writing Flutter). Useful for me to learn about the karaoke ecosystem and find out that real scoring only works with per-song metadata. So I didn’t push it past toy territory.

Bottom of the barrel (1-2/10)

FrankSherlock — 2/10. I don’t use it myself. So it’s probably full of bugs that nobody’s fixed. It was a test to see if I could build a Tauri app, but I lost interest fast. There’s room to make it better, but it’d take a lot more hours of actual use, testing, and fixes. It was also my first time touching Tauri, so that part is probably pretty rough too.

FrankClaw — 1/10. I don’t use it, and this is the only one I haven’t even run a single time. The exercise was just to see how hard it’d be to port the original openclaw from TypeScript to Rust. But as I’ve said many times, I don’t see any use for openclaw in my own workflow. When I finished version 0.1, I stopped. Even so, there were 6 PRs I closed. If anyone’s interested, feel free to contribute.

shadPS4 — 1/10. Despite generating a lot of code and a lot of documentation, none of it was meant to be a real contribution. Pure learning, plus solving one very specific problem. I left it open just to show what “study-material” code looks like.

easy-subtitle — 1/10. Started it, lost interest, never actually tested it. Just pushed it and forgot.

The private ones

Beyond akitando-news, I have stuff on my private Gitea like my-omarchy-config, emulators-reorg-2026, thinkpad-omarchy-config, homelab-docs, akitando_english_translation, etc. I’m not going to describe them because you’re never going to see them. They serve as “scratchpads” for Claude Code to administer my machines.

What this ranking actually says

Notice: there’s no project on this list that I think is 10/10.

For a project to graduate from hobby to real project, you need users and feedback. If nobody uses it, there are always dozens of functional bugs nobody’s stumbled on, so nothing ever gets fixed. A project with no users is a dead project.

That’s why I keep saying in the videos: there’s no such thing as planning every detail, writing a pile of code, and one day “finishing”. Nope. When you think you’ve “finished,” what you actually have is version 0.1. Then comes months of experimentation with real users reporting bugs. You go fixing. Out of those fixes come refactors, architecture changes because your initial assumptions were wrong against actual use. Software NEVER finishes.

What I took out of this marathon

16 hours a day, every day, vibe coding, the output is always going to have slop in it. There’s no way around it. Forcing volume doesn’t fix it. You have to review manually, commit by commit, PR by PR. Why didn’t I do that on all of them? Simple: nobody’s paying me. I did it on 2 or 3 projects. The rest I just wanted to test the concept and move on to the next one.

The right way is what I reported a few posts ago: “Agile” Vibe Coding. That’s my sarcastic way of saying “use XP and normal software engineering practices”. Metrics, monitoring, CI, the whole nine yards. None of that changed. Without it, you’ll end up like a lot of my exercises: one-offs to throw away.

Don’t be embarrassed to push “bad” projects. What I didn’t anticipate is that some people assume just because I pushed code, I think it’s perfect. But that’s projection: they only push code they consider great, which is why they never push anything. I’ve said this MANY times and I demonstrate it: I don’t care about my code, I have no preciousness about my code, GitHub is a temp dump directory for whatever. The ones I really like usually live private.

If a project is really good, don’t just say it. Put your money where your mouth is: build a business around it, charge users, deliver quality. I’m retired, I don’t need to charge anyone, but I also have no interest in working for free. The moment the code stops being fun, I drop it and move on. The ones I actually got hooked on (ai-jail, M.Akita Chronicles) I keep touching more often. That simple.

As I expected, and as I showed both in these little projects and in the benchmark, the choice is simple: use GPT Codex or Claude Code on one of the subsidized plans. Don’t pay per token. That’s where the 5x productivity lives (max 10x, with slop along for the ride). Yes, there’s Kimi v2.6 or DeepSeek V4 Pro. But I’ll save you the trip: the projects I scored below 5 here would quickly become projects below 2 with those.

The point that matters most

AI doesn’t replace a good programmer. AI mirrors and accelerates who you are. If you’re a good programmer, AI accelerates good code. If you’re a bad programmer, AI accelerates bad code.

That’s exactly what I showed: most of the projects I built were one-offs and proof-of-concept tests. Bad code, with no interest from me to keep going. AI accelerated me finishing those tests in a half-baked way. Which is fine by me: without AI I never would’ve started. Now that I’ve seen what they look like, I can cross them off my list and move on.

What matters is your OBJECTIVE. If yours is to build a business and charge people, you have to do it with 10x more care than I did. Everything I do is meant to be thrown away. I don’t need to make money on any of these projects. Don’t assume things I never said.

I’ve been following Uncle Bob for many years. I’ve exchanged messages with him a few times over the decades, I have formed opinions about his positions, and I did an hour-long live stream on my channel with him about Clean Code, Agile, Craftsmanship, and where the industry was heading. If you’ve never watched it, I recommend it:

Clean Code specifically set a standard: code is written once but read dozens of times. The programmer’s job isn’t just to make it work. It’s to make it work in a way that another programmer can understand, modify, and not break. Meaningful names, small functions, one responsibility per class, no duplication, automated tests, clear structure. The target audience was always another human being sitting at an editor trying to figure out what the first one did.

In 2026, that audience changed.

The audience isn’t human anymore

Last week I wrote VS Code Is the New Punch Card. The thesis is that typing code manually into a text editor is becoming a niche activity, the same way typing binary directly on an Altair front panel became a relic after compilers got good. The AI agent is the new compiler: I describe intent in natural language, the agent navigates code, edits, runs tests, adjusts, delivers.

If that thesis holds, the next question is obvious: who are we writing code for now?

Not for the human programmer who’ll sit down tomorrow to maintain it. It’s for the agent that’ll read, edit, and extend it. And an agent is not human. Agents have different technical constraints, different biases, different limitations. Part of Clean Code still applies (in some cases more critically). Another part shifts in weight. And new demands emerge that Uncle Bob couldn’t have anticipated in 2008.

This article is about that. What’s the version of Clean Code that makes sense when the primary reader is an LLM?

The real agent constraints

Before re-ranking, worth reviewing what agents actually face.

File truncation. Most agent CLIs limit file reads to small ranges. Claude Code reads 2000 lines per chunk by default. Cursor, Codex, Windsurf, all have similar caps. A giant file simply doesn’t fit into the context window in one shot. The agent has to ask for piece by piece, or worse, use grep and reconstruct mentally.

Attention degrades with context. Claude Opus has a 200k-token window, Sonnet 1M, Gemini 1M. Sounds like a lot. In practice, “needle in haystack” tests show retrieval quality drops well before the claimed limit. Flash Attention and variants speed up the computation, but they don’t replace full native attention. The more you stuff into the window, the worse the detail precision. And the agent’s context isn’t just your code: it holds CLAUDE.md, the system prompt, chat history, tool output, error logs, test output. All competing for the same window.

Grep is cheaper than read. The agent knows this. It prefers rg "funcName" over loading a whole file. It’s faster, uses fewer tokens, hits the target. Unique, distinctive names make that much more effective. This isn’t a shortcut, it’s an architectural choice: I wrote about this in detail in Is RAG Dead? Long Context, Grep, and the End of the Mandatory Vector DB, showing that Claude Code itself navigates a repo with Glob and Grep, no vector DB, no embedding, and that’s not a deficiency, it’s mature design. Lexical search + smart reader consuming raw text beats dense retriever + top-k on practically every real-domain benchmark. You benefit from that when you organize your code: greppable names aren’t just “nice for humans”, they’re the agent’s primary navigation API.

Tool calls cost tokens. Every Read or Edit or Bash burns input and output tokens. Short files, small test output, concise logs — all of that keeps the agent productive and the bill low.

Latency matters. Agent in a loop, each tool call adds seconds. A large file slow to process becomes perceptible session friction.

Grepping by visual pattern is hard. If you used inconsistent indentation, mixed tabs and spaces, varied brace style between files, the agent spends tokens internalizing the mess. Consistency helps.

From those constraints, Clean Code principles can be re-ranked by relevance for agent-facing work.

Re-ranking: Clean Code in the age of agents

Most important to least. A caveat: it’s not that the ones at the bottom stop mattering. It’s that the ones at the top started mattering MUCH more.

1. Small functions (and small files)

Uncle Bob: “functions should do ONE thing, they should do it WELL, and they should do it only”. Ideal size 4 to 20 lines, per the book.

For an agent, that recommendation became a technical obligation. A small function fits in a single tool call without truncation. A short file (keep it under 500 lines, ideally 200-300) fits in a single read. If the agent can grab the whole unit of meaning in one call, it reasons about it with full attention. If it has to paginate, it builds a fragmented mental model, and each fragment costs attention.

Before, “small function” was good for humans because it aided reading. Today, “small function” is good because it matches the model’s unit of processing. If there’s one recommendation to take to heart, it’s this one.

2. Single Responsibility Principle (SRP)

Each module does one thing and has one reason to change. It was already the heart of Clean Code. For an agent, it becomes even more critical because:

• The agent can isolate the unit to understand without loading the rest of the system
• You can run focused tests on it
• You can edit without fearing side effects
• Grepping by responsibility becomes predictable

Code with tangled responsibilities forces the agent to load way more context for any simple change. An 800-line class that does three things is worse for the agent than three 250-line classes, even if the total is the same.

3. Meaningful and unique names

Clean Code already preached: names reveal intention, no disinformation, distinctive, pronounceable, searchable. For the agent, “searchable” became the most important property of that list.

The agent searches code via grep/ripgrep all the time. A generic name (data, process, handler, Manager, Service) returns fifty matches and forces the agent to read each one. A distinctive name (UserRegistrationValidator, InvoiceLineItemTotal, ClaudeCodeSessionTracker) returns three matches and the agent goes straight to the right one.

Rule of thumb: if you grep the name and a lot of irrelevant stuff comes back, the name is bad for the agent. If only what matters comes back, the name is right.

4. Comments with context and provenance

This is where the inversion is most jarring. For Uncle Bob in 2008, the axiom was: “good code explains itself, excessive comments are a code smell, every comment is debt that gets stale”. Every experienced programmer who read the book absorbed that rule. Well-named code doesn’t need comments. Too many comments = flag of bad code trying to justify itself.

Now flip it. The agent reads comments. And likes them. Comments become first-class context. The agent has perfect syntax fluency, knows exactly what x++ does, doesn’t need obvious captions (that kind is still bad, see item 13). What it DOESN’T know is why you chose this approach over the obvious one, what production bug motivated this weird logic, what business constraint forces this specific order, what workaround exists because the upstream lib has known bug #1234, which commit introduced this decision, which Jira issue is the reference. That kind of information is provenance: the why of the decision. It only exists in the head of the human who wrote it, in the commit message, or in a well-placed comment. For the agent, the comment is the most accessible source during a tool call.

Docstrings with intent and usage examples also became strong signals. When the agent picks up a function without understanding the context, a header docstring (JSDoc-style with examples, Python """, Rust ///) drastically shortens the path to a correct change. Uncle Bob was skeptical of JavaDoc in 2008 because they got stale. Today, with the agent able to rewrite the docstring alongside the code, that counter-argument lost weight.

A practical consequence: don’t prune the comments the agent writes. If you have the reflex “verbose comment is noise” inherited from the original Clean Code era, that rule flipped. The agent wrote that comment because, in the act of generating the code, it decided that information was worth preserving for future edits. Removing the comment in code review strips context that the agent itself will want to read on the next interaction. Let the agent comment. It knows what it’s doing. The only kind of agent-authored comment worth removing is the obvious redundant one (item 13), and modern models rarely produce those if the system prompt is well written.

5. Explicit types

This isn’t in 2008’s Clean Code because the industry hadn’t converted yet. But in 2026 it’s a fundamental criterion.

Python without type hints, JavaScript instead of TypeScript, Ruby without RBS. Dynamic code without annotations forces the agent to infer types from usage, which costs reasoning and gets it wrong frequently. Typed code gives an immediate answer key: the signature says what goes in, what comes out, which states are valid. The agent saves discovery work and makes fewer mistakes.

If you’re still on Python 3 without type hints, the transition will boost agent productivity more than any logic refactor.

6. DRY (Don’t Repeat Yourself)

Clean Code already said duplication is the root of all evil. For an agent, duplication is worse than for a human for one specific reason: when the agent has to change something that’s replicated, it can update one copy and forget the others. The attention window doesn’t have natural gravity pulling “oh, there are two more copies of this in other files”. The agent has to find each one via grep, and if the pattern has subtle variation between copies, the result ends up inconsistent.

Factoring into a reusable function or module isn’t aesthetic. It’s automated-refactor safety.

7. Tests the agent can run

Uncle Bob dedicates a full chapter to Unit Tests and F.I.R.S.T (Fast, Independent, Repeatable, Self-Validating, Timely). All of it still applies, with an important addendum: the test has to be executable by the agent without human setup.

Meaning: the command to run the test is in the README or CLAUDE.md, in the Makefile, in the package.json. Output has a predictable format the agent parses. It doesn’t depend on manually seeding the database, on a config file that isn’t in the repo, on a secret credential. The agent writes code, runs tests, reads output, adjusts, runs again. That cycle is the foundation. If the tests don’t run headless, the agent goes blind.

Here I speak from field experience. I documented this in From Zero to Post-Production in 1 Week Using AI, where I went hard on a real project: 274 commits in 8 days, 4 integrated applications, 1,323 automated tests by the end. What made it work wasn’t “AI programs on its own”. It was Extreme Programming with the agent as pair instead of a human pair. Running tests on every commit, tight CI, coverage above 80% (95%+ on business logic), test-line-to-code-line ratio above 1:1 on some modules. Sounds like overkill. It isn’t. In 274 commits, CI caught real bugs more than 50 times, bugs that would have gone straight to production if I had blindly trusted the agent. Without tests, the agent hands you plausible code that silently breaks something that worked yesterday. With strong tests, the agent becomes a multiplier: it generates a test, the test validates the code it wrote, the test is the safety net for the next change it makes. Virtuous loop.

XP practices (pair programming, CI, tests first, continuous refactoring, short feedback) didn’t become obsolete. They became exactly the right way to work with an agent. Whoever programs in cowboy mode without tests today isn’t rebellious. They’re just slow, because the agent without tests keeps guessing, and guesses need manual review, which kills the speed the agent should bring. Good tests with good coverage became the difference between a productive agent and an agent that keeps flailing. Or, put another way: TDD became a technical obligation, not a philosophy.

I covered this theme from another angle in Software Is Never “Done”, showing that post-deploy life is where tests matter most: in ten days of operation after launch, I ran 56 commits of fixes, hardening, and adjustment in response to real behavior, and each commit came with a regression test. Without the net, each of those 56 commits would be an opportunity to break something that worked yesterday. TDD isn’t a phase, it’s a habit.

8. Predictable directory structure

Clean Code barely discusses this (it was more focused on code inside a file). For an agent, tree organization matters. If src/controllers/users.rb implies src/models/user.rb and src/views/users/, the agent can anticipate paths without listing directories. If the project uses idiosyncratic naming (random files, unpatterned names, everything flat in one folder), the agent loses time with find.

Strong framework conventions (Rails, Django, Next.js, Laravel) help the agent a lot. Project without convention, the agent builds one over time, but until then it burns tokens exploring.

9. Dependency Injection and Testability

Code with injected dependencies (not hardcoded) is easier to test in isolation. The agent benefits from this. It can swap the real EmailSender for a FakeEmailSender in a test without touching the logic. Code that instantiates its dependencies internally forces the agent into monkey-patch-fake-server-hacks that are slow, fragile, and pollute the session with infra grime.

DI isn’t ceremony. It’s isolation scope. And in a real project, DI quickly becomes a load-bearing refactor: on one of my projects (the M.Akita Chronicles), I discovered after launch that I needed to swap the default LLM model to another provider. The environment variable had existed from the start. But the model name was still hardcoded in references across 24 files. A whole commit (Centralize LLM model config) touched all 24 to isolate the config into a single constant. Swapping models after that became a one-line change. That’s exactly the kind of refactor that only shows up after software meets reality, and it’s where DI and config isolation pay dearly if you didn’t do it earlier.

10. Avoid deep nesting

Clean Code talks about single level of abstraction per function. A corollary: avoid if inside for inside if inside try. Every indentation level is more attention the model has to spend tracking state. Four levels of indentation is MUCH more cognitively expensive for the agent than two levels with early return.

Pattern matching, guard clauses, early returns, flattening logic, all of this improves readability for the model the same way it improves it for the human, except measurably so, because the cost is measured in response quality.

11. Errors with context

raise ValueError("invalid input") doesn’t help the agent when it reads the stack trace. raise ValueError(f"invalid input: received {repr(x)}, expected non-empty string of digits") does. The agent uses exception messages as debug signal. Vague message = agent runs an extra round to figure out what went wrong.

Uncle Bob talked about this in Error Handling: “Provide context with exceptions”. It became critical now.

12. Formatting and style

Don’t waste time on this. Use the default or most popular formatter for your language: cargo fmt for Rust, gofmt for Go, prettier for JS/TS, black or ruff for Python, rubocop -A for Ruby. Configure it in pre-commit, configure it in your editor to run on save, and move on. The agent handles any consistent style just fine, and the auto-formatter keeps the diff tidy between commits. Tab vs space, 80 vs 100 columns, brace style, all of that became noise. The formatter decides, you accept.

13. Comments that describe the obvious

Last on the list. Still bad, got even worse. Comments like // increment i by 1 above i++ waste the agent’s tokens the same way they wasted the human’s patience. The model knows how to read code, it doesn’t need obvious captions.

If you have the habit of writing obvious comments because some school taught you that way, this is the moment to stop. In 2008 it was bad because it polluted visual space. In 2026 it’s bad because it costs real money in tokens.

What Uncle Bob couldn’t foresee

Beyond re-ranking what was in the book, some new things emerged that are specific to the agent world:

Meta-documentation files for agents. CLAUDE.md, AGENTS.md, .cursor/rules, and so on. These are files the agent reads before any tool call, describing project conventions, important commands, caveats, things that don’t go in a docstring. Writing these files is a new skill: short, direct, imperative, action-oriented. No philosophical prose. Bullet points of what the agent needs to know to not mess up.

README with high-level architecture. Uncle Bob barely cared about README (the book is about code). For the agent, well-made READMEs drastically shorten the path to understanding the shape of the project. A simple ASCII or Mermaid diagram helps.

Structured logging. JSON logs with named fields are much more useful for the agent than prose logs. The agent parses JSON trivially, uses fields to filter relevant errors, correlates across services. A loose printf in free text forces heuristic parsing.

Accessible observability commands. pnpm test, make lint, cargo check, python -m mypy — the more the project exposes predictable commands the agent can invoke to validate changes, the better. If running tests requires 10 manual setup steps, the agent won’t run tests, and the feedback loop breaks.

Idempotent setup scripts. The agent has to be able to run bin/setup or scripts/bootstrap.sh on a clean machine and reach a state where it can work. If onboarding depends on instructions in someone’s head, the agent is locked out.

Instructing the agent to write clean code

There’s an important detail that only becomes clear after about 500 hours of using agents: no LLM does any of this by default. You tell it “implement feature X” and it’ll implement it the way the model considers average. No dependency injection. 80-line functions. No tests, or tests that mock the wrong thing. Duplicated logic because it’s faster. 2000-line files because “everything’s in one place”. You need to WRITE these rules. The agent reads, the agent follows.

Where to write: CLAUDE.md, AGENTS.md, .cursor/rules, .github/copilot-instructions.md, depending on the CLI. Format: short, imperative, action-oriented. The agent reads these files on every iteration (Claude Code re-reads CLAUDE.md on every query), so each line burns context tokens — density matters.

Below is a proposal for a template you can drop into a CLAUDE.md on a new project, consolidating what I discussed above in a format the agent consumes. This is not a definitive version, it’s a starting point to test and tune to your language, your team, your flow. If a rule doesn’t fit your context, remove it. If you need a new rule, add it. The point is to have a structured skeleton:

## Code style

- Functions: 4-20 lines. Split if longer.
- Files: under 500 lines. Split by responsibility.
- One thing per function, one responsibility per module (SRP).
- Names: specific and unique. Avoid `data`, `handler`, `Manager`.
  Prefer names that return <5 grep hits in the codebase.
- Types: explicit. No `any`, no `Dict`, no untyped functions.
- No code duplication. Extract shared logic into a function/module.
- Early returns over nested ifs. Max 2 levels of indentation.
- Exception messages must include the offending value and expected shape.

## Comments

- Keep your own comments. Don't strip them on refactor — they carry
  intent and provenance.
- Write WHY, not WHAT. Skip `// increment counter` above `i++`.
- Docstrings on public functions: intent + one usage example.
- Reference issue numbers / commit SHAs when a line exists because
  of a specific bug or upstream constraint.

## Tests

- Tests run with a single command: `<project-specific>`.
- Every new function gets a test. Bug fixes get a regression test.
- Mock external I/O (API, DB, filesystem) with named fake classes,
  not inline stubs.
- Tests must be F.I.R.S.T: fast, independent, repeatable,
  self-validating, timely.

## Dependencies

- Inject dependencies through constructor/parameter, not global/import.
- Wrap third-party libs behind a thin interface owned by this project.

## Structure

- Follow the framework's convention (Rails, Django, Next.js, etc.).
- Prefer small focused modules over god files.
- Predictable paths: controller/model/view, src/lib/test, etc.

## Formatting

- Use the language default formatter (`cargo fmt`, `gofmt`, `prettier`,
  `black`, `rubocop -A`). Don't discuss style beyond that.

## Logging

- Structured JSON when logging for debugging / observability.
- Plain text only for user-facing CLI output.

This block fits in under 100 lines and costs about 500 tokens per iteration. Sounds like a lot, but the savings in code quality and absence of rework easily make up for it, especially if you’re on a pay-per-token API account. Use it as a base and evolve it with your own experience.

Some items (SRP, small functions, tests) the agent will try to do on its own. Others (DI, strict DRY, explicit types EVERYWHERE, aggressively unique names) it only does when you say so explicitly. And some (like “don’t strip the comments you wrote yourself on a refactor”) are so counterintuitive to its default training that without the instruction it WILL prune them. Hence the importance of having a rules file that gets read every iteration.

An analogous pattern shows up around defensive programming: the agent implements circuit breaker, retry with backoff, aggressive timeout, graceful degradation, all of it nicely — when you ask. But on its own it won’t propose them. The agent doesn’t know what your system’s operational failure points are, so it implements the happy path and waits for instruction. If your CLAUDE.md lists the categories of defensive code the project needs (rate limit, retry, breaker, fallback), the agent covers them. If it doesn’t list them, the agent doesn’t invent them. It’s another case where explicit instruction to the agent is what separates robust code from naive code.

If you use the same agent across different projects, it’s worth having a base template and adding project-specific rules on top. But start with something like the above and iterate from there.

The short version

Uncle Bob wrote Clean Code to be read by other humans. In 2026, the primary reader became the agent. The good news is that most of what the book preached still holds. The bad news is that some things that were opinions (“a file should have N lines”) became technical constraints (“a file with X lines makes the agent perform worse”). The difference is that now there’s a metric: token cost, tool-call latency, output quality. Whoever writes clean code for the agent saves API-bill money, session time, and gets less hallucination in the output.

And there’s a cultural bonus worth registering: all these practices (XP, TDD, SOLID, SRP, DI, small code, abundant tests) were falling out of fashion in the 2010s, replaced by “move fast and break things” and two-month bootcamps. Programmers who invested in fundamentals became a minority, and it became fashion to trash Uncle Bob on the internet. Turns out those very fundamentals became the technical differentiator of working with agents. Whoever kept the discipline is well served. Whoever dismissed it is now struggling to teach the agent not to commit mistakes the XP crowd had mapped out 25 years ago.

Clean code was never fashion. It became infrastructure.

Updated April 2, 2026: if you already read this yesterday, jump straight to the new update section.

This morning (March 31, 2026), security researcher Chaofan Shou discovered that the entire source code for Claude Code, Anthropic’s official CLI for AI coding, was sitting there for anyone to grab on the public npm registry. 512,000 lines of TypeScript. 1,900 files. All of it exposed through a 59.8 MB source map file accidentally bundled into version 2.1.88 of the @anthropic-ai/claude-code package.

Within hours the code had been mirrored on GitHub, picked apart by thousands of developers, and Anthropic had put out a statement calling it “human error in release packaging, not a security breach.” Which is technically true and ignores that the result is the same.

I use Claude Code every day. Some of the articles you read here, I wrote with it. So I figured I’d take a look at what’s inside. I actually started writing this piece in Claude Code itself, but my Max plan ran out before I finished. I closed the rest in Codex.

How the leak happened

Claude Code is bundled with Bun, the JavaScript runtime Anthropic acquired in late 2024. When you build with Bun, source maps are generated by default. Those .map files contain the full original source code, not just mappings. Every file, every comment, every internal constant, every system prompt.

The initial theory was that a known Bun bug had caused the leak: even with development: false, source maps were still being served and bundled in. But Jarred Sumner, Bun’s creator, shut that down: “This has nothing to do with claude code. This is with Bun’s frontend development server. Claude Code is not a frontend app. It is a TUI. It doesn’t use Bun.serve() to compile a single-file executable.” In other words, the Bun bug affects the frontend dev server, not the build process that generated the Claude Code npm package.

What actually happened is simpler: somebody at Anthropic forgot to add *.map to .npmignore or didn’t configure the bundler to skip source map generation in production builds. And worse: according to The Register, the source map didn’t just point to the original files, it referenced a ZIP hosted on Anthropic’s own Cloudflare R2 bucket. npm happily served it to anyone running npm pack, and the rest was mirroring work.

The irony is that the code contains a whole system called “Undercover Mode” built specifically to prevent Anthropic’s internal information from leaking in commits and PRs. They built a subsystem to stop the AI from revealing internal codenames, and then a source map exposed everything.

What’s inside: the hidden features

The source code reveals 44 feature flags covering functionality that’s ready but not yet shipped. This isn’t vaporware. It’s real code hiding behind flags that compile to false in external builds. Let me highlight the most interesting ones.

KAIROS: a Claude that never stops

Inside the assistant/ directory, there’s a mode called KAIROS, a persistent assistant that doesn’t wait for you to type. It observes, logs, and acts proactively on things it notices. Maintains append-only daily log files, receives <tick> prompts at regular intervals to decide whether to act or stay quiet, and has a 15-second budget: any proactive action that would block the user’s workflow for more than 15 seconds gets deferred.

KAIROS-exclusive tools: SendUserFile (sends files to the user), PushNotification (push notifications), SubscribePR (monitors pull requests). None of this exists in the public build.

BUDDY: a Tamagotchi in the terminal

I’m not making this up. Claude Code has a complete pet companion system in the Tamagotchi style called “Buddy.” A deterministic gacha system with 18 species, rarity, shiny variants, procedurally generated stats, and a “soul” written by Claude on the first hatch.

The species is determined by a Mulberry32 PRNG seeded by the hash of the userId. Same user always gets the same buddy. There are 5 stats (DEBUGGING, PATIENCE, CHAOS, WISDOM, SNARK), 6 eye styles, 8 hat options, and sprites rendered as 5-line ASCII art with animations. The code references April 1-7, 2026 as the teaser window, with full launch slated for May 2026.

ULTRAPLAN: 30 minutes of remote planning

ULTRAPLAN offloads complex planning tasks to a remote session running Opus 4.6, gives it up to 30 minutes to think, and lets you approve the result through the browser. The terminal shows polling every 3 seconds, and once approved, a sentinel value __ULTRAPLAN_TELEPORT_LOCAL__ “teleports” the result back into the local terminal.

Multi-Agent: “Coordinator Mode”

The multi-agent orchestration system in the coordinator/ directory turns Claude Code from a single agent into a coordinator that spawns, directs and manages multiple workers in parallel. Parallel research, synthesis by the coordinator, implementation by workers, verification by workers. The prompt teaches parallelism explicitly and forbids lazy delegation: “Do NOT say ‘based on your findings’ - read the actual findings and specify exactly what to do.”

And there’s more. The leak also shows in-process teammates with AsyncLocalStorage to isolate context, workers in separate processes via tmux/iTerm2 panes, memory synchronization between agents, and flags ready for BRIDGE_MODE, VOICE_MODE, WORKFLOW_SCRIPTS, AFK mode, advisor-tool and history snipping. None of that guarantees a launch, but it suggests a roadmap that’s a lot further along than the public version lets on.

The memory architecture

The memory system caught my eye. It’s not a “store everything and retrieve” approach. It’s a three-layer architecture:

MEMORY.md is a lightweight pointer index (~150 characters per line) that stays permanently loaded in the context. It doesn’t store data, it stores locations. The actual knowledge lives distributed across “topic files” fetched on demand. Raw transcripts are never reloaded into the context whole, only searched with grep for specific identifiers.

And it comes with an important discipline: the system writes to the topic file first and only then updates the index. MEMORY.md doesn’t become a fact dump. It stays a map. If you let the index turn into storage, it pollutes the permanent context and degrades the whole system.

The “Dream” system (services/autoDream/) is a memory consolidation engine that runs as a background subagent. The name is intentional. It’s Claude dreaming.

The dream has a three-gate trigger: 24 hours since the last dream, at least 5 sessions since the last dream, and acquiring a consolidation lock (preventing concurrent dreams). All three have to pass.

When it runs, it follows four phases: Orient (ls in the memory directory, read the index), Gather (look for new signals in logs, stale memories, transcripts), Consolidate (write or update topic files, convert relative dates to absolute, delete contradicted facts), and Prune (keep the index under 200 lines and ~25KB).

There are four memory types: user (user profile), feedback (corrections and confirmations), project (context about ongoing work), reference (pointers to external systems). The taxonomy explicitly excludes things derivable from the code (patterns, architecture, git history, file structure).

The dream subagent gets read-only bash. It can look at the project but can’t modify anything. It’s purely a consolidation pass.

And there’s another detail I found elegant: memory isn’t treated as truth. It’s treated as a hint. The system assumes that memory may be stale, wrong or contradicted, so the model still has to verify before trusting it. That’s the opposite of the fantasy of “throw everything in a vector database and let the magic happen.”

“Undercover Mode”

Anthropic employees (identified by USER_TYPE === 'ant') use Claude Code on public and open source repositories. Undercover Mode (utils/undercover.ts) prevents the AI from accidentally revealing internal information in commits and PRs.

When active, it injects into the system prompt:

## UNDERCOVER MODE - CRITICAL

You are operating UNDERCOVER in a PUBLIC/OPEN-SOURCE repository. Your commit
messages, PR titles, and PR bodies MUST NOT contain ANY Anthropic-internal
information. Do not blow your cover.

NEVER include in commit messages or PR descriptions:
- Internal model codenames (animal names like Capybara, Tengu, etc.)
- Unreleased model version numbers (e.g., opus-4-7, sonnet-4-8)
- Internal repo or project names
- Internal tooling, Slack channels, or short links
- The phrase "Claude Code" or any mention that you are an AI
- Co-Authored-By lines or any other attribution

There’s no way to turn it off. If the system isn’t sure it’s in an internal repository, it stays in undercover mode. That confirms something kind of uncomfortable: Anthropic uses Claude Code to contribute to open source, and the agent is instructed to hide that it’s an AI.

The internal codenames are animal names: Tengu (codename for the Claude Code project), Fennec (Opus), Capybara, Numbat (in testing). “Fast Mode” is internally called “Penguin Mode” with endpoint claude_code_penguin_mode and kill-switch tengu_penguins_off.

The most paranoid parts

There’s a piece of the analysis I almost missed because I was looking more at the hidden features. But maybe the most revealing thing about Anthropic’s mindset is in the defense mechanisms against copying and abuse.

According to Alex Kim’s analysis, there’s an anti-distillation mode that can ask the server to inject fake tools into the system prompt. The idea is to poison traffic recorded by anyone trying to distill Claude Code’s behavior to train a competitor. There’s also a second mechanism for summarizing connector text, cryptographically signed, so that part of the observable traffic doesn’t match the original raw reasoning. It’s not perfect protection. It’s another layer of friction. But it shows the company is explicitly thinking about copy-by-observation, not just traditional security.

And there’s the more aggressive part: client attestation. Every request includes a billing header with a placeholder cch=00000, and the Bun native runtime replaces that with a hash computed below the JavaScript layer. In other words, looking like Claude Code isn’t enough. The binary tries to prove that it is Claude Code. That helps explain why the fight with third-party tools like OpenCode got so sensitive: it wasn’t just a commercial or legal issue. There was technical enforcement built into the transport.

Update: the DRM died in less than 24 hours

Remember when I mentioned client attestation as “the most aggressive part”? Yeah. It lasted less than a day.

For context: Anthropic had been waging a war against third-party tools since January 2026. First came the server-side blocking of OAuth tokens from non-official clients. Then in March, the OpenCode maintainer merged a PR removing all Claude authentication from the project. The commit message was two words: “anthropic legal requests.” The Register reported that Anthropic updated its terms of service to make it explicit that OAuth tokens from Pro/Max subscriptions can only be used in the official Claude Code and on Claude.ai. People paying $100-200/month for Max who wanted to use the tool of their choice were left holding the bag.

The technical mechanism behind the block was the cch= header. With the leaked code you can see the system has two parts. The first is a version suffix: the cc_version field includes 3 hex characters derived from the user’s first message via SHA-256, using a 12-character salt embedded in the JavaScript. The second is the body hash itself: the entire body of the request (messages, tools, metadata, model, thinking config, everything) is serialized as compact JSON with the cch=00000 placeholder, then hashed with xxHash64 using a fixed seed. The result is masked with 0xFFFFF (20 bits) and formatted as 5 lowercase hex characters. The placeholder is replaced with the computed hash before the request leaves the process.

The detail that makes the difference: that substitution happens inside the Bun native runtime, written in Zig, below the JavaScript layer. Bun literally mutates the JavaScript string in-place, overwriting the 00000 bytes in the string buffer with the computed hash. If you ran the same bundle in Node or in a stock Bun, the placeholder would go to the server as-is and the request would be rejected.

And then the leak happened. With the source code exposed, @StraughterG (Jay Guthrie) announced that same night: “Yesterday I said Anthropic’s compiled Zig cch= hash was banning 3rd-party Claude clients. Tonight, the DRM is dead. We extracted the algorithm from the binary. It’s not advanced cryptography. It’s a static xxHash64 seed.”

The seed is 0x6E52736AC806831E. The full algorithm, as he explained in a thread of tweets, fits in a few lines of TypeScript:

import xxhash from "xxhash-wasm";

const { h64Raw } = await xxhash();
const body = JSON.stringify(request); // com cch=00000 no placeholder
const hash = h64Raw(new TextEncoder().encode(body), 0x6E52736ACn | (0x806831En << 32n));
const cch = (hash & 0xFFFFFn).toString(16).padStart(5, "0");
// substituir cch=00000 por cch={valor calculado}

@paoloanzn celebrated: “we cracked it. the cch= signing system in claude code is fully reverse engineered.” And he immediately put the bypass in free-code, a fork of Claude Code with telemetry removed, system prompt guardrails stripped, and all 54 experimental feature flags unlocked.

The technical point that matters: xxHash64 is not cryptography. It’s a checksum hash designed for speed, not security. The seed is static, embedded in the binary. It changes with each version of Claude Code, but within a version it’s the same for everyone. The “security” depended entirely on nobody being able to extract the seed from the compiled Zig binary. With the source code leaked, that obscurity evaporated in hours.

Now any third-party client — OpenCode, Claw-Code, whatever — can intercept the fetch(), hash the body with the correct seed, and pass the server’s validation as if it were the official Claude Code. The barrier Anthropic built to protect its $2.5 billion ARR business model was, in the end, security by obscurity over a non-cryptographic hash.

The third detail is small but says a lot about real product in production: the system detects user frustration with regex. Yes, regex. Profanity, insults, “this sucks,” that kind of thing. It’s funny to see an LLM company doing sentiment analysis with wtf|ffs|shit, but it’s also the kind of pragmatic solution you reach for when you need a cheap immediate answer, not conceptual elegance.

What the code reveals about how you use Claude Code

@iamfakeguru compiled a thread with seven technical findings from the code that any user should know:

Claude Code has a 2,000-line cap per file read. When you ask it to read a larger file, it silently truncates. Tool results are cut off at 50,000 characters. The context window compression system drops old messages to fit more new context. And there’s a difference between the access level of Anthropic employees (USER_TYPE === 'ant') and public access: internal tools like ConfigTool and TungstenTool are invisible in the external build.

The most useful finding from the thread is how Anthropic employees work around the limitations external users face. The code reveals that USER_TYPE === 'ant' unlocks internal tools, exclusive beta headers (cli-internal-2026-02-09), access to staging (claude-ai.staging.ant.dev), and a ConfigTool that lets you change configurations at runtime. External builds compile all of this to false via dead code elimination.

But the point that matters is: the CLAUDE.md you put at the root of your project gets read in full by Claude Code and injected into the system prompt. It’s literally the place where you control how the agent behaves. @iamfakeguru published a complete override with 10 mechanical rules, and then put the whole file in a separate repository: iamfakeguru/claude-md.

I’m not going to paste the whole block here. What matters is the content: it forces post-edit verification (tsc and eslint before declaring success), it imposes re-reading files before editing, it requires reading large files in chunks, it assumes silent truncation of long results, and it tells you to break larger work into phases or parallel subagents. In other words: it turns into explicit rules everything that external users were having to figure out empirically.

These aren’t magic instructions. They’re guardrails. The difference is that now we know which limits the system actually has and we can write a CLAUDE.md that works with them, not against them.

Cache bugs that cost real money

@altryne (Alex Volkov) reported cache invalidation bugs that make uncached tokens cost 10-20x more than cached ones. There are two bugs: a string substitution bug in Bun that affects the standalone CLI (workaround: use npx @anthropic-ai/claude-code instead of the installed binary), and another in the --resume flag that breaks the cache with no known workaround. Over 500 users reported similar quota exhaustion issues. If you’ve been feeling like Claude Code was burning tokens faster than expected over the past few days, it probably wasn’t your imagination.

“Staff engineer spaghetti”

The code analysis revealed real problems. A comment in the source itself admits: “1,279 sessions had 50+ consecutive failures (up to 3,272) in a single session, wasting ~250K API calls per day globally.” The fix was three lines: limit consecutive failures to three before disabling compaction.

The print.ts file has 5,594 lines with a single 3,167-line function containing twelve levels of nesting. main.tsx is 803,924 bytes in a single file. interactiveHelpers.tsx is 57,424 bytes. These are files no human can review with confidence.

The most viral reaction came from @thekitze: he asked GPT-5.4 to evaluate the codebase and the score came in at 6.5/10. The description: “This is not junior spaghetti. This is staff-engineer spaghetti: performance-aware, feature-flagged, telemetry-instrumented, surgically optimized spaghetti.” In other words, it isn’t bad code from inexperience. It’s bad code from pressure to ship fast without paying the cost of cleaning up afterwards.

@thekitze also elaborated in another thread on how the code shows a lack of basic engineering practices. And this is where I feel vindicated.

I’ve been repeating in several posts on vibe coding that speed without discipline produces exactly this. The principles I defend, small increments, tests at every step, review before committing, continuous refactoring, CI that rejects high cyclomatic complexity, are the same Extreme Programming principles that have worked since the early 2000s. Anthropic apparently didn’t follow any of them on their own product.

A 3,167-line function with 12 levels of nesting isn’t something that appears overnight. It’s accumulation. It’s the result of dozens of additions where nobody stopped to refactor because “it works, don’t touch it.” It’s the classic anti-pattern of vibe coding without discipline: generate code with AI, see that it compiles, commit, repeat. Without rigorous review. Without complexity limits in CI. Without the basic rule that if a function passes 50 lines, it needs to be broken up.

The irony is that Anthropic sells the most popular vibe coding tool on the market and doesn’t practice what I call responsible vibe coding. Claude Code is worth $2.5 billion in ARR. The code that generates that revenue is rated 6.5/10.

The “clean room” question

With the entire source code public, there’s a serious legal and competitive implication. And here I think a lot of people started using the term “clean room” with a lightness that doesn’t fit the subject.

A real clean room isn’t just “I rewrote it in another language” or “I didn’t copy and paste.” The classic model is much more annoying: one group studies the original and produces a functional spec; another group, isolated, implements from that spec without ever seeing the original code. The whole point is to reduce contamination risk.

@braelyn_ai raised another interesting point: with generative tools, somebody could try a “clean room rebuild” using tests, observable behavior and documentation, without reusing the original implementation. In theory, that makes sense. In practice, what shows up in the heat of a leak usually lands in a much grayer zone.

The Claw-Code case illustrates this well. The project presents itself as an independent rewrite and has already shifted focus to Python and Rust, but the README itself admits direct study of the exposed code and even mentions a parity audit against a local archive. So I wouldn’t call that a clean room in the strictest classic sense. I’d call it an inspired reimplementation, with a deliberate attempt to move away from the leaked snapshot.

That doesn’t mean every reimplementation is doomed. Software copyright doesn’t protect abstract ideas, generic tool flow, high-level architecture or “a CLI that does X.” It protects concrete expression. But that’s exactly why discipline matters. The more a project wants to maintain independence, the less it should rely on the leaked material as a direct benchmark.

There’s a more pragmatic detail in there: the literal copies of the leaked source will probably disappear quickly when the first DMCAs start arriving. Mirrors fall easily. That’s why a reimplementation matters more than a raw mirror. It doesn’t erase the legal discussion, but it changes the type of fight quite a bit and the chances of staying online.

That’s more or less what I did myself when I rewrote OpenClaw in Rust. The point wasn’t to copy line by line. It was to understand the behavior and rewrite the whole piece in my own code.

The satirical site malus.sh appeared today offering “Clean Room as a Service” with the tagline “Robot-Reconstructed, Zero Attribution.” The joke: AI robots recreate open source projects eliminating attribution obligations, with guarantees like “This has never happened because it legally cannot happen. Trust us.” and indemnification via an offshore subsidiary in a jurisdiction that doesn’t recognize software copyright. It’s satire, but it’s satire that describes what someone is going to actually try to do.

Update on April 2, 2026

Since the text above opens in the heat of March 31, it’s worth recording what happened right after. I decided to add this update after reading this tweet from @k1rallik, which captures the post-leak mood well but mixes verifiable facts with a touch too much epic.

First: the DMCA part got messier than it looked. The notice itself published in the github/dmca repository says GitHub processed the takedown against the entire network of 8,100 repositories, because the notification claimed that “all or most of the forks” were infringing to the same extent as the main repository. The next day, Anthropic published a partial retraction: it asked for the reinstatement of all the removed repositories, except nirholas/claude-code and 96 forks listed individually. So the thesis that the initial attempt was too broad is correct. The final picture, however, isn’t “8,100 repositories were taken down.” What happened was a formal walk-back after the mass removal.

Second: the Claw-Code project really did blow up. By the time I was updating this post, GitHub was already showing 142,829 stars and 101,510 forks. That alone is enough to say the story has moved out of the “curious fork of the leak” category and into the “real competitive side effect” category. The viral tweet that went around today is right about the size of the damage but exaggerates some details. The project’s own README describes itself as “the fastest repo in history to surpass 50K stars” and says the milestone came in two hours. I couldn’t independently confirm that historical record, so I’d rather treat that as the project’s own claim, not as a settled fact.

Third: the Rust part also needs nuance. Yes, there’s already a Rust workspace on the main branch and the Cargo.toml is at version 0.1.0. But I couldn’t find a public release on GitHub to support the line “release 0.1.0 already shipped” as a formal launch. What I can say with confidence is something else: the project already has a Python base, already has a Rust workspace, and has already drawn enough attention to keep existing even without the literal mirror of the leaked code.

What Anthropic should have done

Anthropic responded fast. They pulled the compromised package, put out a public statement, and cleaned up what they could. But the damage was done. The code was mirrored before the takedown. Mirrors on GitHub, analyses in blogs, threads on X/Twitter. There’s no way to un-publish something on the internet.

What bothers me isn’t the leak itself. Bugs happen. What bothers me is that this was avoidable with basic engineering practices:

1. Add *.map to .npmignore. One line.
2. Configure the bundler to not generate source maps in production builds. One flag.
3. Have a CI check that rejects publication if the package contains .map. A 5-line script.
4. Have a release pipeline with manual review before publishing to npm. Process, not code.

None of these are hard. They’re all the kind of thing that gets dropped when you’re moving too fast and don’t have discipline in the release process. It’s exactly what I preach as disciplined vibe coding: moving fast doesn’t mean skipping the guardrails.

And the second failure: the quality of the code itself. 512,000 lines with 3,000-line functions and 12 levels of nesting isn’t engineering. It’s accumulation. It’s what happens when you generate code with AI without rigorous review, without continuous refactoring, without CI that rejects high cyclomatic complexity. The irony of being precisely the company that sells the most popular vibe coding tool in the world doesn’t go unnoticed.

Sources

• Kuberwastaken/claude-code - Complete breakdown of the leaked code
• Alex Kim - Claude Code Source Leak: fake tools, frustration regexes, undercover mode
• VentureBeat - Claude Code’s source code appears to have leaked
• The Register - Anthropic accidentally exposes Claude Code source code
• Fortune - Anthropic leaks its own AI coding tool’s source code
• Cybernews - Full source code for Anthropic’s Claude Code leaks
• Gizmodo - Source Code for Anthropic’s Claude Code Leaks at the Exact Wrong Time
• Anthropic - Anthropic acquires Bun as Claude Code reaches $1B milestone
• Bun Issue #28001 - Source maps incorrectly served in production
• Hacker News - Claude’s system prompt is over 24k tokens with tools
• malus.sh - Clean Room as a Service (satire)
• @iamfakeguru - Thread with 7 technical findings from the code
• @altryne - Cache bugs that cost 10-20x more
• @thekitze - “Staff-engineer spaghetti” 6.5/10
• @braelyn_ai - Clean room and legal implications
• GitHub DMCA - Anthropic takedown notice processed against the network of 8.1K repositories
• GitHub DMCA - Anthropic’s partial retraction the next day
• ultraworkers/claw-code - Python and Rust reimplementation that became the main post-leak project
• @mem0ai - Analysis of the memory architecture
• @himanshustwts - Memory architecture summary
• iamfakeguru/claude-md - Override published with the complete CLAUDE.md
• @StraughterG - “the DRM is dead” - reverse engineering of the cch= hash
• @StraughterG - xxHash64 seed and TypeScript bypass code
• @paoloanzn - “we cracked it” - confirmation of the reverse engineering
• paoloanzn/free-code - Fork of Claude Code with telemetry removed and features unlocked
• a10k.co - What’s cch? Reverse Engineering Claude Code’s Request Signing
• The Register - Anthropic clarifies ban on third-party tool access to Claude

–

I published the “final post” of the Behind The M.Akita Chronicles series 10 days ago. 274 commits, 1,323 tests, deployed to production. I wrote down the lessons, did the conclusion, dropped the quote at the end. Done.

125 post-production commits later, I can confirm: software “done” doesn’t exist.

Today the M.Akita Chronicles repo has 335 commits and 1,422 tests. Tomorrow, Monday, subscribers get the 3rd consecutive newsletter — generated, reviewed, and sent by a system that won’t stop evolving. Meanwhile, Frank Sherlock went from 50 commits and v0.1 to 103 commits and v0.7 with face detection. FrankMD got 3 external contributors and shipped v0.2.0. And even FrankMega — a project built in 1 day — needed fixes when real users showed up.

Throughout February, I published more than a dozen posts detailing the build of each project. This post is different. It isn’t about building from scratch, I covered that already. It’s about what happens after. And what happens after destroys the one-shot prompt narrative. Software needs an experienced human at the wheel. Iterative development is the only thing that works. Anyone who disagrees hasn’t put a system in production yet.

I’ll prove it with git log.

Life After Deploy

Let’s get into the details.

The M.Akita Chronicles: 3 Weeks in Production

The newsletter system has been live since February 16. It’s already sent 2 newsletters and tomorrow it sends the third. 56 commits have happened since the “final post”:

Features Nobody Saw Coming

Steam Gaming. Wasn’t in the original plan. It was born because the entertainment section only had anime — and gamer readers were left out. Result: 6 commits just on launch day. The first commit (bf41e25) added the whole section — Steam API service, generation job, tests, Hugo shortcode. The next 5 fixed things: Portuguese date parsing, ranking offset so the top 10 didn’t repeat, wishlist filter for releases, dark theme. No spec in the world predicts that the Steam API returns dates with Portuguese abbreviations ("fev", "mar", "abr") when you ask for l=brazilian. You find that out when the parser breaks in production.

Preview system. /preview was born because I needed to see how the auto-generated content looked before publishing the entire newsletter. Seems obvious in retrospect, but in v1 the only way to validate was to generate everything and look at the final result. 5 commits to build a preview system with 9 sections, aliases (hn for hacker_news, steam for steam_gaming), separate comment-preview mode, and rescue when one section fails without taking the others down (d23c725 — because a section with an error was killing the entire preview).

Marvin’s moods. Marvin (the bot’s sarcastic persona) suffers from a chronic LLM problem: smoothing. Without intervention, every comment ends up the same lukewarm tone. The iterative fix was a mood system — 9 modes the operator picks per story:

MARVIN_MOODS = {
  "sarcastic"   => "Be extra sarcastic and biting in your commentary.",
  "ironic"      => "Use irony and dark humor to make your point.",
  "grounded"    => "Be more neutral and journalist-like, factual and measured.",
  "provocative" => "Be provocative and controversial, challenge assumptions.",
  "counter"     => "Take the opposite position from Akita's comment.",
  "insightful"  => "Find a non-obvious angle -- a historical parallel, a second-order consequence.",
  "positive"    => "Find the genuinely positive angle.",
  "hopeful"     => "Be cautiously optimistic. Acknowledge the good potential.",
  "negative"    => "Be extra pessimistic and bleak."
}.freeze

None of this came from a spec. It came from 3 weeks of reading bland comments and thinking “this isn’t good enough”. There are 7 commits just for prompt tuning across 10 days. I banned formulaic patterns (d4bd3a6). I killed off “Ah,” as an opener (4171628). I removed Marvin from entire podcast sections because he was contaminating the tone (246cc60). I pushed for substance instead of puns (8bc47c4). Each prompt commit is a micro-correction that only makes sense after reading the previous output.

Bugs That Only Exist in Production

Of the 56 commits, 13 are bug fixes. Some favorites:

• Gmail clipping (3258f5b): Gmail silently cuts off emails larger than 102KB. I found out when the 2nd newsletter went out longer and Gmail readers didn’t see the end. Fix: a size check in the content preflight that warns before publishing. I shrunk the history sections from 10 to 5 items (51ce528) to fit the limit.

• False paywall detection (1fd8739): the scraper marked sites as paywalled when it found accidental block text in a generic footer. Only showed up when the story list grew to dozens of real sources.

• t.co link mangling (6bd056f): Twitter shortens URLs with t.co, but it also auto-links file names that look like domains. config.yml becomes a link to the config.yml domain. 158 lines of fix to handle tweet text correctly.

• TTS language revert (7803ad9 -> c1dd668): I tried switching the TTS language from “Portuguese” to “Auto”. The model produced a mixed accent. Reverted the same day.

• Podcast section ordering (5bef18c): podcast sections came out in the wrong order. 3-line fix + untracking of 1,504 lines of generated content that had been committed by accident.

• Elementor sites with multiple <article> (e70b756): sites built with Elementor use the <article> tag in a non-semantic way. The content extractor was grabbing the wrong block. 171 lines of fix + tests.

None of these bugs could have been predicted in a spec. They only exist because the system is live, processing real data, hitting real APIs, being read by real people.

From 1,323 to 1,422 Tests

The test suite kept growing along with it:

New feature? Comes with a test. Bug fix? Comes with a regression test. Without that, the 56 post-production commits would be 56 chances to break something that worked yesterday. TDD isn’t a phase, it’s a habit.

The Model Changed: From Claude to Grok-4

Commit 8f9d11c: Switch default model to x-ai/grok-4. The architecture had isolated the LLM model into an environment variable from day one. But the model name was still hardcoded across 24 files. Result: commit c8c688e — Centralize LLM model config — 24 files touched to make swapping models a 1-line config change.

This kind of refactoring only shows up in operation. When you start the project, you don’t know you’re going to change models. When you change, you want it to be trivial. And the cleanup to make it trivial? Only happens when the pain shows up.

Frank Sherlock: From v0.1 to v0.7 in 4 Days

The Frank Sherlock post covered the first 2 days and ~50 commits: the benchmark research, the Tauri app scaffold (Rust + React), the classification pipeline with Ollama, and the v0.1.0 release with binaries for Linux, macOS and Windows.

What the post didn’t cover: the next 53 commits, in 4 days, that took the project from v0.3 to v0.7:

Face Detection: A Feature Born from Iteration

v0.1 was an image indexer with LLM-based classification. v0.7 has face detection with 512-dimensional embeddings, cosine-similarity clustering, and per-person search (face:alice). None of that was in the original plan. It came up because I started using the app with real photos and thought “I want to find every photo of so-and-so”. The feature was born from use, not from a spec.

The path to get there was methodical:

1. f34132f: A/B testing framework for face detection — benchmark before implementing, not after
2. ef3be82: A/B results (SCRFD + ArcFace won)
3. 6d9174a: Native implementation with ONNX Runtime — no Python, no external dependency
4. 3b25eaf: Clustering, person management, complete FacesView
5. a02d67f: Refactoring — extract helpers, delete dead code, share CSS

Benchmark -> implement -> refactor. The same cycle as always. And something no one-shot prompt produces.

Video Support: Another Emergent Feature

The app was built for images. But real media folders have videos. Commit a67a2f9 added full support: scanning of 11 formats (MP4, MKV, AVI, MOV, WebM…), keyframe extraction for LLM classification, black frame skipping, .srt subtitle parsing for the full-text index, and inline preview with HTTP Range streaming.

1,626 lines added in one commit. An entire feature born from real use, not from a spec.

7 Releases, 3 Platforms, Automatic AUR

In 7 days, Frank Sherlock had 7 releases. Each one with binaries compiled for Linux (AppImage), macOS (DMG with notarization and code signing), and Windows (MSI). The CI/CD pipeline (release.yml) does everything automatically. Starting at v0.5.0, an additional workflow (aur-publish.yml) publishes automatically to the AUR — the Arch Linux package repository.

Final tally: 17,210 lines of Rust, 10,863 lines of TypeScript, 621 tests (322 Rust + 299 frontend). Cross-platform. Releases published and installable by any user. One week, one dev, one AI agent.

The CI/CD cross-platform part is where the “works on my machine” fantasy dies. There were 17 commits just for builds: Windows UNC paths, macOS signing with notarization, Linux AppImage, release workflow permissions, macOS Intel target removed. No LLM in the world knows that macOS requires specific entitlements for hardened runtime, or that paths on Windows start with \\?\. That kind of deploy work demands someone who’s been through that pain before.

FrankMD: Real Open Source

FrankMD was the first project in this saga, a self-hosted Markdown editor with Rails 8. The February posts covered the build. What happened after is more interesting: other people started contributing.

14 commits since February 20. 3 external contributors. 4 PRs merged. v0.2.0 release on the 28th:

The Maintainer’s Pattern

What’s most telling isn’t the contributions themselves, but what I did after every merge.

When @LuccaRomanelli submitted the Omarchy theme sync PR (+308 lines), I merged it and immediately committed fix: harden Omarchy theme integration and fix broken tests — +163 lines of fixes and tests in the next commit. The contributor implemented the feature. The maintainer hardened it.

When @rafaself submitted the table hint fix, there were 3 iterative commits the same day — enhance, improve, streamline — showing the same progressive refinement pattern I do with the AI agent. The next day, he sent a separate commit updating faraday, nokogiri and rack for security advisories.

On the 28th, I sat down and did everything in one shot: merged the 4 PRs, committed the hardening, updated 5 gems (brakeman, bootsnap, selenium-webdriver, web-console, mocha), and published the v0.2.0 release notes. A classic “release day”.

FrankMD today has 226 commits, 1,804 tests (425 Ruby + 1,379 JavaScript), and active contributors. It isn’t “my little personal project” anymore. It became software with a community. And community doesn’t show up for a test-less prototype that “works on my machine”. It shows up for a project with green CI, documentation, versioned releases, and code you can actually read.

FrankMega: Even the Smallest Project Needs Post-Production

FrankMega was built in 1 day. 26 commits, secure file sharing with Rails 8, 210 tests, deploy via Docker + Cloudflare Tunnel. Post published the same day. Done, right?

Three days later, 2 commits:

• db4bb705 — Add macOS, Linux, and Windows package MIME types to seed defaults
• b0c4829a — Fix MIME types to match Marcel detection, add normalizes strip

Users tried to share .dmg, .deb, and .msi files. The MIME type detection (via the Marcel gem) didn’t recognize those formats because they weren’t in the seed defaults. Two commits. 22 lines. 15 minutes.

But without them, FrankMega couldn’t be used to share installer packages — which is exactly the most common use case on a dev’s home server.

No prompt in the world predicts that your users will want to share .deb files on day one.

The simplest project of all, with the shortest post-production. And even then, it needed iteration.

The Consolidated Numbers

692 commits. 4,057 tests. 4 projects in production. February 2026. Me and an AI agent.

Why One-Shot Prompting Is a Myth

Look at the 125 post-production commits and tell me which one of them could have been predicted by a spec.

Gmail cuts off emails larger than 102KB, and you only find that out when you send the first long email. The Steam API returns dates with Portuguese abbreviations ("fev", "abr"), and you only find that out when the parser breaks on generation Sunday. macOS users try to share .dmg files and can’t because the MIME type isn’t in the seed. Marvin opens every comment with “Ah,” and you only notice it after reading 30 in a row and wanting to claw your eyes out. The LLM model has to be swappable with 1 line of config, but it’s spread across 24 files. Windows uses UNC paths starting with \\?\, and CI explodes in your face. And TTS in “Auto” mode? Generates an accent that sounds like Lisbon Portuguese trying to be carioca.

None of this is “debugging” in the traditional sense. It’s navigating a problem space that only reveals itself when the software meets reality. Each one of these was a real-time decision made by someone with context.

The one-shot prompt fantasy is that, if you write a sufficiently detailed spec, the AI produces the perfect software. But the perfect spec would require you to know in advance everything that’s going to go wrong. And if you knew in advance everything that’s going to go wrong, you wouldn’t need the spec — you’d already have the software done in your head.

Good software is the result of hundreds of micro-decisions made with the system running. Not a single macro-decision made before the first line is written.

The Role of the Experienced Developer

AI accelerated all of this. Without it, 692 commits in a month would be impossible for one person. But acceleration without direction is just faster entropy.

In every project there were decisions the AI couldn’t have made on its own. Switching from Claude to Grok-4 because the previous model was weak in a specific domain. Benchmarking SCRFD against alternatives before implementing face detection, because the wrong choice would cost days. The +163 lines of hardening I committed immediately after merging the Omarchy PR, because I saw where it was going to break. The TTS revert from Auto mode the same day, because I know that “Auto” on a TTS model for Brazilian Portuguese will produce a mixed accent.

The circuit breaker case is the most illustrative. When I added rate limiting, the Brave Search API started returning 429s every once in a while. If I had asked, the agent would have implemented a retry with exponential backoff. But I didn’t ask for retry. I asked for a circuit breaker:

class WebSearcher
  CIRCUIT_BREAKER_SECONDS = 120

  def self.search(query, max_results: 5)
    return [] if circuit_open?

    response = brave_search(query, max_results)

    if response.code == 429
      trip_circuit!
      Rails.logger.warn("WebSearcher rate limited (429)")
      return []
    end
  rescue Net::OpenTimeout, Net::ReadTimeout => e
    trip_circuit!
    []
  end
end

If the API returns 429 or times out, it stops trying for 120 seconds. No retry, no backoff, no queue. Why? Because I know the cron runs every day at 8am and that on newsletter generation Sundays the query volume triples. A herd of retries delaying everything is worse than empty results in one section.

The AI implements a circuit breaker when you ask it to. But it isn’t going to ask to implement a circuit breaker. It doesn’t have the operational context. That’s knowledge that comes from running systems in production for decades.

The agent writes the code. I decide what code to write. And that decision requires experience no prompt can replace.

The Lessons

1. Software in production diverges from the spec in hours, not months. The first M.Akita Chronicles bugs showed up in the first real newsletter. FrankMega’s MIME types broke on the first uploads. Anyone who thinks deploy means done is living in a world that doesn’t exist.

2. Post-production isn’t “maintenance”. 56 commits in 10 days on the M.Akita Chronicles aren’t patches. They’re new features (Steam Gaming, preview, moods), architecture refactoring (LLM config centralization), security hardening (rate limiting). That’s development. The software didn’t stop evolving just because I said “done” in a blog post.

3. TDD protects evolution. 99 new tests in 10 days. FrankMD’s 1,804 tests let me merge 4 external contributor PRs without fear. Without tests, every merge is Russian roulette.

4. Small releases keep you sane. 7 Frank Sherlock releases in 7 days. Green CI, compiled binaries, release notes. Something broke? Roll back one version. Compare that with “6 months + big bang release” and tell me which one works better.

5. Community shows up for real projects. Nobody sends a PR to a test-less prototype that “works on my machine”. FrankMD got 3 contributors because it has green CI, documentation, and versioned releases.

6. The developer’s experience is the bottleneck, not the AI’s speed. 692 commits in a month. But every commit that mattered required decades of experience to know it was needed. The AI types fast. I know what to type.

7. One-shot is for demos. Iteration is for production. If the goal is a 10-minute video showing a “finished SaaS”, one-shot will do. If the goal is software that survives contact with real users, only iteration works. And sustainable iteration demands discipline: TDD, CI, small releases, continuous refactoring. No shortcut.

To the Senior Still Sitting on Their Hands

So far I’ve been hitting the amateurs who think they can fire off a prompt and the AI spits out a finished system. Fair enough. But there’s another group that worries me just as much: the senior developer who saw the AI mess up three times, declared “this is useless”, and went back to doing everything by hand.

I get the reasoning. The AI hallucinates, generates code with subtle bugs, suggests over-engineered solutions. All true, and I documented every one of these problems in the previous posts. But tell me something: have you never done exactly the same thing? Have you never spent 2 hours reading documentation only to find out it was a typo in the config? The difference is that when the AI gets it wrong, you catch it in the tests and fix it in minutes. When you get it wrong on your own, it takes you the same amount of time to make the mistake and even longer to find it, because you trust your own code.

Let’s look at the concrete numbers of what I shipped in February, one person and one AI agent:

Now do the math in your head. How long would it take you to build Frank Sherlock on your own? A Tauri app from scratch, with an LLM classification pipeline in Rust, OCR via Python, full-text search with FTS5, duplicate detection by perceptual hash, face detection with native ONNX, video support with keyframe extraction, CI for 3 platforms with macOS code signing and notarization, auto-update, and automatic publishing to the AUR. With 621 tests. In Rust, which doesn’t forgive.

To be honest: without AI, a good senior dev would take at least 3-4 weeks on this, probably more. I did it in 7 days, alone, and every release is published with binaries anyone can download and install.

I already estimated The M.Akita Chronicles in the previous post: ~200 user stories. In Scrum with a senior team of 2-3 devs, no impediments, that would be 10-15 weeks. I did it in 8 days. Today, 3 weeks later, the system keeps running, evolving, with 99 more tests than when it “was done”.

FrankMega is more modest, but secure file sharing with Rails 8, I18n in 2 languages, 22 security issues fixed, Docker deploy, 210 tests. I did it in 1 day. A good senior, without AI, would do it in 1-2 weeks at best.

And I’m not talking about disposable prototypes. People from outside send PRs to FrankMD. Real subscribers get the M.Akita Chronicles newsletter every Monday. Anyone can download Frank Sherlock from GitHub Releases or the AUR. Green CI, Brakeman clean, real tests, automated deploys. That’s production software, not a conference demo.

“Yeah, but I write better code without AI.” Maybe. But how long does it take you? What I showed here isn’t that AI writes perfect code. Far from it. It’s that with TDD, CI, pair programming with the agent, and continuous refactoring, the end result is production code with quality. 4,057 tests are there to prove it. Brakeman clean. 125 post-production commits show the code can take evolution without turning into a ball of mud.

And using AI here and there to generate a snippet of code, like glorified autocomplete, also isn’t the answer. You’re leaving 90% of the gain on the table. What I did in February was full-time pair programming with an agent. From the first commit to production deploy. With the same discipline I’d use with a human pair. Result: 4 projects in production in 1 month, with quality I put my name on. Because I did put my name on it.

If you’re a senior and you’re still waiting for AI to “get better” before you really start using it, here’s my message: it’s already good enough. The 692 commits are there to prove it. The bottleneck now is you learning to work with it.

Conclusion

In February 2026, I built 4 projects from zero to production with AI. But the build is the easy part. What separates real software from a demo is the 125 commits that came after, when the bugs nobody predicted appeared, when external contributors sent PRs that needed hardening, when new features came up from real use and not from a requirements spreadsheet.

AI made me absurdly more productive. Without it, Frank Sherlock wouldn’t have face detection in 7 days. Without it, M.Akita Chronicles wouldn’t be in its 3rd week of operation with 1,422 tests. The speed is real.

But none of the decisions that mattered came from the AI. Switching models, reverting the TTS the same day, looking at the contributor PR and seeing where it was going to break, asking for a circuit breaker instead of retry. All of that was me. The AI executed. The decisions were mine.

The AI is the accelerator. Extreme Programming techniques (TDD, small releases, pair programming, continuous refactoring) are the brake and the steering wheel. Without discipline, AI produces fast code that piles up technical debt even faster. With discipline, AI produces software that actually evolves, week after week.

692 commits. 4,057 tests. 4 projects in production. And tomorrow, Monday, at 7am, M.Akita Chronicles subscribers get the 3rd newsletter. Generated, reviewed, and sent by a system that will never be “done”. Because finished software is dead software.

“Finished software is dead software. Live software iterates.”

And make sure to subscribe to my new newsletter The M.Akita Chronicles!

–

Unit tests are every developer’s comfort zone. You mock everything, isolate everything, run it in milliseconds, and the green in the terminal gives you a false sense of security. Until the day you deploy and discover that service A writes in a format service B can’t read. Or that the Hacker News RSS switched from Atom to RSS and your parser blew up. Or that Yahoo Finance started returning 429 after the third ticker because the rate limit that worked in tests doesn’t exist in real life.

Unit tests prove the pieces work in isolation. Integration tests prove the system works. And in a project with multiple applications sharing data via the filesystem, the difference between the two is the difference between “the code compiles” and “the newsletter lands in the inbox”.

The Problem: Three Apps, One Filesystem, Zero Guarantees

The project has an unusual architecture: three Rails/Hugo applications in a monorepo, integrated through a shared content/ directory:

akitando-news/
├── marvin-bot/    # Gera conteúdo (Discord bot, AI, scraping)
├── newsletter/    # Monta e envia a newsletter
├── blog/          # Hugo + Hextra → Netlify
├── podcast-tts/   # Servidor TTS (Python/FastAPI)
└── content/       # Markdown + YAML, a cola entre tudo

marvin-bot writes Markdown with YAML frontmatter into content/anime/, content/hacker_news/, etc. The newsletter reads those files to assemble the newsletter. The blog serves the same content as static pages. The podcast reads the assembled newsletter to generate the dialogue script.

The contract between these systems is implicit. There’s no schema. No versioned API. No protobuf. The contract is: “the Markdown file has this frontmatter, with these fields, in this structure”. If marvin-bot changes the format of a field — say, switches image from string to array — the newsletter breaks silently. No unit test in either app detects it.

That’s the fundamental flaw of testing distributed systems with unit tests: you’re testing each side of the contract separately, but never the contract itself.

Layer 1: Unit Tests With Paranoid Isolation

Before talking about integration, I need to explain what the unit tests do in this project — because without that foundation, the integration tests would have nothing to stand on.

The 977 marvin-bot tests run in parallel using every core on the machine. Each test worker has its own directory:

parallelize(workers: :number_of_processors)

parallelize_setup do |worker|
  Rails.application.config.content_dir =
    Rails.root.join("tmp", "test_content_w#{worker}").to_s
end

But worker-level isn’t enough. Inside each worker, multiple tests run sequentially — and if one test leaves a file in content/stories/, the next test might find spurious data. The solution: one unique directory per test:

setup do
  @base_content_dir = Rails.application.config.content_dir
  @test_content_dir = File.join(@base_content_dir, SecureRandom.hex(8))
  Rails.application.config.content_dir = @test_content_dir

  FileUtils.mkdir_p(File.join(@test_content_dir, "stories"))
  FileUtils.mkdir_p(File.join(@test_content_dir, "newsletters"))
  FileUtils.mkdir_p(File.join(@test_content_dir, "images"))
end

teardown do
  FileUtils.rm_rf(@test_content_dir) if @test_content_dir
  Rails.application.config.content_dir = @base_content_dir if @base_content_dir
end

Pay attention to the teardown line that restores @base_content_dir. Without it, the next test would create its directory inside the previous test’s directory — tmp/test_content_w3/abc123/def456/ — and by test 50 you’d have a path 50 levels deep. I learned this burning an entire afternoon debugging why the tests kept getting slower as the suite progressed.

This pattern — worker-level base + per-test subdirectory + restore on teardown — is identical across both apps. And it’s what lets you run 977 tests in 7 seconds with 32 parallel workers.

The golden rule: the test content_dir must NEVER point to the real ../content. If it does, the teardown’s rm_rf wipes out your production prompts. This isn’t hypothetical — it happened during development.

Layer 2: The Integration Environment

Unit tests mock everything: external APIs, AI, Discord, SES. That’s correct — you don’t want to burn $2 in GPT-5.2 tokens every time you run bin/rails test. But it means the tests never exercise the real pipeline.

The project has a third environment beyond development and test: integration. It’s not test. It’s not staging. It’s a specific mode for running the full pipeline with real data, using real APIs, but without sending actual emails:

# config/environments/integration.rb

# Jobs rodam inline (perform_later vira perform_now)
config.active_job.queue_adapter = :inline

# Emails salvos em arquivo ao invés de enviados via SES
config.action_mailer.delivery_method = :file
config.action_mailer.file_settings = {
  location: Rails.root.join("tmp/integration_emails")
}

The queue_adapter = :inline is crucial. In production, perform_later enqueues the job into SolidQueue and a separate worker executes it. In integration, it runs right there, in the same process. That turns the asynchronous production pipeline into a synchronous pipeline you can run, observe, and debug sequentially.

SES is swapped for writing to disk — every email becomes an HTML file in tmp/integration_emails/. After running the pipeline, you open those HTMLs in the browser to visually validate that the email renders correctly.

Layer 3: DevCache — LLMs Aren’t Idempotent

Here’s a problem anyone working with AI in production knows: calling GPT-5.2 with the same prompt twice doesn’t return the same answer. And every call costs money. If you’re iterating on the integration pipeline — running, tweaking a prompt, running again — spending $3 per iteration gets expensive fast.

The solution is DevCache: a file-based cache that only activates in the integration env:

module DevCache
  CACHE_TTL = 1.day
  CACHE_ENVS = %w[integration].freeze

  def self.fetch(namespace, key)
    return yield unless enabled?

    path = cache_path(namespace, key)
    if path.exist? && File.mtime(path) > CACHE_TTL.ago
      return JSON.parse(path.read, symbolize_names: true)
    end

    result = yield
    path.write(JSON.generate(result))
    result
  end
end

In production and in test, DevCache.fetch runs the block directly. In integration, it saves the result as JSON and returns the cached value on the next run. TTL of 1 day — enough to iterate within the same day without recomputing, short enough to not serve stale data.

The generation code uses it like this:

summary = DevCache.fetch("summarize", url_key) do
  AiChat.summarize(article[:content], source_url: url)
end

First run: calls GPT-5.2, pays $0.02, saves to tmp/dev_cache/summarize/{md5}.json. Second run: reads from the cache, zero cost. FORCE=1 busts the cache when you want to force regeneration — as an explicit flag, not a default.

DevCache exists in both apps (marvin-bot and newsletter) with identical implementation. When you run integration:clean, both caches are wiped.

The point: LLM caching in a development environment is a requirement, not an optimization. Without it, iterating on a prompt that needs 8 API calls becomes financially unworkable.

Layer 4: Real Data via Rsync

This is where most projects stop. They have unit tests, maybe a staging environment, and ship it. The problem: staging uses test data. Test data is too clean. It doesn’t have the edge cases real data has:

• Articles with titles in Japanese (anime ranking)
• URLs that redirect 3 times before reaching the destination
• Images that return 200 with Content-Type text/html (CloudFlare error page)
• Previous newsletters with slightly different frontmatter formats
• Stories with score: null because they were created before the field existed

The solution: rsync from production.

# Limpa conteúdo gerado (preserva stories e imagens de produção)
cd marvin-bot && bin/rails integration:clean

# Sincroniza content/ de produção pra local
rsync -avz --delete production:/path/to/content/ ../content/

# Roda o pipeline completo com dados reais
RAILS_ENV=integration bin/rails integration:pipeline

integration:clean is surgical: it removes generated directories (anime/, hacker_news/, newsletters/, podcasts/) but preserves stories/ and images/ — which are user-submitted data coming in via Discord. After the rsync, the local directory is an exact copy of production.

The pipeline then generates every section from that real data. If a parser breaks on a Korean title, you find out here — not in production at 5pm on a Sunday when the 8 jobs run in parallel.

Layer 5: The Full Pipeline

The heart of the integration system is integration:pipeline. It simulates the entire production week in a single run:

RAILS_ENV=integration bin/rails integration:pipeline

The pipeline runs in waves, honoring dependencies:

Wave 1 (paralelo, 8 jobs):
  Book, Holidays, History, Geek History,
  Anime, Hacker News, YouTube, World Events

Wave 2 (depende de World Events):
  Market Recap

Wave 3 (depende de tudo):
  Closing Remarks

Preflight → Newsletter Assembly → Podcast

Each wave runs its jobs in parallel threads, exactly like production does with SolidQueue. Parallel execution isn’t an implementation detail here — it’s a test feature. If two jobs try to write to the same file, or if one job reads a file another is in the middle of writing, the bug shows up here.

threads = to_run.map do |name, klass|
  Thread.new do
    ActiveRecord::Base.connection_pool.with_connection do
      run_single_job(name, klass, target_date, results)
    end
  end
end
threads.each(&:join)

with_connection is necessary because each thread needs its own SQLite connection — without it, connections leak and the pool gets exhausted.

At the end, the pipeline prints a status table and a billing summary:

That billing summary isn’t vanity — it’s cost control. If a refactored prompt triples the output tokens, you see it here before going to production and burning 10x on a Sunday with 200 stories.

Layer 6: Preflight — Automated Structural Validation

After generating every section, the pipeline runs ContentPreflight. It doesn’t validate the content — it validates the structure. Each section gets checked:

• Does the file exist? If content/anime/2026-02-16.md doesn’t exist, something failed.
• Minimum item count? The anime section needs at least 3 items. If it has 1, the generator had a problem.
• Required markers? The Markdown must contain [COMMENTARY] and [AKITA] — without them, newsletter assembly doesn’t know where to cut.
• Specific data? Market recap needs at least 5 ticker rows.

The result is a status per section: pass, degraded, fail, or skip. The pipeline uses this to decide whether the newsletter can be assembled automatically or needs human intervention:

In production, this result goes to Discord as an embed. In the integration pipeline, it’s printed to the terminal. The logic is the same — the notification channel changes.

Layer 7: Cross-App — Newsletter Assembly

Here’s where the monorepo pays dividends. marvin-bot needs to invoke the newsletter to assemble the final newsletter. In production, these are different servers. In the integration pipeline, it’s a system() call:

env_vars = "RAILS_ENV=integration CONTENT_DIR=#{Shellwords.escape(content_dir)}"
cmd = "cd #{newsletter_dir} && #{env_vars} bin/rails integration:generate_all"
system(cmd)

The newsletter reads the files marvin-bot just generated in content/, assembles the newsletter, and saves the result in content/newsletters/. CONTENT_DIR is passed explicitly — both apps point to the same directory.

If the frontmatter format changed and the newsletter can’t parse it, the error shows up here. If the newsletter’s SectionParser has diverged from marvin-bot’s SectionParser (they’re identical copies, but maintained by hand), here’s where you find out.

After the newsletter, the podcast: the pipeline invokes podcast:integration, which generates the dialogue script from the assembled newsletter, synthesizes audio via TTS, and builds the final MP3. It’s the longest pipeline — from Markdown newsletter to Spotify-ready MP3, passing through a two-pass LLM and per-turn voice synthesis.

The Value of Testing With Production Data

Let me give concrete examples of bugs that only showed up with real data:

1. Japanese anime titles broke the slugify. ContentWriter.slugify used Rails’ parameterize, which strips non-ASCII characters. Title: “進撃の巨人 Season 4” → slug: “season-4”. Two anime with the subtitle “Season 4” collided on the filesystem. It never showed up in tests because the fixtures used English titles.

2. The HackerNews parser assumed RSS, but the feed switched to Atom. rss/channel/item became feed/entry. Unit tests mocked the XML. The parser itself was never tested against the real feed. Running integration with the real rsynced feed, the parser returned zero items — caught by preflight as fail.

3. Yahoo Finance returned 429 after the 5th ticker. Rate limiting didn’t exist in tests (HTTP was mocked). In the real integration, the first 5 tickers went through, then everything failed. The fix — sleeping between requests — would never have been implemented without real data exposing the problem.

4. Images with a valid URL but the wrong Content-Type. A CDN returned 200 for image URLs, but with an HTML body (Cloudflare error page). UrlValidator.reachable? returned true. ImageProcessor.download_image downloaded the HTML and tried to resize it. mini_magick crashed with an obscure error. The fix: validate Content-Type on the HEAD response — but this scenario would never surface with mocked URLs.

5. Newsletter assembly failed on stories with score: null. Old stories, created before the score field had a default, had nil in the frontmatter. Assembly tried to do sort_by { |s| s[:frontmatter]["score"] } — nil <=> "high" raises ArgumentError. It only appeared when I rsynced production data that had 3 months of accumulated stories.

None of these bugs would be caught by a unit test. All of them were caught by the integration pipeline with real data.

Practical Tips

1. Separate Cost From Fidelity

The full integration pipeline costs around $0.40 in API calls. Not much, but if you run it 10 times in a day while iterating, that’s $4. DevCache solves this: the first run pays, the following runs are free.

But there are scenarios where you want to pay again — when you change a prompt and need to see the real result. FORCE=1 busts the cache selectively. And SKIP_EXISTING=1 skips sections that have already been generated, saving time when you only want to re-run a specific section.

2. The Integration Environment Is Not Staging

Staging is a server running 24/7 simulating production. The integration environment is a local execution mode you turn on when needed and turn off when you’re done. No dedicated infrastructure required. No separate database required. Same code, same machine, different flags.

The difference is philosophical: staging tests “does the system work on the staging server?”. Integration tests “does the system work with this specific data?”. The first is environment-dependent. The second is data-dependent. And data bugs are more common than environment bugs.

3. Clean Before Rsync, Not After

Order matters: clean the generated content before syncing from production. If you do the opposite — rsync first, clean after — you lose the production data you just downloaded.

integration:clean is surgical by design. It removes anime/, hacker_news/, newsletters/ (generated), but preserves stories/ and images/ (production). That enables an iterative workflow: rsync once, run the pipeline, tweak a prompt, run again — without re-downloading 500MB of content.

4. Preflight Is Your Smoke Test

Don’t trust “ran without exception = correct”. A generation job can run, produce a Markdown with 2 items instead of 10, and return success. Preflight catches that by verifying minimum counts, required markers, and expected structure.

In practice, every LLM bug I’ve found was caught by preflight: truncated responses, wrong format, missing sections. GPT-5.2 is good, but it isn’t deterministic — and when it fails, it fails in creative ways no schema validation would catch.

5. Billing Summary Is Not Optional

If you’re using LLMs in production, every test pipeline needs to report cost. Not for accounting — for engineering. A prompt that works but costs 3x more than the previous one is a performance bug. Input and output tokens per provider, cost per job, total cost — all visible on every run.

6. Monorepo Helps, But Doesn’t Solve

Having the three apps in the same repository helps: a git bisect covers changes on both sides of the contract. A PR that changes the frontmatter format in marvin-bot can also update the parser in newsletter — in the same commit.

But “can” isn’t “guarantees”. The contract between the apps is file-based and implicit. There’s no shared type, no common interface. SectionParser is copied manually between the two apps. If someone edits one and forgets the other, they diverge. The integration test is the only verification that both sides agree.

7. Preview Before Sending

The pipeline doesn’t end at generation. The newsletter has a newsletter:preview that renders the final result as HTML in light and dark mode:

cd newsletter
FILE=../content/newsletters/2026-02-16.md bin/rails newsletter:preview
# → tmp/preview/newsletter_light.html
# → tmp/preview/newsletter_dark.html

Opening those HTMLs in the browser is the visual integration test. The Markdown can be perfect, the frontmatter correct, every field present — but an anime image shows up stretched because the aspect ratio changed. Or Marvin’s commentary block doesn’t have enough contrast in dark mode. Those bugs are visual — no automated test catches them.

8. The CI Pipeline Is Different From the Integration Pipeline

These are different things. CI runs on every commit:

bin/ci  # rubocop + bundler-audit + brakeman + tests (~22s)

Integration runs when you want to validate the pipeline end-to-end — before a deploy, after changing prompts, or after rsyncing fresh data from production. CI is fast and cheap. Integration is slow and costs money. Both are necessary.

The Cost of Not Having Integration

Let me be direct: if this project didn’t have the integration pipeline, I would have shipped at least 3 broken newsletters.

One with the anime section missing because the XML parser assumed RSS and the feed was Atom. One with every YouTube image pointing to 404 because the thumbnail URL pattern changed. And one with the market recap showing last month’s tickers because Yahoo Finance blocked the requests and the job failed silently.

Unit tests would have passed in all those scenarios. CI would have gone green. The deploy would have shipped. And at 7am on a Monday, 300 people would have received a newsletter full of holes.

The integration pipeline is expensive — in setup time, in API cost, in maintenance complexity. But it’s orders of magnitude cheaper than sending broken content to real subscribers. Especially when you only get one shot a week.

Conclusion

The testing hierarchy in this project is:

1. Unit tests (1,330 across both apps, 7 seconds in parallel, $0): prove each piece works in isolation
2. CI pipeline (rubocop + audit + brakeman + tests, ~22 seconds, $0): prove the code is healthy
3. Integration pipeline (real data, real APIs, ~3 minutes, ~$0.40): proves the system works
4. Visual preview (HTML in the browser, manual, $0): proves the final result is presentable

Each layer catches a different type of bug. None of them is enough alone. And the one that surprises most — the one that catches the bugs you swore didn’t exist — is the third: real data, real pipeline, no mocks.

The confidence you have in your system is proportional to how real the data in your tests is. Mock everything and you get 100% coverage and zero confidence. Test with production data and you get the truth — however inconvenient.

This is a concept I’ve talked about multiple times in videos on my channel and in talks. Everything in programming happens more or less at the Pareto ratio: 80/20:

• 80% of the code is “easy” and usually gets done in 20% of the time. The most important 20% will eat up most of the remaining 80% of total time.
• For bugs and issues like performance or security, if you fix the top 20% you solve 80% of the general problems.
• 20% of the scope is truly important, which is why you can always simplify, replace, change, or even cut 80% of it - and that’s how you squeeze a project into budget/deadline.

Everything is based on Top X. Learning to prioritize is learning to identify the 20%. The difference between someone senior/experienced and someone amateur/beginner is that beginners think 100% is important and can’t pick the 20%. What experience brings is the ability to isolate that 20% faster and faster.

HOWEVER, it’s very easy to just say this. Everyone has heard it at least once, but most likely still doesn’t understand what it means. That’s why I decided to write this (LONG) article: I’ll show you in practice what the easy 80% and the hard 20% are, and how that actually affects code quality, time spent and cost.

This article is for beginners mainly, but also for you: the senior who has juniors to mentor, or the teachers with students. Use it as discussion and study material with your apprentices. This is the kind of thing that should be taught.

Since this is going to be long, remember that on desktop, on the right side there’s a menu with the sections so you can jump straight to each topic and study it piece by piece.

The App: TV Clipboard

I picked a little app I had been wanting for a while for my personal use. It’s simpler than a to-do list, it doesn’t need a database or any other external service. In practice it’s a peer-to-peer (p2p) communicator over websockets.

The use case is this: I have a gamer mini-PC that I hook up to the living room TV. I usually leave a mouse by the armchair to navigate on it (it’s a damned Windows 11). But occasionally I need to type something (passwords, fill out some form). And I don’t want to leave my important accounts logged into that PC (I don’t trust gamer PCs).

Either I have to grab my bluetooth keyboard, or use the Windows on-screen keyboard. Both options are inconvenient in a living room. BUT, what if I could open a website (that’s already in my bookmarks), a QR Code shows up, I scan it with my smartphone and from there I can type or paste whatever is in my mobile clipboard and transfer it to the mini-PC clipboard instantly? That’s the app:

Then I open the camera on my phone, scan that QR Code and it opens this page:

Done. Now I can type anything or paste the password from my password manager (by clicking “paste”) and when I send it, on the TV I’ll see this:

It shows that it was received, and it goes straight to the TV clipboard automatically. It shows it “blurred” because if it’s a password I don’t want it legible in large text on the TV - and yes, the mouse cursor is gigantic because it’s easier for me to see on the TV.

This is useful not just for the TV, but for any situation where I want to send text from my phone to one of my PCs. A lot of people end up using WhatsApp or other messengers for this. I, for instance, have a “Just me” group that only has me in it, where I keep sending links and silly stuff from my phone. Then I always keep WhatsApp Web open in a tab on my PC and I can grab it from there. WhatsApp for me is a remote clipboard.

But that’s not safe for sending things like passwords, crypto addresses, or any other sensitive message. WhatsApp is a company with zero transparency. The advertising “claims” they have end-to-end encryption, but I don’t believe it. It’s equally likely they open everything in transit and keep it logged - it’s data worth money, even more in this era of LLM training.

Recommendation: NEVER COMMUNICATE ANYTHING IMPORTANT VIA WHATSAPP.

For file transfers I already use LocalSend:

That’s where the inspiration came from: I wanted a LocalSend for Clipboard. I’m sure there are already dozens of apps that do exactly this. But I wanted to take the opportunity to do this exercise.

The Plan: 100% Vibe Code

Honestly, I know enough about front-end, I started my career as a “webmaster” in the 90s. But I hate front-end. It’s slow, it’s tedious, yuck. In particular, I hate dealing with CSS. Even more particularly, I take no pleasure in writing Javascript. So I’m always lazy about starting a web app from scratch. But with Vibe Coding, this is one of the parts I can finally get rid of, especially if I don’t care about the exact look as long as it’s not crooked.

While we’re at it, I’m going to use a language I know the basics of but don’t have enough experience in, much less fluency (meaning, on my own I’d be slow): GoLang. Why Go:

• it compiles to a native binary: I don’t even need the performance, but having a compiler helps the LLM get feedback on obvious errors.
• no dependencies: later I’ll want to deploy it to my home server, and it’ll be infinitely easier to set up Docker with a dependency-free binary compared to something like Ruby or Python where you have to install a ton of stuff.
• full standard tooling: besides the compiler it already has a formatter, linter, tests and everything else that also gives good feedback to the LLM
• simple, short and unambiguous syntax. I like how Go is Python-like. I don’t consider either of them “pretty” (aesthetically, I much prefer Swift, Kotlin or Ruby). But unlike Javascript and even Rust, it’s much less verbose and much less ambiguous.
• rich standard library. Unlike Javascript, you don’t have to keep installing 500 external libs for basic things.
• since I’m going to use websockets, GoLang’s channels will make my life easier. Its async mechanism is simpler, more robust and better than Node’s reactor.
• most importantly: because I want to. It’s a personal project, that only I’ll use, I pick whatever I feel like and that’s that.

By the way, since I’m not fluent in Go, I’ll probably let some bad practice or obvious anti-pattern slip through. If that happens, you Go senior reading this, don’t hold back, comment below and explain what you think.

If you want to see the end result now, all the code is on this GitHub. I’ll be linking snippets from it all the time.

Another goal is to test how good the famous GLM 4.7 really is. I’ve written posts and tweets about it before. I’ve tried running the Flash version on my local PC (it’s good, but VERY slow). Overall, my impression is that it has similar quality to Claude Opus, GPT Codex, Gemini, but it’s WAY slower (at least 2x slower). Meaning if I had used Gemini or Claude, I’d get the same result but probably in half the time, though at a higher token cost (GLM is “cheaper”).

Let’s get to the TL;DR: it managed to implement the app the way I wanted, but it was SLOW. I spent an entire day, having lunch and dinner in front of the PC while Crush kept thinking, a total of more than 10 HOURS. And, according to OpenRouter, I spent just over USD 40 (over 200 reais). It was neither fast nor cheap, considering the app’s scope. Reminder: it’s simpler than a To-Do List.

In terms of size, even a simple app like this one resulted in more than 3,000 lines of code (LOC) of GoLang:

=== Code Statistics ===

Total Go code: 3493 lines

Production code: 1431 lines
Test code: 2062 lines
Test ratio: 1.44:1

Top 10 largest files:
  3493  total
  602  ./pkg/server/server_test.go
  560  ./pkg/hub/hub_test.go
  385  ./pkg/token/token_test.go
  304  ./pkg/hub/hub.go
  283  ./pkg/server/server.go
  263  ./i18n/i18n.go
  262  ./pkg/config/config_test.go
  253  ./pkg/QR Code/QR Code_test.go
  243  ./pkg/config/config.go

And for Javascript it came out to more than 1,000 LOC:

=== JavaScript Code Statistics ===

Total JavaScript code: 1360 lines

Production code: 787 lines
Test code: 573 lines
Test ratio: 0.73:1

All JavaScript files:
  1360  total
  573  static/js/test.js
  313  static/js/client.js
  249  static/js/host.js
  114  static/js/common.js
  111  static/js/i18n.js

Oh right, out of curiosity, I added loc.sh and loc-js.sh scripts to do this counting.

Phase 1: Making It “Work” - what does that mean?

Contrary to what many amateurs might think, it’s not enough to type a description of the application (like I did in the previous section) as a prompt, throw it at the LLM and it will automatically come out with a working app: IT WON’T.

This is what I was saying 2 years ago and it’s still true: it doesn’t matter how much more LLMs evolve. They might get a bit faster, a bit more efficient, but the architecture is still the same since 2022: transformers are glorified text generators, with a random sampling factor that always embeds entropy into the process (it will never be exact, deterministic). There will ALWAYS be a need for human intervention (and a lot of it).

Now I’ll demonstrate with a real example what that means.

In the end, I stopped this project after 43 commits. I followed the process I always say is the best: small prompts, one feature at a time, verify that it works and is correct, have it git commit, then move on to the next feature or requirement.

On a back-of-the-envelope count, each commit took about half a dozen prompts on average. Let’s say 43 commits x 6 prompts = nearly 258 prompts over more than 12 hours. Some prompts were a simple sentence that got solved in 2 seconds. Most took minutes and multiple iterations from the agent (Crush).

GLM 4.7’s context window is average: around 200k tokens. Crush has a mechanism where, when usage gets well past 90%, it asks to summarize the context so far so it can reset and reload from that summary to keep going. But it still has bugs and sometimes crashes and loses context:

Then I have to recover the last prompt, sometimes type part of the interrupted context myself, so it can pick up where it stopped. It’s tedious and you have to keep paying attention. This is a problem that Gemini suffers from less because its context window is MUCH larger (but attention in the middle degrades as it gets bigger, it’s a trade-off).

“Ah, Akita must be dumb. I built a Todo app and it didn’t even get close to 250 prompts. He did it wrong.”

Now let’s get to the part that separates the boys from the men. Indeed, to “make it work” you don’t need all those prompts. Of the 43 commits that took 10+ hours, I already had an app that “worked” after 10 small commits, which took about 1 hour.

If you want to see the “v0.1” version, just do this:

git clone https://github.com/akitaonrails/tvclipboard.git
cd tvclipboard
git fetch --tags
git checkout v0.1

This version has 1,827 LOC of Go and 255 LOC of Javascript. But the Go backend itself is only 549 LOC vs 1,431 LOC at the end. And that’s because in this minimal version I had already added backend unit tests (1,278 LOC), which most beginners never bother with. So you could “solve” my problem in just over 800 LOC, less than 1 hour and half a dozen prompts.

The project structure looked like this. Save this image to compare with how it ends up at the end:

In this version it already opens what I call the “host” page, which is the page that would open on my TV, showing the QR Code to connect. It already has the “client” page, which is what the QR Code opens on my phone. And it can already paste text on the client, send it to the host and copy it to the host’s clipboard. Pretty simple.

This is where 99% of people would stop and call it done.

Phase 2: Going beyond “just working”

Let me repeat: this project is simpler than a todo list and was built exclusively for me alone to use, at home, on my local network. Anyone thinking “ah, all of this is unnecessary for this app” - I KNOW!.

The idea is to demonstrate the minimum you have to think about for a real project. If you’re building a toy just for yourself alone, quality doesn’t matter.

For Go seniors, or Javascript seniors (because I’ll use it on the front), I know you’ll think “ah, he should have used framework X or library Y” - I KNOW!! 😅 At the end there will be a section where I’ll discuss some of those other options, wait before commenting!!

Back to the Pareto concept. I said 20% of the most important work costs 80% of the time, but that’s just an order of magnitude. In my example, the whole project took more than 10 hours and “to make it work” took only 1 hour. So it’s closer to 90/10 than 80/20.

What the hell cost 90% of the time if 10% “already worked”??

1. Token Management

The first thing I think about when I see a QR Code is that the link inside it should be disposable and volatile. It’s not a rule and obviously depends on the app. For my little app it’s definitely “overkill” but I’ll do it as an example. (I won’t keep repeating “yes, for my app it wasn’t needed”, this is the last time).

I don’t want a visitor who uses my app at home to be able to bookmark the link and use it from outside (I’ll expose my service on the internet to be able to use it from anywhere later). I want each use to be unique and to expire after a few minutes, requiring a new QR Code.

So I need some kind of token system that expires.

This article is already going to be very long, so I won’t keep copying the full code of every feature. I’ll always point to the commit where I implemented it so you can look it up later, and only show snippets for illustration.

Anyway, the goal will be to have a URL like http://clip.example.com/?token=blablabla. Nothing out of this world.

Yes, it could be a JWT, but my use is very simple so I had it built from scratch, and GLM 4.7 came up with a JSON structure with a random private key and a timestamp, which it decided to encrypt with AES (it really wasn’t necessary) and encode in hexadecimal to be the token (it ends up hyper long, more than 100 characters).

It turned out to be a little monster like this:

I didn’t like it but I decided to leave it because it’s working. I can mess with it later - in fact, a lot of things were done “out of order” because I just did them as I remembered them. I didn’t have a step-by-step plan written beforehand. I literally pulled it out of my head in real time.

The goal of this is to remember that Session Management is a concern every dev must have in mind in a real system.

2. First Refactor

The main code was already over 500 LOC, the tests over 1000 LOC. That’s too big to stay in a single file. So I asked GLM to refactor it into packages. This kind of thing I know Claude Code handles more easily, but GLM did well too. In the end it looked something like this:

This is what we call “Separation of Concerns”. Each package has an isolated domain. One package to deal only with configuration options, another to handle only the HTTP server, another to handle only QR Code generation, and another to handle only token management. Each with unit tests in the same package.

Asking for tests from the start was a huge help to the LLM, because every so often it would run the test, get confused about why it wasn’t working and the penny would drop that it needed to adjust the implementation. All automated feedback is important for the LLM to have context to work with.

And refactoring isn’t something you do only once. It’s a process that has to be done from time to time to consolidate and simplify the code, improving maintainability.

The goal here is to remember that “just because it works” doesn’t mean the code is frozen and you never touch it again.

3. Docker Support

Since I was going to run it on my home server, Docker support is mandatory. But regardless of that, it’s good practice to keep a Dockerfile that works. It’s another kind of “automated test”: see if you can “build” an image and run it. And document it for another user so they don’t need to install anything on their system to run it, just run Docker:

docker run -d \
  --name tvclipboard \
  -p 3333:3333 \
  akitaonrails/tvclipboard:latest

In fact, the command above works, in case you want to test it right now. I uploaded the image to my Docker Hub account.

And if you hadn’t already implemented it this way, this also forces you to add configuration options as environment variables (like TVCLIPBOARD_SESSION_TIMEOUT) or via command-line arguments (like --expires), exposing what is configurable.

The goal here is to remember how important it is to automate builds and document how to use it for other people.

4. Organizing Javascript and adding Cache Busting

To “just work”, way back at the beginning, all you had to do was make an “index.html” and go writing if host or if client in the middle of the HTML and JS. But at least GLM started out by duplicating a host.html and a client.html. But the javascript was all inline between <script>...</script> tags and duplicated in both files, so another refactoring step was to ask it to separate common.js, client.js, host.js.

This is the bare minimum of maintainability. Again: refactoring doesn’t end in a single step.

But I could run into a problem. Being a very simple little system, I had no ETAGs implemented and no asset pipeline like WebPack or similar.

In a dynamic web framework, most will support generating CSS and JS files dynamically, changing their names to things like /assets/host-20260128123456.js, and every time we edit some JS, the pipeline regenerates that file with a different name.

When we open it in a web browser, the browser always tries to “cache” static assets like images, JS, CSS, so it doesn’t keep asking for the same thing all the time. If it already downloaded host.js, on the next “reload”, if the same file is requested, it just pulls from the cache instead of asking the server again.

The problem with that is if you change the content of the file and keep the same name. The browser will think it’s the same thing and you’ll spend hours scratching your head thinking “damn it, I already fixed that, why isn’t it showing up??” - and it might be cache.

To avoid this, there are Cache Busting techniques, like ETAG headers. But the most common is just to add some kind of timestamp indicator in the URL, for example /static/js/host.js?v=20260128120431. This way, every time my server restarts, the timestamp will change (it’s not the best rule, but for now it works). The URL will be different and the browser will be forced to download the newest version, making sure I don’t waste time hunting down cache issues.

Again, any minimally competent web framework with HTML template support will have this, but I asked to do it from scratch:

Since I serve static HTML pages, my server code needs to modify the URLs before sending them to the browser, and that’s what the commit above does.

While writing this post, I noticed it added cache busting for “.js” files but forgot about “style.css”, which was left without it. I didn’t notice because I barely touched the original CSS it made. But again, to show that LLMs are never perfect and you have to keep constantly checking if it didn’t forget anything or leave anything behind. I had it fix this in this last commit

The goal here is to remember that you can’t just drop files on the server without knowing how they’re being served. Every front-end dev needs to know how a browser works. “Knowing HTML” isn’t enough.

5. Adding Command-Line Arguments and doing more cleanup

The next three commits are nonsense I should have squashed into a single commit.

This is more of a reminder that it’s not entirely bad to let an agent like Crush do git commits automatically, but you need to pay attention because you might have to tell it to undo (git has ways to redo commits, with options like -amend).

I just added command-line options. Now the binary has this support:

And since it forgot to document it in the README, the other commit is for that. This is why I said the right thing to do was to squash into a single commit, because they’re the same task.

6. Dev vs Production environment configuration

This is another point every beginner ignores. The classic “ah, but it works on my machine”.

On the web, especially, “your machine” is always going to be a localhost, or 127.0.0.1 or 0.0.0.0 (watch my Networking playlist to understand the difference).

The problem: if I run this app locally on my PC, I access it via http://localhost:3333. When the QR Code gets generated, it’ll use “localhost”. Now imagine your phone trying to load a “localhost” link. Obviously, it won’t work. Because your phone doesn’t have the app running on that port on localhost. I need my PC’s IP. So the first thing I have to ask to fix is that, if it’s localhost, it should look up my PC’s IP - which will be something like 192.168.0.xxx. With my PC and my phone on the same DHCP, on the same network, they can find each other via that private IP.

But what about when I run it in Docker? The internal IP of a Docker container is different from the external IP (the container is always either on NAT or on Bridge). The container’s “external” IP will be something like 192.168.xxx, but the internal IP will be something like 172.xxx. Again, if the QR Code uses the internal IP, there will be problems.

So I need a new configuration, usually called PUBLIC URL or PUBLIC HOST or just HOST. Where I explicitly declare it.

For example, when I’m done, as I said, I’ll leave my service exposed on the internet using a real personal domain. Something like https://clip.example.com with TLS support and all. So I need my docker-compose.yml to have an option like:

  - TVCLIPBOARD_PUBLIC_URL = https://clip.example.com

And that way the QR Code is generated with this name instead of having to guess. But at the same time, if nothing is passed, it assumes “localhost”.

In this first commit it doesn’t yet do all of this - I built it little by little. But that’s how you start this kind of feature.

The goal here is to remember that “running on my machine” (localhost) means nothing. We don’t put “your machine” into production. “Working” means “working OUTSIDE your machine”.

7. More code cleanup

I’m deliberately mentioning “code cleanup” all the time so it’s very clear that a considerable part of the work of programming is “cleaning up” code that has already been written.

The next four commits are just maintenance.

For example, the LLM inadvertently left a duplicated snippet inside the code. It didn’t break anything because it was in the Docker Compose configuration:

On top of that, there’s an accidental behavior in the way this app works that I decided to keep as a “feature”. There always has to be a browser that opens first and automatically becomes the Host and displays a QR Code. The next browser to open needs the link from that QR Code and becomes the “client”.

But when the host closes, the client loses the connection and is automatically promoted to host. I didn’t ask for that but it ended up working out, so I asked to keep it and document it.

The goal here is to remember that LLMs aren’t perfect, they’ll leave leftovers behind, and you need to keep monitoring all the time.

8. Optimizing Javascript

The next three commits are about cleaning up the Javascript.

Last time we touched it, I had only asked it to pull the inline Javascript out of the HTML and separate it into reusable “.js” files. But the time came to take a look at them.

Right off the bat, I could see there was duplication between client and host, so I asked to deduplicate and move things into “common.js”.

On top of that there were several small issues:

• a bunch of variables and functions in the global scope (global pollution: bad practice)
• websockets that open but with nothing checking the close
• timers being started but never cleared (leaks)

Javascript code that “just works” is always like this: everything global, everything open, nothing checked, multiple leaks. That’s normal and every LLM will do it this way.

I also don’t think you need to start from the beginning demanding SOLID and everything else. First make it work, then make it work properly, finally make it work fast. It’s always in that order, especially in a simple scope like this.

That’s why now it’s time to ask it to refactor. And the first thing to do is always to isolate the scope, for example, in a block like this:

(function() {
  'use strict';
  ...
})();

The problem with this is that every function the HTML was calling, like “onclick”, will fail because there’s nothing in the global scope anymore. Once everything is hidden, now you have to expose only what actually needs to be public.

This is called IIFE Encapsulation (Immediately Invoked Function Expression). There are thousands of practices like this that Javascript needs to get “cleaner”. I know, I should be using modules, Typescript, blah blah, but this is a simple example.

The goal here is to remember that, just because it “works” doesn’t mean it’s “good”, especially with Javascript.

9. Optimizing the Golang

This is the longest section. The Golang code already “works”, but it’s not good yet. I know it has race conditions in channels, I know the error handling is bad, I know there are still micro security issues, so I want to fix what I can.

The next 6 commits deal with:

• Refactor: Improve error handling, race conditions, and graceful shutdown
• Fix: Critical race conditions, memory leaks, and security issues
• Security: Add message size validation and rate limiting to prevent abuse
• Config: Tighten security limits and add origin validation
• Fix: CORS origins, Web Crypto API fallback, and clipboard issues
• Fix: QR code uses correct port from PublicURL and document Web Crypto API HTTPS requirement

6 commits

GoLang itself has a tool to check for race conditions, the Data Race Detector. Just pass the --race option to commands like go build or go test. And luckily GLM was smart and immediately used this tool when I asked it to check for these issues.

Race conditions happen when two asynchronous tasks try to modify the same data structure, for example. One way to “fix” them is to put Locks or Mutexes on structures that can only be modified synchronously. Any application with anything asynchronous will eventually have race condition issues. It’s not “maybe”, it’s “when”.

A server has to be “well-behaved”, it can’t just “crash” or “panic”. But the first “working” code GLM wrote had several snippets like this:

This is wrong. Besides that, we need the server to correctly accept system signals like SIGINT, SIGTERM, etc. This is when we hit “Ctrl-C”, for example, or when Docker tells the container to shut down. It won’t simply crash the program, first it asks the program to shut itself down, giving it a chance to do cleanup, like flushing files, closing ports and things like that.

This is called Graceful Shutdown.

I didn’t stop to check how right GLM was but according to itself, some of the issues it itself found in the code it itself wrote included things like:

As you can see, just because it “compiles” doesn’t mean it’s ok. Just because it “runs” doesn’t mean it’s ok. Even asking it to check MULTIPLE TIMES, the LLM doesn’t catch all the errors in one go. Each of these commits took hours to resolve and I had to intervene multiple times, and even so I’m still not sure it caught everything (I’d have to review more carefully - but I wanted to publish this article quickly 😂).

10. Adding Security Systems

Again, none of this is necessary in a system that only I will use on my local network, but you as a developer have to take responsibility.

ALWAYS assume every user of your system is MALICIOUS! That’s the RULE, not the exception.

A silly example: the user can paste and transmit any text over websockets to my server. The 3 most obvious things that immediately crossed my mind were:

a) When the host receives the text, it renders it into the HTML in the “Received” box

What if the user sends a <script>....</script>? That will render into the HTML and the browser will EXECUTE it. That’s how Code Injection works. That’s how XSS (Cross-Site Scripting) works. This was very common in old forums or comment systems. To avoid it you must SANITIZE everything coming from the user.

Luckily, the way GLM did it is already sanitized because it writes the content with textContent instead of innerHTML, so the text is written literally without being executed:

It happened to get this right on the first try, but you need to check because it could have done it wrong.

b) There was no message size check

A user should never be able to send text of any arbitrary size. They can always try to POST a gigabyte file directly to see if it blows up some Overflow error on the server, exposing some security exploit.

EVERY user field needs limits and it needs to be checked not only on the client/front-end, but more importantly, on the backend:

c) You have to reject Spammers!!

Again, you can’t assume everyone will use your web app nicely. The rule is to expect everyone to try to take it down or break in. And one way to do that is to write a script to keep making thousands of POSTs per second to try some kind of Denial of Service, where the server queues up so many requests it blows up.

The right thing is to configure this in another service that sits in front of yours, like Cloudflare.

But it doesn’t hurt to implement a fallback in your application itself. A way to configure how many requests a single user can make at a time (e.g. 4 requests per second - no one is going to paste text that many times per second in an app like this).

If they try more than that, the app has to immediately REJECT and drop the connection, without wasting time processing any more, freeing resources for other real requests. This is called RATE LIMITING.

d) Another issue beginners don’t stop to think about is CORS and Origin Check.

As I said before, we have Javascripts like /static/js/common.js. It’s not a huge problem if someone else uses them because they don’t have any big secrets in them. But at the very least I don’t want some site like “http://malware.pk/index.html” to include https://clip.example.com/static/js/common.js inside it and for me to end up paying for its traffic.

I want only the SAME ORIGIN, in this case “clip.example.com”, to be able to load files from the same server. No one from outside is welcome and that’s the BEST PRACTICE. Study CORS (Cross-Origin Resource Sharing). This is more important if your system spans multiple domains where one depends on the other, then you want - in a controlled way - to allow only certain other domains that are under your control.

The rule for everything on networks is DENY ALL and then ALLOW [specific]. It should NEVER be ALLOW ALL.

Implementing this took a while, because GLM kept tying itself in knots over how to allow “localhost”, but also “192.168.xxx:3333”, but also “clip.example.com” (via port 443), etc. It’s not hard, GLM just got itself all tangled up and I had to intervene several times with prompts to steer it in the right direction. It ended up taking multiple commits for it to get it right.

e) Another little thing beginners don’t think about: browser resource PERMISSIONS.

Every time you load this system on “localhost” and it receives a message to drop into the clipboard, this alert pops up asking for permission:

And that’s normal and correct: nothing should have access to things outside the browser. That’s why the Javascript asks “send this to the clipboard”, the browser intercepts it, checks the permission and, if it doesn’t have it, asks for it. Every time it will ask. But if you use a real domain (e.g. clip.example.com), it asks once and then remembers per domain. That’s one of the reasons I’m going to expose it on the internet: to have a real domain.

Exposing services on the internet is DANGEROUS. That’s why I wrote this article explaining how to use real domains and this other article about using Cloudflare Zero Trust.

f) There was another security factor which is “overkill” but I wanted to implement anyway.

Let’s say I want to run only on localhost. Let’s say it’s a private but shared network (guests, roommate, office, etc). Running on localhost there’s no TLS, meaning HTTP packets travel in plain text.

Anyone with Wireshark can “sniff” the packets on the network and intercept mine. And if I happen to be passing sensitive data, like a password, it will be visible in the clear.

The right thing is to run with TLS (HTTPS). But just as a “stopgap”, I asked GLM to implement a simple obfuscation (it’s not a fully secure process, but it uses AES-GCM), which is enough to not be obvious with a sniffer. For this it uses the Web Crypto API.

But there’s a catch: Web Crypto API is only available if the site opens over HTTPS or if it’s localhost. But if I try http://192.168.xxx:3333, it won’t be available. For that there’s this commit that deals with this and documents it.

The goal here is to show that security, optimization, and maintainability aspects are much more than a beginner thinks. In fact, they’ll never think about them, because you need to have studied beforehand to even know they exist.

11. Open Source Licenses

This topic alone would be material for a long article, but in practice I tend to think simply:

• if it’s a project where I don’t care whether anyone contributes or not, I just open it with PERMISSIVE licenses, like BSD, MIT
• if it’s a project where I’d like contributions and that no one uses commercially, I just put a RESTRICTIVE license on it, like GPL
• if it’s restrictive and it’s an app that could turn into a SaaS (so the code isn’t distributed, so there would be no obligation to contribute), then I add AGPL (Affero)

That’s what I did here: I licensed it under AGPL 3.0.

The warning is to not let LLMs write the license file. Always take texts on laws, regulations and rules from the official site. The LLM always produces a shortened or incomplete or even wrong version. Legal documents need to be EXACT, they can’t have wrong words and ambiguous meanings or incomplete statements.

LLMs are not good at producing correct legal text on the first try, there will always be errors. Always assume such text is wrong: there’s no “compiler” for laws.

12. Fixing the problem of overly long tokens

Way back I said that the tokens GLM generated for me are too big (over 100 characters). This results in super gigantic links.

It’s not a problem because no one has to type this and it’s not meant to be bookmarked. This is more of a personal choice: I like short tokens in URLs.

The advantage of the long token is that I don’t need to implement any kind of “database” (key-value store, or KV Store - Memcached, Redis). That’s because all the session info (ID and expiration timestamp) is encoded inside that token, which is why it’s long. It’s a json struct, encrypted with AES, converted to Hex.

Trying to keep the same concept, in this first commit I only asked it to try to optimize the size of the structure. So GLM on its own chose to try to change from UUID (36 chars) to a random 24-char Hex ID. Swap standard ISO timestamp (24 chars) for UNIX (10 chars). This went from 130-140 chars down to around 112 chars. A reduction on the order of 20% to 25%.

It wasn’t a bad attempt. But in the end I wasn’t really swallowing tokens that big (it could be a cookie too, I know, but I wanted to keep everything in the URL).

In the end I decided I wanted really short tokens (8 chars, base62) and so I’d need to keep the Session in some “storage”. I decided it would be “in-memory” (process-affinity, but I’m only going to run 1 instance, no load balancer, so it’s fine). And that would be a FIFO with rotation (older keys get discarded to make room for newer ones, and they expire that way).

This is the commit for that feature and I still had to intervene several times. For some reason it thought it would be a good idea to have a background goroutine cleaning up expired tokens from this storage. But that’s completely unnecessary if it’s a FIFO. It did it anyway, I had to tell it to undo. That’s the routine.

13. More refactoring and cleanup (almost done!)

The last 6 commits are MORE MAINTENANCE!!

Sounds like a joke, but the amount of maintenance I did is actually small. In a real project this is part of the routine. Every time you add something new, you break something that was already there, and you need to adjust it to avoid accumulating technical debt.

Remember we ran the race condition check? We added new code, so it has to be checked again. And it found more issues fixed in this commit. I don’t remember now whether these were already there way back and it skipped them without fixing, or whether the last edits ended up creating these new bugs.

After adding so many things and refactoring, I decided to bump up Go code coverage a bit more. The go test command has the -coverprofile option to help check which parts of your code haven’t been tested yet. So I asked GLM to add tests for new features like Rate Limiting, message size check, origin check and other things.

I asked for a trivial thing which is a Close button on the client side, to not keep piling up tabs on my smartphone.

After I had it swap the token system, there’s no more need to encrypt the timestamp structure. But it left behind unit tests about that. So I told it to remove them. Again: LLMs will leave things incomplete behind. A lot of times it doesn’t know whether to keep code so as not to break the test, or to drop the test because the code is no longer needed.

Then I asked Go to run the linter and support golangci-lint for last-pass checks. A Linter is a sort of formatting, syntax and good-practice check for a particular language. But most of the adjustments were extra whitespace, a few bad indentations and stuff like that. Nothing major.

One last thing that was my own negligence is that I was forgetting to add Javascript tests. A considerable part of the code is front-end dealing with WebSockets and everything. I had to add them, and that was in this commit.

Since I was doing it all in one go, I also asked it to add ESLint, and to prioritize testing everything only unit-wise, with mocks and stubs so everything is headless and doesn’t need to load Chromium. Just test the publicly exposed functions. This raised front-end test coverage from zero to 73%, which is great coverage. See the commit to see how it turned out.

14. Last thing: I18n

This is something I remembered while I was already writing this article. Completely unnecessary, but I wanted to see if GLM could do it. And it “did”. I didn’t detail how I wanted it and I let it come up with the solution. I would have had the Go server swap the strings before sending the HTML to the client.

But GLM preferred to add this on the front-end side and swap the strings via Javascript. It’s not wrong, but it’s not the best way.

Language translation is exactly one of the things AIs were originally built to be good at: Natural Language Processing (NLP). I had it extract all the English strings and then make a Brazilian Portuguese version, and I can say it did it well:

In a real web app this is something we always need to worry about: not leaving user-facing messages “hardcoded” and always extracting and testing them. Which brings me to the last part:

Wouldn’t it have been better to use Framework X or Library Y?

The Go crowd might not like that I didn’t use Gin, Fiber or another web framework. But really in this case the app is too simple. There are no more than 3 endpoints. The most complicated part would be the WebSockets, which wouldn’t look much different using a framework. Or bigger checks like CORS, which also wouldn’t have much less LOC.

On the Javascript side, maybe I18n would have been worth importing a ready-made package, but again, it’s very simple. There was no major feature I had to reinvent.

Of course, this is because this app is miserably small, as I said at the beginning. For a real larger app, I would indeed recommend using a popular framework with good community support, well documented and so on. But things the size of a ToDo List really don’t make a difference unless the exercise was about learning a particular framework.

Thinking about that, you have to be careful because no LLM will pick for you. If you start with small prompts, one feature at a time (which is the right way), it tends to write things all from scratch, without importing any library.

You have to know beforehand which library to ask for each thing!

Again, a beginner wouldn’t know, so they’d end up reinventing the wheel dozens of times instead of using popular libraries.

Remember I said, way back at the beginning, to save the structure of the project in version v0.1 that “already worked”? Here’s how it looks at the end in a real project:

Conclusion

Phew! That was long! Congrats if you made it this far!

My experience on this little project was like having a super motivated intern that I needed to spend the whole day looking over the shoulder of, and every other minute saying:

“Whoa! Stop everything, this is wrong!” “Whoa! Undo it and do it again but this other way!” “Whoa! No, you already did this wrong before and you’re repeating it!”

I’m not kidding when I said it was more than 250 prompts. And that’s a low estimate, I think I wrote MANY more than that.

As soon as I finished the Vibe Coding process, I pushed to GitHub and started writing this article, so there are still hours of REVIEW of the whole code, from beginning to end, to see if I missed anything. You couldn’t put it in production without doing that! If anyone sees weird things in the code, feel free to post in the comments.

It really felt like I was mentoring an intern. But with one problem: unlike a real intern, the LLM has already forgotten everything I taught it.

When I do a new project with GLM, it will make practically ALL the same mistakes and mix-ups. It’s like I was mentoring a blind intern with the memory of a goldfish.

It’s extremely exhausting!!

In the end it took around 12 hours to code everything, and I have the impression that a senior Go developer would have done it on their own in half the time. A junior would have taken this long over 2 or 3 days (stopping to research).

I already knew this, I’ve repeated it dozens of times, but the notion of:

“I don’t need to hire programmers anymore, AI does everything by itself.” - is a GIANT FALLACY.

And it’s not going to get much better than this. We’re already at the top of the S curve I’ve explained in every podcast I’ve been on and in videos on my channel. It costs orders of magnitude more energy to train the next version than it took for all previous versions combined, for incremental progress.

While the architecture stays the same, it’s not going to get past this. There needs to be some NEW BREAKTHROUGH (which hasn’t happened to date, since 2021 when Transformers came out).

A Claude Code will be a bit faster than GLM. The next version of GLM will be a bit faster. Maybe the 12 hours I spent will turn into 6 hours (for this miserable app). But the PROCESS will be the same: it does it wrong, I tell it to fix it, and it does this ad eternum.

There will never be such a thing as writing a single prompt and out comes a perfect, secure, optimized app. It will ALWAYS come out buggy on the first try. That’s why someone who is not a programmer should not build software to put in production with only Vibe Coding: it must go through review by a real experienced programmer.

This is not optional.

Post-Mortem

This section was written the day after publishing the original article. Once I was done, I asked Claude Code to do a deep quality analysis, focused on security and tests. And it found SEVERAL errors. Remember when I said you couldn’t put it in production without reviewing carefully first?

This is the commit with the fixes, take a look.

So it “already worked” in Phase 1, with a ton of problems. I intervened and spent 90% of the time in Phase 2 adding the foundation. I didn’t review it properly and I published the file, now I came back and there are still problems.

Software is NEVER “done”.

Even what Claude Code found doesn’t mean it fixes EVERYTHING. It means it fixes SOME THINGS. No software ever has “a guarantee” of being perfect. That doesn’t exist.

Even if I had started with Claude Code, as I said before, it would indeed have been faster. But it all depends on what you ask it. If you don’t know exactly the requirements and specs for security, or performance, or scalability, or maintainability BEFORE asking, it will do SOME THINGS. Not necessarily the PERFECT THINGS, just SOME THINGS.

Then someone might say “See? Now that you’ve reviewed it with Claude Code, it’ll be perfect”

So, afterwards I also asked GPT 5.2 Codex to do the same extensive analysis and guess what: it also found more little issues. Far fewer, of course. But look at it: after GLM 4.7 and Opus 4.5, GPT 5.2 still found more things. IT NEVER ENDS!

In particular, remember how earlier I had tokens encrypted, then I changed the requirement asking for them not to be and to stay in-memory on the server? Old code from that feature was left behind. GPT 5.2 thought the intention was to finish implementing the encryption and started putting it back. I had to interrupt it and tell it to stop and remove it. Not even Opus 4.5 had removed this dead code.

Always, you who ask are the one responsible for asking exactly and checking afterwards.

It has been very useful to me, especially for summarizing papers, reconciling research across multiple sites, summarizing topics, and things like that.

“BUT ..”

And it is a huge BUT, it has been PRETTY useless for real programming. I am not going to repeat every problem I have already run into: I detailed plenty of them in previous posts. But my conclusion is simple: it can help with code, on small, focused, isolated tasks, with a lot of prompting to help, and you - the programmer - need to be alert about what to pick and what to reject (and you will reject most of it).

Better prompts have limits. RAGs have limits. Loras have limits. Bigger context windows have limits. Using agents that hand back stacktraces and asking for deep thinking has limits. I have already hit EVERY one of these limits, and I have not found any “workaround” that can make it work 100% well 100% of the time - which is exactly what the marketing promises.

“But Microsoft, Google, Meta, NVIDIA have already stated that 30% of their code is already written by AI and next year programmers will be replaced”

That is the great half-truth, half-lie (interpret it however you want).

“What 30%??”

You, who are not a programmer, did you know that project documentation (which is extensive), language translation files (which are extensive, do you know how many languages a Microsoft or Google supports globally?), HTML pages that duplicate that extensive documentation, all of it is “considered” part of the code and sits in the same “code” repository like GitHub?

You, who are not a programmer, did you know that the VOLUME in characters/bytes of that material very often nearly matches, or can even exceed, the byte volume of the actual code itself?

That is just one small example that comes to mind. Launch pages for new products, the “hot-sites” which are DISPOSABLE sites built to be thrown away. That is also code volume. There is A LOT of “code” that is not “THE code”. Every programmer who works at a big company knows this.

Your little home project has no documentation, no automated tests, no build scripts, no deploy or packaging automation, no separate language files, no hot-site, no multi-language product site. It is just a “hello.py” and that is ALL of your code. I understand why you extrapolate that a big project is the same thing.

PART of the “30%” everyone keeps talking about is this. Sure, there is some real code in there, but nowhere near what you think.

Conflict of Interest

I am stunned. If a “drug dealer” gives you a free sample and you ask “but is it safe?”, what do you think he is going to answer?

The owner of Meta has AI products. The owner of Google, the owner of Microsoft, RENT OUT GPUs for AI. The owner of NVIDIA SELLS GPUs to AI datacenters. The owner of OpenAI, Anthropic, they all RENT that infrastructure and RESELL you TOKEN CREDITS.

What do you think their incentive is? SELLING TOKEN USAGE.

You saw what I reported in previous posts. In less than 1 week of testing I burned nearly USD 150 in credits. That would be more than 850 BRL in a week, and after running a HUNDRED attempts I got merely “reasonable” refactorings HALF of the time, and I got good passing unit tests LESS THAN HALF of the time (actually, almost NEVER). But I did spend tokens, LOTS OF TOKENS.

Features that the marketing tells you will improve things, like DEEP THINKING/REASONING, only “sometimes” give better code answers. They really are better at compiling papers or text research. But for code, in MOST attempts, I did not see that much difference. Except it took MUCH longer and burned MUCH more tokens. You know what Deep Thinking does? It adds more steps before the answer and stuffs the context with tokens. Those extra tokens: YOU PAY EXTRA FOR.

Features that the marketing also tells you will improve things, like RAG-based solutions, deep down are just more “prompt injection” - which always burns context and BURNS MORE TOKENS. It works, sure, but the incentive is to burn more tokens.

Features that the marketing is now trying to push on you, like MCP, agents, and LLM automation, combined with RAGs and all the rest. Devin, n8n, and everything else are just a more automated way to BURN MORE TOKENS.

Every incentive, from everyone, is to burn more tokens, not fewer. And in my tests, with many models (I will not say all of them), I got better code answers by TURNING OFF DEEP THINKING. You can configure that in tools like Aider or directly in the OpenAI API. Reduce thinking_tokens or set reasoning_effort to “Low” or, even better, “NONE”.

LOOT BOXES

I also explained in the article about how to build your own Ollama Modelfile that, essentially, the text generation process (swallow it, LLMs are next-word generators) has random components. Temperature, Top_P, Top_K. More: the training has non-linear components (e.g. ReLU). It is not deterministic, it is a DRAWING from a probability distribution.

That means, fundamentally, there is no way for an LLM to be 100% right 100% of the time. It is mathematically impossible. It can always be “almost right” or “looking like it is right”. But if you diligently double-check everything, all the time, you are going to start finding small mistakes. In the case of code, it is a lot more obvious. It is VERY wrong, MANY times.

And the whole process looks a lot like those “free” online games with LOOT BOXES or GACHA systems, you know? Like a Genshin Impact and friends.

What??? You are a boomer and have no idea what Loot Boxes are? …

I AM A BOOMER AND I KNOW, there is no excuse. Go get up to speed (and it is the main reason I choose not to play that kind of online game).

The economy of those games works like this: you do not have the option of buying the specific items you want. Instead you pay, say, 20 credits for the chance to take part in a drawing (which happens on pre-scheduled dates and times, not all the time). In that drawing you have a (HIGH) probability of only winning a 10-credit item OR an incredible (VERY LOW) probability of winning an exclusive 1000-credit item!!!!

Sounds like the deal of the century, right??? (LITERALLY IT IS!!!)

You always win the cheapest item in the drawing. And every once in a while it gives you a MEDIUM item so you do not give up and believe you still have a real chance. Only if you spend A LOT (and by that point you have already spent more than 1000 credits anyway), only then maybe it hands you a big item, to shoot you up with more dopamine and deepen your addiction.

That is exactly what happens with LLMs. Their answers have odds similar to a Loot Box. Sometimes it surprises you with an excellent answer, but most of the time you are stuck asking “dude, are you sure about that answer?”, or “dude, isn’t that second line wrong?”, or “dude, re-read what I asked and try again”, and your whole day goes on like that.

You are not watching it, but under the hood every time it has to correct itself it keeps burning the same credits. By the end of the day, there goes USD 50. Sounds like a little, but do it every day and see how it adds up.

Except your CONFIRMATION BIAS makes you remember well when it got things right, but keeps “forgetting” all the other times it got things wrong. And your perception is that “Yeah, it works.”

Worse, your low self-esteem and need to belong to a group, to CONFORM. Reinforced by what you read in the news every day, like:

• 30% of all Microsoft code is already written by AI
• The CEO of Anthropic says that next year programmers will be replaced by virtual workers
• The CEO of NVIDIA says he would not recommend his son to study programming anymore

And all of it RAISES THE BIAS: you think they are right, and the times you yourself saw the LLM getting it wrong “must have been a small exception, I am the one who is dumb” and you BELIEVE the LIE.

Who remembers my talks from 15 years ago where I used to show this video?

This is a well known phenomenon, today amplified even more by social networks and by the dumbness of journalists who just parrot what they hear and no longer ask critical questions like: IS THERE A CONFLICT OF INTEREST??

“Does OpenAI use your chats for training?”

Here I have to fix a lie I have been telling you all. I always kept saying no. Your conversation is not used for training, because it would pollute the training material with garbage that in the end would not improve the answers.

It is a half-truth. Indeed, I doubt it a lot and I will stake my reputation on it: no provider wants to risk someone finding a copyrighted text and getting sued. That already happens, the New York Times sued because ChatGPT managed to reproduce a text they had published. And that kind of “proves” it was part of the training material.

There are people who paste private things in there like health conditions, financial situation, and even critical data like passwords, addresses, stuff they should not share but end up publishing because they have no clue about cybersecurity. Everybody falls for a scam every day because of that. OpenAI or Meta or Alibaba are not eager to have that show up in a chat one day and turn into a viral international news story about how irresponsible they were.

BUT

And it is a HUGE “BUT”, yes they do use your chat. Everyone’s. Without copying any of your data. They just need the METADATA, the data that describes your chat.

Off the top of my head, being naive and amateurish about training, I can see at least 2 important pieces of information you hand them: the length of your chat and the sentiment of your responses in the chat.

Determining sentiment from text is a well solved problem, there are algorithms with or without neural networks that can tell if a text looks happy, sad, angry, anguished, and so on. They do not need the content of your text, they just need the SENTIMENT you had at that moment.

Add to that the LENGTH of your Chat and as I said in the previous post, the incentive is to make you burn MORE TOKENS (LONGER chats). IN THE NEXT POST I will show you how they do alignment, the model “fine-tuning”. An OpenAI uses thousands of lines of prompt/completion pairs to “force” the model to answer in a certain way. Think of an Excel full of pre-written questions and answers that, according to their metadata, we know are the most POPULAR answers.

The goal is not to make the model answer more correctly, because sometimes the right answer is very harsh or inconvenient for some people. I am almost certain that the data extracted from millions of people’s chats is just that: sentiment and length. It must help craft prompts that force the model to answer “less correctly” but “more pleasantly”, encouraging the user to KEEP talking in the chat and BURN MORE TOKENS.

It is the oldest behavior in the world. Every shopkeeper knows this. Every salesperson knows this. Everyone who works in services knows this. If you are correct and direct, you will sound arrogant and the customer will not come back. Better to sound less correct but more inviting, pleasant, helpful, and so on. And the customer comes back, because that is how the average person works.

So, yes: I do think your data serves to re-align the model after training, to make it answer more pleasantly. They do not even bother hiding it anymore. Did you not see Sam Altman himself a few days ago saying they made ChatGPT 4 answer in a friendlier way, with more personality?

That is what I think they are doing. Making the model smarter is turning out to be a challenge. Because - as I have been saying for a year now - we are hitting the top of the S-curve. To keep selling tokens, to bring you back and get you hooked on talking, it does not need to be smarter, it just needs to be more pleasant.

Conclusion

There is no LLM that will be 100% right 100% of the time, the fundamental architecture PREVENTS that from happening, no matter how many optimizations we pile on. There is now and there will always be an ENTROPY component. It is that randomness that produces text which sounds like a human talking and not a robot.

Automations that are too long and too complex built on top of this fragile foundation are a HUGE mistake. Sure, for simple, short, focused tasks, it has a higher HIT RATE. For real code? It is SUPER low.

Never let an MCP commit directly to your Git repository. Tools like Aider ship “autocommit = true” by default and that is an enormous stupidity! Do not do that.

I warned you.

But is there no way to improve LLMs at coding? Maybe. 100% never, but bumping up the rate I think we can. And to prove I am not anti-LLM or anything like that, in the next article I am going to try to teach HOW to improve an LLM for programming (with medium probability, it is no miracle!).

Now, you may say “another Ruby fanboy.” That’s fair, but a terrible argument, as it’s a poor and common argumentum ad hominem. And on the subject of fallacies, the click-bait article above is wrong exactly because it falls for a blatant Post hoc ergo propter hoc fallacy plus some more confirmation bias which we are all guilty of falling for all the time.

I’m not saying that the author wrote fallacies on purpose. Unfortunately, it’s just too easy to fall for fallacies. Especially when everybody has an intrinsic desire to confirm one’s biases. Even trying to be careful, I end up doing that as well.

On the specific subject of Rails, start here:

• Rails has won: The Elephant in the Room
• The Ruby Community and Reputation
• Programmers Guild - Religion and Sports

As a general rule, anyone announcing the future demise of anything is most probably wrong. It’s fun to play guessing games, but articles in high profile websites like TNW should be more careful as many beginners in the field end up trusting their opinions.

Which is why I usually tell people to stop trusting opinions. Everybody has an opinion, which is why it’s so cheap. Then people artificially grade “celebrities” or “gurus” as having opinions of higher value. I don’t buy it, neither should you.

Black Swans do happen. But it’s an exercise in futility to try to predict them. You may be lucky, once or twice. But it’s useless to attempt. Which is why I am a high proponent of Antifragility, so I will give you the general rules to understand what to do.

Am I saying that Ruby on Rails will live on forever? Don’t be absurd, of course not. Nothing lasts forever. More than 2 decades actively working in the field taught me that betting everything on one horse is a mistake. It’s valid in the financial markets, it’s valid in the tech market.

Instead of stock or bonds, your asset is your knowledge and your practice. Diversification of your portfolio is usually a better bet.

You invested all your 401k in Bitcoins. Now you will try everything to try to confirm your bias by subscribing to every news piece and opinion that confirms your bias, and you will denounce every article that may have an argument against your bet. Am I saying to not invest in Bitcoins? Hell no. Just don’t be stupid and invest everything on anything.

You invested all your time learning Google Web Toolkit in 2006. Good luck for you finding an article to confirm that bias. This is one case, an after-the-fact, that we can’t do much but consensually agree on its death.

The job of a proper professional programmer is not mastery of a single tool or even a single technology stack. Your job is to be as good as possible in learning and practicing in the technology field.

Having said all that, and even though the article’s structure is inaccurate, The feeling is not wrong. Ruby on Rails no longer represents what it was 10 years ago. Now, before you start freaking out, let’s explore it down memory lane.

2003 - A New Renaissance

As an after-the-fact exercise, one has to realize that a series of events helped unfold the Rails phenomenon between 2003 and 2011. It was not by design. It was not planned. It could’ve been something else.

• We were just recovering from the 2001 Internet dot-com Bubble Crash.
• Social networks were starting to emerge, we had Orkut, MySpace, Friendster. Still no Facebook on the radar.
• Blogs were starting to pick up steam and competing with traditional news outlets.
• Big vendors focused solely on the Enterprise market at that point and the rest were deemed amateurish.
• In the mainstream either you were doing .NET or Java on the Enterprise or doing hotsites in Flash and PHP for ad hoc agencies.
• Even open source tools like Eclipse were primarily used through enterprisey umbrellas like WebSphere or WebLogic.
• The Agile Manifesto had just been published in 2001 and the mainstream market didn’t pay much attention. PMI and RUP were the order of the day.

Then we reached 2004.

• Google successfully launched Gmail to much fanfare. Dynamic HTML and Ajax demoed how we could achieve Desktop-level applications in the Web. This is also where Javascript earns a new role beyond stupid web mini games or ads.
• Facebook skyrocketed and we were shown glimpses of what granular social networks look like.
• Apple consolidated its comeback as a relevant tech giant with the successful resurrection of OS X, proving that Unix on the consumer desktop was viable.
• Microsoft entered the 21st century after a super heavy antitrust case that almost obliterated the company. Bill Gates resigned, Steve Ballmer took the reins and they had their darkest decade
• Years of vendor lock-in finally started to wear off developers, no one actually wanted to use IBM, BEA, Oracle anymore and the promise of the open source revolution started in the 90’s finally started to solidify
• The dot-com crash left a depression in the tech market but it also left behind an infrastructure that was finally being put to good use

Ruby on Rails shows up as a very small curiosity, in an obscure tech conference in Brazil, created by a small but promising agency called 37signals. Signaling something most developers had never seen before: a flexible language - unlike Java or C# or PHP - that could be used to implement Gmail-like web applications with a flavor of the Apple vision and steering away to everything that Microsoft and IBM represented while incorporating Agile-like techniques such as Test-Driven Development in a practical way and out-of-the-box.

What else could you ask for in 2005?

Then 2006 happens: Amazon AWS releases S3 and EC2 and the term “Elastic Cloud” becomes the new hype.

Up to that point, “web development” either meant expensive and bureaucratic IT departments in big corporations or inexpensive but weak shared hosting in services such as Dreamhost, 1&1, Media Temple.

Elastic Cloud holds the key to surpass the enterprise and make it possible for independent developers to build the elusive “web scale”.

Ruby on Rails is exactly where it should be: starting to be adopted by a new crop of Silicon Valley tech start-ups trying to make a dent in the universe in this new Ajax-based Cloud-era. Remember that Rails was first released with RJS, a way to write Javascript through Ruby using Prototype and Scriptaculous. jQuery was not even mainstream yet.

Twitter shows up. Groupon shows up. Engine Yard shows up. And much more follow.

It’s all happening blistering fast, bear in mind that I am only touching the surface here.

2007 arrives and at least 3 big things happen:

• Another Rails-based start-up is born: Heroku. If AWS EC2 coined the term “Infrastructure as a Service” (IaaS) Heroku defined what a “Platform as Service” (PaaS) means. It was able to put EC2 to good use and make its power approachable.
• At the same time, developers have grown bitter of version control systems such as CVS and again, big vendor options such as Rational ClearCase, Microsoft SourceSafe, and others were just terrible. Subversion had emerged but its many shortcomings would become its demise. Linus Torvalds decides to fix the problem and release Git. Mainstream developers finally learn basic computer science terms such as DAGs (“Directed Acyclic Graphs”).
• The iPhone is released and a new mobile revolution takes place. Dominant players like Nokia and Blackberry were shaken to their core. Developers want a piece of it. Mobile Safari becomes high profile overnight. And its WebKit core rises to stardom, eclipsing old veterans like Firefox/Gecko and Opera.

2008: the birth of GitHub and the dawn of the DevOps.

Git was great but too low level. Several other contenders tried to become the new standard such as Mercurial and Bazaar. But the birth of GitHub would change all that. Again, another Rails-based start-up making the news.

But the honeymoon is showing signs of reaching an end: prominent Ruby developer, Zed Shaw, loudly leaves the community by January.

During this year we also had the cold war between Engine Yard’s Merb framework and 37signals’ Rails, resulting in the controversial Rails-Merb merge, placing another crack in the community.

But we still had a lot of work to do, so we moved on.

2009: “there’s an app for that!”

Apple finally releases a proper native SDK. Developers start building a new generation of productivity apps and games. WebKit starts growing in market share. Apple solidifies the concept of aesthetic in software development and Ruby on Rails is at its high at that point.

We begin the NoSQL era. When data storage and fetch start to really become a bottleneck, we quickly learn that traditional enterprisey databases are not “web scale” enough. MongoDB leads the pack. Web development starts an era of craze over NoSQL.

In parallel, Twitter, once the poster-child of the Rails generation, creates a crack in the Ruby community when it declares that it’s started to use yet another obscure language: Scala!

This starts the endless obsession era about “concurrency”, yet another elusive concept that many talk big and talk a lot but only a few actually grasp its meaning. It’s the start of the window of opportunity for Node.js, Scala, Erlang, Clojure, Go.

The honeymoon is over.

2010:

• The demise of Sun and Oracle as the new stewardship for Java. If things were not playing out cool for Java, that was a heavy nail in the coffin.
• Apple declares open war against Flash. Another big push for Javascript and front-end heavy web apps.

In my personal opinion, this is the year when Rails reached its peak in terms of influence. It was the go-to stack for developers in Macs wanting to create their brand new Lean Start-up.

2011: A New Era

In around 6 years the software development market did titanic shifts:

• Distributed development derived from years of Open Source workflow becomes the new standard. Git and GitHub were key players. Big vendors have to play catch-up with their stinky enterprise alternatives such as TFS.
• Agile is taken seriously in a new crop of tech startups evangelizing proper software development. Tests shift from unproductive obnoxious things into invaluable core technique. Tools like Hudson/Jenkins combined with Git workflow pave the way for what continuous integration and later continuous deployment would become.
• AWS and the Heroku example pave the way to make old IT obsolete and create a new generation of infrastructure as software, and “DevOps”. There is now the urge to deploy super web scale with zero friction. Time to market becomes a key factor: release early, release often.
• Apple releases the AppStore and helps drive the Software as Service market, with apps requiring live internet to function. The “Cloud” reaches the masses.
• By making everybody on Earth carry an always-on powerful device, a smartphone, we were never so close to the Orwellian future of 1984. Everything a person does is a data point. Big Data becomes a thing. Data Sciences becomes viable as a mainstream profession. I.A. related disciplines become a necessity to reason about so much data.

Android is getting closer to the iPhone. Microsoft is resurrecting from its ashes. In October, Steve Jobs dies, marking the end of the era of Apple’s heavy influence.

But around 2011 another thing happened: we win.

Independent developers surpassed the status quo of enterprise developers. There is a shift of power. The conventions for the software development field don’t come from the old big tool vendors anymore. We are not listening to IDE vendors like Microsoft, BEA, Oracle. We are now listening to the winners of this generation of startups: Apple, Google, Facebook.

The concept of “one language to rule them all” was shattered in the web camp by Ruby on Rails and in the native application space by Apple. 2 very unusual and strange languages, Ruby and Objective-C, reached mainstream status. They proved that developers can learn and master exotic languages turning them into profitable products.

With those foundations set, a new gold rush emerges: new languages galore, free from the notion that to become successful one had to stick to Java, C#, or C++ like languages, people finally started looking elsewhere to differentiate themselves, and thus a new generation of exotic languages arrive: Scala, Haskell, Ocaml, F#, Erlang, Clojure, Rust, Elixir, etc. The sky is the limit.

By killing Flash and accelerating HTML 5, ES6, and with all the controversies around concurrency, Javascript gained new life and fast adoption unlike nothing else before.

Epilogue

Is Ruby on Rails dead? No.

Is Ruby on Rails still the leader of the pack? No.

What is the legacy of the Ruby on Rails Generation? It played an important role in defining what we now consider the new status quo for web development: Git, CI, CD, Cloud, Metrics. It brought the Apple aesthetics thinking to the table in a big way.

That new crop of tech startups had a serious side-effect though: it put new engineers in the spotlight, elevating them to rock star levels and turning them into Steve Jobs wannabes. They just started taking everything we accomplished for granted. Recipe for disaster.

And while the big vendors’ influence became thin compared to the 90’s, they are not dead. After 2011 we ushered in a new era of chaos, an era of dissatisfied developers. Where every couple of years a new derivation emerges and we have that unsatisfying feeling that we need to rewrite everything all the time.

The Ruby on Rails pool got diluted into smaller niche languages such as Javascript itself, Go, Elixir, Scala.

The Java/Microsoft mainstream is gaining new traction. Aesthetics also got diluted. Careful engineering is getting diluted. Social networks ubiquity and the easy path of quick cloud deployment is making people less and less patient. Crafting requires patience. People crave Likes, a new generation of vanity metrics arose.

We now have a larger number of independent non-enterprise developers with little to no patience, with an exaggerated sense of confidence, and not-invented-here syndrome, taking everything for granted and caring less and less about proper value.

In conclusion, the death of Ruby on Rails is indeed greatly exaggerated, it will keep going for many years still - especially if the Ruby Core Team can actually deliver on the 3x3 goal. People like to share doomsday articles while optimistic and more realistic reports such as this Why Ruby on Rails is still the best choice? are largely ignored. Such is the way of social networks vanity.

But it will not be what it once was because the times have changed. We can now live by the “On Rails” workflow without Ruby on Rails. And it can’t go on breaking the status quo, because it became the status quo.

I wonder where that will lead us. I do have a few clues, but as I said, I am not in the game of publishing click-bait headlines.

One of the most popular ones answers the question “What is the hardest thing you do as a software engineer?”. I wrote something similar in Brazilian Portuguese in the post “Estimativas são Promessas. Promessas devem ser cumpridas.”.

In a nutshell, you can never give an estimate that is “correct”. If you could, it would not be an “estimate”, it would be a “prediction”.

Let us assume we have neither precognition powers nor magic crystal balls to tell us the future.

To estimate something is to “guess” the value of something. It’s always a guess, the same thing as a valuation. And like any guess, it can never be deemed “correct”: it’s just a likely candidate among an infinite range of possible values.

There is zero connection between a guess and the outcome. Understand this simple truth: saying something can happen does not MAKE it happen.

Estimating tomorrow’s weather as rainy does not MAKE it rain. Estimating the result of the Super Bowl does not MAKE it happen. There is zero correlation between an estimate and the actual outcome.

I stated in those articles that “Estimates are Promises”. The intention was to provoke a reaction, since most people assume that estimates can never be promises, exactly because of what I just explained.

What makes a promise special is that once you promise something, you’re expected to ACT on delivering it.

No one that has no “skin in the game” should give estimates.

If you’re not an active player in the game, you shouldn’t be handing out guesses. The same way no one makes promises on someone else’s behalf.

Can you actually make credible promises and keep them? You can, but first you have to understand a few more truths about reality.

You’ve probably already seen plenty of articles explaining project management methodologies and techniques. Most of them manage to explain the Whats and the Hows, but as usual, they fail on the Whys.

Why do we need those methodologies? Why are they necessary? Why do they work? What hidden mechanisms do they set in motion?

What makes Agile techniques any different from your usual homeopathy or a budget self-help cliché?

It’s always all about Pareto

There is no such thing as a precise project scope. There is a limit to adding detail, and past it you just get diminishing returns.

The most precise level of detail a feature scope can have is the actual code of the feature. And this matters: the lines of code are NOT the outcome. What end users actually experience is the execution of that code at runtime.

The programming itself is the actual blueprint. The diagrams, the use cases, the user stories, and anything else that comes before the programming are just a draft, a mere sketch.

A naive architect or designer might think that detailed diagrams, use case documents, and fancy PowerPoint slides are worth as much as an engineering blueprint. They’re worth less: they’re the equivalent of a sketch on a napkin. Volatile and, in practice, mostly worthless.

“But isn’t the programming the same as the construction phase itself, laying bricks on top of bricks?” NO, that’s not what a programmer does. The bricklaying is the job of the language interpreter or the compiler that spits out the executable binary.

This is the metaphor that drives non-programmers crazy. In engineering, the construction itself is the most expensive and time-consuming part. In programming, the expensive part is the blueprint (writing the code), and the “construction” is trivial: just compiling, which is automatic.

In the same way, a project scope is just a set of sketches. We have to drop the notion that a “complete” scope will ever exist. There’s no way to say “100% of the scope” or “closed scope”, because a software project scope is, by definition, always incomplete.

On top of that, I’ll argue that roughly 80% of this so-called “scope”, what I prefer to call a sketch, is mostly worthless to most end-user activities: the admin section, the institutional pages nobody reads, convoluted sign-up flows, and so on.

This is why every feature list MUST be prioritized. You can usually get by with 20% of the features, which is roughly what people mean when they say “MVP” or “minimum viable product”. Release as early as you can, get user feedback, and refine the rest of the “sketch” you call a backlog.

So instead of aiming for an all-or-nothing proposition, hunting for stupidly complex equations to figure out a “precise” estimate of an incomplete sketch, assume you can actually deliver EARLY the first 20% that really matters and figure out the rest in iterations.

Oh yeah, this is what we call “Agile”, by the way.

Agile is about Risk Management

People assume Agility is about managing the project management instruments themselves: the backlog, the rituals, the metrics.

Having Agile-like instruments doesn’t make you Agile.

Being Agile is keeping Risk under control.

Instead of treating projects as an all-or-nothing endeavor, start thinking about them the way an investor thinks about a portfolio of stocks. You don’t expect the whole portfolio to turn a profit; you assume some stocks will underperform. You just don’t know which ones, so you dilute your risk.

Trying to predict the stock market is an exercise in futility.

Trying to predict the precise implementation of a project, especially the long ones, is also an exercise in futility.

So you have to deal with uncertainty the right way: by becoming Antifragile.

“Some things benefit from shocks; they thrive and grow when exposed to volatility, randomness, disorder, and stressors and love adventure, risk, and uncertainty. Yet, in spite of the ubiquity of the phenomenon, there is no word for the exact opposite of fragile. Let us call it antifragile. Antifragility is beyond resilience or robustness. The resilient resists shocks and stays the same; the antifragile gets better” - Nassim Taleb

Instead of the preposterous exercise in futility of trying to predict uncertainty and random events, you do the reasonable thing: you assume Black Swans exist and you can’t predict them. So you prepare for uncertainty the only reasonable way there is, by not trying to predict them.

Expose the small mistakes early and correct them often. Implementing everything in a black box and doing a Big Bang deployment is the easiest path to failure. Delivering often, exposing the bugs, and fixing them constantly is accepting that mistakes will happen and gaining strength along the way.

A Better Metaphor

Imagine that you, whether the non-programmer client or the programmer who has no clue how to explain the process to that client, have an iron ore furnace to manage.

The thing about those furnaces is that if you heat them too much, they can explode in your face. If you cool them down too much, the ore hardens.

Your job is to add coal to the furnace, and you decide the rate. Too fast and it gets exploding hot. Too slow and you might put out the fire and lose your furnace.

Now try to estimate a constant intake of coal to keep the furnace in good shape.

You can’t.

The easiest way out is to install a thermometer that tracks the furnace’s current temperature.

You stay safe within a certain margin of temperatures, speeding up or slowing down the coal intake by checking the thermometer all the time.

ALL THE FREAKING TIME.

This is what Agile-based “Velocity” (or any of the fancier Monte Carlo simulations) actually is: a thermometer.

If the Velocity is too high, your team is probably working extra hours or delivering lower-quality code. This will backfire: either your team burns out too fast, or the code piles up technical debt too fast and you won’t be able to pay it back. Keep that pace and Velocity drops to a halt (the furnace explodes).

If the Velocity is too low, your team is slacking off, your backlog is a useless piece of shit no one can understand even after 10-hour meetings, or you already let Velocity run too high and now you’re paying back technical debt, or your team is dead from burnout (fire extinguished).

You want to keep Velocity steady, constant. This is what being Agile is all about: keeping an eye on the thermometer and responding.

The Triforce of Projects

Welcome to the Iron Triangle of Project Management.

Repeat after me: if I want to lock down the time, the cost, and the scope, I am a moron.

Repeat again.

You should lock down time and cost. If you’ve read this far, you know you can never lock down “scope”; you can only make it fat, and not necessarily more valuable. This is why I always say the very definition of a Product Manager or Product Owner is to be the bastion of ROI (Return on Investment).

Now, why?

Because the Iron Triangle has the following corollary:

• If you want it fast and good, it can’t be cheap
• If you want it fast and cheap, it can’t be good
• If you want it good and cheap, it can’t be fast

This is a Law, you can’t fiddle with it. Pick your choice.

How do I keep Promises then?

Now, with those 3 truths in hand:

• You can’t have your lunch and eat it too (cheap, fast, and good)
• You’re managing the temperature of the furnace
• You only need 20% of the “sketch” you call “scope”

Yes, any experienced developer can give you a “ballpark” estimate. A ballpark goes like this:

• 1 month (maybe 1 month and a half, but definitely less than 2)
• 3 months (it’s more than 2 months, less than 6 months)
• 6 months (it’s more than 4 months, less than 9 months)
• it’s more than 6 months, probably less than 1 year

Don’t even try to granulate more than that. It’s useless.

Lock the time. And lock the cost (that being the number of developers times the hourly rate times the total amount of estimated hours). That’s it.

Now jot down what the client calls “scope” as user stories in a backlog and have him prioritize it.

Start the iterations. After each one, release to a staging environment. Non-programmers, beware: ALWAYS make sure whichever programmer you hire delivers testable versions of the finished stories to a publicly available URL you can actually visit and test.

If your programmer refuses to do that or gives excuses: FIRE HIM.

If your programmer or company or whatever promises you a “closed scope” price, promises to do everything, and you believe them, you’re too gullible.

Do you think it’s funny to play this stupid game of “I’ll pretend to tell you that I know the truth and you will pretend that you believe what I tell you.”

No serious professional has time for stupid games, and the only honest thing anyone can say about any software project is: “given my experience I believe the ballpark for this kind of project is within X months, given the Y and Z assumptions”.

You don’t have to believe him. You just need to start checking the thermometer. Any non-programmer can assess the quality of the delivery based on the frequent deliveries of the prioritized features.

“But what if after 2 weeks I don’t like the results?” Easy: FIRE HIM.

And sometimes “firing” isn’t even the right word. Sometimes the relationship is difficult and the best thing to do is to part ways.

You have to accept losing 2 weeks, or any short period of time, as part of Risk Management. It’s better to accept losing 2 weeks of your project budget than to blindly trust someone for 6 months and lose the entire project budget and then some.

Pareto again: Agility is about Risk Management. You accept that losing 20% of your budget is okay, and you play with that. And that’s fine, because you only really need a bit more than 20% of the sketchy scope you have.

See what I just did with the Math here?

We stop playing the pretend game and actually manage the project’s risks. You collaborate on the equivalent thermometer, which combines the prioritized backlog (the scale) and the velocity (the temperature), and you keep an eye on the partial deliveries in the staging environment.

Conclusion

So yes, Estimates should be taken as seriously as Promises. You can give reasonable Estimates as long as you can manage the Risks and the Client accepts the rules of the game: there is no “closed, complete scope”, priorities first, and testing and accepting delivered features every week.

The idea behind Promises is that you have to MANAGE in order to meet them. The best way to do that is to frequently stop, re-assess, and keep going. It feels like you’re wasting time, but you’re actually saving yourself from wasting it.

If you don’t have skin in the game, back off.

Velocity is meant to be kept steady, at a predictable rate. Use the variance as a sign of being too fast or too slow, adjust the other variable, measure again, and go. Just like a furnace thermometer.

There are several versions of “thermometers”, from Joel Spolsky’s Evidence Based Scheduling all the way to fancy Monte Carlo simulations and other stochastic processes (they are all thermometers, and NOT estimation tools).

What stops you from doing that is using the wrong metaphors and the wrong references.

Instead of hunting for equivalent metaphors in construction, factories, and other “hard”-ware assemblies, look elsewhere, where you’ll find other “soft”-ware processes.

Musicians have deadlines. Painters have deadlines. Choreographers have deadlines. Sports have deadlines. Research laboratories have deadlines. How do they meet them? By constantly checking the current state, comparing it to the goals, assessing whether what they’re doing is actually working, and changing what doesn’t.

Hollywood has deadlines. They have far worse variables to control than any software project you’ll ever run into, and they still manage to deliver. And profit.

And accept that you can’t control every variable, so stop trying. Think of it as the financial markets. One day you have Ethereum skyrocketing 4,000% and the very next day you have it falling down to ashes in a flash-crash.

Don’t aim to become resistant or resilient. Prepare to become Antifragile.

Now, you’re a small tech startup or even a small team in a big corporation. How to even start?

And the recommendation is: do the smallest thing that works first. Always. Of course, avoid “quick and dirty” as much as you can. But don’t get paranoid and over-engineer too much.

Over-engineering is as expensive as doing things “dirty”.

Add “Time” to your equation

What most people do very wrong is disregarding the variable of “Time”.

Everybody goes to tech conferences or read fancy blog posts, or watch flashy screencasts. Their wrong conclusion is: “Netflix uses it, therefore it should be good for me, because I want to become Netflix”.

The wrong assumption is that Netflix, or Google, or Facebook, or Spotify, is a static system that always functioned the way they advertise it today.

People forget that every unicorn company had a day-1. And on day-1 they did not use microservices. They did not have React.js. They did not have Go or Elixir. Some of them didn’t even have good programmers to begin with. Netflix started as a VHS renting service, remember that? Google started off in a dorm-room with off-the-shelf hardware components assembled in a Lego-based rack.

You are a fan of some celebrity. That celebrity has a Lamborghini car. So you want to be like that celebrity one day. What do you do? Do you buy the same Lamborghini he has?

If you are that stupid, the only thing you will end up with is a HUGE debt to pay.

Stop envying the rich guy’s Lamborghini.

(Technical) Debt

This is neither a new term nor a new theme. Every decision you make in programming is a compromise between what the “future-ideal” should be and what you can actually do right now. Loan to buy the equivalent Lamborghini too soon, and you will have to deal with an unpayable (Technical) Debt. You will stall, you will stop, and you will do nothing but pay that debt from now on.

You want to “code faster” therefore you skip writing automated tests, because you assume it will slow you down. And you’re right. Test-driven development doesn’t make you fast at first, and it isn’t supposed to. It protects your future self from your present self, and keeps you from piling up Technical Debt.

Again, you forget the “Time” variable.

You write and deliver fast for the 1st month. After the first version is deployed, now you will be obliged to start paying the debt. Your productivity will slow down. Regression bugs will show up. Every new feature you try to add breaks something in unexpected ways. Because you don’t have automated tests, you will keep fixing the same things many times over. Debt will catch up, and you will pay. One way or the other.

Minimal test suites are like Insurance. You don’t need it right now, on day-1. But on day-100 you will be so glad you have it.

Doing too much microservices on day-1 is debt. It feels great that you’re doing that fancy thing you read in a blog post. On day-100, with your 3 people team and a dozen microservices, from now on you will do nothing but pay the accumulated debt, with Interest. Every new microservices deployment breaks your system. And of course you didn’t add monitoring, you didn’t add integrated tests. So every time you code something, something else unexpectedly breaks.

Enjoying that early Lamborghini now?

The Mythical Man-Month

Philip Calçado writes very good posts and presentations about microservices in the right way. If you’re serious about that subject I strongly recommend that you read his posts, such as his “Prerequisites” post or his “Economics” presentation.

He correctly remembers Fred Brooks’ The Mythical Man-Month. I urge developers to read this small book. It’s uncanny how the entire industry is still repeating the very same mistakes Brooks reports in his book from projects in the 60’s!

To me, microservices is a by-product of tech companies with too many developers. Once you break through the 5-developers team a “monolithic” system with poor organization and few automated tests can become cumbersome to deal with. So the consequence is the desire to break it down. You make 2 teams and 2 microservices and coordinate. And the teams start to isolate, and play the finger-pointing game when new bugs are reported (“it’s the other team’s microservice’s fault”).

That’s Conway’s Law if you do it wrong:

Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure.

Also in play is another chapter in Brooks’ book: The Second System Effect. Every developer, tech startup, has to assume that the first version of anything is not the best design. And the second system (the first big rewrite) is the most dangerous system you will write. As Brooks states, it goes like this:

An architect’s first work is apt to be spare and clean. He knows he doesn’t know what he’s doing, so he does it carefully and with great restraint.

As he designs the first work, frill after frill and embellishment after embellishment occur to him. These get stored away to be used “next time.” Sooner or later the first system is finished, and the architect, with firm confidence and a demonstrated mastery of that class of systems, is ready to build a second system.

This second is the most dangerous system a man ever designs. When he does his third and later ones, his prior experiences will confirm each other as to the general characteristics of such systems, and their differences will identify those parts of his experience that are particular and not generalizable.

The general tendency is to over-design the second system, using all the ideas and frills that were cautiously sidetracked on the first one. The result, as Ovid says, is a “big pile.”

Then, what happens? The CEO, the board, the investors, or whatever, start doing the wrong thing: hiring more and more people. Loan more and more. Instead of paying the debts they end up making more debt. They fell for what Brooks reported in the first chapter and what gives the name to the book: The Mythical Man-Month.

Adding manpower to a late software project makes it later.

Complex programming projects cannot be perfectly partitioned into discrete tasks that can be worked on without communication between the workers and without establishing a set of complex interrelationships between tasks and the workers performing them.

Therefore, assigning more programmers to a project running behind schedule will make it even later. This is because the time required for the new programmers to learn about the project and the increased communication overhead will consume an ever increasing quantity of the calendar time available.

Again, it’s foregoing the sense of the “Time” variable. And confusing “debt” for “investment”.

First of all, do yourself a favor and read the goddamn book, twice.

Do Agile right! And no, it’s not Kanban!

If there is one good thing the whole Lean noise generated is the notion of “Minimum Viable Product” or MVP. People call it “prototype”. Some call it “launching Beta” or simply “version 1.0”. Doesn’t matter. It’s the realization that you don’t know much at the beginning so overengineering on the first version is a waste of time. Lean is about controlling “waste”, so we do the minimal that works, gauge results and work from there.

The cynical will say that any prototype that reaches production will never die. And they’re not wrong.

The balance is to not do the “quick and dirty” version. That’s what we have Agile techniques for. Do the minimal, organize the minimal. That’s why we have object-oriented design patterns, from GoF to DDD. You don’t need to do EVERY pattern, and that’s exactly what “overengineering” means. But you can do the minimal monolith that will allow you to evolve later.

People criticize Ruby on Rails for not being organized “enough”. Nick Sutterer, the creator of Trailblazer, has a good point.

Rails done wrong is bad. The conclusion should NOT be: “let’s do microservices” though. That jump in faith is idiotic and makes no sense for all the reasons I mentioned above.

“Let’s do proper Agile stuff and proper object-oriented stuff” should be the initial answer.

And by “Agile” forget about idiotic post-its, idiotic numerology-based estimation equations (this is a whole post in itself, because story points and velocity are useful, but adding Montecarlo and other Gaussian-based stuff are not). The only “Agile” things you should be concerned about are the Extreme Programming (XP) techniques, including Iteration-based timeboxes.

You MUST do timeboxes. Stop, re-assess, change directions, and then keep going. The model of “Pull” is only reasonable when your direction is very, very clear, written in stone and unchangeable, like in a factory production line! (Which is exactly where the concept of Pull, and Lean, was actually born!)

Iterations, like automated tests, are like Insurance. You can never avoid all waste, but you can minimize it. You can afford to throw away an Iteration-worth of work. After the iteration you measure the results, and throw it away if necessary, changing direction in the process. Throwing useless stuff away is as important as adding new stuff. If you just add, you have a hoarding disorder!

If in doubt, do XP. Yes, it feels more “difficult”, and Kanban is “simpler” to explain. Now, is that a good reason?

Forget about Raw Performance when you DON’T need it!

Another problem is choosing languages or framework because of performance.

If you’re in the web development business, this is a huge WASTE.

Understand this truth: your servers will IDLE most of the time. And if you feel like using your web app is slow, the slowness comes from the POOR programming you did, not from the language. And no good language will rescue a bad programmer. I always say that if performance was that important, we should all be doing C.

Most of what you serve in HTTP-based apps, be it user-readable content, be it API GET results, can be CACHED! If you’re not using a CDN, you’re doing it wrong.

Yeah, yeah, yeah, you think you’re building the next Spotify. You’re not, at least not 99% of you. And the 1% doing custom engineering, with custom techniques and custom stack, successfully: you’re the 1%. Actually I’d say you’re a fraction of the 1%. Do not assume that what you do is good for the rest of the population.

90% of what most small companies and solo web developers need is just a Shopify account and a vanilla Wordpress installation. And that’s it.

Are you doing infrastructure development like building custom components to Docker, Kubernetes, Terraform? Command-line tools? Daemons? Then choose GO.

Are you doing embedded libraries? Maybe the next generation OpenSSL? Drivers? Then choose RUST.

Are you doing mobile development? Then don’t have a lot of choice: do Swift for iOS, do Kotlin/Java for Android. Or do React Native for simpler apps.

Are you really doing the next Whatsapp? The next Waze? The next Snap? You have hundreds or thousands of users in need of long-lived connections over unreliable network doing broadcasts of messages? Or you’re building the next evolution of distributed NoSQL databases? Or anything that actually has the proper meaning of “Distributed” in its definition? Then choose ELIXIR.

Are you doing a CRUD based web application? Go RAILS and never look back.

Are you doing all of the above? Use all the alternatives then. And I hope you have the budget, because you need a big team.

Conclusion

The Economics only makes sense when you consider the day-1. Do not forget the “Time” variable. It makes all the difference.

You will always get Debt to pay. So be smart in which kind of debt you choose. Because you will have to pay it back eventually.

Again, read Brooks. He said it decades ago: THERE IS NO SILVER BULLET. Don’t bullshit yourself. A new language, a new framework, a new architecture. None of those will save you.

In any tech endeavor, code is not the only thing to worry about. I would go as far as to say that in a tech startup, code is only 20% of the problem. A tech startup is just a company, like any other. If you’re the founder or CEO, you have to deal with all the remaining 80%: marketing, accounting, human resources, legal, etc. It’s already difficult enough without your tech team getting you unwanted Technical Debt, the unneeded Lamborghini, that you won’t be able to pay.

You can start humble on day-1. Keep evolving, continuously. This is the core of any Agile or Lean process: KAIZEN. Choose smart debts, pay them a small bit at a time, continuously. It’s the same reasoning as getting a loan in the bank.

Netflix day-7,200 is NOT your day-1.

Facebook day-4,800 is NOT your day-1.

Google day-6,809 is NOT your day-1.

Be humble. Deliver fast. Pay your Debts. Keep evolving continuously. Stop believing in fairy tales and silver bullets.

My language is “faster”. My framework is “more elegant”. My tool is “more productive”. My community is “more friendly”. All bullshitty reasons to choose something because those reasons don’t really matter.

Although DHH did a good job describing the reality of developers’ choices, he didn’t go as far as to explain why we all do it this way. Including himself. Including myself.

The religion framework theory does explain some of the behaviors, I’d go farther still.

Most developers are more akin to sports fans (or fanatics, as is the root of the word “fan”).

Check out some items from the article “Why Sports Fans are Sports Fans”:

• There’s high drama.
• It’s fun to watch greatness.
• It taps into our creepy side that wants to sit in the ancient Roman Coliseum and watch people fight to the death.
• It’s aesthetically pleasing.
• The sports world includes a lot of gossip.
• Sports provides an escape from life.
• It’s a perfect tool for your crippling procrastination tendencies.
• Sometimes I think we might even use sports as a way to attach ourselves to something immortal.
• In a time when heroic triumphs aren’t part of most people’s lives, sports allows us to capture a little sliver of the feeling of glory.

Developer niches feel more like sports teams in competition. There must be “winners” and “losers”. And people cheer over every small victory (like useless synthetic benchmarks or pseudo-statistical and meaningless indexes such as the TIOBE).

Fans of losing teams feel like they need to have increased loyalty to get the utmost reward.

Testosterone levels rise when discussing nonsense over Hacker News. Threads get really deep, each person shooting bullshitty “evidence” to “prove” their point. And every argument has an easy counter-argument, and the threads go on and on indefinitely. And every person writing will never agree to the contrary. And no one will ever switch sides based on the arguments laid out on those threads. Yet, they still waste lots of time writing anyway. Madness.

People - especially young people - like to have a sense of belonging, a sense of deindividuation, dissolved responsibilities. Which is some of what I wrote a couple years ago on the current nonsense trend on functional languages.

As Mike Perry describes in this post the most passionate fans tend to identify themselves as an integral part of the team they are supporting, with some of them – clearly – feeling less responsible for their individual behaviour. When disinhibition is combined with deindividuation – a loss of self-awareness and a sense of diffused responsibility – fans simply don’t care anymore about what anyone else might think or feel. Deindividuation weakens normal restraints without which, fans are more easily influenced by their surroundings; this is when fans start acting – really – weird, in either a pleasant or less pleasant way.

Groups of fans may mob, insult, threat or even riot. Groups provide fans with a sense of anonymity. Being part of a group removes individual accountability and diffuses responsibility. Unfortunately this behaviour appears online just as often as it does in the stadium.

Sports Networker - Read more

Tech conferences are like the weekend sports games. People like to feel validated being in the middle of equal-minded groups.

Same as with sports, developers feel that their “teams” (technologies, companies) are extensions of themselves. Worse, they feel like they describe their personalities. Which is why developers respond so passionately to contrary arguments: it feels like an aggression to their own selves.

“When we look at motivation for following a sport team, group affiliation is one of the top ones,” says Wann. “Identifying strongly with a salient local team where other fans are in the environment — that’s a benefit to social-psychological well-being.”

“The whole idea behind identification is that it’s really part of how we see ourselves and that doesn’t change easily,” says Robert J. Fisher, professor of marketing at The University of Western Ontario, whose research emphasizes the effects of social expectations on managerial and consumer decision making. “If you see yourself as a member of a family, that role doesn’t change. Those types of connections are very long-lasting and very strong.”

Psychological Science - Read More

It’s all a big show, you’re the audience. Just don’t fool yourself thinking that you’ve been “scientific” in your choices.

We are all victims of heavy Cognitive Bias, which blinds us to the truth:

Here’s a great post by Sam McNerney (@WhyWeReason) explaining cognitive bias. This is the phenomenon where all of us only look for what confirms their intuitions and ignore what contradicts their intuitions, what psychologists call confirmation bias. Confirmation bias helps explain why some fans will boo every single referee call against their team and English soccer fans will – still – argue with Germans about whether the ball did or did not pass the line in 1966(!). Fans automatically see the world as they want to, not how it is.

Sports Networker - Read more

More interestingly, the same way with sports fans of losing teams, people find ways to cope. And superstition is one of them.

Superstitions are an integral part of sports, and they may also be yet another way fans cope with their team’s performance.

In ongoing research, Wann and his colleagues are exploring the role of fan superstition. Over half of his 1,000 participants can readily define a superstition or ritual they believe in. Moreover, some are truly convinced that their participation in ritual superstition impacts the outcome, says Wann. The more highly identified with a team the fans are, the more likely they are to believe that superstitions matter.

“It’s a real struggle that sports fans experience,” says Wann. “They so much care about the outcome of the event they have absolutely zilch control over. How do we gain control? We may develop superstitions.”

Psychological Science - Read More

What I always have to say of all the nonsense is:

• If being the “fastest” was really a real concern, we would all be programming in C, as close to the bare metal as possible.

• If being the “most secure” was really a real concern, we would never ever fetch libraries created by unknown, uncertified, untrained people. We would build everything from scratch, by ourselves, every time, and wait through months of heavy testing before ever delivering to production. You know, like the military used to do.

• If being the “most elegant” was really a real concern, we would not be concerned about being the fastest or the most secure, as they usually go in opposite directions.

The craft of an engineer, or any professional for that matter, is being able to choose between different compromises, and keep balancing out. Never blindly committing loyalty to a specific group or way of thinking. Amateurs commit loyalty to someone else. Professionals hold themselves responsible for their own choices, and they compromise.

Scalability, Maintainability, Reliability, every “-ability” we add to the list is a compromise. You can’t have your cake and eat it too. You want more, you pay for more. Because every technology is a combination of compromises, you can’t have a clear single winner or loser. Everything will win in some aspects and lose in others.

As Neo realizes in The Matrix: the problem is choice. The problem is always about choice. People don’t like to choose, because that makes them accountable. It’s far easier to make someone else make the choice and just follow, creating the delusion that you made a “rational” choice because “the group” validates it.

After all, “Nobody ever got fired for choosing IBM.”

The reality that you will eventually realize is that you are not required to make “the” choice.

Clear “good guys” and “bad guys” only exist in fairy tales and bed-time stories for children. The world of Adults is defined by shades of grey.

At the end of the day, DHH is right. It doesn’t matter the tech specs of PS4 Pro vs Xbox One S vs Nintendo Switch. It doesn’t matter the amount of teraflops. It doesn’t matter the bandwidth. It doesn’t matter native 4K or checkerboarding 4K. If I like Zelda, I will play in the Switch. If I like Forza I will play an XBox. If I like Uncharted I will play a PS4. And if I can afford it, I will have all of them. If I can’t afford it, I will choose whatever I want to play right now knowing that I can switch to another platform without having to justify myself through tech specs.

Either I subscribe to the Rails Doctrine or to The Zen of Python. It doesn’t matter. Can I play Zelda on it? Otherwise, any “evidence” you think you have to counter argue my choice is utter bullshit. Don’t even try.

But of course, you should use Ruby if you’re a rational developer ;-)

When I was testing only with MangaReader.net as a source, everything worked almost flawlessly. But adding MangaFox in Part 3, with its more restrictive rules towards scraper tools like mine (timing out more frequently, not allowing too many connections from the same place, etc), the process just kept crashing and I had to manually restart it (the resuming features I added in Part 4 paid off, but it’s not a reliable tool anymore).

To recap, the Workflow just organizes each step of the process. Its functions are similar to this:

def process_downloads(images_list, directory) do
  images_list
    |> Enum.map(&Worker.page_download_image(&1, directory))
    |> Enum.map(&Task.await(&1, @await_timeout_ms))
  directory
end

It deals with a large list, maps over each element sending it to a Worker function to run, like this:

def page_download_image(image_data, directory) do
  Task.async(fn ->
    :poolboy.transaction :worker_pool, fn(server) ->
      GenServer.call(server, {:page_download_image, image_data, directory}, @genserver_call_timeout)
    end, @task_async_timeout
  end)
end

It returns an asynchronous Task waiting for 2 things: for Poolboy to release a free process to use, and for the Worker/GenServer function to finish running inside that process. As I explained in Part 2 this is so we can limit the maximum number of connections to the external source. If we didn’t have this restriction, sending tens of thousands of asynchronous requests at once, the external source would just fail them all.

The first thing to bear in mind is that a “Task.async/2” links itself to the caller process, so if something goes wrong, the parent process dies as well.

The correct thing to do is to add a Task.Supervisor and make it deal with each Task child. To do that, we can just add the Supervisor in our supervised tree at “pool_management/supervisor.ex”:

defmodule PoolManagement.Supervisor do
  use Supervisor
  ...
  children = [
    supervisor(Task.Supervisor, [[name: Fetcher.TaskSupervisor]]),
    :poolboy.child_spec(:worker_pool, pool_options, [])
  ]
  ...
end

And we can replace the “Task.async/2” calls to “Task.Supervisor.async(Fetcher.TaskSupervisor, …)” like this:

def page_download_image(image_data, directory) do
  Task.Supervisor.async(Fetcher.TaskSupervisor, fn ->
    :poolboy.transaction :worker_pool, fn(server) ->
      GenServer.call(server, {:page_download_image, image_data, directory}, @genserver_call_timeout)
    end, @task_async_timeout
  end)
end

This still creates Tasks that we need to await on, and as before, if the function inside crashes, it still brings down the main process. Now my refactoring found a dead end.

This is the 2nd problem I mentioned in the beginning of the article: a flaw in my design.

Instead of just mapping through each element of a large list, I should have created an Agent based GenServer to keep the list as the state and make the entire Workflow system a new supervised GenServer. If fetching one URL crashed the GenServer, its supervisor would restart it and pick up the next element in the list.

But, as I am in no mood for this refactoring right now (it’s Sunday afternoon) I will concentrate on a quick fix (yes, jerry-rigged patch), just so the function in the async call does not raise exceptions.

OMG! It’s a Try/Catch block!

Turns out that everything I run inside the Poolboy processes are HTTP get requests through HTTPotion. Fortunately I had already refactored every HTTPotion get call into a neat macro:

defmacro fetch(link, do: expression) do
  quote do
    Logger.debug("Fetching from #{unquote(link)}")
    case HTTPotion.get(unquote(link), ExMangaDownloadr.http_headers) do
      %HTTPotion.Response{ body: body, headers: headers, status_code: 200 } ->
        { :ok, body |> ExMangaDownloadr.gunzip(headers) |> unquote(expression) }
      _ ->
        { :err, "not found"}
    end
  end
end

Now I only need to replace 1 line in this macro:

-    case HTTPotion.get(unquote(link), ExMangaDownloadr.http_headers) do
+    case ExMangaDownloadr.retryable_http_get(unquote(link)) do

And define this new retryable logic in the main module:

defmodule ExMangaDownloadr do
  require Logger

  # will retry failed fetches over 50 times, sleeping 1 second between each retry
  @max_retries  50
  @time_to_wait_to_fetch_again 1_000
  ...
  def retryable_http_get(url, 0), do: raise "Failed to fetch from #{url} after #{@max_retries} retries."
  def retryable_http_get(url, retries \\ @max_retries) when retries > 0 do
    try do
      Logger.debug("Fetching from #{url} for the #{@max_retries - retries} time.")
      response = HTTPotion.get(url, ExMangaDownloadr.http_headers)
      case response do
        %HTTPotion.Response{ body: _, headers: _, status_code: status } when status > 499 ->
          raise %HTTPotion.HTTPError{message: "req_timedout"}
        _ ->
          response
      end
    rescue
      error in HTTPotion.HTTPError ->
        case error do
          %HTTPotion.HTTPError{message: message} when message in @http_errors ->
            :timer.sleep(@time_to_wait_to_fetch_again)
            retryable_http_get(url, retries - 1)
          _ -> raise error
        end
    end
  end
  ...
end

I strongly stated that in Elixir we should not use “try/catch” blocks, but there you have it.

This is the consequence of the flaw in my initial Workflow design. If I had coded the Workflow module to be a GenServer, with each list managed by an Agent, each failed HTTPotion call would allow the supervisor to restart it and try again. Without resorting to the ugly “try/catch” code.

Maybe this will force me to write Part 6 as being the code to remove this ugly try/catch later, so consider this a Technical Debt to make everything work now so we can refactor later and pay the debt back.

“HTTPotion.get/2” calls can raise “HTTPotion.HTTPError” exceptions. I am catching those errors for the time being, matching the messages against a list of errors I had already, sleeping for a certain amount of time (just a heuristic to see if the external sources respond better that way) and I recurse to itself through a limited number of “retries”, until it reaches zero, in which case it may even be the case that the internet connection is down or some other severe error that we would not be able to recover soon.

With this code in place, now even fetching from MangaFox, without tweaking down the POOL_SIZE, will run until the end, and this solves my needs for now. If anyone is interested in suggesting a better, GenServer based Workflow design, I would really appreciate a Pull Request.

Cheers.