LLM Benchmark: Has Kimi K3 Reached Claude Opus Level?

I am always testing new LLMs. It is the only way to separate a release note from code that actually goes up. In the two most recent posts, Grok 4.5 and GPT 5.6 Sol and Sonnet 5, Gemini 3.5 Flash and Sakana, the table moved around quite a bit. Now it is Moonshot’s turn.

The focus is Kimi K3, an alternative that is becoming more and more like Claude in the kind of work it can finish. And a silly but important detail when looking for documentation: its official name is Kimi K3, not “Kimi v3”.

For those who just landed here

The benchmark gives every model the same prompt: build, on its own, a ChatGPT-like chat app in Rails 8, RubyLLM, Hotwire, Docker, with tests and CI. Then I inspect the project it produced, including the gem’s real API, conversation memory, error handling, tests, and production image. The score goes from 0 to 100, in A/B/C/D tiers.

This is not a fill-in-the-function benchmark. It is an agent delivering a small app, with the boring parts that usually break after the demo. The methodology is in Part 3; code, logs, and the rubric are in the benchmark repository.

First, the harness problem

Previous Kimi rounds used OpenCode through OpenRouter. K3 did not get through the door: Moonshot’s strict tool-schema validator rejected the schema OpenCode sends:

when using anyOf, type should be defined in anyOf items instead of the parent schema

A controlled probe with a clean schema got tool calling to work. The problem is in the OpenCode ↔ Moonshot pairing. K2.5, K2.6, and K2.7 worked with the same tooling. Be careful with headlines saying “the model does not support tools”: sometimes the bug is in the fit.

So as not to abandon the test, a first-class runner for Kimi Code CLI was added. Phase 1 calls:

kimi -p <prompt> --output-format stream-json -m <model>

and phase 2 resumes the session with -S <session_id>. The runner code and configuration are in commit 0cdf88a. The original error and investigation were recorded in commits c3ab6d5 and 506357b; the final K3 project landed in 2bf1d7b.

This also puts an honest asterisk on the score. I wanted to compare the whole family under the new harness. The Kimi subscription exposes K3 and managed K2.7-Coding, but not K2.5/K2.6. We reran K2.7 through the CLI and compared it with the existing K2.5/K2.6/OpenRouter artifacts. K2.7 CLI came in at an informal 68/B; K2.7 OpenCode/OpenRouter had scored 86/A. This is harness and snapshot sensitivity, not a clean causal claim that “the harness costs 18 points”: managed kimi-for-coding may not be the same public K2.7 snapshot.

Current ranking: 40 models

K2.7 CLI is informal and unranked, so it is not included. The cost column is the actual cost at the round’s provider/OpenRouter or a subscription estimate, not an abstract rate card.

Kimi evolved quickly. Its price even more so.

K2.5 scored 69/B in February. K2.6 jumped to 87/A in April. K2.7 stayed at 86/A in June. Now K3 reaches 89/A in July. There was a big jump with K2.6, a plateau in K2.7, and K3 pushes the ceiling a bit higher.

From the official spec sheet, only what matters here: K3 has 1M context, MoE architecture (mixture of experts, which activates only parts of the model for each token), and maximum reasoning mode. Useful for understanding the product’s ambition. It does not replace opening the project it wrote.

What K3 actually wrote

There is quite a lot right in the Conversation model. It builds the chat, replays only the prior history, and then calls ask(content); the current message enters the collection only after the response. No double-send. If the provider fails, the exception is raised before append, so a failed turn is not stored:

response = build_llm_chat.ask(content)
append(Message.new(role: "user", content: content))
append(Message.new(role: "assistant", content: response.content.to_s))

It also got persistence better than the global-hash pattern: Rails.cache with a one-day TTL, a key per session, and a 20-message limit. Docker is production-grade, non-root, and validation performed a real chat both locally and in the container. That is real Tier A, not an optimistic README.

But you can still see the signature of a generation. It does not use with_instructions, so there is no system prompt. The LLM call lives inside the domain model. There is no preflight for a missing key.

Each individual message can grow without a ceiling and there is no rate limit. The production cache remains the ephemeral default. The read + mutation + write sequence races between requests. Raw provider errors appear in the UI. The error suite is smaller than it should be.

Against Opus 4.6, K3 clearly wins in this artifact: correct public RubyLLM API, bounded state, and tests for provider failures. Opus 4.6 manipulates chat.messages directly, stores an uncapped cookie, and does not cover provider errors.

Opus 4.8 models preconditions, the system prompt, and replay better. The basic RubyLLM call is comparable. The distance appears in hardening, architecture, and test depth.

Fable 5 is more defensive, but charges for it: original/re-release 94/93, against 89; $10/$50 per million, against $3/$15; measured runs of $11.20/$8.30, against ~$2.10 API-equivalent. One run per cell does not provide statistical certainty. It provides concrete code to compare.

Pricing: Kimi is no longer the absurd bargain

It is important here to separate Moonshot direct API, OpenRouter, and subscription. This is the official direct API rate card, per million tokens, with cached input separated:

Official documentation: K2.5, K2.6, K2.7 Code, and K3. For intermediary pricing and availability, also see the K3 page on OpenRouter.

K3 costs 3.16× more for input and 3.75× more for output than K2.6/2.7. Moonshot left the ultra-cheap niche. The costs in the ranking table are something else: K2.6 ~$1.19 and K2.7 ~$1.23 were recomputed from the provider/OpenRouter and the run tokens; K3 is an API-equivalent estimate of ~$2.10 for a subscription-billed run. Mixing the two without identifying the channel creates a false comparison.

Even so, K3 costs 40% less than Opus 4.8 at official rates ($5/$25) and 70% less than Fable ($10/$50). On the $19 Kimi Moderato plan, this is excellent for interactive solo coding. Just do not automate recklessly: we observed around two heavy rounds per five-hour window and quota recovery in 4h16. For unsupervised batch work, a mid-run 403 throws away work and time.

Moderato is too small for serious work

This is the part the monthly price hides. I used the $19/month Moderato. Its quota is organized in rolling five-hour windows: use a lot now, then wait for the window to slide forward before capacity returns. A benchmark run is not one loose prompt. It is exactly two prompts/phases: build the app, then resume the same session to validate and fix it.

K3 completed that whole run in 26 minutes and 4.83 million tokens, almost all cache-read, about $2.10 API-equivalent. K2.7 Coding completed the same two prompts in 16 minutes and 9.25 million tokens. In the benchmark record, approximately two complete runs at that scale fit before a third received a hard 403 partway through. The partial build was lost. Recovery took 4h16, with seven blocked probes at 20-minute intervals.

One caveat: I did not commit the raw 403/probe transcript. This chronology is a recorded benchmark observation, not something another person can independently audit from raw logs. The documentation says already-running tasks normally finish; our third one did not. That is what I saw in use, not an accusation that the documentation is lying.

Official plans are on the membership page:

K3 on Moderato is limited to 256K context. Allegretto and above unlock up to 1M. Allegretto+ also unlocks HighSpeed, but that mode burns roughly 3x the quota. The 5x/15x/30x are relative credits, not a promise of 5/15/30 times more linear runs. The documentation discloses weekly refresh, the rolling five-hour window, and shared quota across CLI, VS Code, and API-key tools, but does not publish the exact formula for a heavy session or token. Cache, output, and agent behavior change the math a lot.

Moderato is good for one, maybe two heavy sessions in five hours. For five hours of coding a day, it is too small: in a sequence of these tiny benchmark runs, with only two prompts per model, the third attempt already hit cooldown. Cooldown is the wait for the quota window to open again. Allegretto is the minimum upgrade for lighter daily interaction and gets 1M context, but 5x can leave little headroom for someone driving the agent continuously or enabling HighSpeed. For sustained professional use, at least five hours a day, I would go with Allegro: 15x credits and four concurrent tasks. There is still no official guarantee of uninterrupted execution. Vivace is for multiple agents or near-continuous load; for one developer, it is generally overkill.

The other choice is subscription or direct API. The K3 API charges $0.30/M cached input, $3/M uncached input, and $15/M output. It provides explicit accounting, the full 1M context, and does not have the membership weekly quota, although normal API limits still apply. It is the better route for CI, batch work, and long runs worth real money.

A subscription buys something else: fixed monthly cost, convenient OAuth/CLI, near-zero marginal cost within the caps, and a predictable bill. In return, you accept a hidden token envelope, rolling/weekly caps, shared allowance, the risk of stopping midway, and no SLA for unattended automation. The K3 ~$2.10 is only a rough API-equivalent. At that run shape, the Moderato monthly fee arithmetically equals around 9 runs, Allegretto 19, Allegro 47, and Vivace 95 per month. That is invoice arithmetic, not a quota guarantee: real workloads vary wildly with cache and output.

Kimi also offers Extra Usage, which can fall back to paid balance under a spending cap. It helps prevent an abrupt quota stop. The exact USD rate is not published, and it does not turn a consumer membership into an automation SLA.

My plain recommendation: for 5h/day, Allegro. For lighter interaction, Allegretto. For automation and CI, pay-as-you-go API. If you stay on subscription and the work cannot stop, enable Extra Usage with a spending cap.

Separating plan from model: K2.6 remains the best raw-API option for high-volume automation with human review. K3 is a very good option for those coding interactively. Opus 4.8 remains my choice for complex refactors, concurrency, security, and autonomous changes where correctness matters more than the bill. Fable, in this slice, is economically dominated.

Bonus: GLM 5.2 versus Opus

GLM 5.2 is at #9, 87/A, 43 minutes, through a Z.ai subscription. The natural comparison is Opus 4.6, its score neighbor at 83/A and 16 minutes, and Opus 4.8, the quality ceiling at 95/A and 17 minutes.

I read both projects. GLM beats the Opus 4.6 artifact: it uses RubyLLM correctly, applies a system prompt, separates the service, accepts DI, and tests failures. The service even excludes the last message from replay before calling ask, the detail that avoids double-send.

But GLM also leaves skeletons in the closet: uncapped process-local singleton; failed turns are retained; config.hosts.clear; generated secret that invalidates sessions; and npm ci || npm install, which turns the lockfile into a suggestion. It remains clearly behind Opus 4.8 in state, design, tests, and configuration hardening.

Z.ai plans are Lite $18, Pro $72, and Max $160, with a promotion through September 2026: $12.60/$50.40/$112. Against Opus 4.8 at ~$6.40 per run, nominal break-even is approximately 3/12/25 runs per month, or 2/8/18 at the promotional price. But GLM takes 43 minutes where Opus 4.8 took 17. If you already pay for Z.ai, GLM 5.2 is a strong default for routine reviewed work. If correctness and turnaround rule, Opus 4.8.

Conclusion

In this greenfield Rails project, K3 is a good replacement for Opus 4.6 and offers a cheaper alternative to Opus 4.8. Claude still has the edge when the project demands defenses against details nobody puts in the prompt.

There is one run per model and there is harness effect, especially with K3. The benchmark also does not cover database migrations, background jobs, or product tool calls. There is no data here to make promises about those.

My map today: K2.6 for reviewed-volume API automation; K3/Allegretto for lighter interactive solo work and Allegro for a sustained professional routine; Opus 4.8 when the patch is sensitive; GLM 5.2 as a good subscription default for routine reviewed work. No hype. Read the code, run the tests, and treat every ranking as limited evidence, not religion.