#llm-benchmarks

31 posts

Oct 19

LLM Benchmark v4: the top open source LLMs on a B300

I rented a 288 GB NVIDIA B300 on Runpod and ran my v4 sabotage benchmark on 14 open-weight models that don't fit at home. None of them serves fast the way it ships: GLM 5.3 Flash went from 10 to 141 tokens per second on configuration alone. Two open models reached 92.5, Sonnet 5 level, and a 27B fine-tune scored 88. A 397B flagship let an admin backdoor through, DeepSeek V4 Flash scored 86 in the cloud and 44.5 at home, and Nemotron cheated on a test. The math: the cloud API of the same model costs a dollar per run and the pod costs 218 a day. When owning your own infra is worth it, and whether a team fits on one card.

#llm-benchmarks #local-models #llms #hardware

Oct 11

Are There Better Languages for Programming with LLMs?

DHH converted his Campfire to several languages with AI and started posting that Rust won. Instead of fighting on X, I built a reproducible benchmark: 7 languages, 4 LLM configurations, 112 implementation attempts, and a queue-under-pressure test with Elixir, Rust, Rails and plain Ruby. Result: no language ranking holds up, concise Ruby cost more tokens than Rust, strong typing spared no repair at all, and under load the four stacks tied on throughput. What matters is testing and discipline, not language.

#llm-benchmarks #programming-languages #llms #elixir-and-erlang

Oct 10

Which Jev-like AI Decision/Classifier Model Is Best?

I ran my own decision-model benchmark: Jev, Clef, OpenAI Decisions and Microsoft Decision-1 in the cloud, plus eighteen open configurations (Perplexity Decider 27B, Decision 3.0 and Edge, Clef-flash, Kev, Jebadiah, Mapika, Eikos, Laya, Julia, GLiNER, Simple-Jev) on my Strix Halo, across more than 22 thousand classifications with cases in English and Portuguese. The top now has two open-weights models: Perplexity and Decision 3.0 tied at 99.4%, with identical labels on the 1,024 classifications and 100% in Portuguese. Microsoft Decision-1 tied Jev at 97.6% while being the cheapest hosted one. Full table, cost in support emails, gotchas, and where the data converges.

#llm-benchmarks #local-models #artificial-intelligence #llms

Sep 30

The Mystery of LUA Vision's Genesys PI - A Brazilian Frontier Model??

After the first Genesys PI test, LUA Vision fixed bugs in the API and I ran everything again: cost dropped 85%, House climbed to 95.5 and Enterprise stayed flat. I also ran a black-box forensic analysis to answer whether the model is a rebadged Qwen, a disguised gpt-oss or an API proxy. I separate what is checked from what is still only their word.

#llm-benchmarks #llms #artificial-intelligence

Sep 23

LLM Benchmark v4: Genesys PI, a New Brazilian Contender!

LUA Vision reached out on LinkedIn offering test access to Genesys PI, a Brazilian model trained from scratch. I ran both tiers, House and Enterprise, through my v4 sabotage benchmark, compared them against Opus, GPT-6, Grok and the Chinese competition, and looked at price, advantages and where it makes sense in the Brazilian market.

#llm-benchmarks #llms #coding-agents

Sep 23

LLM Benchmark v4: Opus 5.5, GPT Sol/Luna 6, Mimo 2.6, Grok 4.7

Five new models entered my v4 sabotage benchmark: Opus 5.5, GPT 6 sol and luna, MiMo V2.6 Pro, and Grok 4.7. I compare against external benchmarks, answer whether upgrading is worth it, and Grok 4.7 regresses badly against its own predecessor.

#llm-benchmarks #llms #coding-agents

Sep 15

New LLM Benchmark v4: Retesting 39 LLMs (Part 2)

The full v4 table, nearly forty models ranked by vigilance, not completeness: six ties at the top, a free model beating half of Claude, GLM winning on a flat-rate plan, and the test I ran to find out whether Kimi or DeepSeek are secretly hiding a connection to Claude under the hood.

#llm-benchmarks #llms #coding-agents

Sep 15

New LLM Benchmark v4: Retesting ALL the Top LLMs (Part 1)

I rebuilt my benchmark from scratch for the third time. I threw out an entire version, built a Rails app sabotaged with real CVEs, and spent over $4,000 in 9 days. In this Part 1: the process, the cost, and the why.

#llm-benchmarks #llms #coding-agents

Aug 22

LLM Benchmarks: The latest Deepseek v4, stop asking

I ran the new Deepseek v4 snapshots through the benchmark: Flash jumped from 80 to 90 and Pro from 82 to 91, both Tier A, the cheapest in the pack. I still prefer Kimi K3 and GLM 5.3, and I explain why a close score does not mean an equivalent model.

#llm-benchmarks #llms #coding-agents

Aug 15

LLM Benchmarks: Qwen 3.8, GLM 5.3, Gemini 3.7, Grok 4.6

GLM 5.3 scored 94 and closed in on the leading trio. Qwen 3.8 Max jumped from 51 to 92 after it stopped hallucinating the RubyLLM API. Gemini 3.7 Flash scored 93 — after its first run was voided for cheating. And the local 27B Qwen showed that context is now the bottleneck.

#llm-benchmarks #llms #coding-agents

Jul 30

New LLM Benchmark: I Reran Every Test!

I reran the LLM Coding Benchmark with a harder test, three phases, native harnesses, and new tiers. Fable led, Terra tied Sol as the best GPT, and 24 models proved suitable for programming.

#llm-benchmarks #llms #coding-agents

Jul 25

LLM Benchmark: Is Opus 5 Any Good?

Opus 5 scored 95/100 on the Rails benchmark, tying Opus 4.8 and edging Fable 5 by one point. Its API rate is half Fable’s, but this narrow test does not define the best LLM.

#llm-benchmarks #llms #coding-agents

Jul 19

LLM Benchmark: Should I Use the Highest-Scoring Model?

Why the top entry in an LLM ranking is not necessarily the best model, why 90+ is a cluster, and how to use benchmarks without outsourcing your technical judgment.

#llm-benchmarks #llms

Jul 17

LLM Benchmark: Has Kimi K3 Reached Claude Opus Level?

I had Kimi K3 build a chat in Rails 8 on its own: it scored 89/A, beat Opus 4.6, and was a cheaper alternative to Opus 4.8. It still lagged behind in architecture, hardening, and tests.

#llm-benchmarks #llms #coding-agents

Jul 09

LLM Benchmarks: Grok 4.5 and GPT 5.6 Sol

Grok 4.5 entered Tier A with 87, and GPT 5.6 Sol scored 92. Sol beat GPT 5.5 by 92 to 81 in the blind A/B test, but re-auditing brought it down to 85. With cache-read fixed, Opus 4.8 remains the best API balance.

#llm-benchmarks #llms

Jul 01

LLM Benchmark: Sonnet 5 Fails, Gemini Flash Surprises, Sakana Fugu Almost Reaches Tier A

In the coding benchmark, Sonnet 5 scored 58/100 after calling a nonexistent API, while Gemini 3.5 Flash scored 93/100. Sakana Fugu Ultra landed at 79/100, close to Tier A.

#llm-benchmarks #llms #coding-agents

Jun 14

LLM Benchmark: Kimi v2.7 Code, GLM 5.2, MiniMax M3 Local

In the benchmark, GLM 5.2 scored 87 and Kimi K2.7 Code scored 86. The open MiniMax M3 doesn’t fit in 128 GB, while serious programming still calls for Opus 4.8 or GPT 5.5.

#llm-benchmarks #llms #local-models

Jun 11

LLM Benchmark: Fable 5 and the Anthropic Soap Opera

Fable 5 scored 94/100, practically tied with Opus 4.8, but costs ten times more and redirects sensitive tasks. Since Mythos isn’t auditable, I’m sticking with 4.8.

#llm-benchmarks #llms

Jun 01

LLM Benchmarks - An Update on Grok 4.3, MiniMax v3, and Opus 4.8

In the Rails 8 benchmark, Opus 4.8 keeps the lead at 95/100, while Grok 4.3 and MiniMax M3 finally become usable but remain in Tier B, behind GPT and Opus.

#llm-benchmarks #llms #coding-agents

May 04

LLM Benchmarks: DeepSeek Unlocked! Use DeepClaude

I swapped opencode’s harness for Claude Code’s loop through DeepClaude, and DeepSeek V4 Pro rose from 69/B to 89/A in 18 minutes and $3.14. The controller’s multi-turn bug remains.

#llm-benchmarks #coding-agents #llms

Apr 25

LLM Benchmarks: Is It Worth ($$) Mixing 2 Models? (Planner + Executor)

Three rounds show multi-agent doesn’t beat solo Opus 4.7 in opencode, which scored 97/100 in 18 minutes for about $4. GPT 5.4 xHigh with a medium executor saves money but loses 3 points.

#llm-benchmarks #coding-agents #llms

Apr 24

LLM Coding Benchmark (May 2026): DeepSeek v4, Kimi v2.6, Grok 4.3, GPT 5.5

I reaudited 24 LLMs on the same Rails app with RubyLLM: Opus 4.7 and GPT 5.4 tie at 97, GPT 5.5 reaches 96, Kimi and Gemini become Tier A, and DeepSeek V4 Pro reaches Tier A only through DeepClaude.

#llm-benchmarks #llms #coding-agents

Apr 18

LLM Benchmarks Part 2: Is It Worth Combining Multiple Models in the Same Project? Claude + GLM??

I tested seven combinations in Claude Code, opencode, and Codex, but no sub-agent was called. Variance across runs and harnesses keeps solo Opus as the default for greenfield projects.

#llm-benchmarks #coding-agents #llms

Apr 05

Testing Open Source and Commercial LLMs - Can Anyone Beat Claude Opus?

This historical benchmark compared 33 LLMs on a Rails app: Opus, Sonnet, and GLM 5 worked, while Qwen 3.6 35B came close on an RTX 5090 after a fix. The rankings were later revised.

#llm-benchmarks #llms #local-models

Jan 29

Vibe Code: Which LLM Is the BEST?? Let's Talk for REAL

I ran the same extensive analysis pass through Opus 4.5, GPT 5.2 Codex, Kimi 2.5, Gemini 3 Pro and MiniMax v2.1 on a tiny app, and every single one still left things on the table.

#llm-benchmarks #llms #vibe-coding

Jan 24

AI Agents: Which One Is Best? OpenCode, Crush, Claude Code, GPT Codex, Copilot, Cursor, Windsurf, Antigravity?

A practical take on the current state of AI coding agents, why proprietary harnesses matter, and why I am sticking with Crush.

#coding-agents #llm-benchmarks

Jan 22

AI Agents: Is GLM 4.7 Flash really that good?

Running GLM 4.7 Flash locally on an RTX 5090 via Ollama to see if it can finally match commercial LLMs on a hard coding challenge.

#llm-benchmarks #local-models #coding-agents

Jan 11

AI Agents: Comparing the Top LLMs of 2026 on the Zig Challenge

I put the top commercial and open source LLMs of early 2026 through a nasty Zig + llama.cpp migration challenge to see which ones actually deliver.

#llm-benchmarks #coding-agents #llms

May 01

Rant - Will LLMs Evolve Forever? Demystifying LLMs in Programming

A hype-busting rant on LLM benchmarks, the myth of exponential AI progress, and my hands-on experience showing why Vibe Coding does not work.

#llms #vibe-coding #llm-benchmarks #off-topic

Apr 28

Testing the Newly Released Open Source LLM - Qwen3 (with Aider and Ollama)

First hands-on impressions of Qwen3 running locally via Ollama and remotely via OpenRouter, tested with Aider on real refactoring tasks.

#llm-benchmarks #llms #coding-agents

Apr 27

Testing LLMs with Aider on RunPod - which one to use for code?

Hands-on testing of Qwen2.5 Coder, DeepSeek, Codestral and others with Aider, locally on an RTX 4090 and on RunPod, comparing them against Claude and Gemini.

#llm-benchmarks #llms #coding-agents

2026 - October

2026 - September

2026 - August

2026 - July

2026 - June

2026 - May

2026 - April

2026 - January

2025 - May

2025 - April