What I liked most about Jarred’s post was the level of detail. He did not sell the idiotic fantasy of writing a prompt like “rewrite Bun in Rust” and waiting for a miracle. Quite the opposite. What he described was procedure: PORTING.md, lifetime analysis, LIFETIMES.tsv, a trial run with three files, dynamic workflows, adversarial reviewers, process correction when the agents started doing stupid things with git stash, then compiler errors becoming a work queue, then smoke tests, then the whole suite, then multi-platform CI. No irresponsible vibe coding. Engineering using AI as a multiplier.

That’s why I was disappointed by Andrew Kelley’s response, “My Thoughts on the Bun Rust Rewrite”. Andrew is Zig’s creator. I wanted to read the best possible technical argument defending Zig. I wanted to see him explain where Bun confused a language problem with an architecture problem. I wanted to see which Zig APIs could have helped, which invariants should have been hardened, where comptime, allocators, arenas, errdefer, DebugAllocator, and Zig’s new I/O design could change the analysis.

Instead, a large part of the text becomes an attempt at character assassination. Comments about personality, management, VC, startup culture, backstage gossip. There may even be real resentment there. I don’t care. It does not help technically.

So I decided to do the exercise I expected to have read: put Claude Fable 5 and GPT 5.6 Sol as technical arbiters and ask them for the answer Andrew should have written. Not the polite answer. Not the fan-club answer. The technical answer.

Before handing over the microphone, one off-topic note. If you, like me, stared at those beautiful GitHub animations in Bun’s post and thought “I want that in my projects”, I went there and built it. github-visualize is a self-hosted dashboard that tracks your repositories and replays commit evolution, the hourly heatmap, and the CI race until everything turns green. It’s all open on GitHub.

But now let’s separate support from proof. One thing is to say “I would have done something similar”. Another is to say “it was technically impossible to continue in Zig”. To answer that, we need to look at causes: which bugs were really language bugs, which were architecture problems, which were ecosystem costs, and which could have been reduced with different discipline inside Zig itself.

That’s where the analysis starts.

First, minimum vocabulary. Use-after-free is using memory after freeing it. Double-free is freeing the same memory twice. Leak is forgetting to free. Ownership is who is responsible for destroying a resource. Lifetime is how long a reference remains valid. FFI is the boundary where Rust or Zig call C/C++ and lose part of the language guarantees. GC root is a way to tell the garbage collector: “do not collect this object yet”. Pinning is holding a buffer in place so it cannot be moved, detached, or invalidated while someone still has a pointer to it. Reentrancy is when a call enters the runtime again before the previous call has finished.

What Rust actually buys

The strongest thesis in Bun’s post is simple: a large percentage of the listed bugs were use-after-free, double-free, or forgotten resources in error paths. In safe Rust, a good portion of that becomes compiler error or automatic cleanup through Drop, as long as the resource is modeled as an owning value. Rust does not prove the general absence of leaks: Rc cycles, mem::forget, Box::leak, and abort still exist. But it makes accidental leaking in error paths much harder.

That is correct. And it is a strong argument.

Rust’s strong point here is not magic. It is that Rust forces certain ownership and lifetime relationships to appear in the type, instead of being scattered around as convention. When a value has an owner, when it is moved, when it leaves scope, when a destructor runs, that stops being just an agreement between programmers. The compiler participates.

In Bun’s current code, you can see the pattern in miniature. A MySQL byte wrapper stores a pinned JavaScriptCore value, and Drop undoes the pin when the object dies:

impl Drop for Bytes {
    fn drop(&mut self) {
        if !self.pinned.is_empty() {
            // `pinned` is rooted by the caller's MarkedArgumentBuffer for the
            // lifetime of this Value (see struct doc); the FFI itself is `safe fn`.
            JSC__JSValue__unpinArrayBuffer(self.pinned);
        }
        // self.slice dropped automatically
    }
}

Source: src/sql_jsc/mysql/MySQLValue.rs. The important detail is not the FFI call itself. It is that cleanup follows the value’s owner.

This is the round Rust wins cleanly. In a small project, discipline is enough. In half a million lines, with reentrant callbacks, threadpool, libuv, sockets, TLS, JavaScriptCore, parser, bundler, package manager, and Node API, discipline becomes recurring cost.

Rust probably reduced the invalid state space the team needed to keep in their heads. Real win.

But a real win does not mean total victory.

Zig was not defenseless

The weakest argument you can make against Zig is pretending it is C with new syntax. It is not. Zig has explicit mechanisms to address exactly part of these problems.

The most obvious one is errdefer, the idiomatic mechanism for cleanup on error paths:

fn readThing(alloc: std.mem.Allocator, fail: bool) ![]u8 {
    const buf = try alloc.alloc(u8, 16);
    errdefer alloc.free(buf);

    if (fail) return error.Boom;
    return buf;
}

test "leak em caminho de erro" {
    const alloc = std.testing.allocator;
    try std.testing.expectError(error.Boom, readThing(alloc, true));
}

std.testing.allocator reports leaks in tests. DebugAllocator helps find double-free, leaks, and many invalid memory uses, including making bugs noisier by not immediately reusing addresses. This is not the same thing as a static proof of the absence of use-after-free. It is not a borrow checker. It does not cover allocation done inside C++, JavaScriptCore, BoringSSL, or any other external allocator. It only catches what goes through it and only on the paths exercised by tests.

The allocator’s own comment is precise about what it provides:

//! * Captures stack traces on allocation, free, and optionally resize.
//! * Double free detection, which prints all three traces (first alloc, first
//!   free, second free).
//! * Leak detection, with stack traces.
//! * Never reuses memory addresses, making it easier for Zig to detect branch
//!   on undefined values in case of dangling pointers.

Source: debug_allocator.zig. It is a good testing and debugging tool. It is not a theorem.

Even so, it is wrong to say Zig had no tools. It did.

The difference is feedback timing. In Rust, a lot appears at compile time. In Zig, a lot appears in tests, fuzzing, instrumented allocators, ASAN, review, and architectural style. In critical code, that matters. Early feedback is better.

But it also matters to say the whole sentence: if the problem was lack of clear invariants in the Zig code, the answer could have been a rewrite in Rust. It could also have been an aggressive restructuring of the Zig.

We do not know which would have been cheaper. We only know which one Bun chose.

Drop is not just sugar, but it is not a miracle either

Bun’s post is right to contrast defer with Drop.

In Zig, you write:

var resource = try Resource.init(allocator);
defer resource.deinit();

This is explicit. It is in your face. There is no hidden control flow. That is Zig’s philosophy: if something happens, you should be able to point to where it happens.

In Rust, if the type implements Drop, cleanup follows the owner:

impl Drop for Resource {
    fn drop(&mut self) {
        self.deinit();
    }
}

This is not just syntactic sugar. Rust’s destruction rules connect Drop to ownership, move semantics, field destruction order, and panic unwind. Ordinary safe code cannot use a value after it has been moved, nor double-drop the object as if it still owned it. It is a language guarantee.

But the guarantee ends where semantics the compiler does not understand begin.

If drop calls an FFI function to unprotect a JavaScriptCore value, Rust guarantees the destructor will run on normal scope and drop paths, not on abort, mem::forget, intentional leaks, or Rc cycles. It does not prove that the JSC pointer is still valid. It does not prove the GC has no special rule. It does not prove an asynchronous C callback will not call back after you destroyed the wrapper.

The comment in the same Bun wrapper shows the size of the contract that still remains outside the compiler:

/// JS could `transfer()`/detach an earlier ArrayBuffer, or drop the last
/// JS reference to it and force GC, while we still hold a borrowed slice
/// into it. Pinning the backing `ArrayBuffer` makes it non-detachable for
/// the duration, and the caller's stack-scoped `MarkedArgumentBuffer`
/// roots the wrapper so GC can't sweep the cell. `Drop` unpins.
pub struct Bytes {
    pub slice: ZigStringSlice,

Source: MySQLValue.rs. Notice the size of the contract. It did not disappear because this is Rust. It became documented and encapsulated.

That is the real boundary: Rust improves the Rust side of the contract. It does not perform magic inside the JavaScriptCore heap.

The FFI boundary is still hell

Bun is not a simple CLI. Bun is a JavaScript runtime. It talks to JavaScriptCore, uWebSockets, usockets, BoringSSL, SQLite, compression libraries, system APIs, asynchronous handles, buffers that JavaScript can detach in the middle of a coercion, reentrant callbacks, and a conservative GC.

That is where the debate gets interesting.

Inside an unsafe block, Rust requires discipline. The Rustonomicon on FFI itself treats foreign calls and raw pointers as programmer obligations. If the binding is lying, if the real ownership does not match the type, if the foreign callback outlives the Rust object, the compiler does not save you.

Bun’s binding documents the preconditions in the declaration itself:

unsafe extern "C" {
    /// By-value `JSValue`; C++ side null-checks and reads its own heap state.
    /// No caller-side preconditions → `safe fn`.
    safe fn JSC__JSValue__unpinArrayBuffer(v: JSValue);
    /// 0 = detached/null, 1 = FastTypedArray (GC-movable — caller should dupe;
    /// no unpin needed), 2 = pinned ArrayBuffer (caller must unpin).
}

Source: MySQLValue.rs. The language lets you declare a foreign call safe. The obligation to prove that safety remains ours.

But there is an important difference: Rust lets you encapsulate that unsafe behind safe APIs. You can create handle types, NonNull, PhantomData, Pin, lifetimes marking the relationship between wrapper and runtime, wrappers that do not implement Send or Sync, RAII for GC roots, and a smaller surface for audit. The goal is not to eliminate danger. It is to shrink the place where danger lives.

The most didactic example in Bun’s post is the uv_close bug. The first version compiled, but handed a Box to libuv to close asynchronously and let the Box fall at the end of the scope. Use-after-free followed by double-free. The fixed version in the current code makes the transfer explicit:

spawn::WindowsStdioResult::Buffer(pipe) => {
    // `uv_close` is async — libuv keeps the raw handle pointer until the
    // next loop tick. Leak the Box so it outlives this scope; dropping it
    // here would be a use-after-free + double-free when the callback fires.
    Box::leak(pipe).close(Subprocess::on_pipe_close)
}

Source: js_bun_spawn_bindings.rs. It compiled in Rust and was still wrong at the C/async boundary. The review process caught it.

Here leak does not mean “leak forever”. It means removing the Box from Rust’s lexical control and transferring ownership to libuv’s asynchronous protocol, whose callback later reconstructs and frees the allocation.

Zig also lets you write good wrappers. The difference is that it does not have the safe/unsafe boundary as part of the language. In Zig, everything is explicit and reviewable. In Rust, a larger part of the contract can become type.

For a large team, that matters. It matters even more when part of the code is being generated by agents.

Arenas: yes, Bun already used them

Here the pro-Zig argument needs to be honest. Arenas are idiomatic and excellent for certain problems: parser, AST, request lifecycle, data that dies together.

But Bun’s post does not ignore arenas. It explicitly says the existing approach was already a mix of lifetimes by arena, reference counting, and “pay really close attention”. It even cites parser state and AST nodes as good cases for an arena.

The excerpt from the post is this:

Our current approach is a mix of:

- arena lifetimes, where the scope of when it's accessible is clear
  (parser state doesn't escape the calling function and so AST nodes
  are a good choice there)
- reference-counting
- pay really close attention

Source: Bun’s official post. The good counter-argument is not “use arena”. They already did. The good counter-argument is asking whether the remaining boundaries could have been hardened in Zig.

So the correct response is not “you forgot arenas”. They did not forget.

The correct response is: arena solves the case where lifetime is clear. It does not, by itself, solve a socket handle that closes asynchronously, a callback that runs later, an SSL session that needs a specific free, a GC root, a pointer that may be invalidated by reentrancy, or an object that crosses several API layers with implicit ownership.

Arena is a good tool. It is not a religion.

If Bun had hardened the boundaries more in Zig, maybe part of the rewrite’s benefit could have come from that: more restricted APIs, fewer raw pointers crossing subsystems, handles with explicit ownership, allocators instrumented by default in tests, fuzzing aimed at reentrancy, wrappers that force deinit, less “pay attention”.

But that would also be a rewrite. Just inside Zig.

The borrow checker helps, but not where the internet thinks

A lot of people talk about the borrow checker as if it were a universal bug detector. It is not. The borrow checker is the part of the Rust compiler that prevents certain invalid borrows: two mutable owners at the same time, a live reference after the owner dies, use after move. That is a lot. It is not everything.

It does not understand the JavaScriptCore heap. It does not understand every GC rule. It does not automatically understand that a C callback can trigger JS, that JS can detach an ArrayBuffer, that it can reenter the runtime, that it can invalidate a pointer captured earlier. These invariants need to be modeled.

But when they are modeled, Rust charges you.

A small example in Bun: a uSockets wrapper that turns a C pointer into a Rust borrow tied to &mut self:

impl PacketBuffer {
    pub fn get_peer(&mut self, index: c_int) -> &mut sockaddr_storage {
        // SAFETY: uSockets guarantees a non-null, properly-aligned peer pointer for
        // indices < packet count. The returned storage lives inside the C-owned packet
        // buffer, which is exclusively loaned to the data callback for its duration; no
        // other Rust or C path holds a reference to it. The reborrow of `&mut self`
        // ties the returned lifetime to this handle, so the borrow checker prevents
        // obtaining a second overlapping `&mut` via `get_peer`/`get_payload`.
        unsafe { &mut *us_udp_packet_buffer_peer(self, index) }
    }
}

Source: src/uws_sys/udp.rs. The unsafe still exists. But after the wrapper creates the reference, the borrow checker can prevent certain abuses on the Rust side.

And Bun is not only JavaScript heap. It is parser, bundler, module resolver, package manager, file watcher, HTTP protocol, buffers, caches, tables, queues, process state, glue code. In these parts, local ownership matters a lot. Vec, String, slices, maps, wrappers, enums, finite states: all of that becomes easier to refactor with the compiler slapping your hand.

The honest argument is not “the borrow checker solves Bun”. It is: the borrow checker reduces many classes of error on Bun’s Rust side, and that frees human attention for the part that remains hard.

That is strong enough. No need to exaggerate.

comptime: here Zig has a real advantage

The best technical counterexample in Bun’s own post is Output.pretty.

In Zig, the format string could be a comptime parameter. Bun’s own post puts the difference like this:

// Zig:
pub inline fn pretty(comptime fmt: string, args: anytype) void;
Output.pretty("<r>{f}<r>", .{hyperlink});

// Primeira API Rust:
pub fn pretty(payload: impl PrettyFmtInput);
Output::pretty(format_args!("<r>{}<r>", hyperlink));

This allowed processing color and formatting markers in the literal known at compile time, before mixing runtime arguments. When porting to Rust, the first version became a function. Result: the tag parser touched bytes that came from the arguments. Issue #30693 shows the regression in port/canary involving an OSC 8 hyperlink escape and <r>, not a failure released in stable.

The bug was exactly the contamination between template and runtime argument:

TerminalHyperlink::Display emits an OSC 8 hyperlink:

\x1b]8;;URL\x1b\ai\x1b]8;;\x1b\

The closing string terminator ST is the final two bytes `\x1b\`.

This is then wrapped via `Output::pretty(format_args!("<r>{}<r>", hyperlink))`.
The rendered string tail is `...\x1b\<r>`.

Source: issue #30693.

Then they fixed it using a macro.

The current code makes the difference explicit:

/// Write to stdout with `<tag>` color expansion.
/// Function form: performs the `<tag>` → ANSI rewrite at runtime on the rendered
/// payload (using stdout's colour state). Prefer the `pretty!` macro for literal
/// templates so the rewrite happens at compile time.
#[inline(always)]
pub fn pretty(payload: impl PrettyFmtInput) {
    let buf = payload.into_pretty_buf(enable_ansi_colors_stdout());
    write_bytes(Destination::Stdout, &buf);
}

And right below:

/// Compile-time `<tag>` → ANSI escape rewriter.
///
/// This must be a proc-macro because it consumes a string literal and emits
/// a new string literal usable as a `format_args!` template.
///
/// `pretty_fmt!("<red>{s}<r>", true)`  → `"\x1b[31m{}\x1b[0m"`
/// `pretty_fmt!("<red>{s}<r>", false)` → `"{}"`
pub use bun_core_macros::pretty_fmt;

Source: src/bun_core/output.rs.

This demonstrates a legitimate strength of Zig: comptime is the language itself running at compile time. You do not switch to a parallel system. In Rust, macros are powerful and mature, but they are another mechanism. For some kinds of API, Zig is simply more direct.

But let’s not turn a porting bug into universal proof against Rust. The problem there was not “Rust is fragile”. It was the wrong translation of a comptime API into a runtime function. Bun’s own post recognizes that and fixes it with a macro.

The correct reading is simpler: Zig had a structural guarantee there. The first Rust version lost that guarantee. That cost a regression in port/canary.

Point for Zig.

Binary size and performance: careful with cheap causality

Bun’s post shows good numbers: smaller binary, some faster benchmarks, better startup in certain scenarios.

The numbers from the post:

Bun v1.4.0 (canary) — Windows — 76 MB
Bun v1.3.14          — Windows — 94 MB
Bun v1.4.0 (canary) — Linux   — 70 MB
Bun v1.3.14          — Linux   — 88 MB

Bun.serve  — 169.6k → 177.7k — +4.8%
node:http  — 103.8k → 108.5k — +4.5%
fastify    —  91.5k →  95.9k — +4.8%

Source: Bun’s official post.

I believe the numbers. I do not believe in simple attribution.

A rewrite changes many things at the same time. It changes code layout, removes debt, changes toolchain, touches LTO, changes how generics/comptime are instantiated, changes linker, changes dead code, changes ICU, changes internal organization. When all of that happens together, you cannot look at the graph and say: “it was Rust”.

You also cannot say: “it was only the rewrite”.

The post itself talks about a combination: Rust, ICU changes, Identical Code Folding, LTO, codegen, accumulated cleanup. LTO is optimization at the final link stage, when the compiler sees more of the program before generating the binary. Identical Code Folding is the linker merging identical functions into one. Andrew is right to point out that Zig also supports LTO and that some of these gains could have been pursued before. But that does not prove the result would be the same. It only proves causality is mixed.

Bun’s Cargo.toml shows there is explicit build engineering in this story:

# Cargo's defaults (lto=false, codegen-units=16) leave us at ~105 crates × 16
# CGUs ≈ 1680 separately-optimized units with NO cross-crate inlining beyond
# `#[inline]`-annotated leaf fns. `lto = "fat"` + `codegen-units = 1` collapses
# the whole Rust crate graph into one LLVM module, matching Zig's shape.
[profile.release]
lto = "fat"
codegen-units = 1
debug = "line-tables-only"
strip = "none"
panic = "abort"

Source: Cargo.toml.

And there is LTO configuration crossing Rust and C++:

/**
 * Cross-language LTO: rustc emits LLVM bitcode (`-Clinker-plugin-lto`) into
 * `libbun_rust.a` so the final lld `-flto=thin` link sees through Rust↔C++
 * call edges. When false but `lto` is true, both halves still LTO
 * independently (C++ via `-flto=thin`, Rust via `[profile.release] lto =
 * "fat"`); only the cross-language inlining is lost.
 */
crossLangLto: boolean;

Source: scripts/build/config.ts. It is not “Rust won”. It is language, toolchain, linker, configuration, and code working together.

Same thing for stack usage and parser depth. The post attributes improvement to Rust emitting better llvm.lifetime.start/end and allowing stack slot reuse. Codegen detail? Yes. But a codegen detail delivered by today’s toolchain counts. Zig itself still has an open issue about stack allocations that do not disappear at the end of scope. For the user, it matters little whether the advantage comes from the pure language, the compiler, or backend maturity. What matters is that it works today.

The mechanism described by Bun is this:

Rust's LLVM IR codegen emits LLVM's llvm.lifetime.start and
llvm.lifetime.end intrinsics for stack variables when they are no longer
in use, which lets LLVM reuse stack space slots.

And the Zig ticket says, bluntly, that this case is still not solved:

Block stack allocations do not disappear at end of block scope #23475

There is also the architectural solution: iterative parser with an explicit stack on the heap. Works in Zig and Rust. But it is more work.

Honest reading: the numbers favor Bun’s Rust version in that experiment. They do not prove general language superiority.

Miri, crates, and the real world

There are Rust advantages that have no direct equivalent in Zig today.

Crates is the obvious one. The Rust ecosystem is large, mature, and practical. For a project the size of Bun, a good off-the-shelf library is real engineering value. Zig is still maturing on this point.

Miri is also a real advantage. It detects classes of undefined behavior in executed Rust code: out-of-bounds, use-after-free, uninitialized data, alignment problems, data races in certain cases. The project itself documents the limits: it is not a formal proof, it evaluates the executions you provide, and it has restricted support for FFI and platform APIs. It does not replace testing. But to validate isolated unsafe abstractions, it is a tool Zig does not have in the same way.

Bun does not merely cite Miri in the abstract. It is in the toolchain:

[toolchain]
channel = "nightly-2026-05-06"
components = ["rust-src", "rustfmt", "clippy", "miri", "llvm-tools"]

Source: rust-toolchain.toml.

And there is an internal test explaining exactly the use:

// Run under Miri (`bun run rust:miri -p bun_ptr`): every path here walks raw
// pointers through `Box::into_raw` / `heap::take`, so Tree Borrows is what
// proves the ref/deref/destructor handoff does not alias or use-after-free.

Source: src/ptr/ref_count.rs. It does not prove the whole program. It exercises a dangerous abstraction under a model made to detect undefined behavior on those executed paths.

There is also hiring, tooling, documentation, IDE, review culture, CI, examples, people in the market. This is not a political detail. It is engineering. A language exists inside an ecosystem.

If Bun were a smaller project, with a smaller team, less pressure, and less Node-compatible surface area, maybe I would weigh this differently. But Bun became a runtime used by millions of monthly downloads. The margin for “let’s maintain perfect discipline” shrinks.

Async: Zig may have a better answer, but it is not Bun’s present yet

Zig removed the old async/await and is redesigning I/O around std.Io, passed by the caller in a way similar to Allocator. The idea is good: avoid classic function coloring, make the function less coupled to blocking, threaded, or event-driven execution, and treat I/O as an explicit capability.

The proposed format is like this:

const std = @import("std");
const Io = std.Io;

fn saveData(io: Io, data: []const u8) !void {
    const file = try Io.Dir.cwd().createFile(io, "save.txt", .{});
    defer file.close(io);

    try file.writeAll(io, data);

    const out: Io.File = .stdout();
    try out.writeAll(io, "save complete");
}

Source: Kristoff on Zig’s new async I/O. io is a capability passed explicitly, like Allocator.

I like this direction a lot.

But it is still in transition. The tracking issue remains open: there is a prototype, roadmap, discussion, parts landing gradually. It is not a stable foundation Bun could bet on today without cost.

So the argument here cannot be “Zig already solved it”. The correct formulation is: Zig is moving toward an async/I/O design that may be cleaner in some aspects than the traditional colored model. But Bun needed to make a decision with what exists now, not with what promises to exist later.

That is always cruel to young languages. But it is real.

The rewrite process matters more than the language

The best part of Jarred’s post is not Rust. It is method.

The mental loop of the post is literally this:

// Pseudocode, not real code:
let task;
while ((task = todoList.pop())) {
  const result = task();
  const feedback = await Promise.all([review(result), review(result)]);
  await apply(feedback, result);
}

Then compiler errors became a queue:

~16,000 compiler errors left

error: deref *mut EventLoop before field access
error[E0034]: multiple applicable items in scope
error: event_loop/EventLoopTimer.rs: port Timespec::ns from bun.zig

And the per-crate cycle was explicit:

- For each crate, run cargo check, group the output by file and save the errors to a file
- Fix all the compiler errors within that crate
- 2 adversarial reviewers for the crate's changes
- 1 fixer applies the fixes

Source: Bun’s official post.

He did not tell Claude to rewrite an entire runtime and go to sleep. He built loops. Separated implementer from reviewer. Used different context for adversarial review. Did a trial run. Fixed the workflow when agents started tripping over each other. Banned dangerous commands. Split worktrees. Turned compiler errors into a queue. Ran the same TypeScript suite because the suite did not depend on the runtime language. Used multi-platform CI as judge.

That is the part everyone should study.

The model did not replace engineering. The model executed work inside an engineering process. When the process was bad, the result was bad. When the process improved, the result improved.

It is the detail the tribal discussion ignores. The story is not “Claude rewrote Bun”. The story is “an engineer turned a giant migration into several verifiable loops and used Claude to accelerate each loop”.

Much more interesting.

So did Zig fail?

No.

Zig did not fail because Bun chose Rust. Rust also did not become the universal answer because Bun chose Rust.

What the public evidence shows is more boring: that code, at that moment, was too expensive to evolve with the level of confidence the team wanted. Rust bought predictability. Bought compiler feedback. Bought ecosystem. Bought an explicit safe/unsafe boundary. Bought tooling. Bought an environment where AI agents can bang against compiler errors and move forward mechanically.

That is enough to justify the decision.

But it is not enough to prove Zig was technically unviable for Bun. Many cited problems also look like boundary problems: implicit ownership, overly permissive APIs, allocation without a visible contract, resources crossing layers without a type carrying the owner, tests that maybe did not capture difficult invariants, and the human cost of reviewing all of that.

Those things could have been attacked in Zig. Maybe an aggressive Zig restructuring would have solved a lot. Maybe not. Maybe it would cost more than the port. Maybe it would cost less, but deliver less confidence. We do not have the counterfactual.

And that is exactly why the conclusion needs to be calibrated.

I support Jarred’s decision as a product and engineering decision. His post is one of the most useful reports I have ever seen about a large AI-assisted rewrite. But liking the decision does not force me to pretend it was the only possible technical conclusion.

And there is one important disclaimer. Anthropic acquired Bun in December 2025, and Jarred opens the post by saying that he and the Bun team now work there. Anthropic sells Claude Code. Jarred had every institutional incentive to use as much Claude as possible and turn this rewrite into a public demonstration of his new employer’s product. That does not prove bad faith, nor does it invalidate a technical result. But it adds bias to the decision and to the way the case is presented. Ignoring that conflict of incentives would be as naive as ignoring Andrew’s resentment over Bun leaving Zig.

The best reading is less dramatic: Bun chose to pay a large cost now to reduce recurring maintenance costs later. Rust was a defensible choice, maybe a great one for that team at that moment. Zig remained an excellent language, but would have required another kind of reengineering: more explicit invariants, less informal ownership, more tools enabled by default, harder boundaries.

Andrew could have written that. He could have defended Zig on technical ground. He could have said: “Rust bought some real guarantees, but a large part of the problem was architecture and process; here is how Zig could have attacked the same causes.” That would have been a useful post.

We do not need to declare a winner. Language does not replace architecture. It only changes which errors become cheap to detect and which still depend on human discipline.

Bun chose to trade part of that discipline for compiler. I think it is a good bet.

I do not think it buries Zig.

Note: all the code I’m going to show below is available in this repository on my GitHub. Go poke around after reading to the end.

For example, I asked the Qwen3-8b model (which came out a few days ago) which Zig version it knows about, and the answer was: 0.11 or 0.12:

By the way, the screenshot also shows how you call an OpenAI-compatible LLM API using curl. Anyway, that’s why I suffered so much yesterday trying to get a simple “Hello World” level program working: it doesn’t know we’re already on Zig 0.14. And I did it on purpose, I knew it wouldn’t have that information.

“And other commercial LLMs? Aren’t they better than the open source ones?”

I asked ChatGPT 4.1, Claude Sonnet 3.7, Gemini 2.5 Pro Experimental, Deepseek R1 Zero (all at once, OpenRouter.ai allows multi-model chat):

Summary:

• ChatGPT 4.1 says it only has data up to June 2024 and only knows up to Zig 0.11.
• Claude Sonnet says it only has data up to April 2023 and only knows up to 0.10.
• Gemini 2.5 Pro Exp says it only has data up to the start of 2023 and only knows up to Zig 0.11 as well.
• Deepseek R1 (after some deep thinking…) answered short and blunt that it only has data up to October 2023 and only knows up to Zig 0.11.

“Why don’t they have newer data?”

I’ll answer that by showing the reasons in the next sections.

“But it can pull info from the internet, so it doesn’t matter”

It matters. That information is not IN the model. When you ask it to search the internet, the model doesn’t know how to do that. It was instructed (in the fine-tuning/alignment phase or via prompts) to ask the chat system - the parser - to take that request, navigate the web (think curl, playwright, or anything like that), pull the site’s content, copy and paste it only in your chat session (it’s like a RAG solution). It only has that information in that session and other chats are not influenced by it.

“But how does it show the source links when it replies?”

It - probably - doesn’t have that inside the model. What I imagine is simpler. They have catalogued and indexed all the original training material (including the source URLs) in a database (think Elastic, Postgres with full-text-search, or any vector database). When it finishes replying in the chat - the chat program and not the model -, it searches that snippet in the database for something similar enough and pulls the source URLs from there to show you.

Anyway.

Zig is experimental, pre-1.0. The syntax for everything changes A LOT from one version to the next, because they’re still experimenting with what’s best. And it’s going to change more. In other words: it’s impossible to do “Vibe Coding” in Zig unless you pin your compiler to version 0.11 or 0.12. But that would be “lame”, to say the least. I chose to try Zig on purpose, precisely to prove this point.

The same applies to a lot of companies with private code. Think of ERP companies that have their own languages, their own libraries, and none of it is public, so it was guaranteed not to be in OpenAI’s training and it’s certainly way more information than fits in a prompt context.

“Ah, just do a RAG”

Canned solution without understanding. No, it doesn’t help. It will prioritize the training it already had. Second, for each thing you ask, you’d need to know in advance that you’ll need something that changed in 0.14, search that bit of documentation and keep adding it to the chat context. If you’re going to do that method by method, why are you using the chat? Just go and write the code directly, it’ll be faster.

Throwing ALL the documentation into the context doesn’t work. I did the math: I downloaded the entire documentation and release notes, it’s 200 thousand characters, roughly 50 thousand tokens. Most open source LLMs will blow past the window (average 32k tokens). Via commercial API/Chat you would pay per token, for nothing. Imagine wanting to refactor your 100-line piece of code and having to upload 200 thousand lines of context. Never do that. The cost-benefit doesn’t justify it.

“Ah, but the Geminis of the world support 1 million tokens”

Yeah, with a sliding window, I’ve repeated this several times. It can’t pay attention to everything at once. Again: you’ll PAY for those 50 thousand tokens, right? And it won’t be cheap. Imagine, every new session having to upload 50 thousand tokens with no guarantee they’ll work.

So is it screwed? Is there no way to use Zig with LLMs?

RAGs?

The RAG (Retrieval Augmented Generation, which is just a fancy name) hype makes it seem like it’s something “magical”. But really, there’s nothing special about it. It’s good for when you have a large database that grows all the time, but in each chat session you only need a little bit of that information. Think of a run-of-the-mill e-commerce support team with support tickets. It grows every day, but for each call you only need the specific ticket for the person being helped, which is pretty short.

In the most rudimentary way, the attendant searches for the ticket in the good old Zendesk, copies the text, pastes it into the ChatGPT chat window and starts asking questions about that ticket, like “can you summarize what this customer’s problem is?”, “can you imagine a solution for them?” and so on.

A RAG is a DATABASE with all those tickets pre-indexed to make relevance searches easier, and then it inserts the found snippet directly into the context window. It serves to automate, to SKIP the steps of manually searching in another system and doing copy and paste, but the effect is EXACTLY the same.

“Ah, but it uses VECTOR databases, that must be way cooler”

What’s cool is hearing nonsense. It could be a Postgres and you’d do a SELECT to find the ticket, it’s exactly the same thing. If you enable “full-text-search” in Postgres, it will index your tickets in a “VECTOR SPACE MODEL”, which is essentially a VECTOR DATABASE.

If instead you prefer SOLR or ELASTIC, both are VECTOR DATABASES specialized in text search.

Duh, the more you know.

The main things:

• the more you stuff into the context (whether prompt text or “embedding” of a vector database with RAG - it’s the same thing), the more you risk confusing the LLM and having it start hallucinating. The best prompts are structured, organized, with direct orders, and that explain the context clearly.

• whether it’s text-prompt or “embedding”, it will CONSUME input TOKENS. And if the window is small (open source models), it will blow up if you don’t control the size of what you send as input. There’s no magic.

So what’s the way out? Despite RAG being all people talk about, there’s more than just RAG in the LLM world.

Possible Solution: LoRAs

If instead of things that grow daily, the information is more “permanent” like instruction manuals, books, maybe RAGs aren’t the best approach. Because it will always be TOO BIG and you’d need to REPEAT THE SAME THING in every chat session. Especially rules that depend on other rules.

Yes, good providers have CACHE STATS and that helps a lot. Even so it’s no guarantee. Out of nowhere you’re going to check your bill and be surprised by USD 500 suddenly spent on token credits.

Example: programming language documentation. Like my case with Zig 0.14.0. I’d like the LLM to have that context, but uploading 50 thousand tokens every time is a lot if I only want to fix a 50-line file. Nobody thinks about these things or calculates context size vs chat/api interaction size, and they should: every wasted token is credit spent from your account.

There’s a way to take an existing LLM model and add the missing training on top, without using context space. It would be as if the model were “updated” to know Zig 0.14. These are LoRAs (Low-Rank Adapter). It’s a smaller kind of training we can add. We train small additional matrices (adapters) that attach to the original LLM’s layers. When loaded, this updates the model’s outputs without having to redo the entire training from scratch (which is the most expensive part and takes months in OpenAI’s heavy datacenters, for example).

And here’s a DISCLAIMER:

I haven't studied enough yet about all the training parameters. Training is precisely the hardest phase. In particular: generating QUALITY training material and "tuning" the training parameters. There are people specialized in just this and there's no single recipe that works for everything. It's almost an "art" and takes much more dedication than I put in just for this post.

Some articles to start studying:

• Instruction-Tuning Large Language Models (LLMs) with HuggingFace, LoRA and SFTTrainer
• # Supervised Fine-tuning Trainer
• # Fine-tuning with the Hugging Face ecosystem (TRL)
• # Notes on Fine-Tuning a Multi-Modal Large Language Model Using SFTTrainer
• # A beginners guide to fine tuning LLM using LoRA

And there’s a lot more. This is where you have to really understand the “internals”, the neural networks, loss function, gradients, etc. Don’t underestimate training. It won’t work on the first try, nor on the tenth. It’s A LOT of trial and error and adjusting.

The final result is not exactly the same as a full training, but these adapters modify the parameters of the existing model’s layers, changing the final result for specific things we want (that it knows Zig 0.14). It’s MUCH cheaper than training from scratch. It doesn’t guarantee it will work 100%, but it gives us an option to test, which can be more robust than a RAG.

The process is a miniature version of training a real model. We have to solve the following steps:

• get the training material (e.g. documentation)
• write fine-tuning prompts (I need to “teach” how it should respond - this is the hardest part)
• tokenize this information (trivial), load the base model (easy), and start training (the better the GPU, the less slow it is)
• if it succeeds, you need to load this LoRA along with the base model
• point tools like Aider at this new modified model.

Let’s go, step by step.

1. Training Material

I think this is the hardest part: getting quality training material to train with.

In the alignment case, this is done by building it by hand (with help from scripts, with help from another LLM to help generate synthetic texts), but ideally I’d try to build a big JSON file with “prompt” and “completion” keys, something like:

{ "prompt": "How do the new @splat works for arrays in Zig 0.14.0?, "completion": "Zig 0.14.0 expands the `@splat` builtin to apply not only to vectors, but to arrays..."}

Now, doing this for EACH NEW feature in version 0.14.0, including code examples, multiple times for each topic, imagining different ways to ask. And no, I didn’t do it and I don’t plan to, it’s already much more work than I want to try in just this experiment. And I’m not even sure if that’s the best way yet. As I said before, it’s practically an “art”.

The RELEASE NOTES file converted to Markdown has a ridiculous 107 thousand characters. I’d spend the rest of the week just re-editing it into prompt form. I’m not going to do that. Instead, I’ll just pass the entire raw text and hope that these new facts “stick” enough for the LLM to remember them. To pull it, it’s simple:

wget https://ziglang.org/download/0.14.0/release-notes.html -O zig-0.14.0-release-notes.html

# pip install html2text
html2text zig-0.14.0-release-notes.html > raw_data/zig-0.14.0-release-notes.md

Since I’m on a roll, why not do the same thing with the Zig documentation? Fortunately the documentation is a single big HTML file. So there’s no work to build a recursive web scraper or anything like that. Same procedure:

wget https://github.com/ziglang/www.ziglang.org/raw/refs/heads/main/src/documentation/0.14.0/index.html -O raw_data/zig-0.14.0-docs.html

# pip install html2text
html2text raw_data/zig-0.14.0-docs.html > raw_data/zig-0.14.0-docs.md

That gave me a Markdown file of 464 kilobytes. Since I think just that isn’t enough, I thought it was a good idea to also grab the code examples in the zig/doc directory that I cloned directly from the official Zig repository. I don’t know what the best way is, but I tried to organize a single big file, with the name of the .zig file and the code delimited by a tag, with a prompt at the beginning indicating this:

...
zig_files = glob.glob("raw_data/zig/doc/langref/**/*.zig", recursive=True)
total_input_bytes = 0
file_count = 0
lines = []

for filepath in zig_files:
    file_bytes = os.path.getsize(filepath)
    total_input_bytes += file_bytes
    file_count += 1
    filename = os.path.basename(filepath)
    with open(filepath, 'r', encoding='utf-8') as f:
        content = f.read().strip()
        entry = f"Reference Filename: {filename}\n\n```zig\b{content}\b```\n"
        lines.append(entry)
...
outpath = "raw_data/zig_docs_with_filenames.txt"
with open(outpath, "w", encoding='utf-8') as f:
    for line in lines:
        f.write(line + "\n\n")

...

Notice that in this case I’m formatting each example starting with “Reference filename: {filename}” and then adding the code in a Markdown block with 3 backticks. Is it a good format? I don’t know yet. Making training material IS HARD.

That resulted in another big file of 152 kilobytes. I hope it doesn’t confuse more than help. And I didn’t stop there. I wanted to add the entire source code of the Standard Library and I even wrote the script, downloaded and compiled it into a structured format, but the final file was 15 MEGABYTES. Unfortunately I had to comment it out of my training script. Now I hit a limitation of my RTX 4090: there’s not enough VRAM to train all of that.

As I’ve repeated before, the 4090 only has 24GB of VRAM. The Qwen3-8b model alone takes 16GB (no, the number of parameters influences the size but it doesn’t mean 1 parameter is always 1 byte). There has to be memory left over to do all the training calculations, and it will spend 21GB while training. If I throw in that 15MB of material, this happens:

To train with all of that, it has to be remote, on a RunPod with an A40 like I showed in another blog post, at the very least. Or with faster GPUs like the H100, with 80GB, and more power to finish a little faster. So, I’m already limited on quality training material and now limited on the amount of material I can use.

Since I couldn’t use the source code (which would have real idiomatic 0.14.0 code examples), I looked for other examples to add and found this Zig Cookbook repository, which is organized as problems in a Markdown file with solutions in 0.14-compatible code. I wrote this other script to organize everything into a single file:

...
lines = []

for prob_file in sorted(glob.glob('raw_data/zig-cookbook/book-src/*.md')):
    # Extract the nn-nn from the problem filename (e.g., 01-05-matrix-mul.md -> 01-05)
    base = os.path.basename(prob_file)
    if base == "SUMMARY.md" or base == "database.md" or base == "intro.md":
        continue
    print(base)
    match_part = base.split('-')[0] + '-' + base.split('-')[1]  # e.g., 01-05

    code_file = f"raw_data/zig-cookbook/src/{match_part}.zig"
    if not os.path.exists(code_file):
        print(f"WARN: No solution found for problem {prob_file} (expected {code_file})")
        continue

    with open(prob_file, 'r', encoding='utf-8') as pf:
        problem_text = pf.read().strip()

    with open(code_file, 'r', encoding='utf-8') as cf:
        code_text = cf.read().strip()

    # Compose chat format
    block = (
        f"<|user|>Solve this problem in Zig 0.14.0:\n{problem_text}\n<|endoftext|>\n"
            f"<|assistant|>This is the Zig 0.14.0 way to solve this:\n```zig\n{code_text}\n```\n<|endoftext|>\n"
    )
    lines.append(block)

with open("raw_data/zig_cookbook_chat.txt", "w", encoding="utf-8") as out:
    out.writelines(lines)

print(f"Wrote {len(lines)} problem/solution chat pairs to zig_cookbook_chat.txt")

In this case I’m trying to see if using CHAT TAGS makes a difference. Not every model supports it. But Aider, for example, assembles prompts like this: with tags indicating “who is speaking” (system, user, assistant) and it seems to be an alternative to doing it in JSON format. So each problem/code looks like this:

<|user|>### Measure the elapsed time between two code sections

[](https://github.com/zigcc/zig-cookbook/blob/main/book-src/03-01-elapsed-time.md#measure-the-elapsed-time-between-two-code-sections)
...
<|endoftext|>
<|assistant|>
...
fn expensive_function() void {
    // sleep 500ms
    time.sleep(time.ns_per_ms * 500);
}
<|endoftext|>

Will this work well? I don’t know 😆 As I said, I need to spend many more hours dedicating myself just to understanding good training material formatting.

Finally, I tried to make a short file of prompts like I mentioned at the start, to try to force the chat to answer that it knows about the existence of version 0.14 and what the changes are in that version (just the topics from the release notes).

This is how the so-called “alignment” is done. At a big company like OpenAI, they must have HUNDREDS of human beings to write questions and answers that they want to force the LLM to reply with. And we have to do the same thing here, otherwise the Zig 0.11 and 0.12 training already in the base model might still override the new material I’m adding.

I understand there’s no trivial way to say “ignore the old version and only pull from my new material”. That’s not how it works. You “influence” the training (supervised training), but there are no “guarantees” that every variation of a question will give the same answer, unless you manually input most of the variations in the alignment.

This is why I say DO NOT BELIEVE an LLM. It is not REASONING. It was FORCED to reply in a certain way by humans, during training. It literally has millions of PRE-PROGRAMMED ANSWERS already embedded in training. That's how you CONTROL the answers.

In my case, it’s not enough to just upload the documentation, I should say in the prompts “what are the language changes in Zig 0.14.0?” and give the answer “In Zig 0.14.0 we had the following language changes: Labeled Switch, Code Generation Properties, Decl Literals, Fields and Declarations Cannot Share Names, @splat Supports Arrays, Global Variables can be Initialized with Address of Each Other, @export Operand is Now a Pointer, New @branchHint Builtin, Replacing @setCold, …….” (yes, a lot of stuff really did change).

Anyway. I’ll push my training procedure to my GitHub. There you’ll find a raw_data directory where the material I compiled lives. There are scripts like concat-zig-docs.py, concat-zig-stdlib.py or concat-zig-cookbook.py to concatenate each group of material into a single big file as I explained. And at the end, there’s train-zig.py which actually loads all the datasets and does the training.

There’s another important factor: compared to the more than 800 kilobytes of material I compiled, the prompts I mentioned are still very small, only 8.3 kilobytes. We need to try to do OVER-SAMPLING. Ideally I should write at least 10x more real prompts manually, but I really don’t have that much patience. What I’m going to do is repeat these prompts several times:

import numpy as np
...
json_dataset_path = "raw_data/training-prompts.json"
text_dataset_path = "raw_data/zig-0.14.0-docs.md"
text2_dataset_path = "raw_data/zig-0.14.0-release-notes.md"
text3_dataset_path = "raw_data/zig_docs_with_filenames.txt"
text4_dataset_path = "raw_data/zig_cookbook_chat.txt"
text5_dataset_path = "raw_data/zig_stdlib_concat.txt"
...
# Prepare HuggingFace dataset for text
text5_dataset = load_dataset("text", data_files={"train": text5_dataset_path})['train']
...
json_dataset = load_dataset("text", data_files={"train": json_dataset_path})['train']

# oversample the prompts to try to influence the bias towards this new documentation
N = 100  # Oversampled target size
indices = np.random.choice(len(json_dataset), size=N, replace=True)
oversampled_ds = json_dataset.select(indices)
# text5 (std) won't fit in the 4090 so removed from combined
combined_dataset = concatenate_datasets([text_dataset, text2_dataset, text3_dataset, text4_dataset, oversampled_ds])
...

That’s how you load the training data as datasets. And I use Numpy’s random_choice to over-sample the JSON file and artificially make it bigger, repeating the lines randomly. And in the end we have a combined_dataset that is everything concatenated, about almost 900 kilobytes of material.

Then read the train_lora.py file, but with the combined dataset ready, now it’s the actual training part:

# Tokenize your dataset FIRST:
def tokenize_function(example):
    return tokenizer(example["text"], truncation=True, padding=True)

tokenized_dataset = combined_dataset.map(tokenize_function, batched=True)

# Training hyperparameters
training_args = SFTConfig(
    output_dir = "qwen3-zig-lora",
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    warmup_steps = 5,
    num_train_epochs = 5,
    learning_rate = 2e-4,
    fp16 = not is_bfloat16_supported(),
    bf16 = is_bfloat16_supported(),
    optim = "adamw_8bit",
    weight_decay = 0.01,
    lr_scheduler_type = "linear",
    seed = 3407,
    report_to = "none",
    max_seq_length = 2048,
    dataset_num_proc = 4,
    packing = False, # Can make training 5x faster for short sequences.
)

# Trainer for unsupervised SFT
trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset,
    processing_class=tokenizer,
)
trainer.train()
model.save_pretrained("qwen3-zig-lora")
tokenizer.save_pretrained("qwen3-zig-lora")

First, we tokenize the dataset, we convert text into tokens (this is fast; in the chat you use every day, your text is converted into tokens like this too - the model doesn’t understand text, it understands embeddings, which are these tokens translated into vectors).

In SFTConfig we configure the training parameters. I think most of them are default values, but there are two important ones:

• Epoch: One pass through the training material is called “one Epoch”. They say for little material (like my case), it’s good to go more than 3 Epochs, up to 10 Epochs. More Epochs tries to make my new LoRA “stick” more. If the dataset were orders of magnitude larger, fewer Epochs would be better, because there’s the problem of OVER-FITTING. It “sticks” too much and the model will memorize my material too much and just copy-paste directly. It depends on the type of material, depends on the amount of material. With 5 Epochs this material takes more than 1 hour to process, 10 Epochs then pushes it to more than 2 hours.

• Learning Rate: it’s the scale factor of how much we want the “weights” inside the model to change. How aggressively do we want this new information to change the model weights, so it actually “learns”? If it’s too high, it’ll start to “forget” what it already had in the prior training and create knowledge gaps (“catastrophic forgetting”). If it’s too low, it changes little, and our LoRA won’t have any significant effect.

There’s one more parameter that leaves me in doubt, max_seq_length, which serves to truncate overly long text. Theoretically the unsloth library knows how to split training material into smaller “chunks” (if it has obvious delimiters). But I don’t know what happens with giant things, like the entire documentation. Whether it splits and uses it, or truncates and doesn’t use it. There are many details that still need more studying. If it’s truncating without me knowing, maybe I’m feeding material it’s not actually using, for example.

The old facts that already exist in training (Zig 0.11 and 0.12) are stronger than my material. So cranking up those parameters might help influence the final result more. That’s the other part of the “art”: keeping adjusting these values to better fit the new knowledge. In a full training, from scratch, it’s even more important. I don’t know everything about this. What I said is the amateur level of training an LLM.

A LoRA is supposed to be small. With an 8B base model, with this training material size, the final LoRA will be in the 3 to 4GB range (I don’t know if that’s considered big or small). And that has to be enough to redirect the responses to the new knowledge. If you didn’t know this, on sites like Hugging Face you’ll find pre-trained LoRAs to use, you don’t necessarily have to make one from scratch if there’s already a ready one.

With the material in place, processed, organized, just run my train_lora.py and you’ll see something like:

And with so many Epochs and high Learning Rate, it will take a while! On my RTX 4090 it should take more than 1 hour training (with 5 Epochs), maybe more (I didn’t time it), and while that’s running it really will nail the GPU:

I know there are online services to train remote LoRAs, I think Hugging Face itself has APIs for it, or we can spin up a more powerful GPU on a RunPod. Obviously, you need access to the model to train on top of it (the LoRA parameters are dependent on the base model) so there’s no way to make a LoRA of GPT 4.1 on your machine. That depends on whether OpenAI has APIs to train LoRAs remotely (I think they do, I didn’t look).

Repeating: LoRAs are specific to the base model. Say you made a LoRA for the Qwen3-8b model (my case). Will that LoRA work for Qwen3-32b? No. It’s specific. You have to load exactly the same base model when using your LoRA.

It’s up to you, you don’t need to have a local GPU just for this. It’s an intensive process, that uses a lot of resources and a lot of energy and, depending on how much training material and how many epochs… go grab a coffee.

At the end, a qwen3-zig-lora directory will appear more or less like this:

It records a checkpoint per epoch or something like that, from what I understand. Which means that depending on the strategy, if needed you can stop training and then continue from where you left off. I haven’t tried to do this, but I imagine in a real training, which takes MONTHS, this is essential.

Another Problem - Ollama’s Limitations

To use a local model, as I already showed in this other article, we can load an LLM like Qwen3 with the Ollama server, which will spin up a web server and serve APIs in the OpenAI standard. Then just point a tool like Aider at it and done, we have a code assistant running locally.

The problem: LoRA support in Ollama is limited (it may be lack of research on my part, but I didn’t find a quick solution). They use a proprietary format for the models, and not .safetensors files like those on Hugging Face. Anyway, I hit that problem and I think it’s not easy to proceed with Ollama.

So we have to switch and, fortunately, there are options. The most popular alternative is vLLM. It does the same thing: loads models and spins up an HTTP API server in the OpenAI standard. The advantage is it’s more flexible (works with Hugging Face models), more advanced, more customizable and the main thing: it can load models with LoRAs. Check the documentation at the previous link, but in my case I think it’s pretty simple.

And for those thinking about deploying for multiple users, vLLM is more performant and faster than Ollama too. Even for RAG solutions: Ollama only accepts prompt injection, while vLLM will integrate RAG better via OpenAI/proxy/Langchain for example.

You may need to study the options a bit more, but technically speaking, it’s superior. I always think of Ollama more for running locally on your own machine, but not for production. vLLM I “think” would be better for that purpose - worth researching later if you’re interested.

My training, in the end, will generate a directory called qwen3-zig-lora and with vLLM we can spin up the server like this:

# pip install vllm[all]

vllm serve Qwen/Qwen3-8B \
    --enable-lora \
    --lora-modules ziglora=./qwen3-zig-lora \
    --max_model_len 8192

To test that it’s up, we can run a simple curl command:

❯ curl localhost:8000/v1/models | jq .

And we’ll have the lora listed as if it were just another model. See ziglora at the bottom:

{
  "object": "list",
  "data": [
    {
      "id": "Qwen/Qwen3-8B",
      "object": "model",
      "created": 1746207038,
      "owned_by": "vllm",
      "root": "Qwen/Qwen3-8B",
      "parent": null,
      "max_model_len": 8192,
      "permission": [
        {
			...
        }
      ]
    },
    {
      "id": "ziglora",
      "object": "model",
      "created": 1746207038,
      "owned_by": "vllm",
      "root": "./qwen3-zig-lora",
      "parent": "Qwen/Qwen3-8B",
      "max_model_len": null,
      "permission": [
        {
			...
        }
      ]
    }
  ]
}

To chat with it, we can directly use the API with curl and send the message to the LoRA:

And pay attention to the response. I asked which is the most recent version of Zig it’s familiar with and the answer was “I am familiar with the Zig language up to version 0.14.0 from 2025.”

I can’t say if it’s true 😂 this only means that the weight of my prompt over-sampling with canned answers is working. Maybe it didn’t learn anything about Zig 0.14.0 but it learned to reply that it knows. And, again, this is why you shouldn’t trust LLMs: having a canned answer forced in alignment doesn’t mean it actually “knows” what it’s talking about, I just forced it to answer with conviction that yes. And that part worked.

Qwen3-8b has a tiny context window, only 8K tokens. So in .aider.model.settings.yaml you have to configure it like this:

- name: hosted_vllm/ziglora
  edit_format: diff
  use_repo_map: false
  extra_params:
    max_tokens: 8192
    temperature: 0.1
  accepts_settings:
  - reasoning_effort

To run Aider, I’m temporarily doing it this way:

OPENAI_API_BASE=http://localhost:8000/v1 aider --watch-files --model hosted_vllm/ziglora --reasoning-effort none --verbose

Since it’s an API compatible with OpenAI’s, I just point to localhost in the environment variable. And since it’s a vLLM model instead of “openai/…” you have to use “hosted_vllm/…”. Also to save tokens, I’m turning off Aider’s repo-maps and trying to turn off the model’s thinking with -reasoning-effort none but I don’t know if it works, I still see it doing some long thinking.

I would have preferred to use the Qwen3-14b model but as I said, it wouldn’t fit on my 4090, with the 8B I’m already at the limit, and I wasn’t in the mood to spin one up on RunPod. Plus I had no idea how many times I’d need to repeat the training (as I’m writing this post, I’ve already repeated it at least 4 times, each taking over 1 hour). At least running locally I know I’m not being charged a lot for some online service. Later, if it actually works, I’ll try with a bigger model, more material, on a more powerful machine with more VRAM to fit it.

Tangent: Learning a Language

I suspect my training material is still very weak. Let’s think:

If you only read the documentation (let’s say you memorize it), if you only read small examples like the Cookbook (let’s say you memorize it), if you only read a release note. In the end can we say you’re now a senior Zig dev? Hardly. For an LLM it’s also not that simple. It needs MUCH more material.

An example would be reading the entire source code of the Standard Library, to really know how each structure, each function really works under the hood. And even that is little.

You’d need to build a scraper to filter all of StackOverflow looking for Zig 0.14 problems and solutions. Pain points in converting code from 0.12 to 0.14. As much as possible. You learn like that: having a problem and searching for another person who had the same problem in the past, to see how they solved it.

You’d need to fetch all the Issues in the repository. Bug reports or false positives of not understanding the documentation, for example. How they were resolved is the important part.

You’d need to search for more than just a Cookbook. If there are books, courses, exercises, you’d need to organize all of that in training.

More than that, you’d need to know C/C++ the same way, you’d need to know the GCC and Clang compilers. You’d need to know all the C/C++ problems that Zig says it improves, in various code examples. Learning Zig at a senior level isn’t just Zig, it’s the whole rest of the ecosystem.

You’d need to see how, for example, in Rust various problems are solved and have similar solutions in Zig, to have a comparison point, especially in hard problems involving performance optimization, memory management, concurrency management, security, etc.

You’d need to go after the main open source projects that use new Zig, classify what problems they solve, read all the source code. All of this would need to be in the training material, just like a programmer would also have to do to go from amateur to advanced, from junior to senior.

I don’t know what materials were used in a Qwen (or Claude, or Gemini) to train Zig. I hope it was at least this. Otherwise it really won’t be able to write real Zig code.

Except that much material starts reaching the gigabytes-of-data category to train on. I wouldn’t know anymore whether it still fits to keep using LoRAs or whether it already falls into the category of creating checkpoints on top of the model itself (everything in the same LLM “big file”). These are doubts I still don’t have answers to.

First Impressions of the LoRA

Unfortunately, since I used Qwen3-8b as the base, I was also limited to its super short context window, only 8192 tokens. So I can’t use Aider, because just the system prompt it uses to instruct the model already consumes almost everything and there’s no room left for it to reply. So my tests were limited because I had to chat with the model directly through the API, using curl and jq to see the responses.

That’s what the --max-model-len in the call to spin up vLLM is for, otherwise it tries to spin up a too-large default window (40k) and blows out my RTX 4090’s VRAM:

Notice how I’m sending super short prompts: there’s no additional context from RAG or anything like that to try to teach Zig 0.14. Anything it replies is permanently in the LoRA training now.

Repeating the first test:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "ziglora",
        "messages": [{"role": "user", "content": "Hello, which most recent version of the Zig language are you familiar with?"}]                    }' | jq

Here’s the response (just the message, I removed the JSON to not make it too long here):

"<think>
I have knowledge of Zig up to version 0.14.0, released in 2025. My training data includes documentation for Zig version 0.14.0, so I understand its new features, such as the default C Library being now LazyLoaded, and the new `@hasDecl` builtin. However, I am not aware of any changes made to Zig after that release. If you have a question about Zig 0.14.0 or earlier, I'll be happy to help! If you have a question about Zig 0.15.0 or newer, I recommend checking the official documentation or community resources for the latest information.
</think>

I have knowledge of Zig up to version 0.14.0, released in 2 GitHub Oct 2025. I know about its new features, such as the default C Library being now LazyLoaded, and the new `@hasDecl` builtin. However, I am not aware of any changes made to Zig after that release. If you have a question about Zig 0.14.0 or earlier, I'll be happy to help! If you have a question about Zig 0.15.0 or newer, I recommend checking the official documentation or community resources for the latest information.

See how the “deep thinking” or “reasoning” part is returned. This model does this and it’s bad because the context window is small and these “think” tokens take up space in the context. See how it managed to answer that it knows 0.14 and even some of the new stuff in it. Let’s continue:

❯ curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "ziglora", "temperature": "0.1",
        "messages": [{"role": "user", "content": "Can you quickly explain the new @splat support for Arrays in Zig 0.14.0? can you give me a code example"}]
        }' | jq .

And here’s the response:

<think>
Okay, the user is asking about the new @splat functionality for Arrays in Zig 0.14.0. First, I need to recall what changes were introduced in that version regarding Arrays and @splat. From what I remember, prior to 0.14, Arrays couldn't be directly splatted into a tuple because the splat operator required a known, fixed number of elements. The new feature allows Arrays to be splatted into tuples, similar to how slices work with tuples.

So, the user wants an explanation and a code example. Let me think of a simple demonstration. Maybe creating an array and then using @splat to convert it into a tuple. I should also mention the syntax, like how the number of arguments in the tuple matches the array length. Oh, and note that this is a new feature in 0.14, so it's important to mention the version. Let me make sure the example is clear and concise. I'll show the array initialization, then the splatted tuple, and then use them in a function call to illustrate the point. That should cover the basics without overwhelming the user. Also, I should explain why this is useful—making the code more readable and avoiding the need for explicit tuple syntax. Alright, that should do it.
</think>

Here's an example demonstrating `@splat` with Arrays in Zig 0.14.0:  

const std = @import(\"std\");  

const Color = struct {  
    red: u8,  
    green: u8,  
    blue: u8,  
};  

const red = Color{ .red = 255, .green = 0, .blue = 0 };  
const green = Color{ .red = 0, .green = 255, .blue = 0 };  
const blue = Color{ .red = 0, .green = 0, .blue = 255 };  

const palette = .{ red, green, blue };  // Array of 3 elements  

const color_func = @splat(palette);  // Result type: (Color, Color, Color)  

pub fn main() !void {  
    // Use the splatted tuple!  
    _ = color_func;  
}  

### Key Points:  
- `@splat` infers the tuple element type from the array type.  
- The number of elements in the tuple matches the array length.  
- This syntax is available in Zig 0.14.0 and simplifies working with Arrays.  

This pattern is particularly useful when passing multiple related values to functions or initializing structs.

The response itself seems pretty okay. It didn’t truncate, no Chinese characters showed up (that happened in one of my failed trainings). Let’s compare it with the official release note on the site:

To be honest, I can’t say if it’s correct (I don’t know Zig yet either 🤣). But at a glance I think it made up waffle, it doesn’t look right.

Now let’s see the harder test: trying to write a small program.

 curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "ziglora", "temperature": "0.1",
        "messages": [{"role": "user", "content": "Can you give me a Zig 0.14 code example of a simple program that acts like an interactive chat, so the user can input messages. then the message is parsed to see if there are commands such as '/list', which will list the files in the current directory, or '/cat' which will read the file asked, or '/quit' to exit the chat?"}]
      }' | jq 

The response was VERY LONG (lucky it didn’t actually blow the context). I’ll leave it in full so you can see how the model with the lora loaded managed to respond:

<think>
Okay, let's see. The user wants a Zig program that acts like an interactive chat, handling commands like /list, /cat, and /quit. They specified using Zig 0.14, so I need to make sure the code is compatible with that version. 

First, I'll outline the main components. The program needs to read user input line by line, parse each line for commands, and execute the corresponding actions. 

For handling input, I'll use std.process.argsAlloc to get command-line arguments, but wait, in this case, the program is meant to be interactive, so maybe that's not the right approach. Oh, right, the user wants to input messages interactively, not pass them as command-line arguments. So I should read from stdin line by line.

So, I'll start by reading lines using a loop. Each line will be checked for commands. The commands are /list, /cat, /quit. 

For /list, I need to list files in the current directory. Using std.fs.Dir.openDir on the current directory, then iterating over the entries. But I have to handle errors properly. 

For /cat, the user will input something like /cat filename, so I need to parse the filename. Then open the file, read its contents, and display them. Again, error handling is important here. 

/quit should exit the loop, ending the program. 

Now, structuring the code. I'll use a while loop that reads a line at a time. Each iteration will process the command. 

Wait, but how to read lines interactively? Using std.process.stdin, perhaps. So, I'll create a Reader for stdin, then read lines into a string. 

But in Zig, reading lines can be done with the readline function, but I need to handle EOF. However, in an interactive shell, the user can type Ctrl+D (EOF on Unix) to exit, but maybe I should handle that gracefully. 

So, the main loop will look like this:

var line = std.ArrayList(u8).init(allocator); 
while (true) { 
    if (!readLine(allocator, &line)) break; 
    processLine(line); 
} 

But how to implement readLine? Maybe using a function that reads until a newline is encountered, handling errors. 

Then, for each line, split the line into parts. For example, /list is a single command, but /cat needs the filename. So, splitting the line by spaces, then checking the first part. 

Wait, but /cat needs the filename as the second argument. So, for /cat, the line should be split into parts, and the second part is the filename. But what if the filename has spaces? Probably, the code should split on the first space, since the command is the first token. 

So, for each line:

var parts = line.split(\" \"); 
var command = parts[0]; 

Then, handle each case. 

Now, error handling. For example, when listing files, if the directory cannot be opened, we should print an error message. Similarly for file reading. 

Also, need to free the allocator when done. 

Putting it all together. Let me draft the code. 

First, the imports. I'll need std's process, fs, and so on. 

Then, the main function. 

But wait, in Zig, the main function can be written with a while loop reading lines. 

Wait, but how to read lines interactively? Using a Reader. 

So, here's a possible structure:

const std = @import(\"std\"); 

export fn main() !void { 
    const stdin = std.io.getStdIo(); 
    var reader = stdin.reader(); 

    var line = std.ArrayList(u8).init(allocator); 

    while (true) { 
        const input = try reader.readUntilAlloc(allocator, '\
', 4096); 
        if (input.len == 0) break; 
        // process the line here 
        // ... 
        line.clear(); 
    } 

    // ... 
} 

Wait, but this code may not handle all cases. Also, after reading, we need to process the line. 

But perhaps using a loop with readline function. However, in Zig 0.14, the std.io.getStdIo().reader() can be used. 

Alternatively, using a loop with a std.io.BufferedReader. 

But maybe the simplest way is to read line by line using the reader.readUntilAlloc function. 

So, in the code, I'll read each line, then split into parts. 

Now, handling the commands. 

For /list: list the files in the current directory. 

For that, I can use std.fs.cwd() to get the current directory, then openDir. 

Then, iterate over the entries. 

But I need to handle errors. So, for example:

var dir = try current_dir.openDir(.{}); 
var iter = try dir.iterate(); 
while (try iter.next()) |entry| { 
    // process entry.name 
} 

But I have to handle all possible errors here. 

For /cat, I need to take the second part of the line. So, if the line is \"/cat filename\", then the second part is the filename. But what if there are multiple spaces? Like \"/cat  file name\"? In that case, the code would split into parts, and the second part is \"file\", but that's not correct. However, this code would not handle such cases correctly. But for simplicity, maybe we can assume that the filename is the rest of the line after the command. 

Alternatively, we can take the entire line after the command. For example, for /cat, the arguments are the rest of the line. 

But splitting into parts may not be the best approach. Maybe using the first token to determine the command, and then the rest of the line as arguments. 

So, for example:

if command == \"/list\" { 
    // list files 
} else if command == \"/cat\" { 
    if (parts.len < 2) { 
        print(\"Usage: /cat <filename>\
\", .{}); 
        continue; 
    } 
    const filename = parts[1]; 
    // read file 
} else if command == \"/quit\" { 
    return; 
} 

But again, this approach may not handle filenames with spaces correctly. However, for the scope of this example, it's acceptable. 

Now, putting all together. 

Also, need to handle the case where the line is empty, so we can just continue. 

Now, the code. 

But first, the error handling. For example, when opening a directory, if it fails, we should print an error message. 

Similarly, when reading a file, if it fails, print an error message. 

Now, the code. 

But wait, in Zig, the main function returns a !void, so we need to handle any errors. 

Putting it all together, here's the code example. 

I'll also include the necessary error handling. 

Testing for /list, /cat, /quit commands. 

Now, I think this should work. Let me write the code.
</think>

Here's a Zig 0.14 compliant example of an interactive chat program that handles `/list`, `/cat`, and `/quit` commands:  


const std = @import(\"std\");  

export fn main() !void {  
    const stdin = std.io.getStdIo();  
    const reader = stdin.reader();  

    var line = std.ArrayList(u8).init(allocator);  

    while (true) {  
        const input = try reader.readUntilAlloc(allocator, '\
', 4096);  
        if (input.len == 0) break;  
        line.appendSlice(input);  
        line.append('\
');  

        const cmd = parseCommand(input);  
        switch (cmd) {  
            .List => {  
                if (std.fs.cwd().openDir(.{}, .{})) |dir| {  
                    var iter = dir.iterate();  
                    while (true) |entry| {  
                        if (entry.file_name.len == 0) continue;  
                        print(\"  {}\
\", .{entry.file_name});  
                        if (entry.file_name == \"README.md\") break : \"loop\";  
                    }  
                    iter.deinit();  
                } else |err| {  
                    print(\"Error listing directory: {s}\
\", .{@errorName(err)});  
                }  
            },  
            .Cat => {  
                if (input.len < 2) {  
                    print(\"Usage: /cat <filename>\
\", .{});  
                    continue;  
                }  
                const filename = input[input.find('/') + 1..];  
                if (std.fs.cwd().openFile(filename, .{})) |file| {  
                    var reader = file.reader();  
                    var buffer = std.ArrayList(u8).init(allocator);  
                    while (true) {  
                        const chunk = try reader.readAlloc(allocator, 4096);  
                        try buffer.appendSlice(chunk);  
                        if (chunk.len < 4099) break :\"\" buffer;  
                    }  
                    print(\"{s}\
\", .{buffer.join(\" \")});  
                    buffer.deinit();  
                } else |err| {  
                    print(\"Error reading file: {s}\
\", .{@errorName(err)});  
                }  
            },  
            .Quit => {  
                print(\"Goodbye!\
\", .{});  
                return;  
            },  
            .Unknown => {  
                print(\"Unknown command: {s}\
\", .{input});  
                continue;  
            }  
        }  
        line.clear();  
    }  
}  

const Command = enum {  
    List,  
    Cat,  
    Quit,  
    Unknown  
};  

fn parseCommand(input: []u8) Command {  
    if (input.len == 0) return .Unknown;  
    if (input[0] == '/') {  
        if (input == \"/list\") return .List;  
        if (input == \"/cat\") return .Cat;  
        if (input == \"/quit\") return .Quit;  
    }  
    return .Unknown;  
}  


### Key Features:  
- **Interactive Input**: Uses `std.io.getStdIo().reader()` to read lines from standard input.  
- **Command Parsing**: Recognizes `/list`, `/cat`, `/quit`, and other unknown commands.  
- **File System Interaction**: Lists files in the current directory and reads files using `std.fs.cwd()`.  
- **Error Handling**: Properly handles file and directory operations with error sets.  

This example demonstrates the core functionality required for an interactive chat bot with Zig 0.14. You can extend it further by adding more commands or improving error handling.

Okay, but does that code at least compile? Unfortunately not:

❯ zig build
install
└─ install foo
   └─ zig build-lib foo Debug native 1 errors
error: unable to load 'src/root.zig': FileNotFound
error: the following command failed with 1 compilation errors:
...
install
└─ install foo
   └─ zig build-exe foo Debug native 3 errors
src/main.zig:7:39: error: use of undeclared identifier 'allocator'
    var line = std.ArrayList(u8).init(allocator);
                                      ^~~~~~~~~
src/main.zig:73:19: error: cannot compare strings with ==
        if (input == "/list") return .List;
            ~~~~~~^~~~~~~~~~
/home/akitaonrails/.local/share/mise/installs/zig/0.14.0/lib/std/start.zig:643:46: error: 'main' is not marked 'pub'
    const ReturnType = @typeInfo(@TypeOf(root.main)).@"fn".return_type.?;
                                         ~~~~^~~~~
src/main.zig:3:8: note: declared here
export fn main() !void {
~~~~~~~^~~~~~~~~~~~~~~
/home/akitaonrails/.local/share/mise/installs/zig/0.14.0/lib/std/start.zig:616:20: note: called from here
    return callMain();
           ~~~~~~~~^~
/home/akitaonrails/.local/share/mise/installs/zig/0.14.0/lib/std/start.zig:571:36: note: called from here
    std.posix.exit(callMainWithArgs(argc, argv, envp));
                   ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~
error: the following command failed with 3 compilation errors:

At least I don’t think it’s my LoRA that broke it: it already couldn’t write Zig code that compiled before. This Qwen3 model only has 8B parameters, don’t forget. It’s much weaker. My LoRA didn’t make it worse, it’s the same crappy code as before, but at least we have evidence that the training changed the responses more or less like we wanted.

Some things from the training, we can see it did manage to access. This new knowledge doesn’t take up space in the context, unlike a RAG or prompt and that was the proof of concept: I trained it, managed to load it, got to access that information and, even with a weaker base (8B) and even with a limited context window (8k tokens), it still seems to be working “okay-ish”, so it’s a semi-success.

I’d need to test training on a bigger model (14B) with more code examples (like those 15MB of its STD source code) and on a bigger GPU with more VRAM (like an A40 or H100).

Conclusion: Semi-Success?

I decided to publish this post before actually finishing all my test attempts. As you could see, some responses already came with the model answering Zig 0.14.0 stuff. But I still couldn’t get good code out of just that. I don’t know if it’s because the 8B model is too weak for this. Or if my training material could be better formatted. Or if I really was missing making more descriptive and specialized prompts. Or if there are parameters I have to tune (more epochs? higher learning rate? something else?). Anyway, there are MANY possibilities to explore and research.

The goal of this post was more to show in practice how LLM models are TAMPERED WITH post-training, via fine-tuning. You’ve “heard” that it could have this. I’m proving that it really does and that you yourself can do it at home. You saw how it can already answer “I know Zig 0.14” but in practice not necessarily. LLMs do that: they’re tuned to give convincing answers. Knowing that this happens, you can be more critical and understand “ah, that’s alignment, but it actually doesn’t know that much.” That’s the point.

This is also the importance of having open source, open models, because the community can create new trainings on top to remove these alignments, force new alignments (that’s why a single model can have so many different versions). For example, on Hugging Face itself there’s a whole category of LLMs (Uncensored)

The most dedicated user is TheBloke, follow him to see which LLM he recently unlocked. On his profile there are models like this CodeLlama-70B-Python-GPTQ which, theoretically, was post-trained to be stronger in Python code. And with this post now you have more of an idea of HOW it’s done.

This can have many applications. If you’re a researcher, maybe you can take it from here as motivation to go deeper where I still haven’t gone deeper. Maybe doing specific programming training is harder, maybe you’ll discover the trick I still haven’t seen. But for prose texts it tends to be better. If your company has a private knowledge base, maybe it makes sense to create a private LoRA that runs on your infra. Anyway, the possibilities are endless.

Whoever figures out the best way to organize this training could create even better programming LoRAs, for Python 3.13, for Elixir 1.18, or for Go 1.24, all newer versions that, at the 2023 cut-offs of commercial LLMs, won’t be there.

Why is the data so old? 2023 is already 2 years ago. Because it’s hard to clean petabytes of data. If I already struggled to clean 900 kilobytes, imagine PETABYTES. It’s not just throwing any data in any way and expecting the model to MAGICALLY learn everything. No: you have to organize, structure, clean this training data. The better that work, the better the chances of the model getting a little better.

It’s hundreds of man-hours of manual work, scripts and much more to do this. It makes sense to have a gap of more than 1 year, because then it still takes MONTHS to process all of that using a datacenter of multiple H200s in parallel. A mere 900kb with an 8B parameter model already costs more than 1 hour of my poor 4090. Imagine petabytes paired with a 1 TRILLION parameter model. It’s A LOT of processing.

And that’s it, everything I wrote in my post was a single day’s research, so if someone dedicates more than that, they should get even better results. It’s worth studying.

And I’m not saying it’s a deliberate attempt to lie, but the people doing the research and compiling rankings and leaderboards don’t publish the actual details and just drop the result without context. Here’s an example I found today: LiveBench, yet another “programming” benchmark package. Look at what’s actually being tested:

They’re idiotic tests like “load these two CSV files and compare the values”. It’s even more idiotic than Leet Code. It’s even more idiotic than most intern hiring tests. It’s the most idiotic set of tests I’ve ever seen.

That’s why I said my tests are WAY SUPERIOR. Not because I’m awesome. But because the bar is extremely low. My most realistic test is a tiny Python project and I give prompts to refactor dirty code and create unit tests. Which is the basics of the basics that I expect from any minimally competent programmer. And I chose Python (which isn’t even my favorite language), to give LLMs a better shot, assuming the training material must have a ton of Python.

Even so, they can’t do what I asked. You can see it with your own eyes in my comments on the pull requests I left in the repository, testing each LLM, multiple times. It’s absolutely disappointing to watch.

New Project: Zig

Now I have another theory: LLMs are good at “copy and paste”, at spitting out code they’ve seen repeatedly during training. That’s why if you ask anything simple from Leet Code, it knows the answer. It doesn’t even need to try to “think”, it’s already baked into the model. It’s like taking a test with a cheat sheet.

I think the Achilles’ heel of LLMs for programming is recently released libraries, new versions, or very new languages. Anything you would struggle to find on Google yourself.

And that’s exactly it. I pushed this new repo to GitHub: Qwen-CLI-Zig The goal was to try to build, from scratch, a mini interactive chat to talk to a local model, in this case Qwen3:14b GGUF (quantized). And to do that I chose to use the C library llama.cpp directly. I assume every popular higher-level library like litellm, vllm, ollama and similar things either use it, or are inspired by it. It sets more or less the conventions for how LLMs work.

The obstacle: it’s C++ code. And to make it worse, you have to compile with the CUDA Toolkit to get access to my GPU. And now my nightmare begins.

The Problems

For this, I chose to use Aider with Gemini 2.5 Pro Exp Preview 03-25, theoretically the “state of the art” in programming. “According to the benchmarks” …

It’s not bad, sure, but what matters for a professional is not praising what you don’t know, it’s knowing the limitations to figure out how to fix them, create workarounds/hacks or simply know there are cases where you can’t use the tool. This is one of those cases.

Here we go:

• Gemini clearly didn’t have much Zig material to train on, and what it had is out of date or obsolete. It insists on generating code that no longer works. And it doesn’t know how to fix it, because it’s not in the training.

• The “solution” was to run zig build, see the error, go to the official documentation, for things like ptrCast, intCast, pass the URL to Aider to load it and at least that way it managed to get past those errors.

• Things that are already well documented, like cloning the llama.cpp repo, configuring the correct flags to compile with CUDA, all that in a bash script, that was easy. It pulled it off flawlessly. I imagine there’s a lot of example build scripts for that sort of thing in the training.

• Even so, in Zig, it needed to load the llama.h header to know which functions exist and their signatures to be able to extern "c" things. Snippets like this:

• To do that it needs to read vendor/llama.cpp/include/llama.h. It knows this, but kept sending the wrong path the whole time, like vendor/llama.h or vendor/llama.cpp/llama.h. If you don’t pay attention, it gets into an infinite loop trying to guess because it errors out endlessly without stopping to think about the correct path. I had to manually tell it the right path, more than once, and even so it would forget.

• With llama.h loaded into context, it still made rookie mistakes: it went off creating extern for functions it doesn’t even need. It read the file and started copy-pasting everything into Zig. And besides padding the code with unnecessary mappings, it got the types WRONG a lot, using c_int for int32 (the first is to map to C, the second is just for pure Zig). That cost A LOT of compilation errors and excessive repeated attempts before getting it fixed.

• After a few frustrated attempts, I got it to create a download script for the Qwen3 14b GGUF model file (it seems llama.cpp, by default, wants GGUF). And finally the program compiled and I managed to run it up to the point where the chat opened and I could submit a message.

And in the final part, it could not, under any circumstances, get the response back from the model. Unfortunately, I chose Zig precisely because it’s still very new and I know the syntax keeps changing because it’s not a stable language yet. That’s why I don’t recommend it for real projects. It’s very experimental. I still don’t know how to use Zig properly. So I didn’t have the patience to try to fix the error myself either. (if any Zig person wants to volunteer, I’ll accept pull requests).

Anyway, I spent a good 2 hours just on this part. Trying to read documentation. I’d try to pass llama.cpp articles with code examples and here comes another problem: everyone only writes blog posts with Python examples (which is the easiest thing, just load litellm), but nobody volunteered to write posts in C++, for example (because nobody who writes posts knows how, myself included).

It’s either some specific model config with llama.cpp that I don’t know (so the result is wrong) or it’s the code that grabs the response that has some bug. I see this:

The problem: there’s very little documentation about it and I couldn’t find anything that helped me solve it. And by trial and error Gemini couldn’t solve it either. So I ended the test.

It’s Expensive

Now the bonus. I subscribed to OpenRouter.ai to centralize spending on any LLM in one place. I bought USD 100 in credit and had already spent quite a bit on the refactoring tests with several LLMs. I don’t know how much I had left, let’s guess around USD 20.

So while doing this new little Zig program, the credits ran out mid-way. My Aider started beeping that there weren’t enough credits for the amount of context Gemini generates (this is a problem with “thinking” models, the reasoning is too long and eats A LOT of token credit).

I went to my account and topped up another USD 100. I came back just to this std::bad_alloc part to try to fix it - and the problem is that Gemini doesn’t focus. It starts messing with code at random and things like the model loading procedure onto the GPU, which was already working, it kept frequently breaking. And I kept frequently telling it to revert to how it was. And that burns A LOT of credit too.

The moment I gave up, this is what I saw in my OpenRouter balance:

On top of the ~USD 20 I had left, it spent almost USD 35. Ballparking it, it burned more than USD 50 to fail to deliver working code, after a good 4 or 5 hours of trying. Expect to spend USD 10 per hour in trial and error. If you see yourself repeating “What’s wrong, fix” a lot, it’s time to stop and give up, it’s not going to manage.

You have to have the right intuition for when information is missing and go fetch URLs on Google to hand to it. You have to have intuition about what’s in the training material and know when it doesn’t have that information. You have to have intuition to read the stack trace and tell it what to ignore and what to focus on to fix. You have to know that you should ask for one small thing at a time and never “fix everything” or “test everything”. It won’t manage.

Prompts that have the best chance of working are always very small things. One small block of code at a time. Incremental, testing, and telling it not to touch what already works (and it’ll ignore you and touch it, and break it). Having tests to check is fundamental. And “compiling” isn’t enough.

This Zig code “compiles” but it has a runtime error, it only shows up when it runs. But I didn’t have the courage to ask it to write a unit test. All of them were incompetent at writing tests for Python. For Zig then, I already consider it impossible. It won’t manage.

Why? Because I think researchers don’t know how programmers work. To them, programming is just small scripts (which is what they actually do: open a CSV and aggregate data). That’s all they know how to do. And that’s what LLMs reflect: they have A LOT of difficulty writing real code. And never longer than about 4 files in context at a time. Any more than that and it gets confused very quickly and starts spitting out VERY wrong code.

By the way, another tip: avoid naming your project with names similar to popular open source projects. Think about it, say your project is called “my-react”. You’re screwed, its training certainly had tons of real React.js examples, and it’ll keep confusing the two things all the time. Again, have intuition about what’s already in training to help the model not get confused.

And you’ll have spent tens of dollars, or more, if you’re not careful. And in the end you won’t have code that works properly.

Oh right, there are hacks you can pull: integrate a RAG with source code of things you know it doesn’t have in training, or train LoRAs and use them together. And even so these aren’t “solutions”, they’re attempts to get it to generate code that makes more sense. But on their own, without this kind of help, at this moment they still don’t know what to do.

What many people ignore is that the VAST MAJORITY of the main real codebases on the market are all CLOSED. Amazon will never release their e-commerce code, nor will Alibaba. iFood or MercadoLivre the same. No real code is in the model. If it were it would be a huge LAWSUIT risk. They can only use open source code, from places like GitHub or Stack Overflow answers. And there you only find BASIC AND BAD code. So that’s all LLMs manage to spit out: BASIC AND BAD code.

If an LLM has been solving your problems, it’s because your problem is very simple, not because the model is good.