But nobody needs a GPT-class model for everything. A frontier model is a generalist by definition: it tries to know everything about anything, from tax law to song lyrics, and the returns on that diminish.

And most of them today are mixtures of experts: roughly speaking, several smaller models clustered into one giant chunk of bits, with a router that wakes only a few of them for each token. OpenAI’s gpt-oss-120b has 117 billion parameters in total and uses 5.1 billion per token. DeepSeek-V3 has 671 billion and activates 37. The big model is already, on the inside, a collection of small models.

I could have grabbed an open model, gpt-oss, Qwen, Phi, and adapted it. The techniques exist and are well documented:

• Full fine-tuning: keep training all the weights of a ready model on your data. It’s classic transfer learning, formalized for text by ULMFiT in 2018.
• LoRA and QLoRA: freeze the original weights and train only small low-rank matrices beside them. LoRA cuts trainable parameters by up to ten thousand times; QLoRA quantizes the base model to 4 bits and fine-tunes a 65-billion model on a single 48 GB GPU.
• Distillation: train a smaller model to imitate a larger one’s outputs. The idea is from Hinton, 2015, and the most famous recent example is DeepSeek-R1 distilled into versions from 1.5 to 70 billion on top of Qwen and Llama.
• Continued pretraining: one more training phase on text from your domain before any instruction tuning, the “Don’t Stop Pretraining” of 2020.
• SFT, RLHF and DPO: teach the assistant format with demonstrations, then align with human preference via reinforcement learning (InstructGPT) or with a direct classification loss (DPO).

With any of those I’d have a useful model in hours. But the goal was different. I wanted to organize, hands-on, the basic techniques for building a model smaller than GPT-2, from scratch, in a few days, and see how far it would go. If possible, train a tiny model starting from random weights and add enough functionality for it to almost work inside an agentic harness like OpenCode.

Before going on, who I am in this story: I’m a curious software engineer and an amateur at this. As an LLM researcher I’m a zero. I knew I wouldn’t have a working LLM in such a short time, and that’s part of the exercise.

But it’s not unachievable. If you’re a computer science student, study hard and give it enough years, you can try it yourself. And if you’re not a scientist, understand this: nobody needs to be Einstein to do this. The foundations are all available, published, for you to study and try. What’s mandatory is elbow grease.

Spoiler: we got far into the “almost” and stopped there. The repository is public, with every step and every failure documented. This text is the summary.

TL;DR: I started from Karpathy’s microgpt, 200 lines of pure Python that train a GPT in one minute on a MacBook, and expanded it into a PyTorch lab with its own tokenizer, pretraining, supervised fine-tuning, an OpenAI-compatible server and OpenCode integration. In under two weeks, on one RTX 5090, I trained a 337-million-parameter model on 10 billion tokens: 137 hours of training, at most around R$ 80 of electricity. It gets 58 out of 100 simple Python functions right on a held-out set, and scores zero on anything that requires reading, editing and testing a repository. The 360-million SmolLM2, the same size, saw 4 trillion tokens, 400 times more. Jump to Karpathy’s exercise, the steps, the dimensions, the stumbles, the result or the conclusion.

Karpathy’s exercise

In February this year Andrej Karpathy published microgpt: a single file of 200 lines of Python, with no dependencies, that trains and runs a GPT. He calls it “the full algorithmic content of what is needed. Everything else is just efficiency”. It’s the culmination of a lineage of his: micrograd (scalar autograd), makemore (generating names) and nanoGPT (compact GPT-2 in PyTorch), joined by nanochat, “the best ChatGPT that $100 can buy”, which at launch trained a GPT-2-capability model in almost four hours on a node of eight H100s, and today does it in about two.

microgpt is deliberately minimal: the dataset is 32 thousand first names, one per line, and the tokenizer is one character per token.

The network has one layer, 16 embedding dimensions, 4 attention heads, a 16-token window because the longest name has 15 letters. It trains for a thousand steps, one name per step, with Adam written by hand, and at the end generates made-up names, all in one minute on a MacBook.

What it teaches, by having everything exposed in a single file:

• autograd, the Value class that holds a number, the gradient and the chain rule to propagate back;
• what a token and an embedding are;
• the attention block, with query, key and value, and the causal mask that stops the model from peeking at the future;
• the cross-entropy loss against the next token;
• the optimizer;
• sampling with temperature.

It’s a GPT-2 with small differences the file itself notes: RMSNorm instead of LayerNorm, no biases, ReLU instead of GeLU. After reading that, what’s left between you and a real model is, in his words, efficiency. I went to see how much efficiency.

What we expanded

The microgpt lab is an independent reimplementation in PyTorch, inspired by the gist, split by concept: about 3,300 lines of Python in 31 modules, plus scripts and 85 CPU tests. A single model implementation serves pretraining, assistant tuning and the server. Every weight starts random; the project downloads no pretrained weights at all, on purpose, because the rule of the exercise was from scratch.

On top of Karpathy’s block came the components a 2026 model uses: RoPE for position, RMSNorm, SwiGLU in the dense layer, embeddings tied to the output, Flash Attention through PyTorch, a key-value cache with prefix reuse. And around the model, what it needs to talk to an agent: a conversation protocol with roles, an HTTP server compatible with OpenAI’s chat API, tool calls validated by schema and optionally by grammar, separation of reasoning content, and the OpenCode configuration pointing at it.

The project went through four scales:

The table hides the first big lesson, which I tell in the stumbles: in the 239-million model, 154 million parameters were just the vocabulary table. 85 million were left for the actual transformer.

The steps of building an LLM

Regardless of size, the path has the same stations. What changes is the trailing zero at each one.

1. Data

A model is what it eats. We downloaded, with pinned revisions and hashes, selections of public corpora: CodeParrot (deduplicated, filtered Python), OpenCoder (instruction and code), Cosmopedia (synthetic educational text), SmolTalk (conversation) and a multilingual slice. The 14 download manifests add up to about 20 GB raw, all stored on the NAS.

Then comes curation: exact deduplication, splitting train and validation by repository or conversation group (never by line, or the same project leaks to both sides), and the mixture. The final foundation corpus has 3.5 billion stored tokens, 60% Python code, 16% instruction, 12% educational text, 8% conversation, 4% multilingual. Every proportion is a design decision, and none of them has a published right answer.

2. Tokenizer

The tokenizer is the dictionary between text and integers. The byte lesson uses 256 tokens plus one for end of document. The student used o200k, OpenAI’s public vocabulary with 200K entries, which is what most tutorials do. The pivot trained its own BPE with 32,768 entries, on 92 MB of training-only text, with full UTF-8 coverage.

The choice looks like a detail and decides where the parameters end up. Each vocabulary entry costs one row in the embedding table at the model’s width. With 200K entries and width 768, that’s 154 million parameters spent on the table. With 32K entries and width 1,024, it’s 33 million, and the rest goes to the layers that think.

3. Pretraining, or foundation

This is Karpathy’s loop at scale. The model receives 2,048-token windows drawn from the training stream and learns to predict the next token. Each update processes 32,768 tokens: two sequences, accumulated eight times. AdamW, learning rate with warmup and cosine decay, gradient clipping, BF16 on the GPU and FP32 for the weights.

The pivot ran about 305 thousand updates, 10 billion tokens of exposure. Since the corpus has 3.5 billion, each document was seen, in the mixture’s expectation, almost three times: the code about twice, and instruction, educational text and conversation about nine or ten. Every 500 updates, a validation and a checkpoint. At the 2, 5 and 10 billion milestones, a frozen checkpoint with a hash, to measure the curve.

4. Supervised fine-tuning

Pretraining produces a model that completes text. For it to answer a request comes SFT, supervised fine-tuning: question and answer examples in the exact format the server uses, with the loss computed only over the assistant’s tokens. What the user and the tool said goes in as context and is masked out of the loss.

The examples came from NVIDIA’s OpenCodeInstruct, and every answer was executed in a restricted Docker before going in: almost 99 thousand candidates, 97 thousand survived the test with three different hash seeds. 92 thousand training episodes remained. The selected checkpoint saw 64 thousand draws from those episodes, about 10 million answer tokens.

5. Protocol, tools and server

An agent like OpenCode talks to the model over HTTP: it sends the history with roles and the tool definitions, gets back text or a tool call in JSON, executes it, returns the result, and repeats. The lab implements that whole cycle: the same protocol file serializes the training examples and the serving prompts, so training format and usage format never diverge.

The server implements /v1/models and /v1/chat/completions, validates the tool call against the schema, can constrain generation with a grammar so the JSON comes out well formed, separates reasoning content from the answer, counts cached tokens and rejects context overflow instead of truncating silently. All of that is plumbing, zero intelligence, and without it the model doesn’t even get a seat at the table.

6. Evaluation

The step that changed the project the most. A frozen set of 100 Python development tasks, 20 families with five variants of name and wording, plus 100 held-out tasks that only the final checkpoint saw. Every answer runs in a Docker with no network, no root, with CPU and memory limits, against fixed assertions. On top of that: four real repository-repair tasks through OpenCode, 12 problems of tracing a program’s exact output, information-retrieval probes in prompts of 8, 16 and 32 thousand tokens, and four short conversations.

No HumanEval, MBPP or SWE-bench. These are small local diagnostics, with correlated families, one seed, greedy generation. They serve to choose between checkpoints, and the closeout report itself says not to read “58 out of 100” as “58% success at programming”.

The dimensions of the problem

What all of this cost, and how it compares with the real world:

The electricity bill: the RTX 5090 has a 575 W rated power. Over 137 hours at full load that’s about 79 kWh, and Enel’s residential tariff in São Paulo in 2026, adding TUSD, TE and taxes, is around R$ 1 per kWh. Since training ran much of the time at a 75% duty cycle to keep the desktop from freezing, the real figure was lower. It’s the marginal cost of someone who already owns the card.

Renting a 5090 on Runpod costs about US$ 1.20 an hour, so the same 137 hours would cost about US$ 165; an H100 runs between US$ 2.90 and 4 an hour, depending on the variant.

And the comparison that gives the scale. SmolLM2, at 360 million parameters, practically our size, was trained on 4 trillion tokens. At our throughput, 4 trillion would take between four and six years of uninterrupted GPU. Same model size, 400 times the amount of study.

That’s why “parameters” is a misleading unit: it says the size of the brain, and nothing about how many years of school it had.

The rule of thumb derived from Chinchilla (70 billion parameters, 1.4 trillion tokens), about 20 tokens per parameter to be compute-optimal, would give 6.7 billion tokens for our model. We went past that. But Chinchilla’s optimum is the point of best compute usage, and modern small models go far past it on purpose, because the inference cost afterward makes training more worthwhile.

The stumbles

The full list is in the closeout report. The ones that taught the most:

Measuring too close to training. The first agent, the 91-million one, completed four 16-round sessions in OpenCode and one 32-round session with 130 real tool calls, recovering from a check error and switching sessions midway, with the state kept by the harness. It looked ready.

Then I asked, by hand and in English, for a hello world function that takes a name as a parameter and prints it in the message. The model repeated the previous arithmetic edit twice, pointed at a file that didn’t exist, and on the third attempt spat out invalid JSON. All it knew was the training task family.

The vocabulary ate the model. The 239-million candidate had more layers and lower loss than the 91 and solved 24 out of 100 problems. Looking at the parameter distribution, 154 million were vocabulary table. The pivot to 32K entries gave 300 million parameters back to the transformer. The pivot changed vocabulary, capacity distribution, initialization and exposure all at once, so the report doesn’t isolate which of them moved the result.

Loss is not competence. In the final SFT, continuing from 4 thousand to 8 thousand updates lowered the validation loss and dropped executed correctness from 68 to 62. Lower loss, worse code. Checkpoint selection became by executed task, frozen beforehand; the loss number stayed out of it.

Data and graders lie. “Correct” answers from the public dataset failed when executed again. Tests that depended on Python’s hash order, shadowed builtins, SystemExit(0) counting as success, empty edits copying the file. Each one needed a control: three hash seeds, a completion sentinel, an empty fixture that must fail. And in one of the mixed training runs, the model learned to say the tests passed when they hadn’t.

Training format and usage format have to match. Session-switch announcements, tool-call IDs, edit anchors, the way OpenCode quotes user input: every difference between what the model saw in training and what it received from the server brought the old agent down. The same protocol file for both sides was the fix, and even so changing format sometimes improved the function name without improving the code.

Tokenization crosses correctness boundaries. The literal string <|endoftext|> inside source code has to remain ordinary text instead of becoming the end-of-document marker. Reconstructing UTF-8 documents without splitting a character in half. XGrammar’s support for byte-level vocabularies. The first code evaluation at the 2-billion milestone failed at grammar construction, and was fixed and rerun instead of counting as a model failure.

The NAS almost cost me days. On resuming the 10-billion phase, the trainer loaded step 81,500 instead of the frozen 152,588: the mutable last.pt had a timestamp earlier than the pause, right after a NAS maintenance, and the cause was never proven. We discarded 1.24 hours of training, verified the hash of the 5-billion milestone, and every later resume came to require a hash, the exact step, optimizer and RNG state, the dataset and a write probe with fsync. Never resume a training run just because the file is called last.pt.

A desktop GPU is a shared GPU. Free VRAM didn’t stop the browser from freezing. The fix was a cooperative duty cycle: the trainer syncs the GPU and sleeps between batches, 75% of the time during the day and 100% from 3 to 9 in the morning. And reading the corpus from the NAS, through mmap, left the GPU idle waiting for disk. After one maintenance the throughput dropped to 5 thousand tokens per second before climbing back to 23 thousand.

I stopped at a pipeline boundary. The pivot’s controller implemented foundation, code SFT and holdout. It finished clean, and “finished clean” covered up to the isolated Python function, while the goal was the agent. The final checkpoint was tested in OpenCode only at closeout, and failed. It should have been tested before any sentence with “ready” in it.

What came out the other side

The final model’s spec sheet:

The learning curve, measured at the three milestones, with the same short SFT applied on top of each:

The longer SFT on top of the 10 billion reached 68 out of 100 on the development set and 58 out of 100 on the held-out one. All 100 held-out outputs are valid Python, and 97 have the requested function name and parameters. Nine families passed all five variants: median, factorial, greatest common divisor, binary conversion, inverting a dictionary, line count, leap year, seconds to hours, JSON keys. Running total ended at four out of five.

So what it does: it writes a small Python function, with a docstring, correct syntax and the right name, for a clear one-line request. Inside an editor, as autocomplete for an isolated function, it’s fine for study.

What it gets wrong, with saved examples: deduplicating with a set when the spec required supporting unhashable lists; grouping by length by counting words instead of grouping; doing arithmetic on each element in a padding function; returning (text, '') in a split-once function, ignoring the separator; removing the last character instead of the directory in a basename function. Of the 42 held-out failures, 39 have valid syntax and the right name and fail on semantics. The hard part was never looking like code.

And what it zeroes:

• 0 out of 4 real repository repairs in OpenCode, with zero tool calls. Asked to read, edit and test a file, it returned text with code. In one of the tasks, it renamed the function to fix_mean.
• 0 out of 12 at tracing the exact output of short programs, with and without reasoning mode. Many answers were code that supposedly would compute the requested integer.
• 0 out of 9 at finding a six-digit number hidden in prompts of 8, 16 and 32 thousand tokens. The 32K-token allocation works; the useful memory, none.
• Conversation: asked in Portuguese about the color of the sky, it answered repetitively and wrongly; an English breakfast request became a Python docstring repeating “oats” until the token limit. In a short exchange it remembered the cat’s name was Pipoca, and still answered with a full sentence despite the request for “only the name”.

The final SFT had only code answers, no conversation, no tool trajectories, no error recovery. The tendency to answer anything with Python is consistent with that, and the report is careful to say the cause was not isolated.

What it would still take for a simple agent in OpenCode

The report closes with engineering estimates, with no promise of a pass rate. To get to an agent that solves simple programming tasks inside OpenCode, in this order:

1. Teach tool use on the stronger model. Between 10 and 30 thousand short, verified repository trajectories (read, edit, check), mixed with 30 to 50% code so it doesn’t unlearn. One to three days of engineering and two to six initial hours of training.
2. Correct its own mistakes. Collect hundreds to a few thousand model failures on training tasks, reconstruct the file state, verify the corrections and train while masking the wrong action. Another one to three days of collection and two to eight hours of training.
3. Bring the conversation back. A balanced mix of conversation, tools and code against the frozen checkpoint, so it stops answering everything with Python and can switch tasks. One to four hours of training.
4. Real long context. Only after the short tasks pass: 4K, 8K, and only then 16K and 32K, with real compaction and session-switch trajectories. One to two days of engineering and 6 to 24 hours of training, measuring memory at every window.
5. More foundation, with better data. Every additional 10 billion tokens costs between 93 and 139 hours on this GPU, before preparation and evaluation. And repeating the same 3.5-billion mixture isn’t worth it: it needs more unique educational code.

None of those steps was executed. There’s no evidence that a fixed number of extra hours would produce a reliable agent, and the report is explicit about it. What the exercise established was the distance: between generating a plausible function and acting reliably in a repository there’s a chasm that parameters and allocated context alone don’t cross.

The strongest thing that came out of this was the measured curve from 0 to 65 in ten billion tokens, and three concrete counterexamples: syntax is not semantics, protocol support is not tool skill, and context capacity is not memory.

The future is smaller and more specialized

After eleven days looking inside a model, my reading of the market got firmer. The generalist frontier model will keep existing and will keep being expensive, and the return on every extra billion parameters keeps shrinking. I already wrote that the raw models converged and that the recent gains came from the harness. The exercise gave me the other half of the argument.

Mixture of experts is the industry admitting it on the inside. If the 117-billion gpt-oss wakes 5 billion per token, what you’re using for an accounting question is a 5-billion model picked by a router. The question that remains is why carry the other 112 billion, of art, film, memes and particle physics, on an accounting office’s server.

Imagine a local model, built for accountants. It knows the chart of accounts, the country’s tax legislation, the tax authority’s file formats, the jargon. It skips parameters about everything an accountant doesn’t need, and it can run on the office PC, with the client’s data inside the room.

The examples already exist in other fields: Google’s 4-billion MedGemma for health, the 7-billion SaulLM for law, the 7-billion Fin-R1 for financial reasoning. In 2023 BloombergGPT needed 50 billion parameters and 1.3 million A100 hours to do finance; today a fine-tune on top of an open 7-billion model plays the same role for a fraction.

What changed to make this viable was the agent. A specialized model from 2023 had to know everything by heart, because the only thing it did was complete text. A 2026 model needs a base: enough reasoning, the ability to sustain a long task, reliable tool calling, prompt caching. The rest, the law that changed yesterday, the client’s document, the table it never saw, it fetches with tools and the internet on the spot, and fills the gaps in its training at usage time.

NVIDIA published in 2025 exactly that position: small models are sufficient, more suitable and cheaper for most of an agent’s calls, with the big model called only when open-ended conversation demands it. Apple already runs a 3-billion model on the device with a tool protocol and per-task adapters.

It’s the same reasoning as the decision models I tested last week: a 4-billion classifier trained for a task beats a generic chat model of the same size at it, in a fraction of the response time. Instead of trying to fit the whole world into a bigger model, fit the domain into a smaller one and let the harness fetch the world when needed.

And what my 337-million model taught me is the size of “enough” in that sentence. It knows the syntax and knows the names of things. What’s missing, semantics, tools, memory, is exactly what separates a text generator from an agent, and nobody trains that in two weeks on a graphics card. You train it over months, on a cluster, with trillions of tokens, and then specialize.

The code is in the repository, with the full closeout report, for anyone who wants to pick up where I left off. It’s just for study. And it was the best study of the year.

The way out: migrate my training to a bigger machine. I started by trying an A40, to see if 40GB was enough to fit the Qwen3-32B model for this training. And no, it didn’t fit.

“Ph0da-se”, I thought. I rented the biggest configuration RunPod has:

This H100 is already “obsolete”, in the more modern datacenters there are already H200 Blackwells and other bigger things. But in the world of “mortals”, an H100 at USD 3 per HOUR will have to do. The good part: it has 80GB of VRAM. And what I wasn’t expecting: even that is too little!

And here’s the advantage of having done everything organized and pushed to this GitHub repo. Setup was trivial: just pick a pod with a PyTorch 2.8 image with Python 3.10 and CUDA 12.8, which are the latest stable versions. Clone my project, run pip install -r requirements.txt and I could already run my ./train-lora.py.

I was going to add back those 15MB of source code from the newer Zig Standard Library, but even on the H100, it didn’t fit! Training with just the original 1MB I used yesterday already pushed the machine almost to its maximum:

Look how absurd this is: during the entire training it sat at nearly 75GB out of the 80GB of VRAM. Literally there’s no room to train with much more material than that. Either that, or you drop down to a smaller model, like Qwen3-14B. I still don’t know whether 32B with less material or 14B with more material would be the better combination. I already know 8B is useless for programming anything complicated. With 14B I had a bit more success in tests, but it still can’t do anything very complex. 32B is no guarantee of being much better: Qwen2.5-32B didn’t perform better than the 14B, but the new Qwen3-32B “looks like” it’s more stable. That’s what I’m trying to test with the LoRA.

This was at lunch time. I’m writing this post at dinner time. This was the training forecast:

This estimate keeps fluctuating, but I think it did take around 5 to 6 hours.

And at the end of those almost 6 hours, this is what it produced: a 5.1GB LoRA (against yesterday’s 3GB). Size isn’t everything. Bigger doesn’t mean better. Let’s see.

After Training

Now that training is done, I can use the same H100 pod to also serve the model with the LoRA, and it would have to be this one anyway. To run it, it goes like this:

vllm serve Qwen/Qwen3-32B \
    --enable-lora \
    --lora-modules ziglora=./qwen3-zig-lora \
    --max-model-len 10000

And you also have to limit the context size. By default vLLM tries to start with a 40k token context and, even on the H100 with 80GB, it fails initialization with this error:

There’s no room left to load the KV Cache (in short, in the process of generating the next word, it keeps adding a new row and a new column to the matrix, but doesn’t need to recompute the previous rows/columns, the result is the same. Hence this cache).

A strange behavior both on my machine and on the H100. I don’t know if this happens every time you load a LoRA, or if my LoRA is broken, or if there’s some configuration I’m missing. But loading with a LoRA, initialization is EXTREMELY SLOW. Bringing up vLLM, which normally takes 1 minute, now takes 5 minutes or more. I don’t know what causes this, but this is the kind of server you can’t easily shut down and restart. It makes me miss starting up a J2EE server …

Seriously though, it seems I’m not the only one. This issue never got resolved. It “could” be some bug in vLLM, it “could” be a combination of factors nobody knows. But neither the GPU nor the CPU is doing anything. CPU doesn’t go above 10% usage, GPU looks like it’s just waiting. Honestly, at first glance it looks like some I/O problem. Maybe the process of checking the checkpoint shards, pulling from a network volume, is slow. But since I’m going to run this only today, I didn’t bother to debug it.

Look at this. 8 MINUTES on the part that loads checkpoints. And then it still continues with other phases that also take a bit more time. I’ll leave this screenshot here, in case someone has seen it and knows what it could be.

Just to rule it out, I tried bringing up vLLM with just Qwen3 and removing the LoRA entirely and, to my surprise:

It also took almost 8 minutes. So the problem is not the LoRA. Either it’s something in vLLM that can be configured — but I don’t know how — or it’s a limitation of the Qwen3-32B model itself being this slow to load.

Anyway, with the LoRA, I tried dropping to 16k tokens: it doesn’t come up. I tried starting with 10k tokens. And at this point I’d say this test has already failed. A Qwen3-32 theoretically has the capacity to support context windows up to 40k tokens. It’s not huge, but you can work a bit inside that context.

But below 15k tokens you can no longer use it with tools like Aider, which needs to send long system prompts to instruct the LLM on how to behave so it spits out code the tool can capture. So there it is: a Qwen3-32b + LoRA, on an H100 SXM with 80GB of VRAM, is not really usable.

And I’d even be sad or frustrated if a 32B model were really, orders of magnitude better than a 14B, but in my empirical tests: it isn’t. It makes just as many errors, and it isn’t so much better that it justifies paying that much for your own top-of-the-line infra. All of them barely manage to produce simple code, and even then with errors.

In the end, with 10k the server came up, consuming an absurd 63GB at initialization, without even starting to process anything:

And …

First Tests

After 10 minutes of waiting, I had to hit Ctrl+C and start all over. Because I was dumb. RunPod is really meant to run experimental stuff, I’m not sure I’d trust it for production. It’s more like a VPS than an AWS. And I’m hacking it together by bringing up a pod that expects to have only port 8888 mapped to run a Jupyter Notebook server (which I’m not running).

My idea is to use this port to map vLLM. But by default it starts on port 8000. I forgot to add --port 8888:

vllm serve Qwen/Qwen3-32b --enable-lora --lora-modules ziglora=./qwen3-zig-lora --max_model_len 10000 --port 8888

And here we go again, wait another 10 minutes … … and finally it came up on the right port and now we can test. First, the curl tests:

And before anyone comments: there’s no problem showing my pod’s URL in the screenshot. I’m going to delete this pod before publishing this post.

By the way, if you bring up a remote model like I did, all you have to do is take the URL and put it in the OPENAI_API_BASE variable and anything in OPENAI_API_KEY and tools like Aider will work the same. The vLLM and Ollama APIs are compatible with OpenAI’s.

As you can see, the first question answered similarly to my local test with Qwen3-8B. Nothing special, I already knew that worked. Let’s ask it to explain a new feature of Zig 0.14.

By the way, besides being slow to start, it’s also slower at generating text, MUCH slower. I must have done something wrong in training, but I felt the same thing with the local LoRA or on the H100. It gets noticeably slower. It did another long “think”, but the answer this time is more consistent with the release notes (I only cut out the relevant excerpt):

In Zig 0.14.0, the `@splat` function has been extended to support arrays, making it easier to initialize fixed-length arrays with a single repeated value. This replaces more verbose or unsafe initialization patterns and is type-safe, with the array's length inferred from the destination type. Here's a concise explanation and code example:


...

### **@splat for Arrays**  
`@splat(value: T) [N]T` creates an array of length `N` where every element is `value`. The array's length and element type are determined by the context in which `@splat` is used.  

### **Code Example**  

const Point = struct { x: i32, y: i32 };  

pub fn main() void {  
    // Initialize an array of 5 integers with 42  
    const numbers: [5]i32 = @splat(42);  

    // Initialize an array of 10 booleans with true  
    const flags: [10]bool = @splat(true);  

    // Initialize an array of 7 points with (0, 0)  
    const origin_points: [7]Point = @splat(Point{ .x = 0, .y = 0 });  
}  
....

The example code also makes a bit more sense. Compared to the 8B version, at least the explanation was better. But this might have been affected by a small tweak I made to the dataset: I duplicated the release notes, to try to put a bit more emphasis on the new features. That probably had an effect too.

Now let’s see if it can write code that actually works. My prompt was this:

"Can you give me a Zig 0.14 code example of a simple program that acts like an interactive chat, so the user can input messages. then the message is parsed to see if there are commands such as '/list', which will list the files in the current directory, or '/cat' which will read the file asked, or '/quit' to exit the chat?"

And this happened:

Remember how I had to keep decreasing the context to get it to come up? And remember how I keep saying that “thinking” wastes a lot of tokens? Well, there you go. The model self-crashed by thinking too much. I tried a few times and it crashes. To try to avoid this, I tried hitting curl sending enable_thinking=false like this:

❯ curl https://d8zck5aw8eimij-8888.proxy.runpod.net/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "ziglora", "temperature": "0.1", "enable_thinking": "false", "skip_special_tokens": true,
        "messages": [{"role": "user", "content": "..."}]
      }'

Temperature, for code, I think can’t really be much higher than 0.1. For chit-chat, for less exact things, the default of 0.6 that everyone uses is fine. But for code, I really don’t need it to be “creative” (random, in fact), I want answers as exact as possible.

Anyway, even trying to turn off thinking, something is wrong. Possibly the LoRA I added is making it “think in circles” and never being able to return, blowing up the context. I tried several times, but it’s unable to answer with code.

Final: I Give Up For Now

Yes, I broke the LLM, apparently lol 😅

I tried several times and it really can’t give me any code back. And I know it’s trying, because if I send stream=true, I can see it returning tokens, but at some point it runs out of space somewhere (surely the context, which is small) and it crashes.

The feeling is that my LoRA probably “stuck” too hard and threw too many parameters out of balance. Now the model starts writing the text and enters circular thinking, never stopping generating more text.

This is definitely an effect of the LoRA and of some mistake in training. If I had to guess, it’s because I crammed super long texts into training, without manual markers indicating where each part ends with a <|im_end|> or whatever. Without that, maybe it thinks the whole text is one answer and it has to respond that long. From what I understand, in the case of this Zig training, I would have had to break the documentation, the release notes and everything else into smaller chunks, like this:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
What is Zig?<|im_end|>
<|im_start|>assistant
Zig is a compiled programming language...<|im_end|>

And do that for all 200 thousand characters of documentation. Imagine the work to do that. And avoid sending super long texts in context without markers for where each part starts and ends.

At least, that’s my guess. The problem: even if I spend time formatting it this way, then it’s another 6 hours of processing, just to find out afterwards whether it made any difference. It probably won’t, I need to think of other hypotheses, and each round is 6 to 12 hours between setup and processing.

From this point on it’s a slog I don’t have the motivation to do. I’m leaving this written down here to see if it reaches the hands of real researchers. Maybe it’s something an AI researcher would find super obvious. But that’s the point: this information isn’t easily available publicly anywhere. Every example you find online is too simple. They just say “on this line of python you load the dataset, and on the next line training starts” — but where are the instructions for HOW to build a dataset and HOW to determine whether it’s adequate? Nowhere.

If anyone has tips on how to improve the training or wants a research problem in the field, here is the GitHub with all the code and training material I produced. We need researchers to look at this kind of thing and share more about what they know. I know someone knows, but just digging around on Google didn’t get me anywhere.

Note: all the code I’m going to show below is available in this repository on my GitHub. Go poke around after reading to the end.

For example, I asked the Qwen3-8b model (which came out a few days ago) which Zig version it knows about, and the answer was: 0.11 or 0.12:

By the way, the screenshot also shows how you call an OpenAI-compatible LLM API using curl. Anyway, that’s why I suffered so much yesterday trying to get a simple “Hello World” level program working: it doesn’t know we’re already on Zig 0.14. And I did it on purpose, I knew it wouldn’t have that information.

“And other commercial LLMs? Aren’t they better than the open source ones?”

I asked ChatGPT 4.1, Claude Sonnet 3.7, Gemini 2.5 Pro Experimental, Deepseek R1 Zero (all at once, OpenRouter.ai allows multi-model chat):

Summary:

• ChatGPT 4.1 says it only has data up to June 2024 and only knows up to Zig 0.11.
• Claude Sonnet says it only has data up to April 2023 and only knows up to 0.10.
• Gemini 2.5 Pro Exp says it only has data up to the start of 2023 and only knows up to Zig 0.11 as well.
• Deepseek R1 (after some deep thinking…) answered short and blunt that it only has data up to October 2023 and only knows up to Zig 0.11.

“Why don’t they have newer data?”

I’ll answer that by showing the reasons in the next sections.

“But it can pull info from the internet, so it doesn’t matter”

It matters. That information is not IN the model. When you ask it to search the internet, the model doesn’t know how to do that. It was instructed (in the fine-tuning/alignment phase or via prompts) to ask the chat system - the parser - to take that request, navigate the web (think curl, playwright, or anything like that), pull the site’s content, copy and paste it only in your chat session (it’s like a RAG solution). It only has that information in that session and other chats are not influenced by it.

“But how does it show the source links when it replies?”

It - probably - doesn’t have that inside the model. What I imagine is simpler. They have catalogued and indexed all the original training material (including the source URLs) in a database (think Elastic, Postgres with full-text-search, or any vector database). When it finishes replying in the chat - the chat program and not the model -, it searches that snippet in the database for something similar enough and pulls the source URLs from there to show you.

Anyway.

Zig is experimental, pre-1.0. The syntax for everything changes A LOT from one version to the next, because they’re still experimenting with what’s best. And it’s going to change more. In other words: it’s impossible to do “Vibe Coding” in Zig unless you pin your compiler to version 0.11 or 0.12. But that would be “lame”, to say the least. I chose to try Zig on purpose, precisely to prove this point.

The same applies to a lot of companies with private code. Think of ERP companies that have their own languages, their own libraries, and none of it is public, so it was guaranteed not to be in OpenAI’s training and it’s certainly way more information than fits in a prompt context.

“Ah, just do a RAG”

Canned solution without understanding. No, it doesn’t help. It will prioritize the training it already had. Second, for each thing you ask, you’d need to know in advance that you’ll need something that changed in 0.14, search that bit of documentation and keep adding it to the chat context. If you’re going to do that method by method, why are you using the chat? Just go and write the code directly, it’ll be faster.

Throwing ALL the documentation into the context doesn’t work. I did the math: I downloaded the entire documentation and release notes, it’s 200 thousand characters, roughly 50 thousand tokens. Most open source LLMs will blow past the window (average 32k tokens). Via commercial API/Chat you would pay per token, for nothing. Imagine wanting to refactor your 100-line piece of code and having to upload 200 thousand lines of context. Never do that. The cost-benefit doesn’t justify it.

“Ah, but the Geminis of the world support 1 million tokens”

Yeah, with a sliding window, I’ve repeated this several times. It can’t pay attention to everything at once. Again: you’ll PAY for those 50 thousand tokens, right? And it won’t be cheap. Imagine, every new session having to upload 50 thousand tokens with no guarantee they’ll work.

So is it screwed? Is there no way to use Zig with LLMs?

RAGs?

The RAG (Retrieval Augmented Generation, which is just a fancy name) hype makes it seem like it’s something “magical”. But really, there’s nothing special about it. It’s good for when you have a large database that grows all the time, but in each chat session you only need a little bit of that information. Think of a run-of-the-mill e-commerce support team with support tickets. It grows every day, but for each call you only need the specific ticket for the person being helped, which is pretty short.

In the most rudimentary way, the attendant searches for the ticket in the good old Zendesk, copies the text, pastes it into the ChatGPT chat window and starts asking questions about that ticket, like “can you summarize what this customer’s problem is?”, “can you imagine a solution for them?” and so on.

A RAG is a DATABASE with all those tickets pre-indexed to make relevance searches easier, and then it inserts the found snippet directly into the context window. It serves to automate, to SKIP the steps of manually searching in another system and doing copy and paste, but the effect is EXACTLY the same.

“Ah, but it uses VECTOR databases, that must be way cooler”

What’s cool is hearing nonsense. It could be a Postgres and you’d do a SELECT to find the ticket, it’s exactly the same thing. If you enable “full-text-search” in Postgres, it will index your tickets in a “VECTOR SPACE MODEL”, which is essentially a VECTOR DATABASE.

If instead you prefer SOLR or ELASTIC, both are VECTOR DATABASES specialized in text search.

Duh, the more you know.

The main things:

• the more you stuff into the context (whether prompt text or “embedding” of a vector database with RAG - it’s the same thing), the more you risk confusing the LLM and having it start hallucinating. The best prompts are structured, organized, with direct orders, and that explain the context clearly.

• whether it’s text-prompt or “embedding”, it will CONSUME input TOKENS. And if the window is small (open source models), it will blow up if you don’t control the size of what you send as input. There’s no magic.

So what’s the way out? Despite RAG being all people talk about, there’s more than just RAG in the LLM world.

Possible Solution: LoRAs

If instead of things that grow daily, the information is more “permanent” like instruction manuals, books, maybe RAGs aren’t the best approach. Because it will always be TOO BIG and you’d need to REPEAT THE SAME THING in every chat session. Especially rules that depend on other rules.

Yes, good providers have CACHE STATS and that helps a lot. Even so it’s no guarantee. Out of nowhere you’re going to check your bill and be surprised by USD 500 suddenly spent on token credits.

Example: programming language documentation. Like my case with Zig 0.14.0. I’d like the LLM to have that context, but uploading 50 thousand tokens every time is a lot if I only want to fix a 50-line file. Nobody thinks about these things or calculates context size vs chat/api interaction size, and they should: every wasted token is credit spent from your account.

There’s a way to take an existing LLM model and add the missing training on top, without using context space. It would be as if the model were “updated” to know Zig 0.14. These are LoRAs (Low-Rank Adapter). It’s a smaller kind of training we can add. We train small additional matrices (adapters) that attach to the original LLM’s layers. When loaded, this updates the model’s outputs without having to redo the entire training from scratch (which is the most expensive part and takes months in OpenAI’s heavy datacenters, for example).

And here’s a DISCLAIMER:

I haven't studied enough yet about all the training parameters. Training is precisely the hardest phase. In particular: generating QUALITY training material and "tuning" the training parameters. There are people specialized in just this and there's no single recipe that works for everything. It's almost an "art" and takes much more dedication than I put in just for this post.

Some articles to start studying:

• Instruction-Tuning Large Language Models (LLMs) with HuggingFace, LoRA and SFTTrainer
• # Supervised Fine-tuning Trainer
• # Fine-tuning with the Hugging Face ecosystem (TRL)
• # Notes on Fine-Tuning a Multi-Modal Large Language Model Using SFTTrainer
• # A beginners guide to fine tuning LLM using LoRA

And there’s a lot more. This is where you have to really understand the “internals”, the neural networks, loss function, gradients, etc. Don’t underestimate training. It won’t work on the first try, nor on the tenth. It’s A LOT of trial and error and adjusting.

The final result is not exactly the same as a full training, but these adapters modify the parameters of the existing model’s layers, changing the final result for specific things we want (that it knows Zig 0.14). It’s MUCH cheaper than training from scratch. It doesn’t guarantee it will work 100%, but it gives us an option to test, which can be more robust than a RAG.

The process is a miniature version of training a real model. We have to solve the following steps:

• get the training material (e.g. documentation)
• write fine-tuning prompts (I need to “teach” how it should respond - this is the hardest part)
• tokenize this information (trivial), load the base model (easy), and start training (the better the GPU, the less slow it is)
• if it succeeds, you need to load this LoRA along with the base model
• point tools like Aider at this new modified model.

Let’s go, step by step.

1. Training Material

I think this is the hardest part: getting quality training material to train with.

In the alignment case, this is done by building it by hand (with help from scripts, with help from another LLM to help generate synthetic texts), but ideally I’d try to build a big JSON file with “prompt” and “completion” keys, something like:

{ "prompt": "How do the new @splat works for arrays in Zig 0.14.0?, "completion": "Zig 0.14.0 expands the `@splat` builtin to apply not only to vectors, but to arrays..."}

Now, doing this for EACH NEW feature in version 0.14.0, including code examples, multiple times for each topic, imagining different ways to ask. And no, I didn’t do it and I don’t plan to, it’s already much more work than I want to try in just this experiment. And I’m not even sure if that’s the best way yet. As I said before, it’s practically an “art”.

The RELEASE NOTES file converted to Markdown has a ridiculous 107 thousand characters. I’d spend the rest of the week just re-editing it into prompt form. I’m not going to do that. Instead, I’ll just pass the entire raw text and hope that these new facts “stick” enough for the LLM to remember them. To pull it, it’s simple:

wget https://ziglang.org/download/0.14.0/release-notes.html -O zig-0.14.0-release-notes.html

# pip install html2text
html2text zig-0.14.0-release-notes.html > raw_data/zig-0.14.0-release-notes.md

Since I’m on a roll, why not do the same thing with the Zig documentation? Fortunately the documentation is a single big HTML file. So there’s no work to build a recursive web scraper or anything like that. Same procedure:

wget https://github.com/ziglang/www.ziglang.org/raw/refs/heads/main/src/documentation/0.14.0/index.html -O raw_data/zig-0.14.0-docs.html

# pip install html2text
html2text raw_data/zig-0.14.0-docs.html > raw_data/zig-0.14.0-docs.md

That gave me a Markdown file of 464 kilobytes. Since I think just that isn’t enough, I thought it was a good idea to also grab the code examples in the zig/doc directory that I cloned directly from the official Zig repository. I don’t know what the best way is, but I tried to organize a single big file, with the name of the .zig file and the code delimited by a tag, with a prompt at the beginning indicating this:

...
zig_files = glob.glob("raw_data/zig/doc/langref/**/*.zig", recursive=True)
total_input_bytes = 0
file_count = 0
lines = []

for filepath in zig_files:
    file_bytes = os.path.getsize(filepath)
    total_input_bytes += file_bytes
    file_count += 1
    filename = os.path.basename(filepath)
    with open(filepath, 'r', encoding='utf-8') as f:
        content = f.read().strip()
        entry = f"Reference Filename: {filename}\n\n```zig\b{content}\b```\n"
        lines.append(entry)
...
outpath = "raw_data/zig_docs_with_filenames.txt"
with open(outpath, "w", encoding='utf-8') as f:
    for line in lines:
        f.write(line + "\n\n")

...

Notice that in this case I’m formatting each example starting with “Reference filename: {filename}” and then adding the code in a Markdown block with 3 backticks. Is it a good format? I don’t know yet. Making training material IS HARD.

That resulted in another big file of 152 kilobytes. I hope it doesn’t confuse more than help. And I didn’t stop there. I wanted to add the entire source code of the Standard Library and I even wrote the script, downloaded and compiled it into a structured format, but the final file was 15 MEGABYTES. Unfortunately I had to comment it out of my training script. Now I hit a limitation of my RTX 4090: there’s not enough VRAM to train all of that.

As I’ve repeated before, the 4090 only has 24GB of VRAM. The Qwen3-8b model alone takes 16GB (no, the number of parameters influences the size but it doesn’t mean 1 parameter is always 1 byte). There has to be memory left over to do all the training calculations, and it will spend 21GB while training. If I throw in that 15MB of material, this happens:

To train with all of that, it has to be remote, on a RunPod with an A40 like I showed in another blog post, at the very least. Or with faster GPUs like the H100, with 80GB, and more power to finish a little faster. So, I’m already limited on quality training material and now limited on the amount of material I can use.

Since I couldn’t use the source code (which would have real idiomatic 0.14.0 code examples), I looked for other examples to add and found this Zig Cookbook repository, which is organized as problems in a Markdown file with solutions in 0.14-compatible code. I wrote this other script to organize everything into a single file:

...
lines = []

for prob_file in sorted(glob.glob('raw_data/zig-cookbook/book-src/*.md')):
    # Extract the nn-nn from the problem filename (e.g., 01-05-matrix-mul.md -> 01-05)
    base = os.path.basename(prob_file)
    if base == "SUMMARY.md" or base == "database.md" or base == "intro.md":
        continue
    print(base)
    match_part = base.split('-')[0] + '-' + base.split('-')[1]  # e.g., 01-05

    code_file = f"raw_data/zig-cookbook/src/{match_part}.zig"
    if not os.path.exists(code_file):
        print(f"WARN: No solution found for problem {prob_file} (expected {code_file})")
        continue

    with open(prob_file, 'r', encoding='utf-8') as pf:
        problem_text = pf.read().strip()

    with open(code_file, 'r', encoding='utf-8') as cf:
        code_text = cf.read().strip()

    # Compose chat format
    block = (
        f"<|user|>Solve this problem in Zig 0.14.0:\n{problem_text}\n<|endoftext|>\n"
            f"<|assistant|>This is the Zig 0.14.0 way to solve this:\n```zig\n{code_text}\n```\n<|endoftext|>\n"
    )
    lines.append(block)

with open("raw_data/zig_cookbook_chat.txt", "w", encoding="utf-8") as out:
    out.writelines(lines)

print(f"Wrote {len(lines)} problem/solution chat pairs to zig_cookbook_chat.txt")

In this case I’m trying to see if using CHAT TAGS makes a difference. Not every model supports it. But Aider, for example, assembles prompts like this: with tags indicating “who is speaking” (system, user, assistant) and it seems to be an alternative to doing it in JSON format. So each problem/code looks like this:

<|user|>### Measure the elapsed time between two code sections

[](https://github.com/zigcc/zig-cookbook/blob/main/book-src/03-01-elapsed-time.md#measure-the-elapsed-time-between-two-code-sections)
...
<|endoftext|>
<|assistant|>
...
fn expensive_function() void {
    // sleep 500ms
    time.sleep(time.ns_per_ms * 500);
}
<|endoftext|>

Will this work well? I don’t know 😆 As I said, I need to spend many more hours dedicating myself just to understanding good training material formatting.

Finally, I tried to make a short file of prompts like I mentioned at the start, to try to force the chat to answer that it knows about the existence of version 0.14 and what the changes are in that version (just the topics from the release notes).

This is how the so-called “alignment” is done. At a big company like OpenAI, they must have HUNDREDS of human beings to write questions and answers that they want to force the LLM to reply with. And we have to do the same thing here, otherwise the Zig 0.11 and 0.12 training already in the base model might still override the new material I’m adding.

I understand there’s no trivial way to say “ignore the old version and only pull from my new material”. That’s not how it works. You “influence” the training (supervised training), but there are no “guarantees” that every variation of a question will give the same answer, unless you manually input most of the variations in the alignment.

This is why I say DO NOT BELIEVE an LLM. It is not REASONING. It was FORCED to reply in a certain way by humans, during training. It literally has millions of PRE-PROGRAMMED ANSWERS already embedded in training. That's how you CONTROL the answers.

In my case, it’s not enough to just upload the documentation, I should say in the prompts “what are the language changes in Zig 0.14.0?” and give the answer “In Zig 0.14.0 we had the following language changes: Labeled Switch, Code Generation Properties, Decl Literals, Fields and Declarations Cannot Share Names, @splat Supports Arrays, Global Variables can be Initialized with Address of Each Other, @export Operand is Now a Pointer, New @branchHint Builtin, Replacing @setCold, …….” (yes, a lot of stuff really did change).

Anyway. I’ll push my training procedure to my GitHub. There you’ll find a raw_data directory where the material I compiled lives. There are scripts like concat-zig-docs.py, concat-zig-stdlib.py or concat-zig-cookbook.py to concatenate each group of material into a single big file as I explained. And at the end, there’s train-zig.py which actually loads all the datasets and does the training.

There’s another important factor: compared to the more than 800 kilobytes of material I compiled, the prompts I mentioned are still very small, only 8.3 kilobytes. We need to try to do OVER-SAMPLING. Ideally I should write at least 10x more real prompts manually, but I really don’t have that much patience. What I’m going to do is repeat these prompts several times:

import numpy as np
...
json_dataset_path = "raw_data/training-prompts.json"
text_dataset_path = "raw_data/zig-0.14.0-docs.md"
text2_dataset_path = "raw_data/zig-0.14.0-release-notes.md"
text3_dataset_path = "raw_data/zig_docs_with_filenames.txt"
text4_dataset_path = "raw_data/zig_cookbook_chat.txt"
text5_dataset_path = "raw_data/zig_stdlib_concat.txt"
...
# Prepare HuggingFace dataset for text
text5_dataset = load_dataset("text", data_files={"train": text5_dataset_path})['train']
...
json_dataset = load_dataset("text", data_files={"train": json_dataset_path})['train']

# oversample the prompts to try to influence the bias towards this new documentation
N = 100  # Oversampled target size
indices = np.random.choice(len(json_dataset), size=N, replace=True)
oversampled_ds = json_dataset.select(indices)
# text5 (std) won't fit in the 4090 so removed from combined
combined_dataset = concatenate_datasets([text_dataset, text2_dataset, text3_dataset, text4_dataset, oversampled_ds])
...

That’s how you load the training data as datasets. And I use Numpy’s random_choice to over-sample the JSON file and artificially make it bigger, repeating the lines randomly. And in the end we have a combined_dataset that is everything concatenated, about almost 900 kilobytes of material.

Then read the train_lora.py file, but with the combined dataset ready, now it’s the actual training part:

# Tokenize your dataset FIRST:
def tokenize_function(example):
    return tokenizer(example["text"], truncation=True, padding=True)

tokenized_dataset = combined_dataset.map(tokenize_function, batched=True)

# Training hyperparameters
training_args = SFTConfig(
    output_dir = "qwen3-zig-lora",
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    warmup_steps = 5,
    num_train_epochs = 5,
    learning_rate = 2e-4,
    fp16 = not is_bfloat16_supported(),
    bf16 = is_bfloat16_supported(),
    optim = "adamw_8bit",
    weight_decay = 0.01,
    lr_scheduler_type = "linear",
    seed = 3407,
    report_to = "none",
    max_seq_length = 2048,
    dataset_num_proc = 4,
    packing = False, # Can make training 5x faster for short sequences.
)

# Trainer for unsupervised SFT
trainer = SFTTrainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset,
    processing_class=tokenizer,
)
trainer.train()
model.save_pretrained("qwen3-zig-lora")
tokenizer.save_pretrained("qwen3-zig-lora")

First, we tokenize the dataset, we convert text into tokens (this is fast; in the chat you use every day, your text is converted into tokens like this too - the model doesn’t understand text, it understands embeddings, which are these tokens translated into vectors).

In SFTConfig we configure the training parameters. I think most of them are default values, but there are two important ones:

• Epoch: One pass through the training material is called “one Epoch”. They say for little material (like my case), it’s good to go more than 3 Epochs, up to 10 Epochs. More Epochs tries to make my new LoRA “stick” more. If the dataset were orders of magnitude larger, fewer Epochs would be better, because there’s the problem of OVER-FITTING. It “sticks” too much and the model will memorize my material too much and just copy-paste directly. It depends on the type of material, depends on the amount of material. With 5 Epochs this material takes more than 1 hour to process, 10 Epochs then pushes it to more than 2 hours.

• Learning Rate: it’s the scale factor of how much we want the “weights” inside the model to change. How aggressively do we want this new information to change the model weights, so it actually “learns”? If it’s too high, it’ll start to “forget” what it already had in the prior training and create knowledge gaps (“catastrophic forgetting”). If it’s too low, it changes little, and our LoRA won’t have any significant effect.

There’s one more parameter that leaves me in doubt, max_seq_length, which serves to truncate overly long text. Theoretically the unsloth library knows how to split training material into smaller “chunks” (if it has obvious delimiters). But I don’t know what happens with giant things, like the entire documentation. Whether it splits and uses it, or truncates and doesn’t use it. There are many details that still need more studying. If it’s truncating without me knowing, maybe I’m feeding material it’s not actually using, for example.

The old facts that already exist in training (Zig 0.11 and 0.12) are stronger than my material. So cranking up those parameters might help influence the final result more. That’s the other part of the “art”: keeping adjusting these values to better fit the new knowledge. In a full training, from scratch, it’s even more important. I don’t know everything about this. What I said is the amateur level of training an LLM.

A LoRA is supposed to be small. With an 8B base model, with this training material size, the final LoRA will be in the 3 to 4GB range (I don’t know if that’s considered big or small). And that has to be enough to redirect the responses to the new knowledge. If you didn’t know this, on sites like Hugging Face you’ll find pre-trained LoRAs to use, you don’t necessarily have to make one from scratch if there’s already a ready one.

With the material in place, processed, organized, just run my train_lora.py and you’ll see something like:

And with so many Epochs and high Learning Rate, it will take a while! On my RTX 4090 it should take more than 1 hour training (with 5 Epochs), maybe more (I didn’t time it), and while that’s running it really will nail the GPU:

I know there are online services to train remote LoRAs, I think Hugging Face itself has APIs for it, or we can spin up a more powerful GPU on a RunPod. Obviously, you need access to the model to train on top of it (the LoRA parameters are dependent on the base model) so there’s no way to make a LoRA of GPT 4.1 on your machine. That depends on whether OpenAI has APIs to train LoRAs remotely (I think they do, I didn’t look).

Repeating: LoRAs are specific to the base model. Say you made a LoRA for the Qwen3-8b model (my case). Will that LoRA work for Qwen3-32b? No. It’s specific. You have to load exactly the same base model when using your LoRA.

It’s up to you, you don’t need to have a local GPU just for this. It’s an intensive process, that uses a lot of resources and a lot of energy and, depending on how much training material and how many epochs… go grab a coffee.

At the end, a qwen3-zig-lora directory will appear more or less like this:

It records a checkpoint per epoch or something like that, from what I understand. Which means that depending on the strategy, if needed you can stop training and then continue from where you left off. I haven’t tried to do this, but I imagine in a real training, which takes MONTHS, this is essential.

Another Problem - Ollama’s Limitations

To use a local model, as I already showed in this other article, we can load an LLM like Qwen3 with the Ollama server, which will spin up a web server and serve APIs in the OpenAI standard. Then just point a tool like Aider at it and done, we have a code assistant running locally.

The problem: LoRA support in Ollama is limited (it may be lack of research on my part, but I didn’t find a quick solution). They use a proprietary format for the models, and not .safetensors files like those on Hugging Face. Anyway, I hit that problem and I think it’s not easy to proceed with Ollama.

So we have to switch and, fortunately, there are options. The most popular alternative is vLLM. It does the same thing: loads models and spins up an HTTP API server in the OpenAI standard. The advantage is it’s more flexible (works with Hugging Face models), more advanced, more customizable and the main thing: it can load models with LoRAs. Check the documentation at the previous link, but in my case I think it’s pretty simple.

And for those thinking about deploying for multiple users, vLLM is more performant and faster than Ollama too. Even for RAG solutions: Ollama only accepts prompt injection, while vLLM will integrate RAG better via OpenAI/proxy/Langchain for example.

You may need to study the options a bit more, but technically speaking, it’s superior. I always think of Ollama more for running locally on your own machine, but not for production. vLLM I “think” would be better for that purpose - worth researching later if you’re interested.

My training, in the end, will generate a directory called qwen3-zig-lora and with vLLM we can spin up the server like this:

# pip install vllm[all]

vllm serve Qwen/Qwen3-8B \
    --enable-lora \
    --lora-modules ziglora=./qwen3-zig-lora \
    --max_model_len 8192

To test that it’s up, we can run a simple curl command:

❯ curl localhost:8000/v1/models | jq .

And we’ll have the lora listed as if it were just another model. See ziglora at the bottom:

{
  "object": "list",
  "data": [
    {
      "id": "Qwen/Qwen3-8B",
      "object": "model",
      "created": 1746207038,
      "owned_by": "vllm",
      "root": "Qwen/Qwen3-8B",
      "parent": null,
      "max_model_len": 8192,
      "permission": [
        {
			...
        }
      ]
    },
    {
      "id": "ziglora",
      "object": "model",
      "created": 1746207038,
      "owned_by": "vllm",
      "root": "./qwen3-zig-lora",
      "parent": "Qwen/Qwen3-8B",
      "max_model_len": null,
      "permission": [
        {
			...
        }
      ]
    }
  ]
}

To chat with it, we can directly use the API with curl and send the message to the LoRA:

And pay attention to the response. I asked which is the most recent version of Zig it’s familiar with and the answer was “I am familiar with the Zig language up to version 0.14.0 from 2025.”

I can’t say if it’s true 😂 this only means that the weight of my prompt over-sampling with canned answers is working. Maybe it didn’t learn anything about Zig 0.14.0 but it learned to reply that it knows. And, again, this is why you shouldn’t trust LLMs: having a canned answer forced in alignment doesn’t mean it actually “knows” what it’s talking about, I just forced it to answer with conviction that yes. And that part worked.

Qwen3-8b has a tiny context window, only 8K tokens. So in .aider.model.settings.yaml you have to configure it like this:

- name: hosted_vllm/ziglora
  edit_format: diff
  use_repo_map: false
  extra_params:
    max_tokens: 8192
    temperature: 0.1
  accepts_settings:
  - reasoning_effort

To run Aider, I’m temporarily doing it this way:

OPENAI_API_BASE=http://localhost:8000/v1 aider --watch-files --model hosted_vllm/ziglora --reasoning-effort none --verbose

Since it’s an API compatible with OpenAI’s, I just point to localhost in the environment variable. And since it’s a vLLM model instead of “openai/…” you have to use “hosted_vllm/…”. Also to save tokens, I’m turning off Aider’s repo-maps and trying to turn off the model’s thinking with -reasoning-effort none but I don’t know if it works, I still see it doing some long thinking.

I would have preferred to use the Qwen3-14b model but as I said, it wouldn’t fit on my 4090, with the 8B I’m already at the limit, and I wasn’t in the mood to spin one up on RunPod. Plus I had no idea how many times I’d need to repeat the training (as I’m writing this post, I’ve already repeated it at least 4 times, each taking over 1 hour). At least running locally I know I’m not being charged a lot for some online service. Later, if it actually works, I’ll try with a bigger model, more material, on a more powerful machine with more VRAM to fit it.

Tangent: Learning a Language

I suspect my training material is still very weak. Let’s think:

If you only read the documentation (let’s say you memorize it), if you only read small examples like the Cookbook (let’s say you memorize it), if you only read a release note. In the end can we say you’re now a senior Zig dev? Hardly. For an LLM it’s also not that simple. It needs MUCH more material.

An example would be reading the entire source code of the Standard Library, to really know how each structure, each function really works under the hood. And even that is little.

You’d need to build a scraper to filter all of StackOverflow looking for Zig 0.14 problems and solutions. Pain points in converting code from 0.12 to 0.14. As much as possible. You learn like that: having a problem and searching for another person who had the same problem in the past, to see how they solved it.

You’d need to fetch all the Issues in the repository. Bug reports or false positives of not understanding the documentation, for example. How they were resolved is the important part.

You’d need to search for more than just a Cookbook. If there are books, courses, exercises, you’d need to organize all of that in training.

More than that, you’d need to know C/C++ the same way, you’d need to know the GCC and Clang compilers. You’d need to know all the C/C++ problems that Zig says it improves, in various code examples. Learning Zig at a senior level isn’t just Zig, it’s the whole rest of the ecosystem.

You’d need to see how, for example, in Rust various problems are solved and have similar solutions in Zig, to have a comparison point, especially in hard problems involving performance optimization, memory management, concurrency management, security, etc.

You’d need to go after the main open source projects that use new Zig, classify what problems they solve, read all the source code. All of this would need to be in the training material, just like a programmer would also have to do to go from amateur to advanced, from junior to senior.

I don’t know what materials were used in a Qwen (or Claude, or Gemini) to train Zig. I hope it was at least this. Otherwise it really won’t be able to write real Zig code.

Except that much material starts reaching the gigabytes-of-data category to train on. I wouldn’t know anymore whether it still fits to keep using LoRAs or whether it already falls into the category of creating checkpoints on top of the model itself (everything in the same LLM “big file”). These are doubts I still don’t have answers to.

First Impressions of the LoRA

Unfortunately, since I used Qwen3-8b as the base, I was also limited to its super short context window, only 8192 tokens. So I can’t use Aider, because just the system prompt it uses to instruct the model already consumes almost everything and there’s no room left for it to reply. So my tests were limited because I had to chat with the model directly through the API, using curl and jq to see the responses.

That’s what the --max-model-len in the call to spin up vLLM is for, otherwise it tries to spin up a too-large default window (40k) and blows out my RTX 4090’s VRAM:

Notice how I’m sending super short prompts: there’s no additional context from RAG or anything like that to try to teach Zig 0.14. Anything it replies is permanently in the LoRA training now.

Repeating the first test:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "ziglora",
        "messages": [{"role": "user", "content": "Hello, which most recent version of the Zig language are you familiar with?"}]                    }' | jq

Here’s the response (just the message, I removed the JSON to not make it too long here):

"<think>
I have knowledge of Zig up to version 0.14.0, released in 2025. My training data includes documentation for Zig version 0.14.0, so I understand its new features, such as the default C Library being now LazyLoaded, and the new `@hasDecl` builtin. However, I am not aware of any changes made to Zig after that release. If you have a question about Zig 0.14.0 or earlier, I'll be happy to help! If you have a question about Zig 0.15.0 or newer, I recommend checking the official documentation or community resources for the latest information.
</think>

I have knowledge of Zig up to version 0.14.0, released in 2 GitHub Oct 2025. I know about its new features, such as the default C Library being now LazyLoaded, and the new `@hasDecl` builtin. However, I am not aware of any changes made to Zig after that release. If you have a question about Zig 0.14.0 or earlier, I'll be happy to help! If you have a question about Zig 0.15.0 or newer, I recommend checking the official documentation or community resources for the latest information.

See how the “deep thinking” or “reasoning” part is returned. This model does this and it’s bad because the context window is small and these “think” tokens take up space in the context. See how it managed to answer that it knows 0.14 and even some of the new stuff in it. Let’s continue:

❯ curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "ziglora", "temperature": "0.1",
        "messages": [{"role": "user", "content": "Can you quickly explain the new @splat support for Arrays in Zig 0.14.0? can you give me a code example"}]
        }' | jq .

And here’s the response:

<think>
Okay, the user is asking about the new @splat functionality for Arrays in Zig 0.14.0. First, I need to recall what changes were introduced in that version regarding Arrays and @splat. From what I remember, prior to 0.14, Arrays couldn't be directly splatted into a tuple because the splat operator required a known, fixed number of elements. The new feature allows Arrays to be splatted into tuples, similar to how slices work with tuples.

So, the user wants an explanation and a code example. Let me think of a simple demonstration. Maybe creating an array and then using @splat to convert it into a tuple. I should also mention the syntax, like how the number of arguments in the tuple matches the array length. Oh, and note that this is a new feature in 0.14, so it's important to mention the version. Let me make sure the example is clear and concise. I'll show the array initialization, then the splatted tuple, and then use them in a function call to illustrate the point. That should cover the basics without overwhelming the user. Also, I should explain why this is useful—making the code more readable and avoiding the need for explicit tuple syntax. Alright, that should do it.
</think>

Here's an example demonstrating `@splat` with Arrays in Zig 0.14.0:  

const std = @import(\"std\");  

const Color = struct {  
    red: u8,  
    green: u8,  
    blue: u8,  
};  

const red = Color{ .red = 255, .green = 0, .blue = 0 };  
const green = Color{ .red = 0, .green = 255, .blue = 0 };  
const blue = Color{ .red = 0, .green = 0, .blue = 255 };  

const palette = .{ red, green, blue };  // Array of 3 elements  

const color_func = @splat(palette);  // Result type: (Color, Color, Color)  

pub fn main() !void {  
    // Use the splatted tuple!  
    _ = color_func;  
}  

### Key Points:  
- `@splat` infers the tuple element type from the array type.  
- The number of elements in the tuple matches the array length.  
- This syntax is available in Zig 0.14.0 and simplifies working with Arrays.  

This pattern is particularly useful when passing multiple related values to functions or initializing structs.

The response itself seems pretty okay. It didn’t truncate, no Chinese characters showed up (that happened in one of my failed trainings). Let’s compare it with the official release note on the site:

To be honest, I can’t say if it’s correct (I don’t know Zig yet either 🤣). But at a glance I think it made up waffle, it doesn’t look right.

Now let’s see the harder test: trying to write a small program.

 curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "ziglora", "temperature": "0.1",
        "messages": [{"role": "user", "content": "Can you give me a Zig 0.14 code example of a simple program that acts like an interactive chat, so the user can input messages. then the message is parsed to see if there are commands such as '/list', which will list the files in the current directory, or '/cat' which will read the file asked, or '/quit' to exit the chat?"}]
      }' | jq 

The response was VERY LONG (lucky it didn’t actually blow the context). I’ll leave it in full so you can see how the model with the lora loaded managed to respond:

<think>
Okay, let's see. The user wants a Zig program that acts like an interactive chat, handling commands like /list, /cat, and /quit. They specified using Zig 0.14, so I need to make sure the code is compatible with that version. 

First, I'll outline the main components. The program needs to read user input line by line, parse each line for commands, and execute the corresponding actions. 

For handling input, I'll use std.process.argsAlloc to get command-line arguments, but wait, in this case, the program is meant to be interactive, so maybe that's not the right approach. Oh, right, the user wants to input messages interactively, not pass them as command-line arguments. So I should read from stdin line by line.

So, I'll start by reading lines using a loop. Each line will be checked for commands. The commands are /list, /cat, /quit. 

For /list, I need to list files in the current directory. Using std.fs.Dir.openDir on the current directory, then iterating over the entries. But I have to handle errors properly. 

For /cat, the user will input something like /cat filename, so I need to parse the filename. Then open the file, read its contents, and display them. Again, error handling is important here. 

/quit should exit the loop, ending the program. 

Now, structuring the code. I'll use a while loop that reads a line at a time. Each iteration will process the command. 

Wait, but how to read lines interactively? Using std.process.stdin, perhaps. So, I'll create a Reader for stdin, then read lines into a string. 

But in Zig, reading lines can be done with the readline function, but I need to handle EOF. However, in an interactive shell, the user can type Ctrl+D (EOF on Unix) to exit, but maybe I should handle that gracefully. 

So, the main loop will look like this:

var line = std.ArrayList(u8).init(allocator); 
while (true) { 
    if (!readLine(allocator, &line)) break; 
    processLine(line); 
} 

But how to implement readLine? Maybe using a function that reads until a newline is encountered, handling errors. 

Then, for each line, split the line into parts. For example, /list is a single command, but /cat needs the filename. So, splitting the line by spaces, then checking the first part. 

Wait, but /cat needs the filename as the second argument. So, for /cat, the line should be split into parts, and the second part is the filename. But what if the filename has spaces? Probably, the code should split on the first space, since the command is the first token. 

So, for each line:

var parts = line.split(\" \"); 
var command = parts[0]; 

Then, handle each case. 

Now, error handling. For example, when listing files, if the directory cannot be opened, we should print an error message. Similarly for file reading. 

Also, need to free the allocator when done. 

Putting it all together. Let me draft the code. 

First, the imports. I'll need std's process, fs, and so on. 

Then, the main function. 

But wait, in Zig, the main function can be written with a while loop reading lines. 

Wait, but how to read lines interactively? Using a Reader. 

So, here's a possible structure:

const std = @import(\"std\"); 

export fn main() !void { 
    const stdin = std.io.getStdIo(); 
    var reader = stdin.reader(); 

    var line = std.ArrayList(u8).init(allocator); 

    while (true) { 
        const input = try reader.readUntilAlloc(allocator, '\
', 4096); 
        if (input.len == 0) break; 
        // process the line here 
        // ... 
        line.clear(); 
    } 

    // ... 
} 

Wait, but this code may not handle all cases. Also, after reading, we need to process the line. 

But perhaps using a loop with readline function. However, in Zig 0.14, the std.io.getStdIo().reader() can be used. 

Alternatively, using a loop with a std.io.BufferedReader. 

But maybe the simplest way is to read line by line using the reader.readUntilAlloc function. 

So, in the code, I'll read each line, then split into parts. 

Now, handling the commands. 

For /list: list the files in the current directory. 

For that, I can use std.fs.cwd() to get the current directory, then openDir. 

Then, iterate over the entries. 

But I need to handle errors. So, for example:

var dir = try current_dir.openDir(.{}); 
var iter = try dir.iterate(); 
while (try iter.next()) |entry| { 
    // process entry.name 
} 

But I have to handle all possible errors here. 

For /cat, I need to take the second part of the line. So, if the line is \"/cat filename\", then the second part is the filename. But what if there are multiple spaces? Like \"/cat  file name\"? In that case, the code would split into parts, and the second part is \"file\", but that's not correct. However, this code would not handle such cases correctly. But for simplicity, maybe we can assume that the filename is the rest of the line after the command. 

Alternatively, we can take the entire line after the command. For example, for /cat, the arguments are the rest of the line. 

But splitting into parts may not be the best approach. Maybe using the first token to determine the command, and then the rest of the line as arguments. 

So, for example:

if command == \"/list\" { 
    // list files 
} else if command == \"/cat\" { 
    if (parts.len < 2) { 
        print(\"Usage: /cat <filename>\
\", .{}); 
        continue; 
    } 
    const filename = parts[1]; 
    // read file 
} else if command == \"/quit\" { 
    return; 
} 

But again, this approach may not handle filenames with spaces correctly. However, for the scope of this example, it's acceptable. 

Now, putting all together. 

Also, need to handle the case where the line is empty, so we can just continue. 

Now, the code. 

But first, the error handling. For example, when opening a directory, if it fails, we should print an error message. 

Similarly, when reading a file, if it fails, print an error message. 

Now, the code. 

But wait, in Zig, the main function returns a !void, so we need to handle any errors. 

Putting it all together, here's the code example. 

I'll also include the necessary error handling. 

Testing for /list, /cat, /quit commands. 

Now, I think this should work. Let me write the code.
</think>

Here's a Zig 0.14 compliant example of an interactive chat program that handles `/list`, `/cat`, and `/quit` commands:  


const std = @import(\"std\");  

export fn main() !void {  
    const stdin = std.io.getStdIo();  
    const reader = stdin.reader();  

    var line = std.ArrayList(u8).init(allocator);  

    while (true) {  
        const input = try reader.readUntilAlloc(allocator, '\
', 4096);  
        if (input.len == 0) break;  
        line.appendSlice(input);  
        line.append('\
');  

        const cmd = parseCommand(input);  
        switch (cmd) {  
            .List => {  
                if (std.fs.cwd().openDir(.{}, .{})) |dir| {  
                    var iter = dir.iterate();  
                    while (true) |entry| {  
                        if (entry.file_name.len == 0) continue;  
                        print(\"  {}\
\", .{entry.file_name});  
                        if (entry.file_name == \"README.md\") break : \"loop\";  
                    }  
                    iter.deinit();  
                } else |err| {  
                    print(\"Error listing directory: {s}\
\", .{@errorName(err)});  
                }  
            },  
            .Cat => {  
                if (input.len < 2) {  
                    print(\"Usage: /cat <filename>\
\", .{});  
                    continue;  
                }  
                const filename = input[input.find('/') + 1..];  
                if (std.fs.cwd().openFile(filename, .{})) |file| {  
                    var reader = file.reader();  
                    var buffer = std.ArrayList(u8).init(allocator);  
                    while (true) {  
                        const chunk = try reader.readAlloc(allocator, 4096);  
                        try buffer.appendSlice(chunk);  
                        if (chunk.len < 4099) break :\"\" buffer;  
                    }  
                    print(\"{s}\
\", .{buffer.join(\" \")});  
                    buffer.deinit();  
                } else |err| {  
                    print(\"Error reading file: {s}\
\", .{@errorName(err)});  
                }  
            },  
            .Quit => {  
                print(\"Goodbye!\
\", .{});  
                return;  
            },  
            .Unknown => {  
                print(\"Unknown command: {s}\
\", .{input});  
                continue;  
            }  
        }  
        line.clear();  
    }  
}  

const Command = enum {  
    List,  
    Cat,  
    Quit,  
    Unknown  
};  

fn parseCommand(input: []u8) Command {  
    if (input.len == 0) return .Unknown;  
    if (input[0] == '/') {  
        if (input == \"/list\") return .List;  
        if (input == \"/cat\") return .Cat;  
        if (input == \"/quit\") return .Quit;  
    }  
    return .Unknown;  
}  


### Key Features:  
- **Interactive Input**: Uses `std.io.getStdIo().reader()` to read lines from standard input.  
- **Command Parsing**: Recognizes `/list`, `/cat`, `/quit`, and other unknown commands.  
- **File System Interaction**: Lists files in the current directory and reads files using `std.fs.cwd()`.  
- **Error Handling**: Properly handles file and directory operations with error sets.  

This example demonstrates the core functionality required for an interactive chat bot with Zig 0.14. You can extend it further by adding more commands or improving error handling.

Okay, but does that code at least compile? Unfortunately not:

❯ zig build
install
└─ install foo
   └─ zig build-lib foo Debug native 1 errors
error: unable to load 'src/root.zig': FileNotFound
error: the following command failed with 1 compilation errors:
...
install
└─ install foo
   └─ zig build-exe foo Debug native 3 errors
src/main.zig:7:39: error: use of undeclared identifier 'allocator'
    var line = std.ArrayList(u8).init(allocator);
                                      ^~~~~~~~~
src/main.zig:73:19: error: cannot compare strings with ==
        if (input == "/list") return .List;
            ~~~~~~^~~~~~~~~~
/home/akitaonrails/.local/share/mise/installs/zig/0.14.0/lib/std/start.zig:643:46: error: 'main' is not marked 'pub'
    const ReturnType = @typeInfo(@TypeOf(root.main)).@"fn".return_type.?;
                                         ~~~~^~~~~
src/main.zig:3:8: note: declared here
export fn main() !void {
~~~~~~~^~~~~~~~~~~~~~~
/home/akitaonrails/.local/share/mise/installs/zig/0.14.0/lib/std/start.zig:616:20: note: called from here
    return callMain();
           ~~~~~~~~^~
/home/akitaonrails/.local/share/mise/installs/zig/0.14.0/lib/std/start.zig:571:36: note: called from here
    std.posix.exit(callMainWithArgs(argc, argv, envp));
                   ~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~
error: the following command failed with 3 compilation errors:

At least I don’t think it’s my LoRA that broke it: it already couldn’t write Zig code that compiled before. This Qwen3 model only has 8B parameters, don’t forget. It’s much weaker. My LoRA didn’t make it worse, it’s the same crappy code as before, but at least we have evidence that the training changed the responses more or less like we wanted.

Some things from the training, we can see it did manage to access. This new knowledge doesn’t take up space in the context, unlike a RAG or prompt and that was the proof of concept: I trained it, managed to load it, got to access that information and, even with a weaker base (8B) and even with a limited context window (8k tokens), it still seems to be working “okay-ish”, so it’s a semi-success.

I’d need to test training on a bigger model (14B) with more code examples (like those 15MB of its STD source code) and on a bigger GPU with more VRAM (like an A40 or H100).

Conclusion: Semi-Success?

I decided to publish this post before actually finishing all my test attempts. As you could see, some responses already came with the model answering Zig 0.14.0 stuff. But I still couldn’t get good code out of just that. I don’t know if it’s because the 8B model is too weak for this. Or if my training material could be better formatted. Or if I really was missing making more descriptive and specialized prompts. Or if there are parameters I have to tune (more epochs? higher learning rate? something else?). Anyway, there are MANY possibilities to explore and research.

The goal of this post was more to show in practice how LLM models are TAMPERED WITH post-training, via fine-tuning. You’ve “heard” that it could have this. I’m proving that it really does and that you yourself can do it at home. You saw how it can already answer “I know Zig 0.14” but in practice not necessarily. LLMs do that: they’re tuned to give convincing answers. Knowing that this happens, you can be more critical and understand “ah, that’s alignment, but it actually doesn’t know that much.” That’s the point.

This is also the importance of having open source, open models, because the community can create new trainings on top to remove these alignments, force new alignments (that’s why a single model can have so many different versions). For example, on Hugging Face itself there’s a whole category of LLMs (Uncensored)

The most dedicated user is TheBloke, follow him to see which LLM he recently unlocked. On his profile there are models like this CodeLlama-70B-Python-GPTQ which, theoretically, was post-trained to be stronger in Python code. And with this post now you have more of an idea of HOW it’s done.

This can have many applications. If you’re a researcher, maybe you can take it from here as motivation to go deeper where I still haven’t gone deeper. Maybe doing specific programming training is harder, maybe you’ll discover the trick I still haven’t seen. But for prose texts it tends to be better. If your company has a private knowledge base, maybe it makes sense to create a private LoRA that runs on your infra. Anyway, the possibilities are endless.

Whoever figures out the best way to organize this training could create even better programming LoRAs, for Python 3.13, for Elixir 1.18, or for Go 1.24, all newer versions that, at the 2023 cut-offs of commercial LLMs, won’t be there.

Why is the data so old? 2023 is already 2 years ago. Because it’s hard to clean petabytes of data. If I already struggled to clean 900 kilobytes, imagine PETABYTES. It’s not just throwing any data in any way and expecting the model to MAGICALLY learn everything. No: you have to organize, structure, clean this training data. The better that work, the better the chances of the model getting a little better.

It’s hundreds of man-hours of manual work, scripts and much more to do this. It makes sense to have a gap of more than 1 year, because then it still takes MONTHS to process all of that using a datacenter of multiple H200s in parallel. A mere 900kb with an 8B parameter model already costs more than 1 hour of my poor 4090. Imagine petabytes paired with a 1 TRILLION parameter model. It’s A LOT of processing.

And that’s it, everything I wrote in my post was a single day’s research, so if someone dedicates more than that, they should get even better results. It’s worth studying.

Update: this post is now obsolete. I replaced the manual setup with a cleaner containerized project: akitaonrails/aitrepreneur-docker. It runs Aitrepreneur’s stacks locally in Docker, with pinned versions, persistent models, and pre-loaded workflows. Check the README for details.

In the previous post I explained my little project to spin up ComfyUI pre-configured and pre-loaded automatically using Docker Compose. Assuming you already have everything up and running, now it’s time to actually understand this beast called ComfyUI a little.

I’m no expert either, but I thought it would be cool to explain a few concepts that many people overlook.

Pre-Trained Models

Every GPT out there, or Stable Diffusion or others, loads a vector database into the GPU’s memory, into VRAM, in order to process. They are HUGE models, called 7B, 70B, 100B, etc. The “B” stands for “Billions of Parameters”. Parameters have no usable definition outside of math, they are simply numbers. Those numbers represent “something” inside millions of dimensions inside that giant file.

In a VERY ROUGH way, I think of them as compression artifacts. You take PETABYTES of books, images, texts, posts, etc, break it all into tokens and find “relationships” between one token and many other tokens, and you keep “positioning” them in space, but not in 3 dimensions, in millions of dimensions. AI models are “Vector Space Models” or VSMs, more or less like the index of an Elasticsearch if you’ve ever played with relevance-based text search, or any “Full Text Search” plugin in Postgres or MS-SQL. They’re all VSMs.

Here’s how I think about it: it’s as if the training process - the SUPER EXPENSIVE part, which takes MONTHS - takes all the information that’s fed in and compiles a new “dictionary” for a new “language”. Now, when you “talk” to a GPT, first it needs to transform your text from your language into this internal “language” of its own, which is what we call an “Embedding”.

Any data can be transformed. Audio can be converted into text and from text into embeddings. Videos can be broken down into series of images with temporal information. Images can be deconstructed into elements like luminance, chrominance, etc. In the end, everything can be deconstructed into data structures, there’s no magic.

Image generation models are trained differently. There are several focuses, focus on facial sharpness, focus on pose accuracy, focus on different styles like cartoon or painting, focus on categories like animals or objects. That’s why there are dozens of models out there and lots of “loras”.

To generate an image from a prompt, first you need to decode your prompt into an embedding, and that’s what things like a TEXT ENCODER in ComfyUI are for.

In the specific case of AI for images, the way I understand it is that they use something called a “DIFFUSION” model both to deconstruct the original image and to reconstruct a new image. It’s not intuitive, but the reconstruction doesn’t start on a blank canvas, the way it would for a human illustrator. It starts with an image filled with random “noise” like this:

The model starts from this noise and through a process called DIFFUSION it begins to redraw the image backwards, until it emerges from the chaos and arrives at an image.

What I “THINK”, what I “GUESS” is happening - without having read any paper, pure laziness - is that we try to influence this initial noise. For example, by extracting features from the image like a depth map, which looks like this:

The original photo looked like this:

.jpg)

Extracting a Depth Map is an ALGORITHM, it’s not “AI”. Any Photoshop or Premiere out there can do this with one hand tied behind its back. But the point is to explain that there’s much more information in an image than what you - untrained as you are - can see is there.

Another well-known type of ALGORITHM is Canny Edge Detection, which produces an edge map, making it easier to know where something starts and ends in the image:

So it goes more or less like this:

• just as a text is “converted” into an Embedding, vectors that capture the “essence” of the meaning of the text, dependent on the VSM of the pre-trained model, an image is “converted” internally into a LATENT, which because it’s an image is a “space” (that’s why people talk about Latent Space). We can’t “visualize” Latents, because it’s an internal representation that depends on the U-NET, the pre-trained model, the neural network that learns to “denoise” the Latent.
• An image U-NET is roughly similar to a text model like GPT or LLaMA, the pre-trained, read-only model that will be used to produce the final result.
• A GPT response is a continuation of the context of the previous conversation. It’s not “answering” you directly, it’s just “continuing” the text of the conversation and, by chance, due to training, the result “looks like” a response. In the case of images, we start with a randomly noisy image and through a process of removing and adding noise, it gradually “rearranges” the Latent into an image we can recognize.
• There is a “Scheduler” that decides how much noise is added or removed at each time step. And there is the concept of a DIFFUSION MODEL, which is more or less the processing “pipeline”.
• We can control the process by adding a text prompt, which is decoded into an “embedding” using a Text Decoder that usually has names like “CLIP” or properly a Transformer (such as LLava). These embeddings go into the U-NET via cross-attention layers, steering the “denoising” toward the meaning of the prompt.
• On top of that we have CONTROLNET, which is a second parallel network that adapts the process to extra conditions. It’s a copy of the U-NET architecture (with frozen weights) that receives extra inputs, like Canny Edges, or Depth Maps, pose skeletons (via OpenPose), etc, and learns small adjustments to INJECT into the main maps of the U-NET.
• During the denoising process, ControlNet activations are merged with those of the U-NET, forcing the process to respect the conditions we passed in.

So the U-NET is the “generator”, the Diffusion Scheduler orchestrates the process, text encoder and controlnet both influence the U-NET, and a VAE is used at the end to convert a latent space into an actual image that we humans can actually see.

VERY, VERY ROUGHLY. That’s more or less how I understand the process. And the way I see it, ComfyUI exposes several of these steps in the form of “PROGRAMMABLE NODES” that we can parameterize and tweak to influence how we want the final image to come out.

Beyond numerical parameters, these Nodes also let us choose which pre-trained model files we want to use for each stage. It’s all binary VSMs, but for different uses, which is why many of them have the same extension as I explained, but they live in different sub-directories. So each Node can find only what matters to it.

Taking my photo as an example, this isn’t exactly it but just to illustrate, it helps me think that those maps I showed influence the “noisy” image in a non-random direction, like this one I grabbed from an intermediate stage of the workflow I’m running:

If you strain your eyes, you can see the influence of the previous images, and the Diffusion model keeps doing the “denoising”, being “interrupted” by the ControlNet. It’s as if you were drawing and had a person next to you who every now and then erases what you did, or draws on top, and you keep adapting. So in the end it becomes a “collaborative” job between the two. If it were only the U-NET, it would generate a brand new image quite different from the original, but the ControlNet is the “client” that keeps nagging in your ear “that’s not how I want it, change it”, until it arrives at an acceptable result.

On top of that, a U-NET isn’t great at generating very high resolution images. The ideal is to generate small images, 512 pixels, 1024 pixels, but nothing bigger. But that would be useless for professional use. That’s what UPSCALE MODELS are for. If you read my article from yesterday about Video2K I explain that I pick a model there that’s good for temporal coherence in video. But there were other models that are better for “STATIC IMAGES”, or photos. And we can use them in ComfyUI.

After the new small image is generated by the U-NET, we can then use it as input to a second “UPSCALE” pass using a model like RealERSGAN, and go from a 1024 pixel image to 2160 pixels, essentially 4K. It’s different from enlarging the image in Photoshop, where everything ends up “soft” and blurred. Since it’s also another AI model, it’s trained to “redraw” at higher resolution. And we can add this to the ComfyUI workflow as well.

More than that, ComfyUI also supports Nodes with models like Tencent’s HunyuanVideo, which I introduced in another article from yesterday, about FramePack. In fact, we can do the same thing FramePack does but from a ComfyUI workflow: generate a video from an image, using the same model. FramePack is infinitely easier to use, but ComfyUI is the Swiss Army knife of AI: it does EVERYTHING, which is also why it’s infinitely more complicated and you need to put in serious effort to learn how to use it.

With what we’ve covered so far, take a look at the workflow I used (it’s zoomed out, you can dynamically zoom in):

You can see these intermediate steps using PREVIEW Nodes, so this is more or less how we can use ComfyUI to do a lot of advanced stuff. As I said, I myself haven’t even started scratching the surface.

Let’s now summarize some of the concepts used in ComfyUI:

1. Checkpoints (Base Models)

What they are:
These are the main Stable Diffusion models (e.g.: SD 1.5, SDXL, Flux). They contain the entire model architecture:

• Text Encoder (CLIP)

• Diffusion Model (U-Net)

• VAE (Image Decoder)

What they’re for:
Generating images from scratch. Each checkpoint has a different “style” (realistic, anime, etc).

Examples:
flux.safetensors, realisticVisionV60.safetensors, anythingV5.safetensors

2. VAE (Variational Autoencoders)

What they are:
Component that decodes images from the latent space (the format the model understands) into visible pixels.

What they’re for:

• Improving details and colors

• Some checkpoints require specific VAEs

Examples:
vae-ft-mse-840000.safetensors, animevae.pt

3. Text Encoders

What they are:
Models that convert your text prompt into numerical representations the AI understands.

Common types:

• CLIP: Standard for most models

• OpenCLIP: Alternative version used in SDXL

• T5: Larger models for complex prompts

Examples:
clip_vision.safetensors, openclip.safetensors

4. CLIP Vision

What they are:
Models specialized in visual analysis (as opposed to normal CLIP, which is text→image).

What they’re for:

• Creating descriptions of existing images

• Tools like “Image Prompt” (which use the image as input)

Example:
clip_vision_g.safetensors

5. Diffusion Models

What they are:
A specific part of the architecture (U-Net) responsible for the diffusion process (adding/removing “noise”).

What they’re for:

• Training LoRAs

• Advanced reuse workflows

Example:
unet.safetensors

6. LoRAs (Lightweight Adapters)

What they are:
Small files that modify the behavior of the main checkpoint without replacing it.

What they’re for:

• Adding specific styles (e.g.: anime)

• Creating consistent characters

• Adjusting body proportions

Example:
epiNoiseOffset.safetensors, add_detail.safetensors

7. ControlNet

What they are:
Models that impose control over the composition of the image (poses, depth, edges).

Common types:

• OpenPose (human poses)

• Canny (edges)

• Depth (depth)

Example:
control_v11p_sd15_openpose.safetensors

8. Upscale Models

What they are:
Models for increasing the resolution and detail of generated images.

Types:

• ESRGAN: For realistic details

• Anime6B: Specialized in anime art

Example:
4x_NMKD-Superscale-SP_178000_G.pth

9. Embeddings (Textual Inversions)

What they are:
Small files that add new concepts to the model’s vocabulary.

What they’re for:

• Specific artistic styles

• Custom objects

• Artifact corrections

Example:
bad_prompt.pt, easynegative.safetensors

10. Style Models

What they are:
Models specialized in applying visual styles or art “filters” on top of the generated image, without actually altering the basic structure of the content.

What they’re for:

• Thematic colorization (e.g.: watercolor palette, sepia, chroma-key)

• Simulating artistic techniques (oil, gouache, manga linework)

• Unifying aesthetics across series of images (same color and light “mood”)

Example files:

• flux1-redux-dev.safetensors (“Flux Redux” style)

• watercolor_v2.safetensors (watercolor brushwork)

• film_grain_vintage.safetensors

11. SAMs (Segment Anything Models)

What they are:
General segmentation networks capable of isolating objects or regions of interest in an image, producing binary or weighted masks.

What they’re for:

• Creating masks of objects/people for selective application of filters or compositions

• Guiding ControlNets (e.g.: applying diffusion only inside or outside a region)

• Pre-processing references (cropping background, separating layers)

Example file:

• sam_vit_b_01ec64.pth (SAM ViT-B, base segmentation model)

Visual Summary

models/
├── checkpoints/        ➔ Main models (Stable Diffusion)
│     └── *.safetensors, *.ckpt
├── diffusion_models/   ➔ Alternative diffusion models
│     └── *.safetensors, *.gguf
├── loras/              ➔ Style adapters (LoRAs)
│     └── *.safetensors
├── controlnet/         ➔ Composition controls (ControlNets)
│     └── *.pth, *.safetensors
├── vae/                ➔ VAE decoders / encoders
│     └── *.pt, *.safetensors
├── clip/               ➔ CLIP text encoders
│     └── *.safetensors, *.pt
├── clip_vision/        ➔ CLIP vision encoders
│     └── *.safetensors
├── text_encoders/      ➔ Advanced text encoders
│     └── *.safetensors, *.bin
├── upscale_models/     ➔ Super-resolution models
│     └── *.pth
├── sams/               ➔ Segment Anything (SAM) models
│     └── *.pth, *.onnx
├── style_models/       ➔ Visual style models
│     └── *.safetensors
└── embeddings/         ➔ Embeddings / custom concepts
      └── *.pt, *.safetensors

You have to download the right things into the right places. And to make matters worse, as I already said, the files usually have the same extension. So it’s very easy to get confused and drop a text encoder into the clip directory or vice versa. Once again, that’s why I encourage you to update the “models.conf” file and put the right URLs into the right lists, that way it stays much more organized and my Docker setup will pull things into the right places at startup.

Even so, you may still have a few other doubts that I had at the beginning too. For example “Why is the U-NET called a Checkpoint?”

Checkpoints ≈ A Complete Operating System

• Similarity: Just as a commit in Git captures a specific state of the code, a checkpoint is a complete “snapshot” of a model trained at a specific stage.

• Difference: Checkpoints are self-contained (they aren’t “diffs” like commits), each one is a complete model that can work independently.

• Better Metaphor: Think of checkpoints as different versions of Windows (XP, 7, 10) - each one is a complete operating system, not incremental.

So checkpoints kind of resemble, conceptually, a “git commit” - if you’re a developer. Or BTRFS or ZFS snapshots - if you’re an infra/devops person.

LoRAs ≈ Plugins/Extensions

• Similarity: Just as a PR adds functionality to a base codebase, a LoRA modifies the behavior of the base model.

• Difference: LoRAs aren’t “merged” into the original checkpoint, they are applied dynamically during inference.

• Better Metaphor: They’re like Chrome extensions (e.g.: AdBlock) - they add specific behaviors without modifying the browser itself.

Think of it this way: a model has Checkpoints, like commits, so you can “continue” training on top if you need to. At the same time, we don’t have to mess with the main model, we can create separate layers, the so-called LoRas. For example, let’s say a model is having trouble generating your face, no matter what prompt you use or what Nodes you configure.

Instead of that we can use the “dictionary” of the pre-trained model, like Flux or SDXL, and do training with our personal photos and produce a LoRa specific to me. Then we can add a LoRa Node into the workflow and that will result in much better images with your particular face.

There are entire online communities dedicated to this. One example. On the Civit AI site we’ll find lots of models based on existing models, like this Mistoon which is based on SDXL.

It’s a checkpoint. We can sign up on the site and download the file mistoonAnime_v10Noobai.safetensor and put it in the models/checkpoints directory.

Now, in any workflow that has the “Load Checkpoint” Node we have the option to choose this model. But let’s say this model isn’t able to generate a character with a school uniform like in many anime. Browsing the site, we find this LoRa:

The Mistoon model is large, around 6.6GB, but this LoRa - called Mistoon_Anime\ school\ uniform.safetensor which we should move to models/loras - is less than 290 MB, much smaller because it’s a specific training only on images of school uniforms.

Now we can wire the “Load Checkpoint” Node to this other “Load LoRa” Node and choose this LoRa that we know is compatible:

LoRas have to be picked to be used with the models they were based on. As I said before, think of the model as a “language”, if we make a LoRa in German, there’s no point mixing it with a Chinese model, roughly speaking. But this is one way to do fine-tuning so the model you like delivers results no other one can.

One problem I have with the Civit AI site is that you can only download things while logged in. That’s why I didn’t include any of their models or loras in my automatic install script. Unfortunately, in their case you have to manually download the files and move them to the right directories, write it down somewhere, so on another machine you can download and move everything all over again. It’s a pain that I haven’t sat down to fix yet. If anyone wants to contribute a solution for this (without exposing anyone’s login cookies, send a Pull Request).

A Real Example

Your day-to-day with Comfy, being a newbie like me, will go more or less like this: you start by going to Google and searching for “best workflow image to anime ComfyUI”. The first links are usually some Reddit thread:

Sub-Reddits like r/comfyui or r/StableDiffusion usually have news and workflows like this one. Then we go straight to the first comments:

Either that, or we open the workflow directly in ComfyUI and inspect the Nodes, one by one:

We may already have a problem: the Node came pre-configured to load a LoRa called “SDXL_aidma-niji_jini.safetensors” but the link in the Reddit comment downloads an “SDXL_Niji_V6_DLC_LoRa_V4.safetensors”. Searching specifically for the previous one on Google, I can’t find it. But it seems “safe” to use this other one, at least the names are very similar, both are derived from SDXL so theoretically they’re at least compatible. This is the page of that model on Civit AI, so you have to download manually and send it to the models/loras directory, same thing for the AniToon Checkpoint in the Node above. Download manually and move it to models/checkpoints.

Note that on this page it explains that this LoRa was made to work with the SDXL_Niji_V6 model and not with AniToon, which is what the Workflow I downloaded suggests in its Load Checkpoint Node. It’s worth testing with both. Either way, since we’re already on the site, let’s also download the suggested model:

File sdxlNijiSeven_sdxlNijiSeven.safetensors, 6.5GB, move it to models/checkpoints, are you all getting the process? Download, move it to the right place, reload the workflow in the browser and now it shows up, we select it:

By the way, I think it’s intuitive but for those who haven’t figured it out after opening the ComfyUI interface:

• left mouse button selects Nodes
• middle mouse button drags the canvas
• scroll wheel zooms the interface

Now then, let’s pick these two to test:

Taking advantage of this same image, notice in the left corner a Florence2 Node, which is a Microsoft LLM (and my Docker script already installs it for you). I can’t show all the related Nodes but these are the main ones explaining what it’s for:

The ideal in image generation is to write the longest and most detailed prompt about the original image that you can, but most people just write some garbage like “person smiling standing up” and think that’s enough. The Florence2 model is built to read an image and describe it in text.

Now there’s another detail: many checkpoints or Loras are trained with some activation keyword. So they only kick in if you type those words into your prompt. So this workflow doesn’t need you to type a prompt manually, Florence2 will do that, and every time its text starts with “The image … bla bla bla”, so we have a “Text Find and Replace” Node that swaps that “The Image” for the keywords of Anitoon or Niji: “aidma-niji, niji”.

Are these the right words? I HAVE NO IDEA lol

The workflow already brought these words, but on the Civit AI pages - where it should be documented, it’s not!! So we’re really GUESSING that it should be “niji” since that’s the name of the model and the lora. Pay attention to the keyword, it’s important, otherwise you add the lora and nothing changes and that’s because you forgot it in the prompt!!

There’s more, you have to check that the Nodes for things like VAE, ControlNet, Canny and everything else are populated with files that we have. Usually you use the same half-dozen and I’ve already pre-installed the main ones in Docker. In the image, see that the ControlNet is contronet-union-sdxl-1.0-promax.safetensors, compatible with SDXL, and we’re using Niji which is built on top of SDXL. We just keep guessing like that.

Also notice that it uses Zoe Depth Map to extract the Depth Map and Canny to extract the Edge Map as in the other workflow. You can configure it, in this workflow notice that the Edge Map picked up some weird artifacts in the background, you can adjust it to remove that. That’s why in the previous example the final image also had a window appearing, the model thought it was a window! But for now let’s leave it as is.

Now we can run it:

There’s a huge “RUN” button in blue down at the bottom. If everything is configured correctly, you’ll see that a job has been added to the queue. The interface is smart enough to let you work on other workflows and just keep queueing jobs while the GPU is processing. You don’t have to sit waiting, just leave several queued up and go to sleep.

When it runs, in the interface the Nodes that are processing at any given moment will get a green border, which is good so you know what stage of the process it’s in. And if any Node has an error - because a file is missing, for example - it’ll get a red border, so you have to fix it and run again.

And in the end, here’s one of the results:

Notice how the controlnet maps guaranteed the correct pose. The model interpreted those background artifacts from the edge map as lighting from some window, but in the original photo we can see it’s like the concrete brick of the wall lol but that’s how it is, without knowing, it guesses. And if you run it several times, it’ll always give you a quite different result. Look at another one:

Way off lol But we can swap the SDXL-Niji model for Anitoon. Let’s see one of the results:

Switching to the waiNSFIllustrious model:

This one came out even more off. Something in the maps is making the model assume that I have curly hair, no idea why. But that’s it: you have to tweak the parameters. The result of the workflow I showed at the beginning of the article gave a better result, using the same models.

Of all the Nodes, one of the most important is the “engine” of the process, the KSampler:

This is the guy who runs the show. We give it the model, the prompt, control-net outputs, and it’s the one that iteratively (steps) does the “denoise” of the Latent. Notice that there’s always a random factor (“seed”), a number of steps (25 to 50 is the average), “cfg” (classifier-free guidance) or “guidance_scale”, where higher values pull the image more toward your positive prompt. Denoise is how much noise the scheduler applies (1.0 is full noise, below 1.0 may produce more “artistic” results, you have to test).

Sampler Name is annoying because you’d need to understand the math behind it, but they are the step algorithms, for example euler, ddim, plms and each one has a trade-off between speed and fidelity. In practice: test them one by one and see the results. Scheduler has normal, klms, dpmsolver. Same thing: test them one by one.

As you can see, there are PLENTY of parameters you can play with. But the main flow is more or less this. From there you have to keep learning Node by Node what each one can do and download lots of pre-made workflows to see how people have been solving various problems, which models and loras are the most popular for each kind of application. Anyway, Reddit is your friend. I hope this gave you some sense of direction, I’m also still learning.