I bought a Minisforum MS-S1 Max with the new AMD Ryzen AI Max+ 395 chip for one specific reason: this chip supports up to 128GB of unified RAM, and I can allocate 96GB of it as VRAM for the iGPU. That gives me more VRAM than any consumer gaming card, including the RTX 5090 (32GB). And that changes what I can run locally.

Why ditch the Intel NUC

The NUC was a fine Docker server for two years. But the limitation was clear: without a GPU with enough VRAM, I couldn’t run LLMs locally in any usable way. Frank Yomik, my automatic manga translation system, needed CPU-based OCR (slow) and would connect remotely to the Ollama running on my desktop (AMD 7950X3D + RTX 5090) for translation. It worked, but it meant my desktop had to be on for the server to do its job.

With the Minisforum, Frank Yomik now runs entirely on the server. The worker uses ROCm for OCR on the iGPU, and Ollama runs locally with 96GB of VRAM. Zero dependency on the desktop.

You can get a sense of the size from the photo. The NUC is the tiny cube on the left. The Minisforum is bigger but it’s still a mini-PC. It fits on the rack shelf under my Synology NAS without any trouble.

The specs

The chip is the AMD Ryzen AI Max+ 395: 16 cores / 32 threads Zen 5, with an integrated Radeon 8060S iGPU and 128GB of unified LPDDR5X. In the BIOS, I set the UMA Frame Buffer Size to 96GB, which leaves ~30GB of RAM for the operating system and containers. Plus the kernel parameters for TTM (without them, ROCm only sees 15.5GB even with the BIOS allocation in place).

The operating system is openSUSE MicroOS (more on that in the next post). The whole machine pulls under 100W, which is absurd if you’re used to dedicated GPUs that draw 450W+ on their own.

Minisforum vs my desktop: benchmarks

I ran a set of benchmarks comparing the Minisforum against my desktop (AMD 7950X3D, 96GB DDR5, RTX 5090 32GB GDDR7). The results are clear.

CPU

Mixed results. The AI Max+ 395 is better at pure parallelism (multi-core sieve), probably thanks to lower latency in the unified memory architecture. The 7950X3D wins at float and crypto because of its higher clocks and the 3D V-Cache.

LLM inference (models that fit on both)

This is where it gets interesting. For models that fit in the 32GB of the RTX 5090, the comparison is purely about memory bandwidth:

The RTX 5090 is ~7x faster. The explanation is simple: GDDR7 has ~1,792 GB/s of bandwidth. LPDDR5X has ~256 GB/s. The ratio (7x) lines up almost exactly with the measured speed difference (6.7x). LLM inference is a problem dominated by memory bandwidth. Whoever reads weights faster, generates tokens faster.

And what about prompt processing?

Prompt processing is even worse: 7-11x slower. That makes sense, because the prompt has to be processed in full before generating the first token, and it’s an even more bandwidth-intensive operation.

Where the Strix Halo wins: large models

Now we get to the reason I bought this PC. Models that don’t fit in the RTX 5090:

qwen3.5:122b with 81GB of weights running at 19 tok/s. On a mini-PC. That’s simply not possible on an RTX 5090. On the NVIDIA card, that model would have to offload layers to system RAM, dropping to 2-3 tok/s. In practice, unusable.

The difference between MoE and dense models is brutal. qwen3.5:35b runs at 43 tok/s because, despite having 35B total parameters, only ~4B are active per token. A dense 72B model like qwen2.5:72b has to read 40GB+ of weights per token, and at 256 GB/s of bandwidth, the theoretical maximum is ~6.7 tok/s. The 4.5 measured represent ~67% efficiency, which is what you’d expect for an iGPU (overhead from shared bus and drivers).

Summary: when to use which machine

A ROCm bug that’s still around

Not everything works. Models like deepseek-r1:70b, llama3.3:70b and llama4:scout crash with a ggml bug (GGML_ASSERT(ggml_nbytes(src0) <= INT_MAX) failed). The embedding tensor of these models exceeds 2GB and the ROCm copy kernel uses a 32-bit integer for the size. On CUDA (NVIDIA) it’s already been fixed, but on ROCm it hasn’t. Waiting for the fix in Ollama 0.20.0+.

LPDDR5X vs GDDR7: why this difference exists

The next question is: why is LPDDR5X so much slower?

GDDR7 is dedicated GPU memory. It’s soldered onto the graphics card, connected by a wide bus (384 or 512 bits on the RTX 5090) at high clocks. Its only job is to feed data to the GPU. LPDDR5X is unified memory that serves everything: the operating system, applications, and the GPU all at once. The bus is narrower and shared.

In practice: GDDR7 delivers ~1,792 GB/s dedicated to the GPU. LPDDR5X delivers ~256 GB/s that still need to be split between CPU and GPU. LLM inference is basically “read all the model weights from memory, multiply by the current token, generate the next token, repeat.” Whoever reads faster, generates faster. There’s no shortcut.

The Strix Halo’s advantage isn’t speed. It’s capacity. 96GB of VRAM in a 100W chip that costs a fraction of a professional GPU. The RTX 5090 is 7x faster, but it’s stuck at 32GB. Models that don’t fit, don’t run.

The alternatives: who else does this?

If 96GB isn’t enough or you want more speed, the options are limited.

The Framework Desktop uses the same AI Max+ 395 chip with up to 128GB of RAM. Same platform, same performance, but with the differential of being modular and repairable (it’s Framework, after all). In practice it’s equivalent to the Minisforum on specs and price.

Above that, the alternative is a Mac Studio with M3 Ultra. The M3 Ultra chip supports up to 512GB of unified memory, with ~819 GB/s of bandwidth (more than 3x the Strix Halo). Apple manufactures the memory chips on the package, so latency and bandwidth are superior. You could potentially allocate ~400GB as VRAM and run models that don’t fit anywhere outside of professional GPU servers.

Apple’s internal NVMe is also another level: ~7.4 GB/s of sequential read on the M3 Ultra, compared with ~14 GB/s on the Crucial T700 (PCIe 5.0). The T700 is faster on raw throughput, but Apple’s NVMe latency tends to be lower on random I/O thanks to the SoC integration.

The Brazilian price is the elephant in the room. The Mac Studio’s max config costs $9,999 in the US. With import taxes (~60% + state ICMS), it goes past R$ 110,000. The Minisforum with 128GB lands at R$ 12,000-15,000. A nearly 8x price gap buys you a lot.

If you need more than 96GB of VRAM for truly enormous models (DeepSeek-V3 with 671B parameters fits in ~400GB Q4, for example), the Mac Studio with 512GB is the only consumer option. The alternative would be professional NVIDIA A6000 GPUs (48GB VRAM, ~$6,000 each, and you’d need several in NVLink). For everything that fits in 96GB, the Minisforum gets the job done at a fraction of the cost.

And projects that promise to run big LLMs on small GPUs?

There’s a concept called “layer offloading” that projects like llama.cpp already support. The idea: if the model doesn’t fit fully in VRAM, keep some layers on the GPU and the rest in system RAM. The GPU processes the layers it has, hands off to the CPU to process the rest, and back.

In practice, it doesn’t work well. The bottleneck is PCIe: transfer speed between system RAM and GPU VRAM is ~32 GB/s (PCIe 5.0 x16). Each generated token needs to transfer data back and forth. The result is that you drop from 150 tok/s (everything in VRAM) to 2-8 tok/s (partial offload). It’s too slow for interactive use.

VRAM is the fundamental limitation because LLM inference is memory-bandwidth-bound, not compute-bound. The GPU has compute to spare. What’s missing is the ability to read the model weights fast enough. When part of the weights live in system RAM through PCIe, the entire pipeline waits on the transfer.

That’s why unified memory (like in the Strix Halo or Apple Silicon) makes a difference. There’s no PCIe in the middle. CPU and GPU access the same physical memory. The Strix Halo’s 256 GB/s is slow compared to GDDR7, but it’s 8x faster than offloading through PCIe.

Advances in LLM optimization (up to 2026)

To understand why some models run so much better than others on the Strix Halo, you have to understand what’s changed in the ecosystem over the last two years.

Mixture of Experts (MoE)

If you run local models, MoE is the advance that matters most. An MoE model has high total parameters (e.g. 122B in qwen3.5:122b), but only activates a fraction of them per token (e.g. ~4B). The inactive weights stay in VRAM but aren’t read on every token, which drastically reduces the bandwidth needed.

In the Strix Halo benchmarks, MoE models run 3-10x faster than dense models of the same size. qwen3.5:35b (MoE, ~4B active) runs at 43 tok/s while qwen2.5:72b (dense, 72B active) runs at 4.5 tok/s.

DeepSeek and training optimization

DeepSeek V3 (December 2024) showed that it was possible to train 671B parameter models at a cost an order of magnitude lower than predicted. They combined MoE with FP8 quantization during training (not just inference), multi-stage training with curriculum learning, and several inter-GPU communication optimizations. The impact: everyone copied. Qwen, GLM, MiniMax, all of them adopted variations of the technique.

Quantization: from FP16 to Q4 without losing much

Quantization compresses model weights from 16 bits (FP16) to smaller formats: 8 bits (Q8), 4 bits (Q4), or even 2 bits. A 70B model that would take ~140GB in FP16 fits in ~40GB at Q4_K_M. Quality loss exists, but in modern formats (GGUF Q4_K_M, AWQ, EXL2) it’s small enough for practical use.

GGUF (the llama.cpp format) became the standard for local inference. AWQ and GPTQ are alternatives with more sophisticated calibration, but the ecosystem converged on GGUF because it works on CPU, CUDA and ROCm without recompilation.

Distillation: smaller models that know more

Distillation is training a small model using the responses of a large model as the teacher. Microsoft’s Phi-4 (14B) was trained with distillation from GPT-4 and competes with 70B models on several benchmarks. Qwen3 did the same: qwen3:14b is surprisingly capable for its size.

Flash Attention and optimized KV Cache

Flash Attention (Tri Dao, 2022) changed how attention is computed: instead of materializing the full attention matrix in memory, it processes in blocks keeping the data in the GPU’s on-chip SRAM, reducing memory consumption from O(n²) to O(n). Without that, contexts of 128K+ tokens would be impractical. It already went through versions 2 and 3, with optimizations for FP8 and async operations on H100. PagedAttention (vLLM, UC Berkeley) did the same for the KV cache during serving: applies virtual memory concepts to the cache, eliminating fragmentation and improving throughput by 2-4x.

In Ollama, I set OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0 on the server. The first activates flash attention, the second uses 8-bit KV cache instead of fp16, cutting the bandwidth needed per token in half. These are zero-hardware-cost optimizations that improve throughput measurably.

What Qwen, Kimi, MiniMax and GLM are doing

Qwen (Alibaba) has consistently been the best price/performance in open source models. Qwen3:14b is dense and strong; Qwen3.5:122b is MoE and runs surprisingly well in 96GB. GLM-4.7 (Zhipu AI) is notable for offering bf16 full precision versions that fit in 96GB. MiniMax experimented with long contexts (up to 4M tokens). Kimi (Moonshot AI) focused on large context windows with linear architectures.

What runs well in 96GB of VRAM

With 96GB on the Strix Halo, the models that work well for daily use:

Dense 70B+ models (deepseek-r1:70b, llama3.3:70b) are still blocked by the ROCm bug I mentioned. When it gets fixed, they should run at ~4-6 tok/s, usable for batch but not for interactive chat.

Conclusion

I bought the Minisforum to run models that don’t fit in any gaming GPU. For that, it works. It’s not fast. 19 tok/s on a 122B model isn’t the experience you get with Claude or ChatGPT. But it’s local, it’s private, and it runs on my shelf consuming less power than an old lightbulb.

For people asking about the Mac Studio: if you have the budget, it’s the best machine for running local LLMs. 512GB of unified memory, 819 GB/s of bandwidth, mature Metal/mlx ecosystem. You can run DeepSeek-V3 in full Q4. But in Brazil, with import duties, it crosses R$ 110k. The Minisforum with 128GB at R$ 12-15k is the realistic option.

And for people who think you can work around the VRAM limitation with layer offloading: you can’t. PCIe is too slow. The model has to fit fully in VRAM for inference to be usable. It’s the reason gaming GPUs with 32GB of ultra-fast GDDR7 are still capped on model size, and why the unified memory of the Strix Halo and Apple Silicon changed the equation.

In the next post I tell the story of how I migrated the entire home server to the Minisforum using Claude Code, the problems I ran into, and how openSUSE MicroOS behaves as a Docker server operating system.

Now sit down, here comes the story.

I built a pretty decent PC for my girlfriend: Intel i7 12th Gen 8 cores, 32GB DDR4-2400, NVIDIA RTX 3090, MSI Edge z790 motherboard, NVME and I even hooked up a Synology DS1621+ NAS on a 2.5Gbps network - because she’s a content creator and has A LOT of video to edit. Modesty aside, it turned out pretty nice:

Everything worked very well for a long time, but eventually, for no apparent reason, everything started feeling kind of slow. Not completely slow, but sometimes web pages seemed to stutter while loading. I’d click on the Windows Start menu and it seemed to freeze up. Sometimes Explorer would hang and I’d need to go to Task Manager and restart the process manually. Several super inconvenient things but none of them throwing any explicit “error”. It just felt “weird”.

I’m a Linux user. My main PC is Manjaro Linux. I only use Windows on my gaming mini-PCs, exclusively to run Steam and some emulators. I avoid installing anything on them at all costs. Because I know how much of a pain it is to diagnose Windows after some weird program invades it.

I went after the usual suspects. She had installed games for streaming, like Valorant, Genshin Impact and stuff like that. I always suspect anti-cheats, the crap from Riot Vanguard or Easy or VAC. I started by uninstalling all the games. But I always think anti-cheat is like malware: once installed, it will never leave the system. I uninstalled everything, but that feeling of kind of freezing, kind of slow, intermittent, remained.

I tried the next suspects: peripheral software, like Elgato, Logitech, Razer stuff that install a bunch of garbage on your machines, and recently I was reading about the malware that was shipping with all of them. Watch this GamersNexus video to understand the house of cards that the whole peripheral industry with RGB has become:

Even uninstalling all of them, still the same behavior: kind of slow, kind of freezing, unstable.

Let’s try to go deeper, and for that you have to install tools like Sysinternals Autoruns to see if there’s anything starting up (things hidden in the Registry for example). I tried to disable everything I thought was suspicious.

Nothing. By that point I had also made sure that the newest MSI BIOS firmware was installed. I checked that nothing had been turned off in the BIOS, XMP profile was ok, no fast-boot and stuff like that.

Is It the Internet?

Since most of the usage is internet and it seemed “slow”, I thought it might be network problems. Vivo’s router? The plan? Wifi with interference?

Running out of patience I decided to go nuclear on the internet idea. In fact, opening the Cloudflare Speed Test page I could see that the speed itself was ok, but the latency was VERY high, jitter high and, mainly, PACKET LOSS was over 15%. That’s not normal, it has to be 0%.

Taking advantage of a trip to Miami I brought back a TP-LINK BE9300 router for Wi-fi 7, and I also bought a compatible Wi-fi 7 PCIe card. I turned off Vivo’s Wi-fi router and bumped the plan up to 800Mbps. And indeed, the internet improved a lot:

Latency dropped, jitter dropped, packet loss went to 0%.

But this didn’t fix the problem. Out of nowhere the browser would pause when loading some page. That feeling of kind of slow, kind of freezing. And yes, I tried switching from Edge to Chrome, to Firefox, to everything. Nothing made any difference. The internet really did get faster, but Windows itself didn’t seem to improve much and while browsing, it was getting slower and slower. So the problem wasn’t the internet.

I was running out of options. Everything that could be manually cleaned in Windows, I cleaned. There was almost nothing left to delete - and I delete things without mercy. But I always had it in my head that it was some malware, something that loads before boot, some “ROOTKIT”. Anti-cheats are Rootkits by definition. They load even before the kernel and from that point on, the compromised system can no longer be trusted, because no tool will be able to find it.

Assuming at that point it had to be a compromised system, I decided to go with the nuclear option: reinstall everything from scratch. We backed up the data and off I went to reinstall from scratch.

Reinstalling DID NOT WORK

Now it got worse.

With Windows 11 Pro, freshly installed, guaranteed with no third-party software installed, it still had the same behavior of kind of slow, kind of freezing. The start menu wouldn’t let me click on anything. I had to open task manager and kill Explorer manually to get it to work again.

I spent some time staring at Task Manager, but there wasn’t any process running that I could suspect further: it was bare-bones Windows. I had even run a Debloater to uninstall unnecessary Microsoft apps like Teams.

But lo and behold, from staring at task manager so much, I saw that I had been ignoring an important piece of data. See if you can spot it:

The image is kind of messed up because I took a photo of the monitor with my phone. The refresh rate clashes with the camera’s and you get this “grid”, but I think you can still read it.

Even when you have almost no program running, it’s not natural for the CPU to be at such a low clock, 0.50Ghz - HALF A GIGAHERTZ. This Intel i7 can go above 4Ghz and the base clock is 2Ghz.

I opened browsers, opened tabs, kept monitoring, and it never went above 1.5Ghz. An occasional spike to 2Ghz for 1 second and then it dropped below 1Ghz again. Then I thought: could the AIO be failing??

Looking at Event Viewer you could see when the Start menu (which is part of Explorer.exe) stopped working and froze: it logs it as “Hanging Events” code 1002.

See all those Errors? All Hanging Events, and “hanging” is exactly the behavior: something seems to “hold”, “freeze” and only by killing the process does it come back to working.

And this was happening before and also AFTER reinstalling everything from scratch. Something seemed fundamentally wrong with the hardware itself. With the super low clock, this behavior starts to make sense: it’s so slow it can’t even process basic things. Click several times on the Start Menu (which will be digging through recent documents, updating the list of applications, opening the search bar, etc) and there’s no clock to process all that while there’s still Windows Defender in background, browser tabs processing.

Could It Be Thermal Throttle??

Since Windows was wiped clean, it can’t be anti-cheat anymore, it can’t be some weird anti-virus, it can’t be malware, it’s not the internet, it had to be something EXTERNAL, that was what made sense.

I opened the PC and left it open so I could feel if the heatsink was getting absurdly hot. If it was, it would mean my AIO (Coolermaster Watercooler) had started to fail.

It’s normal for the CPU, when it overheats, to drop the clock to avoid burning the chip. This is called “THERMAL THROTTLING”.

And again, weird: it was COLD to the touch!!!

No way. I opened the BIOS to re-check fan profiles, but everything was normal:

That’s when it finally hit me:

**LET ME CHECK THE POWER SETTINGS**

On a normal PC/laptop, it should show something like this:

By default, Windows always boots in “Balanced Mode”.

But on my girlfriend’s PC, I have no idea why, there was no Power Mode option!!! And I’m on a Windows 11 Pro with a paid license!!

That’s weird, but I know that in the old Control Panel, there’s a Power Options entry:

AND THERE’S THE PROBLEM:

It was on “Power saver”!!!

How did that happen? I have absolutely no idea. REMEMBER: I had reinstalled Windows from scratch. This is right after reinstalling!!

On a laptop this profile makes sense, to save battery when you don’t have an outlet. But on a desktop PC, which is my case, it makes no sense at all!! I switched to “Ultimate Performance” and look at the task manager now:

NOW WE’RE TALKING!! Above 4Ghz!!!

The NORMAL range for this CPU is around 4Ghz, with Turbo Boost of 5Ghz. But in Power saver mode, it NEVER goes above 2Ghz and idles at 500Mhz. In other words, everything was TEN TIMES SLOWER, sometimes it tried to boost and was still FIVE TO TWO TIMES SLOWER. And it’s intermittent!!!

Testing installing apps, browsing, etc, everything is now working normally, no freezing, no feeling of slowness. Start menu doesn’t hang anymore - because now there’s enough CPU to process the crap it needs in background.

Conclusion

CHECK POWER OPTIONS IN CONTROL PANEL!!!!

Again: I had JUST REINSTALLED WINDOWS. It was wiped clean and, by default, it came up in Power Saver mode.

I have no idea why it did that, I don’t know if it always does that, I don’t know if it failed to recognize the MSI motherboard, I don’t know what combination of hardware factors can make Windows fall back to this.

But it’s ALARMING that it does this without warning. There’s no error message. There’s no notification. It doesn’t show up in Event Viewer. In other words, even a Power User like myself has to randomly suspect “hmm, let me check the power profile”.

Lots of people with Windows must be feeling this slowness and freezing and thinking “hmm, my PC is just crap, I need to save up to buy another one”, without knowing that maybe it’s just crippled by this power profile and in reality their PC is 2x to 5x, even 10x faster and they don’t know it! That’s not a small difference!

POWER SAVER MODE is a huge piece of garbage, it has no practical use and it shouldn’t even exist. It’s one of those legacy things that give Windows a terrible experience - because it doesn’t warn you.

Remember those 1002 hanging events error codes? If you go searching on Google, every forum will tell you to run diagnostic programs, uninstall stuff, or reinstall Windows. None of them mention anything about the power profile.

I hope this helps someone else!

The way out: migrate my training to a bigger machine. I started by trying an A40, to see if 40GB was enough to fit the Qwen3-32B model for this training. And no, it didn’t fit.

“Ph0da-se”, I thought. I rented the biggest configuration RunPod has:

This H100 is already “obsolete”, in the more modern datacenters there are already H200 Blackwells and other bigger things. But in the world of “mortals”, an H100 at USD 3 per HOUR will have to do. The good part: it has 80GB of VRAM. And what I wasn’t expecting: even that is too little!

And here’s the advantage of having done everything organized and pushed to this GitHub repo. Setup was trivial: just pick a pod with a PyTorch 2.8 image with Python 3.10 and CUDA 12.8, which are the latest stable versions. Clone my project, run pip install -r requirements.txt and I could already run my ./train-lora.py.

I was going to add back those 15MB of source code from the newer Zig Standard Library, but even on the H100, it didn’t fit! Training with just the original 1MB I used yesterday already pushed the machine almost to its maximum:

Look how absurd this is: during the entire training it sat at nearly 75GB out of the 80GB of VRAM. Literally there’s no room to train with much more material than that. Either that, or you drop down to a smaller model, like Qwen3-14B. I still don’t know whether 32B with less material or 14B with more material would be the better combination. I already know 8B is useless for programming anything complicated. With 14B I had a bit more success in tests, but it still can’t do anything very complex. 32B is no guarantee of being much better: Qwen2.5-32B didn’t perform better than the 14B, but the new Qwen3-32B “looks like” it’s more stable. That’s what I’m trying to test with the LoRA.

This was at lunch time. I’m writing this post at dinner time. This was the training forecast:

This estimate keeps fluctuating, but I think it did take around 5 to 6 hours.

And at the end of those almost 6 hours, this is what it produced: a 5.1GB LoRA (against yesterday’s 3GB). Size isn’t everything. Bigger doesn’t mean better. Let’s see.

After Training

Now that training is done, I can use the same H100 pod to also serve the model with the LoRA, and it would have to be this one anyway. To run it, it goes like this:

vllm serve Qwen/Qwen3-32B \
    --enable-lora \
    --lora-modules ziglora=./qwen3-zig-lora \
    --max-model-len 10000

And you also have to limit the context size. By default vLLM tries to start with a 40k token context and, even on the H100 with 80GB, it fails initialization with this error:

There’s no room left to load the KV Cache (in short, in the process of generating the next word, it keeps adding a new row and a new column to the matrix, but doesn’t need to recompute the previous rows/columns, the result is the same. Hence this cache).

A strange behavior both on my machine and on the H100. I don’t know if this happens every time you load a LoRA, or if my LoRA is broken, or if there’s some configuration I’m missing. But loading with a LoRA, initialization is EXTREMELY SLOW. Bringing up vLLM, which normally takes 1 minute, now takes 5 minutes or more. I don’t know what causes this, but this is the kind of server you can’t easily shut down and restart. It makes me miss starting up a J2EE server …

Seriously though, it seems I’m not the only one. This issue never got resolved. It “could” be some bug in vLLM, it “could” be a combination of factors nobody knows. But neither the GPU nor the CPU is doing anything. CPU doesn’t go above 10% usage, GPU looks like it’s just waiting. Honestly, at first glance it looks like some I/O problem. Maybe the process of checking the checkpoint shards, pulling from a network volume, is slow. But since I’m going to run this only today, I didn’t bother to debug it.

Look at this. 8 MINUTES on the part that loads checkpoints. And then it still continues with other phases that also take a bit more time. I’ll leave this screenshot here, in case someone has seen it and knows what it could be.

Just to rule it out, I tried bringing up vLLM with just Qwen3 and removing the LoRA entirely and, to my surprise:

It also took almost 8 minutes. So the problem is not the LoRA. Either it’s something in vLLM that can be configured — but I don’t know how — or it’s a limitation of the Qwen3-32B model itself being this slow to load.

Anyway, with the LoRA, I tried dropping to 16k tokens: it doesn’t come up. I tried starting with 10k tokens. And at this point I’d say this test has already failed. A Qwen3-32 theoretically has the capacity to support context windows up to 40k tokens. It’s not huge, but you can work a bit inside that context.

But below 15k tokens you can no longer use it with tools like Aider, which needs to send long system prompts to instruct the LLM on how to behave so it spits out code the tool can capture. So there it is: a Qwen3-32b + LoRA, on an H100 SXM with 80GB of VRAM, is not really usable.

And I’d even be sad or frustrated if a 32B model were really, orders of magnitude better than a 14B, but in my empirical tests: it isn’t. It makes just as many errors, and it isn’t so much better that it justifies paying that much for your own top-of-the-line infra. All of them barely manage to produce simple code, and even then with errors.

In the end, with 10k the server came up, consuming an absurd 63GB at initialization, without even starting to process anything:

And …

First Tests

After 10 minutes of waiting, I had to hit Ctrl+C and start all over. Because I was dumb. RunPod is really meant to run experimental stuff, I’m not sure I’d trust it for production. It’s more like a VPS than an AWS. And I’m hacking it together by bringing up a pod that expects to have only port 8888 mapped to run a Jupyter Notebook server (which I’m not running).

My idea is to use this port to map vLLM. But by default it starts on port 8000. I forgot to add --port 8888:

vllm serve Qwen/Qwen3-32b --enable-lora --lora-modules ziglora=./qwen3-zig-lora --max_model_len 10000 --port 8888

And here we go again, wait another 10 minutes … … and finally it came up on the right port and now we can test. First, the curl tests:

And before anyone comments: there’s no problem showing my pod’s URL in the screenshot. I’m going to delete this pod before publishing this post.

By the way, if you bring up a remote model like I did, all you have to do is take the URL and put it in the OPENAI_API_BASE variable and anything in OPENAI_API_KEY and tools like Aider will work the same. The vLLM and Ollama APIs are compatible with OpenAI’s.

As you can see, the first question answered similarly to my local test with Qwen3-8B. Nothing special, I already knew that worked. Let’s ask it to explain a new feature of Zig 0.14.

By the way, besides being slow to start, it’s also slower at generating text, MUCH slower. I must have done something wrong in training, but I felt the same thing with the local LoRA or on the H100. It gets noticeably slower. It did another long “think”, but the answer this time is more consistent with the release notes (I only cut out the relevant excerpt):

In Zig 0.14.0, the `@splat` function has been extended to support arrays, making it easier to initialize fixed-length arrays with a single repeated value. This replaces more verbose or unsafe initialization patterns and is type-safe, with the array's length inferred from the destination type. Here's a concise explanation and code example:


...

### **@splat for Arrays**  
`@splat(value: T) [N]T` creates an array of length `N` where every element is `value`. The array's length and element type are determined by the context in which `@splat` is used.  

### **Code Example**  

const Point = struct { x: i32, y: i32 };  

pub fn main() void {  
    // Initialize an array of 5 integers with 42  
    const numbers: [5]i32 = @splat(42);  

    // Initialize an array of 10 booleans with true  
    const flags: [10]bool = @splat(true);  

    // Initialize an array of 7 points with (0, 0)  
    const origin_points: [7]Point = @splat(Point{ .x = 0, .y = 0 });  
}  
....

The example code also makes a bit more sense. Compared to the 8B version, at least the explanation was better. But this might have been affected by a small tweak I made to the dataset: I duplicated the release notes, to try to put a bit more emphasis on the new features. That probably had an effect too.

Now let’s see if it can write code that actually works. My prompt was this:

"Can you give me a Zig 0.14 code example of a simple program that acts like an interactive chat, so the user can input messages. then the message is parsed to see if there are commands such as '/list', which will list the files in the current directory, or '/cat' which will read the file asked, or '/quit' to exit the chat?"

And this happened:

Remember how I had to keep decreasing the context to get it to come up? And remember how I keep saying that “thinking” wastes a lot of tokens? Well, there you go. The model self-crashed by thinking too much. I tried a few times and it crashes. To try to avoid this, I tried hitting curl sending enable_thinking=false like this:

❯ curl https://d8zck5aw8eimij-8888.proxy.runpod.net/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "ziglora", "temperature": "0.1", "enable_thinking": "false", "skip_special_tokens": true,
        "messages": [{"role": "user", "content": "..."}]
      }'

Temperature, for code, I think can’t really be much higher than 0.1. For chit-chat, for less exact things, the default of 0.6 that everyone uses is fine. But for code, I really don’t need it to be “creative” (random, in fact), I want answers as exact as possible.

Anyway, even trying to turn off thinking, something is wrong. Possibly the LoRA I added is making it “think in circles” and never being able to return, blowing up the context. I tried several times, but it’s unable to answer with code.

Final: I Give Up For Now

Yes, I broke the LLM, apparently lol 😅

I tried several times and it really can’t give me any code back. And I know it’s trying, because if I send stream=true, I can see it returning tokens, but at some point it runs out of space somewhere (surely the context, which is small) and it crashes.

The feeling is that my LoRA probably “stuck” too hard and threw too many parameters out of balance. Now the model starts writing the text and enters circular thinking, never stopping generating more text.

This is definitely an effect of the LoRA and of some mistake in training. If I had to guess, it’s because I crammed super long texts into training, without manual markers indicating where each part ends with a <|im_end|> or whatever. Without that, maybe it thinks the whole text is one answer and it has to respond that long. From what I understand, in the case of this Zig training, I would have had to break the documentation, the release notes and everything else into smaller chunks, like this:

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
What is Zig?<|im_end|>
<|im_start|>assistant
Zig is a compiled programming language...<|im_end|>

And do that for all 200 thousand characters of documentation. Imagine the work to do that. And avoid sending super long texts in context without markers for where each part starts and ends.

At least, that’s my guess. The problem: even if I spend time formatting it this way, then it’s another 6 hours of processing, just to find out afterwards whether it made any difference. It probably won’t, I need to think of other hypotheses, and each round is 6 to 12 hours between setup and processing.

From this point on it’s a slog I don’t have the motivation to do. I’m leaving this written down here to see if it reaches the hands of real researchers. Maybe it’s something an AI researcher would find super obvious. But that’s the point: this information isn’t easily available publicly anywhere. Every example you find online is too simple. They just say “on this line of python you load the dataset, and on the next line training starts” — but where are the instructions for HOW to build a dataset and HOW to determine whether it’s adequate? Nowhere.

If anyone has tips on how to improve the training or wants a research problem in the field, here is the GitHub with all the code and training material I produced. We need researchers to look at this kind of thing and share more about what they know. I know someone knows, but just digging around on Google didn’t get me anywhere.

On Linux, this is a massive pain in the ass, because the SMB protocol (built for Windows) doesn’t properly carry concepts like ownership (chown) or permissions (chmod). It doesn’t understand things like execute permission (everything is executable). For simple stuff like a Downloads, Videos, or Photos directory, it kind of doesn’t matter.

But let’s say you want to edit source code in projects with Git. It’s going to be a nightmare, because SMB overrides the permissions regardless of what’s underneath, and every single time it’s going to conflict with Git, which will think files were modified (changing permissions counts as a change it tracks), and it’ll keep nagging you to commit that, and that will pollute your whole repository (never use Git projects on SMB over the network).

Instead, I’m on Linux, my NAS is Linux. The right call is to use a Linux protocol. That protocol is NFS and I showed how I set it up in last week’s post.

Anyway, I’m using NFS and I have way fewer headaches around file permissions. But then I hit another problem: if you saw my recent posts, you saw I’ve been experimenting A LOT with Docker. Spinning up new images like there’s no tomorrow. My local 2 TB NVMe filled up fast and started complaining about lack of space. So I thought, “Ah, I’ll just move the Docker storage directory, /var/lib/docker, to a directory mounted via NFS on my NAS”

Doing that is very easy:

sudo mkdir -p /mnt/nfs/docker
sudo chown root:docker /mnt/nfs/docker
sudo chmod 771       /mnt/nfs/docker

That creates a new directory inside my NAS and applies the correct permissions.

sudo systemctl stop docker
sudo mv /var/lib/docker /mnt/nfs/docker
# sudo rm -Rf /var/lib/docker (this will free everything up for me)
sudo ln -s /mnt/nfs/docker /var/lib/docker

That will free up all the occupied space. I don’t need to worry about the images because I can rebuild everything from my Dockerfiles. And I don’t need to worry about volumes because I had nothing important that wasn’t mounted externally, in real directories. Rule: never store important things in Docker volumes. It’s bad practice!

# edit /etc/docker/daemon.json
{
  "data-root": "/mnt/nfs/docker"
}

sudo systemctl start docker

Done, this changes the configuration from /var/lib/docker to the new /mnt/nfs/docker and after restarting the service, it’ll start writing everything there. So the post is over, right?

This is as far as most people go. But if you work with this setup for 2 minutes, you’ll feel something is VERY wrong.

I tried building a new image. And it was taking ABSURDLY long. It turns out NFS is indeed faster than SMB, but both were built for file servers, not for operating systems. That makes a huge difference.

These protocols are FILE-BASED, every operation is based on files. And that’s extremely inefficient.

If you watched my videos on file systems, you should already know they are BLOCK-LEVEL. At the filesystem level, you work with fixed-size blocks of bytes, organized in some variation of B-TREE structures (depending on the filesystem). Operations happen at the block level. Files are abstractions over collections of blocks.

When you run a chmod -R, which is a recursive permission change, inside a Dockerfile build, for example, even an empty tree will cause recursion on every entry inside, be it hidden files, ACLs, etc. And it needs to look up every single permission of everything in there. That costs network time (because NFS/SMB are network protocols).

That causes latency. Every permission on every file is an individual remote call, an RPC call. Even on the fastest network, like my 10Gbps one, even if each RPC is under 0.1 ms, it quickly becomes a bottleneck. And that’s exactly what started happening.

I used sudo nfsiostat 1 10 to check. It was showing this:

write: ops/s=2417  kB/s≈295 MiB/s  avg exe=106 ms  avg queue=102 ms

An average of 106 ms per RPC. That’s an eternity. Average wait time in the queue of 102 ms. Even if I make just 100 RPC calls, it will cost more than 10 seconds in latency alone. And in an OS build, installing packages with thousands of files, that will literally cost an eternity. Running Docker build on NFS/SMB is simply not viable.

iSCSI

Obviously I’m not the first to run into this problem and make the mistake of using a file-based protocol for services with high-volume file churn like the Docker cache.

The right move is to create an iSCSI virtual drive on my NAS. If you’ve never heard of it, SCSI is an advanced protocol for drives. Cheap PCs in the 90s and 2000s used IDE/ATA, which is a simpler and cheaper protocol. But the first Macs and UNIX workstations from the 80s and 90s always used SCSI, which was vastly superior.

It was asynchronous, with queue control. You could send dozens of commands at once and the protocol was smart enough to reorder those commands for better efficiency (for example, skipping a file read if a delete command came before it). IDE was blocking, one command at a time. A single SCSI BUS could handle up to 16 devices. Parallel ATA was 2 per cable. Ridiculous. SCSI had advanced error recovery and reporting systems. It had “enterprise” features for hot-swap, power management, timeout behaviors, recovery, clustering, etc. And on top of all that it was agnostic to the transport medium, it could be fiber optic, SAS, or even iSCSI, which is SCSI over Ethernet, which is what we’re going to use.

IDE/ATA had none of that. It was the protocol for poor-person PC disks. Real workstations, SUN, IRIX, Silicon Graphics, Macs, used SCSI.

More importantly, iSCSI is a block protocol, not a file protocol. Anyway, this varies by NAS manager, but in the case of Synology DSM, which is what I use, it has an application called SAN Manager, which lets you create an iSCSI drive:

Just follow the wizard, give it a name, choose the size, enable CHAP if you need extra security (I don’t, I’m on a controlled local network). That creates the iSCSI drive (think of it like a remote USB drive) and a LUN (Logical Unit Number). A LUN is simply one of many potential “virtual disks” that a target can present. A LUN is like a numbered slot in a target’s storage array. A target is a disk server. On a SAN you can have several virtual disks and the way to organize that is with LUNs. In our case there’s no real need to go deep on LUN details.

It takes a while, but when it finishes you just go back to my Manjaro Linux and run:

yay -S open-iscsi
sudo systemctl enable --now iscsi 

That brings up the service to start automatically on every boot. Now to discover my drive on the network:

❯ sudo iscsiadm -m discovery -t sendtargets -p 192.168.0.xx
192.168.0.xx:3260,1 iqn.2000-01.com.synology:TERACHAD.docker.7af6e...
[2804:1b3:....:fe18:3f7d]:3260,1 iqn.2000-01.com.synology:TERACHAD.docker.7af6e...
[fe80::9209:....:3f7d]:3260,1 iqn.2000-01.com.synology:TERACHAD.docker.7af6e...

Every virtual drive has an IQN which is like a “URL” for the drive. Now we log in:

❯ sudo iscsiadm -m node --login
Login to [iface: default, target: iqn.2000-01.com.synology:TERACHAD.docker.7af6e9116c1, portal: 192.168.0.21,3260] successful.

Since I didn’t enable CHAP it doesn’t ask for a password or anything (in a company it should be enabled, obviously). And done. When you do this I hear on GNOME the same little sound as when you plug a pen drive into the PC and it automatically shows up as a disk. If you run lsblk, it’ll show up as a normal /dev/sdX drive. In the GNOME Disks app (or in Windows Disk Management) it really does show up as if it were any other hard drive or SSD. It doesn’t behave like a shared folder, it behaves like a real drive. You even have to format it, just like a new pen drive.

The formatting itself takes quite a while, because in the process it has to check and map block by block. Since it’s over the network, it’s slower than on a real drive, obviously. But it’s a one-time thing. Even on my 10Gbps network it’ll format at around 120 MB/s, and on a 2TB drive that comes out to around 2 hours for a fast format. I forgot you could format with “lazy”:

mkfs.ext4 -v -E lazy_itable_init=1,lazy_journal_init=1 /dev/sdX

Leaving that for next time, now I wait 2 hours. To make sure it mounts automatically at boot, you need to check its node.startup:

❯ sudo iscsiadm -m node -o show | grep -E 'Target:|node.startup'

node.startup = manual
node.startup = manual
node.startup = manual

It’s set to “manual”, to switch it to “automatic” you do:

❯ sudo iscsiadm -m node \
  -T iqn.2000-01.com.synology:TERACHAD.docker.7af6... \
  -o update -n node.startup -v automatic

With the iSCSI daemon running and after logging in, besides /dev/sda, if you run ls /dev/disk/by-path you’ll find something like this:

ip-192.168.0.xx:3260-iscsi-iqn.2000-01.com.synology:TERACHAD.docker.7af6exxxxx-lun-1 -> ../../sdX

Since I’m using the AUTOFS service and it “steals” my /mnt directory, I’m going to mount the iSCSI drive via fstab at /media/docker by adding this line at the end of /etc/fstab:

...
/dev/disk/by-path/ip-192.168.0.xx:3260-iscsi-iqn.2000-01.com.synology:TERACHAD.docker.7af6e9xxxx-lun-1  /media/docker  ext4      _netdev,nofail,x-systemd.automount,x-systemd.requires=iscsid.service,noatime,nodiratime,commit=60  0 2

Then just reload and mount:

sudo systemctl daemon-reload
sudo mount -a

And that’s it. From now on I have a “virtual drive” on my remote NAS. Now, let’s switch Docker over to start using it. I can reconfigure my Docker to cache images and volumes and everything else directly on that drive. Just edit the /etc/docker/daemon.json file with the mount point:

{
  "data-root": "/media/docker/docker"
}

Don’t forget to create the directory on the new drive with the right permissions (and it does accept permissions, because it’s a drive formatted with ext4 like any other):

sudo mkdir -p /media/docker/docker
sudo chown root:docker /media/docker/docker
sudo chmod 771 /media/docker/docker

Finally, just restart the Docker service, wipe out /usr/lib/docker to free up space on my main NVMe, and done. It should be WAY faster than NFS, though not as fast as a local disk, of course. Even at 10Gbps, the network introduces latency, there’s no way around it. But the heaviest part is really the build process, which has a lot of writes. For loading an already-built image, it should be blazing fast. But at least by using a block protocol instead of a file protocol, latency should drop by at least an order of 10x or more.

And is it really faster? YES. In this screenshot of the docker build running, on NFS the times on the right were HUNDREDS OF SECONDS for many commands. Now they’re back below 1 second like they should be. For all intents and purposes, I’m not feeling any difference from what a USB 3.2 Gen 2 external HD would give me.

The MASSIVE difference that a BLOCK-LEVEL virtual drive makes, compared to a FILE-LEVEL NFS/SMB protocol, is brutal, the protocol is gigantic overhead. Incidentally, this is more or less how it works when you go to AWS and hire EBS storage (Elastic BLOCK Store), that’s why it’s called “BLOCK”, because it’s a BLOCK protocol like iSCSI and not NFS. I hope you learned how much of a difference it makes to understand how protocols and filesystems work. If you haven’t watched my videos, there’s this playlist that explains BLOCKS in detail.

But yes, iSCSI has downsides: being a drive, it cannot, or should not, be shared (think of two PCs sharing the same pen drive, it would be a huge sh*tshow). Block devices were built to be used by a single device. Precisely for sharing, that’s why protocols like SMB or NFS exist: to manage concurrent access to the same file over the network.

Another downside: the virtual drive is one big BLOB of bits. Without “mounting” it, there’s no way to see the files. On the NAS server, it doesn’t show the files. It’s like a VirtualBox .vhd file. You need to “turn it on” and “mount” it on the “PC”. That’s why I’m still going to use NFS for the usual stuff, like Videos or Downloads. And the virtual drive is exclusively for the Docker cache, in this case. That works well. No other PC accesses it. If I had a second Linux machine that needed a Docker cache, I’d have to create a second LUN just for it. Remember, it’s a drive. Virtual or not, it behaves like a real drive to your system.

For collectors, the problem is that a lot of anime from the 90s and earlier never came out, and never will, on Blu-Ray (1080p) or UHD (4K). Only the most famous ones get the “remaster” treatment (taking the original master tapes and recapturing them at higher resolution). So a ton of old anime is stuck in the DVD era (480p) or VHS (480i).

Not every old anime can be upscaled, it can end up looking too “washed out”. Anyone who downloads torrents has probably seen badly done upscales. But I wanted to be able to test it myself, on my local machine, with my old files. Here’s a small example clip:

It’s a piece of the 1st episode of the legendary Rurouni Kenshin from 1996. Yes, today we have a remake, which they’re redoing from scratch. But for those of us who feel nostalgic about the original, there’s no replacement. The DVD quality isn’t horrendous, you can still watch it just fine today as is. But I’m curious to see how it looks in 4K.

I didn’t do any long research, but a year ago I had stumbled onto Real ERSGAN. That project has been discontinued for about 4 years, so I don’t know what the most modern option is today. But their pretrained models are still useful.

There are still projects that use this model, the one I know of is Video2K. On Windows it apparently even has a nice little graphical interface, but the advantage for Linux is that it runs in Docker. An example would be like this:

docker run --rm --gpus all \
  -v "$PWD/videos_in":/input \
  -v "$PWD/videos_out":/output \
  ghcr.io/k4yt3x/video2x:6.4.0 \
  -i /input/old_anime.mp4 -o /output/old_anime_4K.mkv \
  -p realesrgan -s 4 --realesrgan-model realesr-animevideov3

If you have your NVIDIA set up with the Container Toolkit, CUDA and all that (ask ChatGPT), Docker can map the device straight into the container. This makes it much easier to keep my Linux from getting dirty with dependencies and loose configs scattered around. I don’t do anything of this kind outside of containers anymore.

But I ran into a problem. If you’re using an AMD GPU, just doing this should already work, because this Docker image actually uses Vulkan and not CUDA. In my case I have a primary integrated AMD GPU and a secondary NVIDIA RTX 4090. I tried passing it directly:

❯ docker run --gpus all --privileged \
  --device=/dev/nvidia0 \
  --device=/dev/nvidiactl \
  --device=/dev/nvidia-uvm \
  --device=/dev/nvidia-modeset \
  --runtime=nvidia \
  -e VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/nvidia_icd.json \
  -e VK_LAYER_PATH=/usr/share/vulkan/explicit_layer.d \
  -v "$HOME/Downloads/Video2K/videos_in":/input \
  -v "$HOME/Downloads/Video2K/videos_out":/output \
  --rm -it \
  ghcr.io/k4yt3x/video2x:6.4.0 \
  -i "/input/(B-A)Rurouni_Kenshin_-_01_(A5D3CB57).mkv" \
  -o "/output/(B-A)Rurouni_Kenshin_-_01_(A5D3CB57)_upscaled.mkv" \
  -p realesrgan -s 4 --realesrgan-model realesr-animevideov3

But it just won’t work. If I only say “–gpus all” it grabs the iGPU by default. Upscaling a video ends up consuming 100% of my CPU, all 32 threads, but my NVIDIA GPU just sits there watching, doing nothing. If I try to pass “/dev/dri/card0” or “/dev/nvidia0”, it errors out and can’t even start.

So from here on the solution is only for those who have NVIDIA:

CUDA Dockerfile

This time I decided not to be lazy and organize everything into a little project on Github. Clone it directly from there Batch Anime Upscaler Video2K Docker CUDA. Just do this:

git clone git@github.com:akitaonrails/Batch-Anime-Upscaler-Video2K-Docker-CUDA.git
cd Batch-Anime-Upscaler-Video2K-Docker-CUDA
mkdir input # put your old videos here
mkdir output

docker build --build-arg HOST_UID=$(id -u) --build-arg HOST_GID=$(id -g) -t anime-upscaler:latest .

If nothing goes wrong, this will build a Docker image with everything you need, then it’s just a matter of placing your videos in the subdirectory I mentioned, or any other you want, and configuring the run command:

 docker run --gpus all --rm \
  -v "./input":/input \
  -v "./output":/output \
  -e HOST_UID=$(id -u) -e HOST_GID=$(id -g) \
  anime-upscaler:latest

And that’s it. It takes a while. With my Ryzen 7950X3D plus RTX 4090, it processes at a rate of around 13 frames per second. So less than half the playback speed. If a video is 20 minutes long, it’ll take more than 40 minutes to process each one. It’s a good idea to set aside plenty of time, and that’s also why this is best reserved for videos you really want to watch.

The original Video2K script supports several parameters, which I left exposed as environment variables in the Dockerfile:

MODEL="${MODEL:-realesr-animevideov3}"
SCALE="${SCALE:-4}"
TILE="${TILE:-0}"
DENOISE="${DENOISE:-1.0}"
NUM_PROC="${NUM_PROC:-1}"

Then you can change them like this:

docker run --gpus all --rm \
  -e DENOISE=0.5 \
  -v ... \
  anime-upscaler:latest

As you can see, by default we use the pretrained model “realesr-animevideov3”, which is less detailed than the 6 billion parameter model but is much more stable between frames, which matters more for video. Most of the other models it supports, like “RealESRGAN_x4plus” or “ESRGAN_x4”, are meant for photos and images and not for video. Video demands temporal coherence, which is quite hard.

At most it’s worth trying the “realesr-general-x4v3” model, which they say is a bit lighter, in case your machine is struggling too much to run the default. But then you’ll have to modify the Dockerfile to download that model and rebuild:

# === OPTIONAL: Preload additional Real-ESRGAN models ===
# Uncomment any models you want available in the container at build time

# Anime 6-block model (better for anime stills, line art)
# RUN wget -q https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/RealESRGAN_x4plus_anime_6B.pth \
#     -O /opt/Real-ESRGAN/weights/RealESRGAN_x4plus_anime_6B.pth

# Original RealESRGAN 4x general-purpose model
# RUN wget -q https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/RealESRGAN_x4plus.pth \
#     -O /opt/Real-ESRGAN/weights/RealESRGAN_x4plus.pth

# RealESRGAN 2x model (lighter upscale)
# RUN wget -q https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/RealESRGAN_x2plus.pth \
#     -O /opt/Real-ESRGAN/weights/RealESRGAN_x2plus.pth

# ESRNet (non-GAN model, ultra-smooth)
# RUN wget -q https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/RealESRNet_x4plus.pth \
#     -O /opt/Real-ESRGAN/weights/RealESRNet_x4plus.pth

# General-purpose lightweight video-friendly model (with denoise control)
# RUN wget -q https://github.com/xinntao/Real-ESRGAN/releases/download/v0.3.0/realesr-general-x4v3.pth \
#     -O /opt/Real-ESRGAN/weights/realesr-general-x4v3.pth

# Anime video model (default in your setup)
# RUN wget -q https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesr-animevideov3.pth \
#     -O /opt/Real-ESRGAN/weights/realesr-animevideov3.pth

# Original ESRGAN (2018) model
# RUN wget -q https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/ESRGAN_x4.pth \
#     -O /opt/Real-ESRGAN/weights/ESRGAN_x4.pth

I haven’t tested all of them, but for anyone who wants to, I think those are the links. Change it in the Dockerfile, run the build, change the MODEL variable, and test on your videos.

What’s worth tweaking are things like DENOISE, which is set to “1.0”, the maximum, but that can leave the image too “stretched”. It’s worth trying smaller values, like “0.5”. It varies case by case, only by testing.

Another factor is the scale, SCALE, where the default is “4x”, which turns 1080p into 4K or 720p into 1440p. But sometimes you don’t need to go that high, and there’s no point because the original image already has very little detail anyway. Maybe 2x is enough. The script supports 2x, 3x, 4x but it caps at 4K resolution, so there’s no point trying to turn a video that’s already 4K into 8K, for example.

My RTX 4090 and my Ryzen 9 are quite powerful. Processing just one video, I don’t see them breaking a sweat at all:

Clearly you could run more videos in parallel. You have to be careful when it’s unbalanced. If the CPU climbs close to 100% just to separate the frames, having spare GPU capacity doesn’t help. With both having headroom, you can make the script spin up more than 1 process on the GPU and send frames in parallel with the “-e NUM_PROC=2” option. Look at how it now uses more of the GPU and the CPU still has room to spare. Maybe 3 is the maximum, but you have to test, it’s not constant the whole time.

This can dramatically reduce the total time. You can choose between increasing the GPU processes via the script or having two separate video directories and spinning up two parallel containers of this program to run at the same time, whichever works best for you.

Final Result

Let’s see how that first clip turned out with 4x upscale and 1.0 denoise:

The difference is HUGE, but whether it ended up “GOOD” is going to be VERY subjective. Personally I didn’t find the original bad to watch. Maybe tweaking DENOISE down a bit, or SCALE to 2x instead of 4x, would already be good enough, I don’t know.

For comparison, here’s another clip in the original old version:

And again, how it looks after the 4x upscale:

At least it’s working as it should, from here on it’s a matter of doing some tuning. If anyone has more tips or knows of newer open source upscaling models or projects, send them in the comments below!

Before someone asks, why not use NVIDIA’s DLSS 3, which they use to upscale game resolution? Answer: because they don’t expose that functionality as an API you can access, it’s only for their proprietary drivers in game profiles. They turn it on and tune it case by case. There’s no general API in the CUDA lib.

Post-publication Update

I decided to test the RealERSGAN_x4plus_anime_6B model, which is much heavier than the realesr-animevideov3 I uploaded to the repository.

Change the Dockerfile and add this after the wget for the other model:

RUN wget -q https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth \
     -O /opt/Real-ESRGAN/weights/RealESRGAN_x4plus_anime_6B.pth

This model can’t be run in parallel even on my RTX 4090, which stays constantly at 100%. If before it ran at around 13 frames per second, with this one it only runs at 3 frames per second. It’s 4x heavier and only for really beefy GPUs. If you don’t have something like that, stick with “realesr-animevideov3”, or with the even lighter (lower quality) “realesr-general-x4v3”. These models depend heavily on GPU power and especially on the amount of VRAM.

To run it, add “-e MODEL=RealESRGAN_x4plus_anime_6B” to the “docker run” command.

See how the same clip from before turned out using this model:

For having spent 4x more of my machine’s resources to process it, I didn’t find the final result that different, and in some parts I think it even has more defects. So that’s it, the result will vary, you have to test. It depends a lot on the quality of the original anime, but just because a bigger and theoretically “better” model exists doesn’t mean every result will turn out better.

I know every motherboard ships pre-configured with safe and conservative “defaults”, the most stable rather than the most performant. So if you don’t go into the BIOS to tweak things, you’re leaving performance on the table.

I have an AMD Ryzen 9 7950X3D CPU. The processor is a package that works with two “dies”, I’m not sure if “chip” is a way to visualize it. Each one has 8 cores, totaling 16 cores, and each core has 2 threads, so 32 threads possible in parallel.

But there’s a difference: each die has different L3 caches. One of the dies has only 32MB and the other has 96MB, so heavier programs benefit from running on the cores of the die with more L3, and you can configure that.

Besides that, it has a base clock of about 3GHz with the ability to turbo boost up to 5.7GHz. They can’t all run at that speed or it heats up too fast and may “thermal throttle”, and then it drops everything below 3GHz. That’s why, especially in laptops that don’t have adequate thermal capacity, the best option is not overclocking but “undervolting” (cutting power so it runs less fast). Because it’s better for everyone to be a bit less fast than for everyone to be super fast and immediately overheat and slow everyone down. It’s complex.

To complicate things further, I have 96GB of DDR5 6000 RAM. Capable of a theoretical maximum of 6GHz, but that’s theoretical because everything can desync and cause instabilities up to a kernel panic crash. That’s why the motherboard default is the base speed of about 4GHz and the CPU also maxing out at around 4GHz.

But you can push that up a bit without making it unstable. In short, here’s what I did:

PBO - Precision Boost Overdrive

Ryzen CPUs have PBO to turbo boost the core clocks. It usually ships either disabled or in a conservative configuration, so your 5.7GHz CPU may be running at only 4GHz or less and never delivering its maximum.

To adjust this, if you want to get advanced, you need to tweak parameters like PPT (Package Power Tracking) to 200W or more. TDC (Thermal Design Current) to 160A or more. EDC (Electrical Design Current) to 200A or more. And you can set up a curve optimizer, per core, to negative (-10 or -20) to reduce voltage, which can allow higher boost clocks.

But this is very advanced, and you need to know a lot more about electrical stuff before trying it. Better not risk it and not mess with this. On the screen below, don’t leave it on Advanced, leave it on Auto:

Instead, it’s better to mess with PBO Enhancement Presets, which is this screen below:

70/80/90 refers to the maximum temperature we allow the cores to reach. This will depend on whether you have a good cooling solution. Whether it’s an AIO with liquid cooling or a Noctua NH-D15 G2, which is what I use. Noctua fans are extremely silent, I almost never hear them, and they have less maintenance and lower failure potential than liquid cooling. I prefer fewer moving parts to be safe.

[

Now those Level 1 to 5 are the PPT/TDC/EDC tunings I mentioned, but already pre-defined and tested by the manufacturer. 1 is less aggressive, the most stable, while 5 is the most aggressive and can be unstable. The right approach is to start at 90 level 5 and decrease until you find the most stable for your system. This will vary; for instance, the more RAM you have, the less aggressive it seems you can be, especially the faster that RAM is.

In my case, right at the boot of my Manjaro Linux, it gave a kernel panic, which looks something like this:

So I went back into the BIOS and tested different levels and, for my configuration, I could only get up to 90 Level 2. This has been stable and so far I haven’t had any random crashes.

Beyond that, recently (early 2025) motherboard manufacturers released firmware updates that opened access to a feature called X3D Turbo Mode. If you see this, don’t think it’s free extra performance. In reality, unless you use the machine more for games, you’ll actually feel it got slower.

As I explained before, Ryzen 9 has 1 CCD (core complex), which is the die with 96MB of L3 cache (V-Cache). Turbo mode tries to force more programs onto the CCD that does NOT have V-Cache (the 32MB one), which can therefore reach a higher clock boost but has more latency. This is good for games, which usually don’t benefit from having more cache but rather from more clock.

But at the same time, programs that need V-Cache will end up landing more often on the Core with LESS L3 cache and will then perform worse than before. Examples of that are 3D modeling programs like Blender, exactly what I want to use more now. So test it, but in principle you can leave it off.

This option in my firmware is called “X3D Turbo Mode” but elsewhere it may be “3D V-Cache Performance Optimizer”, same thing.

XMP/EXPO RAM

Now, DDR5 RAM boots at the base clock of about 4.8GHz. But depending on the RAM model, it can go much higher, in my case up to 6GHz. All modern memory also has several settings like frequency, latency, voltage, and more. Again, it’s not worth trying to set these by hand. Better to use predefined profiles. Intel calls this support XMP and AMD calls it EXPO.

I forgot to take a photo of my BIOS, but this slightly older photo from Google Images is the same thing. XMP/EXPO ships disabled (conservative) but you just pick the profile that already exists:

Note that just below there are things like Low Latency Support and High Bandwidth Support. Theoretically to reduce latency and increase transfer bandwidth. All of which sounds great, but in my case it doesn’t work. If I try to enable both, my system doesn’t boot. I was forced to clear the CMOS to reset the BIOS without that.

The problem: possibly too much memory (96GB). Maybe if you have less, around 32GB, this works, but it seems the more you have, the more unstable it gets if you try to go too fast.

Another problem, on DDR5, the more memory, the slower the Cold Boot, the initial boot when the PC was off. The BIOS has to do a check on the memory to ensure those parameters, response time and so on. And this happens on EVERY boot. It can take a few minutes and some people might even think the machine is frozen, but it’s just that check.

In the BIOS there’s an option called Memory Context Restore where it uses the parameters it recorded the last time it did that check. But I don’t recommend it; for me it doesn’t even boot anymore after enabling it. Theoretically it would cut down the check time and reuse old data, but that data may not match (the values can fluctuate) and that makes the whole system unstable. Better to wait an extra minute on boot than to live with an unstable system.

In advanced memory settings there’s one more thing we should change: DF C-States, which needs to be DISABLED. In theory this feature is to reduce the latency between CPU and memory. But on systems with a lot of RAM (96GB or more like mine) this can also cause instability for not much noticeable gain. I prefer staying stable in this case.

Another setting to change is Power Supply Idle Control, which has to be set to “Typical Current Idle”, and so far (a firmware fix update may come), it’s mandatory for Ryzen processors (even AM5). This prevents the “Deep Power State” sleep state, which can cause random kernel panics.

Same thing for Global C-State Control, which also needs to be disabled for the same reason: avoid “deep sleep states” (C6/C10) when the CPU sits in “idle” (doing nothing) for too long. These features exist to save energy, but it seems AMD has bugs when the CPU is forced into that state and when it wakes up, it comes back unstable or crashing. So it’ll consume more energy when doing nothing (around less than 20W, rather than near zero), but at least it won’t be unstable. If you want to save energy, the better option is to shut down and actually cut the power when not using it.

BIOS Summary (for me)

• PBO Enhancement: 90 Level 2 (clock boost slightly more aggressive than the default)
• Power Supply Idle Control: Typical Current Idle (prevents the idle crash bug)
• Global C-State Control: Disabled (avoids deep sleep, which causes instability)
• DF C-States: Disabled (also avoids sleep, improving Infinity Fabric latency)
• XMP/EXPO: Profile XMP1 if you have it (raises the memory clock)

There are 2 important things, especially if you’re using discrete GPUs:

• Above 4G Decoding and Re-Size Bar Support. In my case both are on by default; if yours are off, turn them on.

On Linux, you can check if it’s enabled. Run the following:

sudo lspci -vv | grep -A 15 VGA

You should see “Resizable BAR” like this:

Capabilities: [XX] PCI Express, ... Resizable BAR: Supported (Enabled)

If it doesn’t show up, go to the BIOS and look for it to enable. Without this you’re not using the maximum of your video card. I think on iGPUs (which are GPUs already built into the CPU) this option doesn’t exist.

Linux-Zen Kernel

I’ve always used the most stable LTS default kernel, currently Linux66, if I’m not mistaken. But I remembered I can use the Zen kernel, which is the same default kernel Arch uses with performance tunings to improve latency, especially in multi-thread. It switches to the CFS scheduler with improvements. Theoretically, it improves overall system responsiveness. I think it’s worth using. On Arch/Manjaro it’s like this:

yay -S linux-zen-versioned-bin linux-zen-versioned-headers-bin

This will uninstall the old kernel and, if you’re using NVIDIA GPUs, it’ll also uninstall the driver package. So you also need to install the following package:

yay -S nvidia-dkms

And then, with everything installed, run the following command:

sudo dkms autoinstall

This will reinstall the nvidia driver binaries for the new Zen kernel. In Manjaro’s case, you don’t need to mess with GRUB because in my case it’s set to GRUB_DEFAULT=saved in /etc/default/grub. That means when it reboots, it’ll record which kernel I last chose, and on the next reboot, if I don’t choose anything, it’ll load again the last kernel I picked. I prefer it this way over hard-coding which kernel is the default.

Once that’s done, on the next boot just check:

❯ uname -r
6.14.2-zen1-1-zen

Another thing is to check if CPPC is active. That’s Collaborative Processor Performance Control, a feature that exposes to the OS kernel a ranking of cores (which has the highest boost, which has more cache, etc). Remember the X3D Turbo Mode I mentioned earlier? Same concept: to help the OS pick which core to send each program to so it runs better.

Not every motherboard firmware exposes this as an option; in the case of my Gigabyte X670E, it doesn’t exist. That probably means it’s already on by default. To check if it’s on, just go to the Linux terminal and run:

cat /sys/devices/system/cpu/cpu*/cpufreq/amd_pstate_highest_perf

If it returns any result, it means it’s on and the firmware is informing the kernel about the characteristics of each core, so the scheduler has more information to manage where to send each thread.

Another way to check:

ls /sys/devices/system/cpu/cpu0/cpufreq/

It needs to return a list containing names like these:

amd_pstate_highest_perf
amd_pstate_nominal_perf
amd_pstate_preferred_core

Which, again, is a sign that Linux is recognizing CPPC properly.

Manjaro itself, besides the stable kernels, has the “linux-rt” versions which are for real-time systems. Technically, this is meant not just for lower latency, but for stable and more predictable latencies. DO NOT USE THIS KIND OF KERNEL. This is for factory systems or things like that which need processing determinism and have no tolerance for random latency. It doesn’t mean it’ll run more responsively; it might even run slower, but the important thing is that it’s a predictable slow.

Others might mention the Linux Liquorix kernel, which many compare to Zen. But while Zen is based on the same kernel Arch uses, Liquorix’s is based on the Debian/Ubuntu version. So it’s not recommended on Arch/Manjaro. On top of that it’s more experimental and theoretically less stable too. Use it if you’re on Ubuntu or derivatives. On Arch, better stick with Zen.

check_ryzen_perf.sh

If you have a Ryzen system, copy the following script and run it on your machine:

#!/bin/bash

echo "🔍 Verifying Ryzen 7950X3D Tuning on Linux"

echo -n "✅ CPU Driver: "
grep . /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver

echo -n "✅ EPP Mode: "
cat /sys/devices/system/cpu/cpu0/cpufreq/energy_performance_preference

echo -n "✅ CPPC Detected: "
if [ -f /sys/devices/system/cpu/cpu0/cpufreq/amd_pstate_highest_perf ]; then
  echo "Yes"
else
  echo "No"
fi

echo -n "✅ Preferred Core Ranking: "
if [ -f /sys/devices/system/cpu/cpu0/cpufreq/amd_pstate_prefcore_ranking ]; then
  echo "Yes"
else
  echo "No"
fi

echo -n "✅ Running Kernel: "
uname -r

echo -e "\n✅ All key Linux-side Ryzen tuning options appear active.\n"

On my machine it now produces this result:

🔍 Verifying Ryzen 7950X3D Tuning on Linux
✅ CPU Driver: amd-pstate-epp
✅ EPP Mode: performance
✅ CPPC Detected: Yes
✅ Preferred Core Ranking: Yes
✅ Running Kernel: 6.14.2-zen1-1-zen

✅ All key Linux-side Ryzen tuning options appear active.

Saying that, theoretically, all the performance bits seem to be active and working. If one of them isn’t, ask ChatGPT what to do for your setup.

Checking the BIOS Version

Many motherboard manufacturers offer software, for Windows, to automatically download and update the firmware. But in Gigabyte’s case, there’s no Linux support. To check which version is currently installed, without having to enter the BIOS, on Linux there’s this command:

sudo dmidecode -s bios-version

In my case, it now returns F33 (March 2025 version). Now, to compare whether that’s the newest version, I just have to manually go to Gigabyte’s crappy site, which in my case is this link. For your specific model, and manufacturer of course, this link will vary.

To make it worse, it seems the content is loaded asynchronously by javascript, so if you try to use wget or curl from the command line, the content won’t have loaded yet.

I decided to write a dumb script to solve that. First, in case you don’t use mise like I do (see my Github project Omakub-MJ for a setup just like mine), install Node manually like this:

sudo pacman -S nodejs npm

Create some directory for the project and from there do:

npm init -y
npm install -g playwright
npx playwright install

Create a file check_bios_playwright.js with this content:

const { chromium } = require('playwright');

(async () => {
  const browser = await chromium.launch({ headless: true });
  const page = await browser.newPage();

  // Load support page
  await page.goto('https://www.gigabyte.com/Motherboard/X670E-AORUS-XTREME-rev-10/support', { waitUntil: 'domcontentloaded' });

  // Click the BIOS tab manually (it triggers JS to fetch content)
  await page.click('id=bios-count');

  // Wait for BIOS version element to load
  await page.waitForSelector('.div-table-body-BIOS .download-version', { timeout: 60000 });

  // Grab the first BIOS version listed
  const biosVersion = await page.$eval('.div-table-body-BIOS .download-version', el => el.innerText.trim());

  console.log('🌐 Latest BIOS Version on Gigabyte site:', biosVersion);

  await browser.close();
})();

Now create a bash script check_bios_update.sh with this content:

#!/bin/bash

# Get directory of this script (absolute path)
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

# Get current BIOS version
CURRENT=$(sudo dmidecode -s bios-version | tr -d '[:space:]' | tr '[:lower:]' '[:upper:]')

# Run the Playwright JS script in the same directory
LATEST=$(node "$SCRIPT_DIR/check_bios_playwright.js" | grep -o 'F[0-9]\+')

# Output
echo "🖥  Current BIOS: $CURRENT"
echo "🌐 Latest BIOS:  $LATEST"

if [[ "$CURRENT" == "$LATEST" ]]; then
  echo "✅ You are up to date."
else
  echo "⬆  BIOS update available!"
fi

Do a chmod +x to make it executable and, if all goes well, you’ll get a response like this:

[sudo] password for akitaonrails:
🖥  Current BIOS: F33
🌐 Latest BIOS:  F33
✅ You are up to date.

It’s a cannon shot to kill a fly, but this will use headless Chromium to pull the page, load the javascript, and then use CSS selectors to find the BIOS version, compare it to the one on your machine, and see if it’s up to date.

If it isn’t, there’s no way around it: you have to manually go to the site, download the ZIP file, extract it onto a pen drive, and boot into the BIOS to update manually. But at least I can quickly check from time to time with one command line now.

Automatic BIOS Version Check

I decided I don’t want to remember to run that script manually, so the best thing is to create a user systemd service and check automatically every week. If there’s a new version, create a notification to GNOME so I can see it visually and automatically open the download page for me.

First, I need to give my user access to the dmidecode command to check my current BIOS version without having to type a password with sudo. To do that, let’s create a new sudoers rule like this:

sudo visudo -f /etc/sudoers.d/bios-check

It’ll open an empty file where I fill in like this:

sudo /usr/sbin/dmidecode -s bios-version

And don’t forget to set the correct permissions on this file:

sudo chmod 0440 /etc/sudoers.d/bios-check

With that it won’t ask for a password to run that command anymore, since we’re going to put it in a background service running automatically. Obviously it’s not safe to allow running sudo without a password, so we restrict it exclusively to allow it only on that specific binary and nothing else.

In the same directory as the other files with playwright installed and so on, I create a bios-update-check.sh with this content:

#!/bin/bash

# Path to your Playwright BIOS scraper script
JS_CHECKER="$HOME/bios-check/check_bios_playwright.js"
NODE_BIN="$(which node)"

# URL of your motherboard's BIOS support page
URL="https://www.gigabyte.com/Motherboard/X670E-AORUS-XTREME-rev-10/support#support-dl-bios"

# Get current BIOS version
CURRENT=$(sudo dmidecode -s bios-version | tr -d '[:space:]' | tr '[:lower:]' '[:upper:]')

# Get latest BIOS version from Gigabyte site
LATEST=$($NODE_BIN "$JS_CHECKER" | grep -o 'F[0-9]\+' | head -1)

if [[ -z "$LATEST" ]]; then
  notify-send "BIOS Check" "⚠️ Failed to fetch latest BIOS version from Gigabyte"
  exit 1
fi

if [[ "$CURRENT" != "$LATEST" ]]; then
  notify-send -u normal "🔔 BIOS Update Available!" "Current: $CURRENT → Latest: $LATEST

Opening the download page..." --app-name="BIOS Checker"
  xdg-open "$URL"
fi

It’s the same as the previous script but now using notify-send to send messages to Gnome (in KDE it’s different) and using xdg-open to open the URL in the system’s default browser. Configure differently for your system. Don’t forget to make it executable and test:

chmod +x ~/.local/bin/bios-update-check.sh

Now to create the systemd service for my user (different from creating one for the system that runs with more permissions. I only need this running at my user level), so I create the file ~/.config/systemd/user/bios-check.service with the following content:

[Unit]
Description=Weekly BIOS version checker

[Service]
Type=oneshot
ExecStart=/home/akitaonrails/.local/bin/bios-check/bios-update-check.sh

Now the timer service to run every week, called ~/.config/systemd/user/bios-check.timer:

[Unit]
Description=Run BIOS version check weekly

[Timer]
OnCalendar=weekly
Persistent=true

[Install]
WantedBy=default.target

And enable the timer:

systemctl --user daemon-reexec
systemctl --user daemon-reload
systemctl --user enable --now bios-check.timer

By Systemd convention, a timer [blabla].timer will execute the service with the same name [blabla].service. That’s why you need both. If you enable the service directly, it’ll run only once and exit. To know if it’s enabled correctly do the following:

✦ ❯ systemctl --user list-timers

NEXT                          LEFT LAST PASSED UNIT             ACTIVATES
Mon 2025-04-21 00:00:00 -03 2 days -         - bios-check.timer bios-check.service

1 timers listed.
Pass --all to see loaded but inactive timers, too.

And that’s how you can automate some things on your system and get more out of your Gnome.

Conclusion

I only started this journey this time because it occurred to me that it had been months since I’d last updated the motherboard firmware, and then I took the chance to tune some things I hadn’t done before. Things like PBO were on automatic (so I was leaving performance behind; I remember that last time it had been unstable but I didn’t dig in to find out why), things like XMP/EXPO were already on. But now I think I really enabled everything I could.

It had also been years since I tried out the Linux-Zen kernel, so I took the chance to test it now. And as I said, so far it’s stable. Since I’ve started learning Blender and pulling more CPU and memory, I figured it was worth spending time trying to extract the most from my machine. 3D modeling is totally CPU/RAM, more than GPU.

Oh yes, speaking of Blender, one last thing. Remember my Ryzen has 1 CCD, which is that 1 die with 96MB of L3 cache. The best thing to do is create affinity for Blender with that die and prevent it from landing on the core with only 32MB of L3 cache.

On Linux, to do that, here’s the command line:

taskset -c 0-15 blender-4.3 &

The taskset command will hint to the scheduler that blender should stay on threads 0 to 15 (where the V-Cache is) and not 16 to 31 (which only have 32MB). That way, in theory, I’m giving Blender the best chances to run at the best possible performance on this machine. Less important programs like the browser, the terminal, if they land on the 32MB core, no problem.

I do the same thing in libvirt/qemu when I run emulated Windows: I configure libvirt to bring up the VM tied to the 96MB CCD and leave the other core for the Linux host. Speaking of BIOS, by the way, to run VMs you have to enable SVM and IOMMU, which usually ship disabled because virtual machine support steals a bit of the CPU’s performance. Maximum performance is with them disabled. If you’re not going to use VMs, leave them off.

And if anyone noticed that I’m explicitly running Blender 4.3 instead of the newer Blender 4.4, it’s because some addons still don’t work on the newer version, so it’s better to stay one version back to keep more compatibility with addons that don’t update fast.

The real “Year of Linux on the Desktop” is still not here. It’s getting closer as I will explain.

TL;DR this won’t be ‘yet-another-WSL-install-blog-post’, there are already several out there, for example this one or this other one. And you can always Google it.

Back in early 2016 Microsoft announced and launched a very alpha version of their current Windows Subsystem for Linux (WSL). It booted and ran some software, but it was awfully buggy. Fair enough as it was still an Insider preview only (by the way, avoid Insider Preview builds, they BSOD all the time as it’s in active development).

A year has passed, and we got to the Windows 10 Anniversary Update and, finally, the much-anticipated Creators Update. This is very polished in comparison.

If you’re unaware of what the WSL is, I highly recommend that you read the articles at Microsoft’s Developer Blog and Jessie Frazelle’s awesome Windows for Linux Nerds blog post. In a nutshell, WSL wraps the Windows kernel system calls and maps them as Linux Kernel system calls, fooling ELF64 unmodified Linux compatible binaries to run as if they were on a real Linux-based system. The kicker is that there is no Linux kernel running! It also wraps Windows services such as filesystem, networking, into appropriate devices that Linux can access.

So, it’s not any kind of virtualization. There is no Hyper-V involved. It’s enabling Windows to boot ELF64 binaries up directly, without recompilation, without modification.

Unlike in OS X where we need to tweak source code and recompile to run on Darwin (the Mach/BSD-based operating system layer), we can download Ubuntu binaries (or any other distro for that matter, there are Fedora and Suse coming soon to the Windows App Store) and run them directly.

Even more interesting, you can run system level daemons (yes, Postgresql, Memcached, Redis, etc all load and run in the background) and you can even run X (GUI) programs!

Don’t get me wrong, I commend everybody that is working hard at Microsoft to make this happen. It is a technical tour de force in itself.

Take a good look at the screenshot above. This is a sight to behold: native GNOME-VIM running natively side-by-side with Microsoft Edge loading a web site running out of a Rails app running within WSL, accessing PostgreSQL.

But, is it good for real people?

How good is WSL?

TL;DR it’s not nearly as smooth as running a pure Linux-distro, and the integration is not nearly as good as OS X in terms of usability as Windows was not redesigned in any meaningful way to be a UNIX system like OS X.

In an ideal world, Windows should replace its entire NT underpinnings and either build their own BSD-inspired infrastructure (like Darwin) or create their own Linux distro using off-the-shelf Linux kernel as the main driver (probably can’t because of GPL).

But this is unrealistic. Apple could do it in a time when the Mac installed base was tiny, so alienating some people wouldn’t hurt so much. Windows is just too big to fail. It is and will remain a maintenance nightmare of epic proportions.

That being said, WSL in itself is a miracle, so I can’t complain. Microsoft had a really harsh decade in the turn of the century. Windows 10 is a return to form, and 2 big iterations later, it’s finally becoming reasonable again. Maybe we’re a couple of upgrades away from a really usable WSL. Remains to be seen.

Apple had its miracle in the turn of the century, and OS X and iOS are terrific OSes. Best of the breed. And by being able to reboot the entire ecosystem from scratch is not something anyone can do.

Linux-distros had their own ups and downs. No ecosystem can evolve healthily and in a cohesive way being that fragmented. But each major distro was able to brute force their own ecosystems. Canonical has prevailed where many failed but they are still not a definitive winner, and it feels like they’re losing their steam. RedHat enjoyed the craze of the dot-com era and was able to fly high in the Enterprise market. There is no “best” distro. It’s again, a matter of taste and ego, most of the time for most of the people.

And the OSS community still wastes an insurmountable amount of time in crazy disputes over stuff like the Systemd controversy or the time it is taking to get over X and finally get into Wayland. OSS has pros and cons, just like everything else.

But I digress, going back to WSL. Yes, as promised almost anything you can do in a native Ubuntu 16.04 can be done in this new environment. Even the ridiculous cmd.exe was heavily upgraded to the 21st century. Still nowhere near a default Terminal on Linux or OS X, but getting close. Actually, I’d recommend installing WSL Terminal as a replacement, and despite the terrible “Bash on Ubuntu on Windows” moniker, I’d recommend installing ZSH instead and launching it over Bash for extra usability features.

You can Google around for endless tutorials on how to customize your environment, Vim, tweak PostgreSQL and so forth. So again I am not repeating those here. A caveat is that I have to sudo /etc/init.d/PostgreSQL start to actually run daemons. They won’t autoload on a restart, keep that in mind.

And now you can install a Windows X Server such as xming and export DISPLAY=:0 and you now sudo apt install gvim and load the graphical version of Gvim, for instance. Or a full GNOME or XFCE desktop if that’s your thing. You can run X based applications side-by-side to Windows apps, and that’s super neat. I like to install YADR which is one of those super customized dotfiles collection and it works just fine.

I am now using ASDF to install and manage multiple versions of Ruby, Node.js, Elixir, etc. And it also just works.

I was able to move my SSH keys from my previous Arch Linux install. And then I can clone from my Git repositories. Now, here comes the big caveat. I cloned and configured a Rails project. This is the fastest I can get running an entire RSpec suite with PhantomJS acceptance tests:

Finished in 5 minutes 35 seconds (files took 48.53 seconds to load)
888 examples, 0 failures

This is running on my Dell XPS 9550 with Core i7 Skylake (4 cores, 8 threads), 16 GB of RAM and NVME2 SSD. I asked a co-worker to run it on his very old Mac Mini 2012 with Core i5 (probably Haswell), 10 GB of RAM, normal SSD. And this is his test run:

Finished in 3 minutes 37.6 seconds (files took 9.73 seconds to load)
888 examples, 0 failures

Yep, this is shocking. The filesystem emulation is notoriously slow under WSL (at least 4 times slower as you can see in the numbers above).

This other test ran in a slightly newer hardware (still underpowered compared to my Dell XPS) running on Linux Mint:

Finished in 3 minutes 2.4 seconds (files took 22.69 seconds to load)
888 examples, 1 failure

I ran the same suite a couple of times to make sure, but it can’t go below the 5 minute mark and loading files is always around 1 minute while in other OSes it’s 2 to 4 times faster.

If your development workflow is I/O bound, you will suffer. Webpack compilation, which is slow, gets even slower. Npm/Yarn installs take forever. Installing stuff from source is super slow.

It’s not unusable, but this slowness will get old fast.

What about Virtualization?

After I saw the dismal filesystem performance shown above, I wondered “and what about running everything under good old Virtualbox?”

In theory, WSL “should” be a thinner layer than Virtualbox, but the worst offender in terms of overhead seems to be the filesystem abstraction layer, and not necessarily access to CPU or RAM. My instinct would first suggest that a Virtual Drive, which is a single file mounted as a virtual hard drive, would perform much better.

So here I go, installing Virtualbox and the same set of tools (asdf, etc).

Mounting a shared folder directly to the one in WSL doesn’t work properly, seems like the mounting protocol doesn’t understand dot files, so they don’t show up. One has to tar the folder and untar within the machine to preserve hidden files and permissions.

This is running the same Rspec suite over Virtualbox with 4 Cores (out of the 8 my machine has) and 4GB of RAM:

Finished in 2 minutes 3.4 seconds (files took 6.33 seconds to load)
888 examples, 2 failures

So, we’re down from 5:35 min to 2:03 min and file loading from 49 seconds down to 6 seconds! So around 2.5 times faster execution and a whopping 8 times increase in file loading! This is ridiculous and it makes way more sense. My 2016 machine, within Virtualbox, still outperforms 2011/2012 bare hardware. And WSL is in the bottom of the list.

Forget synthetic benchmarks, this is a real-world case mixing CPU and I/O bound executions. Just for the sake of comparison, I tried to see if I could squeeze a bit more performance if I try to run over VMWare Workstation. This is the result:

Finished in 2 minutes 3.3 seconds (files took 6.21 seconds to load)

Almost identical. To me, it says that Virtualbox came a long way in being comparable to commercial, historically more robust solutions such as VMware or Parallels, so I will stick with Virtualbox for the time being.

I tried another test. This is asdf installing the brand new Ruby 2.4.2 from source. First over WSL (running with time):

229.38s user 244.59s system 113% CPU 6:57.51 total

And the same install from Virtualbox:

295,39s user 22,43s system 135% CPU 3:53,87 total

Again, a mixed scenario of CPU and I/O bound executions. And I am at least twice as fast on Virtualbox.

If you do want to see more controlled scenarios running synthetic benchmark comparison, I recommend you read Phoronix’s take on WSL vs Virtualbox

The “Year of Linux on the Desktop” Conundrum

The Lack of commercial software is one problem but one of the reasons is the lack of backward binary compatibility.

For all its problems, Microsoft remained committed to a very difficult concept: you can actually run unmodified binaries from Windows 95 in the current Windows 10. Most of the time they “just work”. In Linux distros, every binary is compiled against very specific library headers such as the kernel itself, Glibc and many others. Whenever one of those change, all binaries must be recompiled to run. Which is why you have the concept of “distros” where they recompile and test a whole set of software. Long-Term Support (LTS) is basically locking those dependencies versions for a couple of years to have software running without the need to recompile so much. But in bleeding-edge distros, you can’t avoid having to recompile a lot of things many times over.

Windows carries the old dependency binaries around. OS X does the same to a degree. And OS X goes one step further by providing the concept of “fat binaries” (or “Universal Binaries”) where you have a package with multiple binaries that target different hardware architectures (32-bit vs 64 bit, Intel vs ARM vs PowerPC).

Mostly because of this support, commercial complex applications like Office or Adobe’s Creative Suite can run on many different machines, across many different OS versions, over many years without upgrading. In Linux distros, it’s a major hassle for binaries. Which is why having source code available is so important and it’s a philosophical discussion over the whole “Free Software” concept, as envisioned by Richard Stallman.

This is an unsolvable dilemma. Linux distros don’t have binary backward compatibility out of the box. Developers must rely on tools such as the ABI Compliance Checker. Try to use symbol versioning from glibc and so forth, but it’s not realistic to expect volunteers collaborating their free time to open source to keep maintaining old versions of software for many years down the road. It is a fair compromise. Do you want the best, faster, most secure? Upgrade often, but expect old, unmaintained software to break.

So, without releasing the source code, you can’t have good compatible binaries for every single major distribution, across many different versions going back many years.

Which is right for you?

There is no point in trying to benchmark OSes. They are all generally good enough.

Macs are more expensive (sometimes by a whopping 1 grand or more). There is no denying that. The compromise is that Apple’s hardware, OS X and the ecosystem are generally the better all around, and the " fit and finish" of the products are really usually better from a user experience perspective.

For the best mix of mainstream software and open source software, you won’t find a better alternative. If you can afford a Mac, get a Mac.

Gamers have to stick with Windows. Again, there is no denying that. The best games are on Windows, period. The best combination of hardware supports Windows. No Linux tweaking will give you the raw juice you need to squeeze out 4K 60fps out of the newest AMD and NVIDIA cards through DirectX. And the Mac hardware is capped, and you can’t add the newest GPUs. If you want to play the latest AAA games the way they were envisioned to perform, get custom hardware and Windows 10.

Web Developers are luckier. You can’t go wrong, it’s literally a matter of taste which OS you choose. You will be able to deliver the same results using the same tools. And unless you’re targeting specific Microsoft infrastructure (SQL Server, Active Directory, etc), you can install any Linux distro and rejoice.

Desktop App Developers must target their audience. If you want to build Windows desktop apps, you have to be on Windows. You can use .NET/Xamarin or Java to build cross-platform compatible binaries, though. But you guys are out of luck because ideally, you have to have all 3 to ensure your software is performing as you want.

Mobile App Developers are also out of luck. You can do Android on Windows or Linux, but you have to have a Mac to develop for the Apple App Store. You can’t escape that, and good luck waiting for Apple to release their tools for other platforms.

Creative Professionals can choose either Windows or OS X. That depends on the tooling. If you’re an Adobe guy, either will do just fine, but you will probably want a custom workstation. Until Apple remodels the ridiculous Mac Pro, you won’t find the best hardware for the buck at Apple. The current iMac 5K is good, but higher than that is not available. For example, if you need the power of 2 NVIDIA Titan or Quadro cards running in parallel, Apple can’t help you there.

If you do require a Final Cut Pro, Logic Pro, Motion 5 workflow, the best option you will get is the iMac 5K.

I am not a full-time developer anymore. And I mix my time running a company, doing presentations, experimenting with tech and doing research. My ideal setup is actually a MacBook Pro. But I am forcing myself to live out of the Apple ecosystems, and it hurts. My web developer side can stay in Linux (Manjaro GNOME is my favorite nowadays) and be happy. But when my Creative side needs tooling, Gimp, Inkscape, Kdenlive, are just not good enough. I am too used to Photoshop, Illustrator, Final Cut/Premiere, Motion 5/After Effects. LibreOffice Impress is terrible to do complex presentations, and those HTML/JS based slides software are good just for simple stuff. Try to do serious graphics and animation there and you will suffer in hell.

If you’re a Creative, you need Color Calibrated screens (Adobe RGB space), you need the raw processing power to do a quick render and advanced post-processing. Adobe video editors will prefer Windows. Final Cut video editors are well served on Macs, even though the new MacBook Pro’s cooling system is terrible:

And if you’re a full-time video editor, you can afford either high-end PC or Mac configurations, so there is not much dispute here.

If you’re a video editor on a budget, you will not have a good 4K editing workflow on Linux, period. There is poor support to unlock the full potential of modern GPUs, no good Cuda or OpenCL support, and most video editing tools don’t use them. Blender - which is primarily a 3D modeling and animation tool - is considered one of the best video editing tools available on Linux, which says a lot. The sad state of GPU support on Linux compromises Creative usage. The only other non-linear video editing tool that can achieve professional level workflows is LightWorks and it’s not open source, requires the proprietary NVIDIA binaries, and will cost you.

If you’re going that way, it’s probably better to stick with either Windows or OS X at this point.

Conclusion

I digressed quite a bit, but my conclusion is simple:

• WSL is not there yet. We have to wait another couple upgrades. When you hear that they improved the filesystem abstraction by a few orders of magnitude, then we can try again. And keep in mind that there are tons of open issues, so your mileage will vary.
• If you’re a Web Developer, you’re safe on any Linux distro.
• If you’re a mixed professional in need of both development toolkits and creative tools, use a Mac or Windows 10 with a fully virtualized Linux distro.
• If you’re a Gamer, Windows.
• If you’re a full-time creative professional, either Macs or PCs will do just fine.

For me, I will stick with my Dell XPS running Windows 10 and Manjaro GNOME over Virtualbox. Virtualbox is fast enough that it won’t hurt my workflow. And the creative tools can appreciate the access to my NVIDIA Cuda cores without any problem, so I can render with Blender and make Premiere render 4K video quickly enough.

José Valim kindly stepped in and posted a valuable comment, which I will paste here:

Have you tried reducing the @max_demand instead? @max_demand is how much data you exchange between stages. If you set it to 60, it means you are sending 60 items to one stage, 60 items to the other and so on. That gives you poor balancing for small collections as there is a chance all items end-up in the same stage. You actually want to reduce max_demand to 1 or 2 so each stage gets small batches and request more than needed. Another parameter you usually tune is the stages: ... option, you should set that to the amount of connections you had in poolboy in the past.

However, I believe you don’t need to use Flow at all. Elixir v1.4 has something called Task.async_stream which is a mixture of poolboy + task async, which is a definitely better replacement to what you had. We have added it to Elixir after implementing Flow as we realized you can get a lot of mileage out of Task.async_stream without needing to jump to a full solution like Flow. If using Task.async_stream, the max_concurrency option controls your pool size.

And, obviously, he is right. I misunderstood how Flow works. It’s meant to be used when you have a bazillion items to process in parallel. Particularly processing units that can have high variance and, hence, a lot of back-pressure not only because there is a lot of items to process, but because their running times can vary dramatically. So, it’s one of those cases of bringing a cannon to kill a fly.

What I wasn’t aware is the existence of Task.async_stream and its companion Task.Supervisor.async_stream if I need to add more control.

Let’s backtrack a bit.

Dealing with collections in Elixir

Erlang is a beast. It provides all the building blocks of a real-time, highly-concurrent, operating system! Really, what it provides out of the box is way more than any other language/platform/virtual machine, ever. You don’t get that much for free on Java, or .NET or anything. You have to assemble the pieces manually, spend hundreds of hours tweaking, and still pray a lot to have everything working seamlessly.

So, you have distributed systems to build? There is no other option, really. Do Erlang, or suffer in hell.

Then, Elixir steps this up a notch creating a very reasonable and simple to use standard library that makes the coding part actually enjoyable. This is a killer combo. You need to do the next Whatsapp? You need to do the next Waze? You need to rebuild Cassandra from scratch? You need to create stuff like Apache Spark? Do Elixir.

In Erlang, you need to solve everything using GenServer. It’s a neat abstraction from OTP. You are required to understand OTP intimately. There is no shortcut here. There is no Erlang without OTP.

That said, you can start simple and scale without so much hassle.

Usually, everything starts with Collections, or more correctly, some kind of Enumeration.

Just like my simple Workflow.pages/1 function which iterates through a list of chapter links, fetch each link, parse the returning HTML and extracts the collection of page links within that chapter, reducing the sub-lists into a full list of page links.

If I know the collection is small (less than 100 items, for example) I would just do this:

def pages({chapter_list, source}) do
   pages_list = chapter_list
     |> Enum.map(&Worker.chapter_page([&1, source]))
     |> Enum.reduce([], fn {:ok, list}, acc -> acc ++ list end)
   {pages_list, source}
end

And that’s it. This is linear. It will sequentially process just one link at a time. The more chapter links, the longer it will take. Usually I want to process this in parallel. But I can’t fire a parallel process for each chapter link, because if I receive 1,000 chapter links and fire them all, it will be a Denial of Service and I will certainly receive hundreds of time outs.

You can run into 2 main problems when you need to iterate through a big collection.

• If your collection is humongous (imagine a gigabyte long text file that you need to iterate line by line). For that you use Stream instead of Enum. All functions look almost exactly the same, but you will not have to load the entire collection into memory and you will not keep duplicating it.

• If your processing unit takes a long time. Now that you solved not blowing off your memory usage, what if you have slow jobs while iterating through each item in the collection? That’s our case, where the collection is rather small, but the processing time is long as it’s fetching from an external source on the internet. It can take milisseconds, it can take a few seconds.

One way to control this is through the use of “batches”, something along these lines:

def pages({chapter_list, source}) do
  pages_list = chapter_list
	|> Enum.chunk(60)
	|> Enum.map(&pages_download(&1, source))
	|> Enum.concat
  {pages_list, source}
end

def pages_download(chapter_list, source)
   results = chapter_list
     |> Enum.map(&Task.async(fn -> Worker.chapter_page([&1, source]) end))
     |> Enum.map(&Task.await/1)
     |> Enum.reduce([], fn {:ok, list}, acc -> acc ++ list end)
   {pages_list, source}, results	
end

This is just for the example, I have not compiled this snippet to see if it works, but you get the idea of chunking the big list and processing each chunk through Task.async and Task.await.

Again, for small lists, this should be ok (thousands) and where each item does not take too much to process.

Now, this is not very good. Because each chunk must finish before the next chunk begins. Which is why the ideal solution is to keep a constant amount of jobs running at any given time. To that end, we need a Pool, which is what I explained in Part 2: Poolboy to the rescue!.

But implementing the proper way to keep the pool entirely filled requires some boring juggling between Poolboy transactions and Task.Supervisor.async. Which is why I was interested in the new Flow usage.

The code does come clean, but as I explained before, this is not the proper use case for Flow. It’s better when you have to iterate over tens of thousands of items or even an infinite stream (you have incoming traffic of requests in need of parallel processing, for example).

So, finally, there is a compromise. The solution between the simple Task.async and Flow is Task.async_stream which works like a pool implementation, where it keeps a max_concurrency of jobs running in a stream. The final code becomes way more elegant like this:

def pages({chapter_list, source}) do
  pages_list = chapter_list
    |> Task.async_stream(MangaWrapper, :chapter_page, [source], max_concurrency: @max_demand)
    |> Enum.to_list()
    |> Enum.reduce([], fn {:ok, {:ok, list}}, acc -> acc ++ list end)
  {pages_list, source}
end

And this is the final commit with the aforementioned changes.

Conclusion

The implementation with Task.async_stream is super simple and the times finally became the same as before.

84,16s user 20,80s system 138% cpu 1:15,94 total

Way better than the more than 3 minutes it was taking with Flow. And this is not because Flow is slow, it’s because I was not using it correctly, probably shooting a big chunk into a single GenStage and creating a bottleneck. Again, only use Flow if you have enough items to put hundreds of them into several parallel GenStages. We are talking about collections with tens of thousands of items, not my meager pages list.

There is a small tweak though. To fetch all chapter and page links I am using a max_concurrency of 100. Timeout is the default at 5000 (5 seconds). That works because the returning HTML is not so big and we can parallelize that much on a high bandwidth connection.

But the images downloading procedure at Workflow.process_downloads I had to cut max_concurrency in half and increase the timeout up to 30 seconds to make sure it wouldn’t crash.

Because this is a simple implementation there is no crash recovery and no retry routine. I would have to replace this implementation with Task.Supervisor.async_stream to regain some control. My original implementation was more cumbersome but I had places to add retry mechanisms. So again, it’s a compromise between ease of use and control, always. The more control you have, the worse the code becomes.

This is a simple example exercise, so I will keep it at that for the time being.

As a quick recap, the exercise is that I want to scrape MangaReader.net for a bunch of images, organized in pages and chapters, and in the end it should compile organized PDFs so I can load them on a Kindle device.

Web scraping is a simple loop of HTTP GETs over a ton of URLs, scraping the HTML, and fetching more URLs to download.

In many simple languages, people usually solve this naively in 2 ways:

• A simple nested loop. One single thread, sequential fetches. So if you have 5,000 links and each links takes 10 seconds to fetch, it’s basically 10 * 5,000 = 50,000 seconds, which is a stupid long time.
• A simple spawn of processes, fibers, threads or parallel I/O in a reactor, all at once. An attempt to parallelize all fetches at once.

Everybody probably agrees that the first option is stupid. Now, the second one is tricky.

The tricky part is CONTROL.

Anyone in Go would say “oh, this is easy, just put a loop and spawn a bunch of goroutines” or anyone in Node.js would say “oh, this is easy, just put a loop, make the fetch - they all will run asynchronously - and add callbacks, a basic async/await.”

They’re not wrong, but this is still too naive an implementation. It’s trivial to trigger hundreds or thousands of parallel requests. Now, what happens if one fails and you have to retry? What happens if the MangaReader has a throttling system that will either start cutting down connections or timing them out? Or if your internet bandwidth is just not enough, and after a certain amount of requests you start having diminishing returns and time outs?

The message is: it’s damn trivial to spawn parallel stuff. it’s damn complicated to control parallel stuff.

This is why, in my first implementation in Elixir, I introduced a complicated implementation using a combination of a custom GenServer, Elixir’s own Task infrastructure for async/await pattern, and Poolboy to control the rate of the parallelism. This is how you control the bottleneck out to slow resources: using pools and queues (which is why every good database has a connection pool, remember C3P0?)

This is one snippet of my older implementation:

def chapter_page([chapter_link, source]) do
  Task.Supervisor.async(Fetcher.TaskSupervisor, fn ->
    :poolboy.transaction :worker_pool, fn(server) ->
      GenServer.call(server, {:chapter_page, chapter_link, source}, @genserver_call_timeout)
    end, @task_async_timeout
  end)
end

Yes, it’s very ugly, and there are boilerplates for the GenServer, the custom Supervisor to initialize Poolboy and so on. And the higher level workflow code looks like this:

def pages({chapter_list, source}) do
   pages_list = chapter_list
     |> Enum.map(&Worker.chapter_page([&1, source]))
     |> Enum.map(&Task.await(&1, @await_timeout_ms))
     |> Enum.reduce([], fn {:ok, list}, acc -> acc ++ list end)
   {pages_list, source}
end

So, inside the Worker module each public method wraps the GenServer internal calls into a Task async and in the collection iteration we add Task.await to actually wait for all parallel calls to finish, so we can finally reduce the results.

Elixir now comes with this very interesting GenStage infrastructure that promises to replace GenEvent and the use case is when you have a producer-consumer situation with back-pressure. Basically when you have slow endpoints and you would end up having to control bottlenecks.

Then, Flow is an easier high abstraction that you can use almost the same way you would use Enum in your collections, but instead of sequential mapping, it takes care of parallel traversing and control of batches. So the code is very similar but without you having to control the parallelization controls manually.

This is the full commit where I could remove Poolboy, remove my custom GenServer, reimplement the Worker as simple module of functions and then the workflow could get rid off the async/await pattern and use Flow instead:

def pages({chapter_list, source}) do
   pages_list = chapter_list
     |> Flow.from_enumerable(max_demand: @max_demand)
     |> Flow.map(&MangaWrapper.chapter_page([&1, source]))
     |> Flow.partition()
     |> Flow.reduce(fn -> [] end, fn {:ok, list}, acc -> acc ++ list end)
     |> Enum.to_list()
   {pages_list, source}
end

The only boilerplate left is the Flow.from_enumerable() and Flow.partition() wrapping the Flow.map, and that’s it!

Notice I configured @max_demand to be 60. You must tweak it to be larger or smaller depending on your internet connection, you have to experiment it. By default, Flow will trigger 500 processes in parallel, which is way too much for web scraping on a normal home internet connection and you will suffer diminishing returns. That’s what I had to do previously with Poolboy, by initiating a pool of around 60 transactions at most.

Unfortunately not everything is as straight forward as it seems. Running this new version on the test mode I get this result:

58,85s user 13,93s system 37% cpu 3:13,78 total

So a total time of more than 3 minutes, using around 37% of the available CPU.

My immediate previous version using all the shenanigans of Poolboy, Task.Supervisor, GenServer, etc still gives me this:

100,67s user 20,83s system 152% cpu 1:19,92 total

Less than HALF the time, albeit using all my CPU cores. So my custom implementation still uses my resources to the maximum. There is still something in the Flow implementation I didn’t quite get right. I already tried to bump up the max_demand from 60 up to 100 but that didn’t improve anything. Leaving it to the default 500 slows everything down to more than 7 minutes.

All in all, at least it makes the implementation far easier on the eyes (hence, way easier to maintain), but either the Flow implementation has bottlenecks or I am using it wrong at this point. If you know what it is, let me know in the comments section below.

There are some things you must have in mind when migrating to Linux systems.

Even if you have a top of the line Intel Core i7 Kaby Lake, 32GB of RAM, 2TB M.2 SSDs, you may still benefit from the optimizations I will talk about.

One of the best resources is Arch Linux’s Wiki page on “Improving Performance”. You don’t need everything there but it’s a comprehensive resource that any enthusiast must read.

It’s about Responsiveness, not performance!

Most people’s concerns are first about performance, and this is a mistake. Linux is plenty fast, but for many people it doesn’t feel that way on the Desktop.

When tuning a server, you’ll really want to tweak for performance and high throughput. That’s where most Linux configurations really shine over the competition: they come better tuned to get the most out of server configurations.

But in a Desktop you don’t want that. For example, you’re copying a 20GB file to your old USB thumb drive, or you’re unzipping a large file, or you’re compiling that large package from source, or you leave Dropbox in the background syncing gigabytes of files from their servers. Or you’re doing “nothing” (in the foreground at least, but Gnome Tracker is heavily indexing your files in the background) and your environment stutters, hangs for a few seconds, and keeps doing that every so often.

And you’re left wondering why Linux is so bad compared to Windows or macOS where you don’t see the same behavior on similar hardware.

There is a term that is still misunderstood: real-time.

Being real-time does not mean “computing super fast”, it means “being predictable”. If something needs to happen in a certain frequency, it doesn’t matter if each cycle takes 1 second as long as it consistently takes the same 1 second - in all deadlines. If you have a “fast” machine that computes faster at 10 milliseconds every cycle, but every so often, randomly hangs for a couple of seconds, this is not real-time. And for media creation, it’s a disaster.

There is hard real-time where one peak or valley can be considered a catastrophic failure, and soft real-time where you can handle a few peaks, but not so much. Hard real time requirements are rare, unless you’re developing systems for nuclear plants you may miss a few deadlines.

Most responsiveness issues are related to soft real-time situations. You can handle a few sparse peaks here and there, but no more than a few. And this is how you should do your research: not Googling for “linux performance” but for “linux real-time” or “linux responsiveness”.

You will also find that there are niche professionals with special distros just for audio recording and editing, for example AVLinux or KXStudio.

macOS is particularly good for media creators precisely because it’s highly tuned for soft real-time, which is critical for software such as Logic Pro. And for the same reason, it’s a poor server OS. You will notice that the default Quicktime screen recording is super smooth, you rarely see stutters.

But you don’t need to use an audio specific distro or a hard real-time kernel. Critical audio distros don’t use PulseAudio, but normal users will not be so concerned about it. We can tune it to find a good balance between responsiveness and performance. If you really want to go hard-core, you may want to read the Linux Audio Wiki on Real Time, but it’s out of the scope of this article.

What are the real Bottlenecks?

A “slow” computer is not necessarily sporting old CPUs. I am doing my tests on a very old Lenovo Thinkcentre Edge 71z tower desktop. It has an old 2nd generation Intel Core 2.4Ghz 4 cores SandyBridge CPU. We just saw the release of the 7th generation Kaby Lake processors, so one might assume that nothing would run on such an old CPU, but you would be wrong.

CPU is usually not a serious bottleneck unless you’re doing really intensive computation, such as video compression, data sciences, genetics, neural networks, etc.

For a casual user or even a heavy-weight developer, any processor better than the 1st generation Intel Core series is plenty.

GPU is also rarely a bottleneck unless you’re doing heavy gaming or 4K renderings. Most of the time you don’t really need a USD 7000 dedicated GTX 1080.

By the way, this is not necessary for most systems, but just to be on the safe side do this:

sudo pacman -S mesa-demos
glxinfo | grep direct

You should see direct rendering: Yes. If not, refer to your distro documentation, because this means you’re not compositing through the GPU and you’re wasting CPU cycles rendering your screen!

If you try to measure your CPU and GPU usages, you will realize that most of the time they are actually idle! That’s right, you’re mostly underusing your machine cores.

The bottleneck usually boils down to I/O.

RAM vs SWAP

Now, you’re opening your shiny Chromium browser. Anyone fooling around for a few minutes will open an average of a dozen or more tabs, without breaking a sweat.

It’s super easy to eat up all 8GB of the average machine. Whenever that happens, the OS will have to start offloading data to disk, which is orders of magnitude slower.

If application data is offloaded to disk and you alt-tab to it later, the OS will reach a “page fault”, and it will have to load from disk, from the swap file/partition. And again, this will have the effect of blocking your actions. The environment may stutter for a second or more, making it unresponsive.

The very first thing you may want to do is install an extension such as The Great Suspender. It will simply close all tabs but the one you’re reading right now. When you change to another tab it will reload it. The effect is that you’re not using RAM if you really don’t need to.

This extension alone can save you a couple of GIGABYTES of RAM, which is no small thing if you have 8GB or less.

The other thing to consider is that Linux comes pre-configured to balance out offloading application data to swap to accomodate filesystem cache. So, if you’re unzipping a large file, some of that data will go to RAM cache and application data will move to the disk. After you finish unzipping, you alt-tab to applications and boom: page faults, unresponsiveness.

So you want to configure the OS to more aggressively keep your application state in RAM, and this is how you do it:

sudo tee -a /etc/sysctl.d/99-sysctl.conf <<-EOF
vm.swappiness=1
vm.vfs_cache_pressure=50
EOF

While on the topic of storage, you will find some older kernels making your machine become unresponsive when dealing with slower storage, such as USB drives or SD cards. This is how you tweak it:

sudo tee -a /etc/sysctl.d/99-sysctl.conf <<-EOF
vm.dirty_background_bytes=16777216
vm.dirty_bytes=50331648
EOF

If you don’t want to reboot right now, you can run this in a Terminal:

sudo sysctl -w vm.swappiness=1
sudo sysctl -w vm.vfs_cache_pressure=50
sudo sysctl -w vm.dirty_background_bytes=16777216 
sudo sysctl -w vm.dirty_bytes=50331648

Don’t go too far on tuning, for example, never disable a file system journaling. It increases performance at the risk of putting your data at risk of corruption.

Schedulers

Why was “Linux not ready for the Desktop” years ago?

Because it took too many years to finally start tackling low latency, inexpensive thread switching, better scheduling. You have to thank Con Kolivas, Ingo Molnár, Thomas Gleixner. The Linux kernel development is known to be super difficult to deal with and Con Kolivas is one of its victims, but his work lives on to allow us to have better Desktop experiences these days.

There are Process Schedulers and I/O Schedulers. The first is responsible to manage the Preemption of the kernel, how it switches between different computational tasks, the low-level equivalent of you “alt-tabbing” through apps, so to speak.

I/O Schedulers deal with sharing slow I/O resources with competing processes needing to read from disk, write to RAM, etc.

The recent history of process scheduler for the Desktop boils down to Con Kolivas’ works on fair scheduling, leading to Ingo Molnár’s Completely Fair Scheduler (CFS) which is the default in most distros nowadays and the continuing work of Kolivas on Staircase, Rotating Staircase Deadline, Staircase Deadline, Brain Fuck Scheduler (BFS), and the most recent Multiple Queue Skiplist Scheduler (MuQSS).

Then there are I/O Schedulers. For the most part you will be dealing with the Completely Fair Queueing (CFQ). Most development on this side is attributed to Jens Axboe, also responsible for the Deadline Scheduler and Noop Scheduler. Then there is the controversial evolution called Budget Fair Queueing (BFQ).

When you have SSDs (and this is why you want SSDs), you will more likely choose NOOP (or Deadline) just because there is no need to waste computation time managing complex I/O queues for SSDs as they can easily handle up to tens of thousands of I/O operations concurrently without breaking a sweat.

But if you have to use mechanical hard-drives, particularly the old and super slow 5.400rpm ones, you will want to manage the I/O queue efficiently, touching the spinning plates as little as possible. And in this case, you will really want to use something like BFQ (or at least leave it at the default CFQ).

You can check which I/O Scheduler you’re running like this:

$ cat /sys/block/sda/queue/scheduler
noop deadline cfq [bfq] 

In the example above you will see that [bfq] is the one active, but you can change it on the fly to test it out if you want.

To take advantage of those newest schedulers to better optimize slow computers, your best bet is to install Linux Zen kernel, a version of Liquorix. It includes the MuQSS scheduler instead of CFS and BFQ instead of CFQ, while also adding more tweaks for responsiveness like proper QoS over TCP to avoid TCP congestion.

In Arch Linux it’s a simple thing to do:

sudo pacman -Sy linux-zen
sudo grub-mkconfig -o /boot/grub/grub.cfg

For Ubuntu, you may want to refer to Liquorix’s Install Page as it depends on your CPU, but most likely you will install on 64-bit machines:

sudo apt-get install liquorix-keyring
apt-get install linux-image-liquorix-amd64 linux-headers-liquorix-amd64

Is GNOME 3 too slow?

I always heard that GNOME and even KDE are too slow, you should just use XFCE (or LXQt, or MATE).

And it always striked me as one of those things people just keep repeating until it becomes the official canon.

As an engineer, I dislike thinking that way. Defying the canon is more like what an engineer should do.

GNOME 3.22 is an ever evolving environment and ecosystem. It’s good looking by default, no need to tweak it too much. And it has several built-in conveniences such as GNOME Online Accounts, Tracker for indexing files and making them easily searchable, GNOME Photos that sync from Google Photos, and so on. Every nicety we like about a system like macOS.

Convenience is a trade-off of performance and responsiveness. So high-end machines will benefit from the convenience and old machines will suffer because of the extra “bloat” in the background.

How do you maintain some of the convenience on old hardware?

Again, you must understand what’s going on. The first thing you must install is htop and iotop. The first is good to see what processes running in the background may be eating up your CPU or RAM. The second is good to see what processes are bloating your I/O queues doing background file/network operations.

What I found out in my system were 2 main offenders: Dropbox and Tracker.

Dropbox is optional, but most people nowadays use it. Out of the box it is a hideous monster, one of the worst pieces of software you’re forced to live with.

The first time you install and it has to download everything, your machine will go down to its knees. Nothing to do about that, just remember to install it Friday night and leave it running in the office over the weekend.

Then, edit the /usr/share/applications/dropbox.desktop and replace the Exec=dropbox line with this:

Exec=ionice -c 3 -n 7 dropbox start -i && cpulimit -b -e dropbox -l 10

This “should” tune down Dropbox to have the least amount of CPU time and only have I/O when the system is idle.

Another way is to install Ananicy. It is a daemon that promises to automatically set NICE and IOCLASS of selected processes just like using ionice and cpulimit above. You can install it in Arch like this:

sudo pacaur -S ananicy-git

And if you cat /etc/ananicy.d/dropbox.rules you will see a rule set like this:

# Dropbox client: https://www.dropbox.com
NAME=dropbox       NICE=19     IOCLASS=idle

Which is basically what we did in the Exec line tweak. I didn’t test Ananicy enough but if it does what’s promised, it’s even easier as it comes with pre-configured rules for applications such as make, VLC, transmission, etc.

Then, there is Tracker. The purpose of this tool is to index your files so you can search them fast and easily through GNOME applications such as the Nautilus File Manager.

Again, the first time you install your new GNOME environment it will be very expensive to build the first index pass, especially if you’re downloading tons of files from Dropbox. Do it on a Friday night!

But you should also tune it down to only run when your system is idle. Run Alt-F2 and type tracker-preferences, then configure it to look like this:

In the same applet, also configure it to ignore log directories and *.log file patterns!

Only those 2 things should make your machine WAY more responsive when using slow mechanical drives. I noticed that gnome-photos keeps running in the background and consuming some I/O, it is probably trying to sync your online photos from Google if you set GNOME Online Accounts.

Dropbox, Tracker, Gnome-Photos, will all cause your initial experience to suck. But if you have patience - and a fast internet connection - they should settle down after the initial heavy sync.

GNOME has other services in the background, namely:

gnome-session
gnome-shell
gnome-settings-daemon
gnome-online-accounts
evolution-data-server
gjs-console

There must be more depending on optional apps you installed. Gnome-Shell and GJS are easily the worst of the bunch. You can’t do much about them because they’re the Core of GNOME. GJS in particular enables Javascript-based extensions and everything Javascript is slow. The only thing you can do is avoid installing too many GNOME Extensions.

By the way, if you install XFCE alongside GNOME, you may be surprised that many of the same background services will start up in the background! Install one or the other, not both in the same system.

On the other hand, a bare-bone Arch install with the XFCE4 package set will start up consuming around 150MB of RAM!! But of course, you lose all the niceties that comes pre-installed with GNOME. And also of course: you start Chromium, open a few tabs, and there goes all the RAM anyway.

If your goal is just to save resources, the choice is not between XFCE or LXQt, it is between having Facebook permanently opened in a browser tab or not. The reality is that the main offender of RAM is the Web as a whole. Half a dozen tabs and you eat up more than 1 Gigabyte, and it keeps going up. That’s why my main first recommendation is to install The Great Suspender.

You will run far worse apps in the foreground. For example, Spotify, Franz, Atom, to name a few. If it’s a hybrid app that loads a browser to load a web application, it is going to be heavy.

And I’d choose a forward thinking desktop manager that already supports Wayland/Weston instead of X11. You do want to throw away the bad X11 legacy as soon as possible.

Summary

As a shorter TL;DR remember to do the following:

• Tune your swappiness and cache pressure settings to avoid page faults when using your foreground applications. It’s a trade-off of performance vs responsiveness.
• Install the Linux-Zen or Liquorix kernels (depending on your distro) to have access to the better MuQSS process scheduler and BFQ I/O schedulers. If you’re using an SSD you will want to check if you’re using NOOP or Deadline I/O schedulers instead. Also check for TRIM support.
• Make Dropbox and Tracker play nice with your system. Configure both to only run when you are not using the system (when idle). Maybe install Ananicy and rejoice.
• Do not choose a Desktop Manager because of performance concerns. If you’re using Chromium or other web based applications, you’re already doomed. So don’t panic and use GNOME 3.22.

When I installed both Fedora 25 and Arch Linux I felt them sluggish. When I used Ubuntu 14.04 for months in a better hardware, I also felt it sluggish compared to macOS in similar hardware, but I never attempted this level of tuning before.

The main reason being the heavy initial sync of applications such as Dropbox, Tracker, Gnome-Photos.

The second reason is the better tuning of I/O scheduling and Swap settings because of the use of a mechanical harddrive. If you use an SSD you probably don’t suffer nearly as much.

Bottomline: if you can, buy a good SSD. If you have PCI Express x4, then do even better and buy a M.2 SSD such as the Samsung 950 EVO M.2. The best thing you can do is have more than 8GB of RAM (16GB being a good sweet spot) and a really fast SSD (preferably with a BUS that doesn’t suffer from SATA 3’s bottlenecks).

With all the tweaks and tunings, I am happy to report that my years old Lenovo tower is running as smoothly as it can, being responsive enough even in higher loads, with this shit slow mechanical hard drive. And as a bonus, if you choose to stay in GNOME 3, install the nice Flat Plat theme, heavily inspired by Material Design, and the Paper or Moka icon sets.

But there are many opportunities to accelerate Rails applications as well. For example, we just saw the release of the gem “faster_path”, this time written in Rust and bridged through FFI (Foreign Function Interface). The author’s claim is that Sprockets has to compute lots of paths and making this library, natively compiled and optimized with Rust, added a huge improvement in the asset pipeline task.

Sam Saffron, from Discourse, also built a very small gem called “fast_blank” which is a tiny library written in C that reimplements ActiveSupport’s String#blank? method to be up to 9x faster. Because Rails digests volumes of strings, checking if they are blank every time, this adds some performance (depends on your app, of course).

The Holy Grail to native-level performance is to be able to write close-to-Ruby code instead of having to hack low-level C or having the high learning curve of a language such as Rust. More than that, I’d like to avoid having to use FFI. I am not an expert in FFI but I remember understanding that it adds overhead to make the bindings.

By the way, it’s important to disclose right now: I am not a C expert by any means of the imagination, far from that. So I have very little experience dealing with hardcore C development. Which is again, why this possibility of writing in Crystal is even more appealing to me. So if you are a C expert and you spot something silly I am saying about it, please let me know in the comments section below.

My exercise is to rewrite the C-based Fast Blank gem in Crystal, add it to the same Gem to compile under Crystal if it’s available or fall back to C, and make the specs pass so it’s a seamless transition for the user.

To achieve that I had to:

• Extend the Gem’s extconf.rb to generate different Makefiles (for C and Crystal) that are able to compile under OS X or Linux (Ubuntu at least) - OK
• Make the specs pass in the Crystal version - Almost (it’s ok for all intents and purposes but an edge case)
• Make the performance be faster than Ruby and close to C - Not so much yet (under OS X the performance is quite good, but under Ubuntu it doesn’t scale so well for large strings)

You can check out the results so far on my fork over Github and follow the Pull Request discussion as well.

Comparing C and Crystal

Just to get us started, let’s check out a snippet of Sam’s original C version:

static VALUE
rb_str_blank(VALUE str)
{
  rb_encoding *enc;
  char *s, *e;

  enc = STR_ENC_GET(str);
  s = RSTRING_PTR(str);
  if (!s || RSTRING_LEN(str) == 0) return Qtrue;

  e = RSTRING_END(str);
  while (s < e) {
    int n;
    unsigned int cc = rb_enc_codepoint_len(s, e, &n, enc);

    if (!rb_isspace(cc) && cc != 0) return Qfalse;
    s += n;
  }
  return Qtrue;
}

Yeah, quite scary, I know. Now let’s see the Crystal version:

struct Char
  ...
  # same way C Ruby implements it
  def is_blank
    self == ' ' || ('\t' <= self <= '\r')
  end
end

class String
  ...
  def blank?
    return true if self.nil? || self.size == 0
    each_char { |char| return false if !char.is_blank }
    return true
  end
end

Hell yeah! If you’re a rubyist I bet you can understand 100% of the snippet above. It’s not “exactly” the same thing (as the specs are not fully passing yet), but it’s damn close.

The Quest for a Makefile to Crystal

I’ve researched many experimental Github repos and Gists out there. But I didn’t find one that has it all so I decided to tweak what I found until I got to this version:

Obs: again, I am not a C expert. If you have experience with Makefiles I know this one can be refactored to something nicer than this. Let me know in the comments below.

ifeq "$(PLATFORM)" ""
PLATFORM := $(shell uname)
endif

ifeq "$(PLATFORM)" "Linux"
UNAME = "$(shell llvm-config --host-target)"
CRYSTAL_BIN = $(shell readlink -f `which crystal`)
LIBRARY_PATH = $(shell dirname $(CRYSTAL_BIN))/../embedded/lib
LIBCRYSTAL = $(shell dirname $(CRYSTAL_BIN) )/../src/ext/libcrystal.a
LIBRUBY = $(shell ruby -e "puts RbConfig::CONFIG['libdir']")
LIBS = -lpcre -lgc -lpthread -levent -lrt -ldl
LDFLAGS = -rdynamic

install: all

all: fast_blank.so

fast_blank.so: fast_blank.o
  $(CC) -shared $^ -o $@ $(LIBCRYSTAL) $(LDFLAGS) $(LIBS) -L$(LIBRARY_PATH) -L$(LIBRUBY)

fast_blank.o: ../../../../ext/src/fast_blank.cr
  crystal build --cross-compile --release --target $(UNAME) $<

.PHONY: clean
clean:
  rm -f bc_flags
  rm -f *.o
  rm -f *.so
endif

ifeq "$(PLATFORM)" "Darwin"
CRYSTAL_FLAGS = -dynamic -bundle -Wl,-undefined,dynamic_lookup

install: all

all: fast_blank.bundle

fast_blank.bundle: ../../../../ext/src/fast_blank.cr
  crystal $^ --release --link-flags "$(CRYSTAL_FLAGS)" -o $@

clean:
  rm -f *.log
  rm -f *.o
  rm -f *.bundle
endif

Most people using Crystal are on OS X, including the creators of Crystal. LLVM is under Apple’s umbrella and their entire ecosystem relies heavily on LLVM. They spent many years migrating the C front-end first, then the C back-end away from GNU’s standard GCC to Clang. And they were able to make both Objective-C and Swift compile down to LLVM’s IR and that’s how both can interact back and forth natively.

Then, they improved the ARM backend support and that’s how they can have an entire iOS “Simulator” (not a dog slow emulator like Android) where the iOS apps are natively compiled to run over Intel’s x86_64 processor while in development and then quickly recompile to ARM when ready to package to the App Store.

This way you can run natively, test quickly, without the slowness of an emulated environment. By the way, I will say this once: Google’s biggest mistake is not supporting LLVM as they should, and reinventing the wheel instead. If they had, Go could already be used to implement for Android and Chromebooks as well as x86 based servers and they could put away all the Java/Oracle debacle.

But I digress.

In OS X you can pass a “-bundle” link-flag to crystal and it will probably use clang underneath to generate the binary bundle.

On Ubuntu crystal just compiles down to an object file (.o) and you have to manually invoke GCC with the “-shared” option to create a shared-object. To do that we have to use the "–cross-compile" and pass an LLVM target triplet so it generates the .o (this requires the llvm-config tool).

Shared Libraries (.so) and Loadable Modules (.bundle) are different beasts, check this documentation out for more details.

Keep in mind that benchmarking binaries built with different compilers can make a difference. I am not an expert but out of pure anecdote I believe Ruby under RVM on OS X is compiled using OS X’s default Clang. On Ubuntu it’s compiled under GCC. This seems to make Ruby on OS X “so slightly” inefficient in synthetic benchmarks.

On the other hand, Crystal binaries linked with GCC feel “so slightly” inefficient on Ubuntu, while Ruby on Ubuntu feels a bit faster, having been compiled and linked with GCC.

So when we compare Fast Blank/OS X/bit faster with Ruby/OS X/slower against Fast Blank/Ubuntu/bit slower with Ruby/Ubuntu/bit faster, it seems to give a wider advantage to the OS X benchmark comparison against the Ubuntu benchmark, even though individual computation times are not so far from each other.

I will come back to this point in the benchmarks section.

Finally, every time you have a rubygem with a native extension, you will find this bit in their gemspec files:

Gem::Specification.new do |s|
  s.name = 'fast_blank'
  ...
  s.extensions = ['ext/fast_blank/extconf.rb']
  ...

When the gem is installed through gem install or bundle install it will run this script to generate a proper Makefile. In a pure C extension it will use the built-in “mkmf” library to generate it.

In our case, if we have Crystal installed, we want to use the Crystal version, so I tweaked the extconf.rb to be like this:

require 'mkmf'

if ENV['VERSION'] != "C" && find_executable('crystal') && find_executable('llvm-config')
  # Very dirty patching
  def create_makefile(target, srcprefix = nil)
    mfile = open("Makefile", "wb")
    cr_makefile = File.join(File.dirname(__FILE__), "../src/Makefile")
    mfile.print File.read(cr_makefile)
  ensure
    mfile.close if mfile
    puts "Crystal version of the Makefile copied"
  end
end

create_makefile 'fast_blank'

So, if it finds crystal and llvm-config (which in OS X you have to add the proper path like this: export PATH=$(brew --prefix llvm)/bin:$PATH).

The Rakefile in this project declares the standard :compile task as the first one to run, and it will execute the extconf.rb, which will generate the proper Makefile and run the make command to compile and link the proper library in the proper lib/ path.

So we will end up with lib/fast_blank.bundle on OS X and lib/fast_blank.so on Ubuntu. From there we can just have require "fast_blank" from any Ruby file in the gem and it will have access to the publicly exported C function mappings from the Crystal library.

Mapping C-Ruby to Crystal

Now, any direct C extension - without FFI, fiddle or other “bridges” - will ALWAYS have a much better advantage.

The reason is that you literally have to “copy” data from C-Ruby to Crystal/Rust/Go or whatever other language you’re binding. While with a C-based extension you can operate directly in the memory space with the data without having to move it or copy it away.

For example. First, you have to bind the C functions from C-Ruby to Crystal. And we accomplish that with Paul Hoffer’s Crystalized Ruby mappings. It’s an experimental repository that I helped a bit to clean up in order for him to later extract this mapping library into its own Shard (shards are the same as gems for Crystal). For now I had to simply copy the file over to my Fast Blank.

Some of the relevant bits are like this:

lib LibRuby
  type VALUE = Void*
  type METHOD_FUNC = VALUE -> VALUE
  type ID = Void*
  ...

  # strings
  fun rb_str_to_str(value : VALUE) : VALUE
  fun rb_string_value_cstr(value_ptr : VALUE*) : UInt8*
  fun rb_str_new_cstr(str : UInt8*) : VALUE
  fun rb_utf8_encoding() : VALUE
  fun rb_enc_str_new_cstr(str : UInt8*, enc : VALUE) : VALUE
  ...
  # exception handling
  fun rb_rescue(func : VALUE -> UInt8*, args : VALUE, callback: VALUE -> UInt8*, value: VALUE) : UInt8*
end
...
class String
  RUBY_UTF = LibRuby.rb_utf8_encoding
  def to_ruby
    LibRuby.rb_enc_str_new_cstr(self, RUBY_UTF)
  end

  def self.from_ruby(str : LibRuby::VALUE)
    c_str = LibRuby.rb_rescue(->String.cr_str_from_rb_cstr, str, ->String.return_empty_string, 0.to_ruby)
    # FIXME there is still an unhandled problem: then we receive \u0000 from Ruby it raises "string contains null bytes"
    # so we catch it with rb_rescue, but then we can't generate a Pointer(UInt8) that represents the unicode 0, instead we return a plain blank string
    # but then the specs fail
    new(c_str)
  ensure
    ""
  end

  def self.cr_str_from_rb_cstr(str : LibRuby::VALUE)
    rb_str = LibRuby.rb_str_to_str(str)
    c_str  = LibRuby.rb_string_value_cstr(pointerof(rb_str))
  end

  def self.return_empty_string(arg : LibRuby::VALUE)
    a = 0_u8
    pointerof(a)
  end
end

Then I can use these mappings and helpers to build a “Wrapper” class in Crystal:

require "./lib_ruby"
require "./string_extension"

module StringExtensionWrapper
  def self.blank?(self : LibRuby::VALUE)
    return true.to_ruby if LibRuby.rb_str_length(self) == 0
    str = String.from_ruby(self)
    str.blank?.to_ruby
  rescue
    true.to_ruby
  end

  def self.blank_as?(self : LibRuby::VALUE)
    return true.to_ruby if LibRuby.rb_str_length(self) == 0
    str = String.from_ruby(self)
    str.blank_as?.to_ruby
  rescue
    true.to_ruby
  end

  def self.crystal_value(self : LibRuby::VALUE)
    str = String.from_ruby(self)
    str.to_ruby
  end
end

And this “Wrapper” depends on the “pure” Crystal library itself like with the snippets for the Char struct and String class extensions I showed in the first section of the article above.

Finally, I have a main “fast_blank.cr” file that externs those Wrapper functions so C-Ruby can see them as plain String methods:

require "./string_extension_wrapper.cr"

fun init = Init_fast_blank
  GC.init
  LibCrystalMain.__crystal_main(0, Pointer(Pointer(UInt8)).null)

  string = LibRuby.rb_define_class("String", LibRuby.rb_cObject)
  LibRuby.rb_define_method(string, "blank?", ->StringExtensionWrapper.blank?, 0)
  LibRuby.rb_define_method(string, "blank_as?", ->StringExtensionWrapper.blank_as?, 0)
  ...
end

This is mostly boilerplate. But now check out what I am having to do in the wrapper, in this particular snippet:

def self.blank?(self : LibRuby::VALUE)
  return true.to_ruby if LibRuby.rb_str_length(self) == 0
  str = String.from_ruby(self)
  str.blank?.to_ruby
rescue
  true.to_ruby
end

I am receiving a C-Ruby String casted as a pointer (VALUE) then I go through the lib_ruby.cr mappings to get the C-Ruby string data and copy it over into a new instance of Crystal’s internal String representation. So at any given time I have 2 copies of the same string, one in the C-Ruby memory space and another in the Crystal memory space.

This happens with all FFI-like extensions but it doesn’t happen to the pure C implementation. In Sam Saffron’s C implementation it directly works with the same address in C-Ruby’s memory space:

static VALUE
rb_str_blank(VALUE str)
{
  rb_encoding *enc;
  char *s, *e;

  enc = STR_ENC_GET(str);
  s = RSTRING_PTR(str);
  ...

It receives a pointer (direct memory address) and goes. And this is a huge advantage for the C version. When you have a big volume of medium to large sized strings being copied over from C-Ruby to Crystal, it adds a noticeable overhead that can’t be removed.

String mapping Caveat

I still have a problem though. There is one edge case I was not able to overcome yet (help is most welcome). When C-Ruby passes a unicode "\u0000" I am unable to create the same character in Crystal and I end up passing just an empty string ("") which is not the same thing.

The way to deal with it is to receive a Ruby String (VALUE) and get the C-String from it this way:

rb_str = LibRuby.rb_str_to_str(str)
c_str  = LibRuby.rb_string_value_cstr(pointerof(rb_str))

If the “str” is the “\u0000” (under Ruby 2.2.5 at least) C-Ruby raises a “string contains null bytes” exception. Which is why I rescue from this exception like this:

c_str = LibRuby.rb_rescue(->String.cr_str_from_rb_cstr, str, ->String.return_empty_string, 0.to_ruby)

When an exception is triggered I have to pass the pointer to another function to rescue from it:

def self.return_empty_string(arg : LibRuby::VALUE)
  a = 0_u8
  pointerof(a)
end

But this is not correct, I am just passing the pointer to a “0” character, which is “empty”. Therefore, specs are not passing correctly:

Failures:

  1) String provides a parity with active support function
     Failure/Error: expect("#{i.to_s(16)} #{c.blank_as?}").to eq("#{i.to_s(16)} #{c.blank2?}")

       expected: "0 false"
            got: "0 true"

       (compared using ==)
     # ./spec/fast_blank_spec.rb:22:in `block (3 levels) in <top (required)>'
     # ./spec/fast_blank_spec.rb:19:in `times'
     # ./spec/fast_blank_spec.rb:19:in `block (2 levels) in <top (required)>'

  2) String treats  correctly
     Failure/Error: expect("\u0000".blank_as?).to be_falsey
       expected: falsey value
            got: true
     # ./spec/fast_blank_spec.rb:47:in `block (2 levels) in <top (required)>'

Ary gave a simple tip later, I will add it to the conclusion below.

The Synthetic Benchmarks (careful on how you interpret them!)

The original Rails ActiveSupport implementation of String#blank? looks like this:

class String
  # 0x3000: fullwidth whitespace
  NON_WHITESPACE_REGEXP = %r![^\s#{[0x3000].pack("U")}]!

  # A string is blank if it's empty or contains whitespaces only:
  #
  #   "".blank?                 # => true
  #   "   ".blank?              # => true
  #   "　".blank?               # => true
  #   " something here ".blank? # => false
  #
  def blank?
    # 1.8 does not takes [:space:] properly
    if encoding_aware?
      self !~ /[^[:space:]]/
    else
      self !~ NON_WHITESPACE_REGEXP
    end
  end
end

It’s mainly a regular expression comparison, which can be a bit slow. Sam’s version is a more straightforward loop through the string to compare each character with what’s considered “blank”. There are many unicode codepoints that are considered blank, some are not, which is why the C and Crystal versions are similar, but they are different from Rails’ version.

In the Fast Blank gem there is a benchmark Ruby script to compare the C-extension against Rails’ Regex based implementation.

The Regex implementation is called “Slow Blank”. It’s particularly slow if you pass a real empty String, so in the benchmark Sam added a “New Slow Blank” that checks through String#empty? first, and this version is faster in this edge case.

The fast C version is called “Fast Blank” but although you can consider it “correct” it’s not compatible with all the edge cases from Rails. So he implemented a String#blank_as? which is compatible with Rails. Sam calls it “Fast Activesupport”.

In my Crystal version I did the same, having both String#blank? and String#blank_as?.

So, without further ado, here is the C Version over OS X benchmark for empty strings, and we exercise each function many times within a few seconds to have more accurate results (check out Evan Phoenix’s “benchmark/ips” to understand the “iteration per second” methodology).

================== Test String Length: 0 ==================
Warming up _______________________________________
          Fast Blank   191.708k i/100ms
  Fast ActiveSupport   209.628k i/100ms
          Slow Blank    61.487k i/100ms
      New Slow Blank   203.165k i/100ms
Calculating _______________________________________
          Fast Blank     20.479M (± 9.3%) i/s -    101.414M in   5.001177s
  Fast ActiveSupport     21.883M (± 9.4%) i/s -    108.378M in   5.004350s
          Slow Blank      1.060M (± 4.7%) i/s -      5.288M in   5.001365s
      New Slow Blank     18.883M (± 6.9%) i/s -     94.065M in   5.008899s

Comparison:
  Fast ActiveSupport: 21882711.5 i/s
          Fast Blank: 20478961.5 i/s - same-ish: difference falls within error
      New Slow Blank: 18883442.2 i/s - same-ish: difference falls within error
          Slow Blank:  1059692.6 i/s - 20.65x slower

It’s super fast. Rails’ version is 20x slower on my machine.

Now, Crystal version over OS X

================== Test String Length: 0 ==================
Warming up _______________________________________
          Fast Blank   174.349k i/100ms
  Fast ActiveSupport   174.035k i/100ms
          Slow Blank    64.684k i/100ms
      New Slow Blank   215.164k i/100ms
Calculating _______________________________________
          Fast Blank      8.647M (± 1.6%) i/s -     43.239M in   5.001530s
  Fast ActiveSupport      8.580M (± 1.3%) i/s -     42.987M in   5.010759s
          Slow Blank      1.047M (± 3.7%) i/s -      5.239M in   5.008907s
      New Slow Blank     19.090M (± 9.3%) i/s -     94.672M in   5.009057s

Comparison:
      New Slow Blank: 19090034.8 i/s
          Fast Blank:  8647459.7 i/s - 2.21x slower
  Fast ActiveSupport:  8580487.9 i/s - 2.22x slower
          Slow Blank:  1047465.3 i/s - 18.22x slower

As I explained before, even checking empty strings, the Crystal version is slower than the Ruby check for String#empty? (New Slow Blank) because I have the string copying routine of the Wrapper mappings. This adds overhead that is perceptible over many iterations. It’s still 18x faster than Rails, but it loses to C-Ruby.

Finally, Crystal version over Ubuntu

================== Test String Length: 0 ==================
Warming up _______________________________________
          Fast Blank   255.883k i/100ms
  Fast ActiveSupport   260.915k i/100ms
          Slow Blank   105.424k i/100ms
      New Slow Blank   284.670k i/100ms
Calculating _______________________________________
          Fast Blank      8.895M (± 9.8%) i/s -     44.268M in   5.037761s
  Fast ActiveSupport      8.647M (± 8.2%) i/s -     43.051M in   5.020125s
          Slow Blank      1.736M (± 3.9%) i/s -      8.750M in   5.048253s
      New Slow Blank     22.170M (± 6.2%) i/s -    110.452M in   5.004909s

Comparison:
      New Slow Blank: 22170031.0 i/s
          Fast Blank:  8895113.3 i/s - 2.49x slower
  Fast ActiveSupport:  8646940.8 i/s - 2.56x slower
          Slow Blank:  1736071.0 i/s - 12.77x slower

Notice that it’s around the same ballpark, but the Rails version on Ubuntu runs almost twice as fast compared to its counterpart in OS X, which makes the comparison against the Crystal library go down from 18x to 12x.

The benchmark keeps comparing against strings of larger and larger sizes, from 6, to 14, to 24, up to 136 characters in length.

Let’s get just the last test case of 136 characters. First with C version on OS X:

================== Test String Length: 136 ==================
Warming up _______________________________________
          Fast Blank   177.521k i/100ms
  Fast ActiveSupport   193.559k i/100ms
          Slow Blank    89.378k i/100ms
      New Slow Blank    60.639k i/100ms
Calculating _______________________________________
          Fast Blank     10.727M (± 8.7%) i/s -     53.256M in   5.006538s
  Fast ActiveSupport     11.600M (± 8.3%) i/s -     57.681M in   5.009692s
          Slow Blank      1.872M (± 5.7%) i/s -      9.385M in   5.029243s
      New Slow Blank      1.017M (± 5.3%) i/s -      5.094M in   5.022994s

Comparison:
  Fast ActiveSupport: 11600112.2 i/s
          Fast Blank: 10726792.8 i/s - same-ish: difference falls within error
          Slow Blank:  1872262.5 i/s - 6.20x slower
      New Slow Blank:  1016926.7 i/s - 11.41x slower

The C-version is consistently much faster in all test cases and in the 136 characters it’s still 11x faster than Rails in pure Ruby.

Now the Crystal version over OS X:

================== Test String Length: 136 ==================
Warming up _______________________________________
          Fast Blank   127.749k i/100ms
  Fast ActiveSupport   126.538k i/100ms
          Slow Blank    94.390k i/100ms
      New Slow Blank    60.594k i/100ms
Calculating _______________________________________
          Fast Blank      3.283M (± 1.8%) i/s -     16.480M in   5.021364s
  Fast ActiveSupport      3.235M (± 1.3%) i/s -     16.197M in   5.008315s
          Slow Blank      1.888M (± 4.4%) i/s -      9.439M in   5.009458s
      New Slow Blank    967.950k (± 4.7%) i/s -      4.848M in   5.018946s

Comparison:
          Fast Blank:  3283025.1 i/s
  Fast ActiveSupport:  3234586.5 i/s - same-ish: difference falls within error
          Slow Blank:  1887800.5 i/s - 1.74x slower
      New Slow Blank:   967950.2 i/s - 3.39x slower

It’s also faster, but just by 2 to 3 times compared to pure Ruby, a far cry from 11x. But my hypothesis is that the mapping and copying of so many strings over adds a large overhead that the C version does not have.

And the Crystal version over Ubuntu:

================== Test String Length: 136 ==================
Warming up _______________________________________
          Fast Blank   186.810k i/100ms
  Fast ActiveSupport   187.306k i/100ms
          Slow Blank   143.439k i/100ms
      New Slow Blank    98.308k i/100ms
Calculating _______________________________________
          Fast Blank      3.517M (± 3.9%) i/s -     17.560M in   5.000791s
  Fast ActiveSupport      3.485M (± 3.8%) i/s -     17.419M in   5.006427s
          Slow Blank      2.755M (± 4.2%) i/s -     13.770M in   5.008490s
      New Slow Blank      1.551M (± 4.3%) i/s -      7.766M in   5.017853s

Comparison:
          Fast Blank:  3516960.7 i/s
  Fast ActiveSupport:  3484575.5 i/s - same-ish: difference falls within error
          Slow Blank:  2754669.0 i/s - 1.28x slower
      New Slow Blank:  1550815.2 i/s - 2.27x slower

Again, the Ubuntu versions of both the Crystal library and the Ruby binary run faster and the comparison shows no more than twice as much faster. And pure Ruby’s String#empty? is in the same ballpark as Crystal’s version.

Conclusion

The most obvious conclusion is that I probably made a mistake in choosing Fast Blank as my first proof of concept. The algorithm is too trivial and a simple check for String#empty? in pure Ruby is orders of magnitude faster than the added overhead of mapping and string copying to Crystal.

Also, any use case where you have a huge amount of small bits of data being transferred from C-Ruby to Crystal or any FFI-based extension will have the overhead of data copying, which a pure C-version will not have. Which is why Fast Blank is better done in C.

Any other use case where you have smaller amounts of data, or data that can be transferred in bulk (less calls from C-Ruby to the extension, with arguments with a larger size, and with more costly processing) is a better candidate to benefit from extensions.

Again, not everything gets automatically faster, we always have to figure out the use case scenarios first. But because it’s so much easier to write in Crystal and benchmark, we can make faster proofs of concept and scrap the idea if the measurements prove that we won’t benefit as much.

The Crystal documentation recently received a “Performance Guide”. It’s very useful for you to avoid common pitfalls that harm overall performance. Even though LLVM is quite competent in heavy optimization, it can’t do everything. So read it through to improve your general Crystal skills.

That being said, I still believe that this exercise was well worth it. I will probably do some more. I’d really want to thank Ary (Crystal creator) and Paul Hoffer for the patience in helping me out through many of the quirks I found along the way.

While I was finishing this post, Ary pointed out that I could probably ditch Strings altogether and work directly with an array of bytes, which is a good idea and I will probably try that. I think I made it clear by now that the whole String copying adds a very perceptible overhead as we saw in the benchmarks above. Let me know if someone is interested in contributing as well. With a few more tweaks I believe we can have a Crystal version that can at least compete against the C version while also being more readable and maintainable for most Rubyists, which is my goal.

I hope the code I published here will serve as boilerplate examples for more Crystal-based Ruby extensions in the future!

• MRI Ruby 2.3.1: 57 seconds
• JRuby 9.1.1.0: 45 seconds
• Crystal 0.17.0: 59 seconds
• Elixir: 14 seconds

I also said how it was an unfair comparison as the Elixir version uses a different - and obviously more efficient - algorithm.

It was the “first-version-that-worked” so I decided to go ahead and improve the implementations. In the Ruby/JRuby version I added the thread gem to have a good enough implementation of a Thread pool that works for Ruby and JRuby. I probably should have used Concurrent-Ruby but I was having some trouble making FixedThreadPool work.

Anyway, now all versions will have a constant pool of requests running and we can make a better comparison.

Another thing that may have skewed the results against Crystal is that it seems to have a faulty DNS resolver implementation, so for now I just added the Manga Reader IP address directly to my /etc/hosts file to avoid getaddrinfo: Name or service not known exceptions.

To begin, let’s test the same Elixir implementation, and as expected the result is still the same:

11,49s user 0,82s system 77% cpu 15,827 total

Now, the MRI Ruby with the “batch burst” algorithm was taking 57 seconds and this new implementation using a ThreadPool runs much better:

12,67s user 0,92s system 50% cpu 27,149 total

In Crystal, it’s a bit more complicated as there is no way to implement the equivalent of a “Fiber Pool”. What we have to do is run an infinite loop until the last process signals a loop break. Within the loop we create a maximum number of fibers, wait for each one to signal that it finished through an individual channel and loop again to create a new fiber, and so on. Compared to yesterday’s 59 seconds, this is much better:

5,29s user 0,33s system 26% cpu 21,166 total

The JRuby version is not so fast though. Still better than yesterday’s 45 seconds but now it’s losing even to MRI:

49,24s user 1,41s system 146% cpu 34,602 total

I tried to use the --dev flag for faster start up time and it does improve a bit, getting it closer to MRI:

22,26s user 0,99s system 76% cpu 30,320 total

Not sure if it can be improved more though, any tips are welcome - don’t forget to comment below.

So, Elixir is still at least twice as fast as Crystal at this point. But this also demonstrates how a different algorithm does make a huge difference where it matters. I can probably tweak it a bit more but this should suffice for now.

Changing the Ruby implementation to use ThreadPool

require "thread/pool"

module MangaDownloadr
  class Concurrency
    ...
    def fetch(collection, &block)
      pool    = Thread.pool(@config.download_batch_size)
      mutex   = Mutex.new
      results = []

      collection.each do |item|
        pool.process {
          engine  = @turn_on_engine ? @engine_klass.new(@config.domain) : nil
          reply = block.call(item, engine)&.flatten
          mutex.synchronize do
            results += ( reply || [] )
          end
        }
      end
      pool.shutdown

      results
    end
    ...

The idea is that we will have a fixed number of spawned native threads (in this case, determined by @config.download_batch_size). As one thread finishes it will pop a new link from the collection Array, essentially working as a “queue” to be depleted.

The results are accumulated in the results Array. Because many threads may want to modify it at once we have to synchronize access through a Mutex.

This way we always have a fixed amount of workers performing requests constantly instead of slicing the collection Array and doing bursts as in the previous version.

Changing the Crystal implementation to simulate a Fibers Pool

The Crystal version became a bit more complicated as I didn’t find a pool library to pull in. I found a rough implementation of a pool in this stackoverflow post and I was able to implement an improved version into a new shard so you can take advantage of it in your projects. Check out the source code at akitaonrails/fiberpool. This is how I added it to my project:

dependencies:
  ...
  fiberpool:
    github: akitaonrails/fiberpool

And this is how I used it. Notice that the logic itself is very close to the MRI Ruby version, but using a “Fiber Pool” instead of a Thread Pool.

require "fiberpool"

module CrMangaDownloadr
  struct Concurrency
    def initialize(@config : Config, @turn_on_engine = true); end

    def fetch(collection : Array(A)?, engine_class : E.class, &block : A, E? -> Array(B)?) : Array(B)
      results = [] of B
      if collection
        pool = Fiberpool.new(collection, @config.download_batch_size)
        pool.run do |item|
          engine = if @turn_on_engine
                     engine_class.new(@config.domain, @config.cache_http)
                   end
          if reply = block.call(item, engine)
            results.concat(reply)
          end
        end
      end
      results
    end
  end
end

But again, this is very I/O intensive and both the Ruby and Crystal versions take advantage of the fact that they can do more work while waiting for one HTTP request to finish.

Reproducing the Tests

I implemented a “Test Mode” in all 3 implementations. You can clone from my repositories:

• Crystal version
• Ruby/JRuby version
• Elixir version

And you can run the test mode like this:

# crystal:
time ./cr_manga_downloadr --test

# MRI:
time bin/manga-downloadr --test

# JRuby (you have to edit Gemfile to uncomment the JRuby engine):
time jruby --dev -S bin/manga-downloadr --test

# Elixir:
time ./ex_manga_downloadr --test

This will run only the fetching of chapters, pages and image links, skipping the actual downloading of the images, optimization through mogrify and PDF compilation. Those skipped parts take too long and don’t say anything about the tested languages.

And if you want to test just the CPU intensive parts and avoid all networking interference altogether, you can turn on HTTP Cache mode and run the tests twice so the first run will cache everything first, like this:

# crystal:
time ./cr_manga_downloadr --test --cache

# MRI:
time bin/manga-downloadr --test --cache

# JRuby (you have to edit Gemfile to uncomment the JRuby engine):
time jruby --dev -S bin/manga-downloadr --test --cache

# Elixir:
time CACHE_HTTP=true ./ex_manga_downloadr --test

So, with all requests already cached these are the results:

Elixir:

7,13s user 0,24s system 331% cpu 2,227 total

MRI Ruby:

5.590000   0.180000   5.770000 (  5.678714)
5,87s user 0,21s system 101% cpu 5,996 total

JRuby:

10.580000   0.180000  10.760000 (  3.184472)
14,54s user 0,44s system 262% cpu 5,716 total

Crystal:

1.610000   0.050000   1.660000 (  1.344123)
1,62s user 0,06s system 124% cpu 1,350 total

The Ruby/JRuby/Crystal versions have internal benchmarks to take away startup time (this is why they have 2 lines of times).

So, Elixir is very fast. It takes roughly 2 seconds to parse all the 1,900 HTML files to find the links.

Ruby is the slowest, obviously. It takes almost 6 seconds.

The JRuby version also takes almost 6 seconds but internally it processes in 3 seconds, the rest is startup time and warming up of the JVM underneath.

And Crystal is the fastest, as you would expect because it’s a super optimized binary doing CPU bound operations, clocking in at a bit more than 1 second.

Conclusion

Even though the algorithms are roughly similar, Elixir is still winning by a very large margin in the total process (with the external HTTP requests).

There is more than just interpreter/compiler speeds, there is more than single-thread, multi-thread, fibers infrastructure.

We also have the maturity of the respective standard libraries (including TCP stack, HTTP client libraries, String/Array/Regex operations, etc) and 3rd party libraries (libXML, Nokogiri, etc). So there is a lot that can interfere with the tests. I’d guess that Crystal’s standard library, especially the networking parts, are not battle-tested enough at this point (pre 1.0!).

So, the summary with the new results is this:

• MRI Ruby 2.3.1: 27 seconds
• JRuby 9.1.1.0: 30 seconds
• Crystal 0.17.0: 21 seconds
• Elixir: 15 seconds

Let me know if you have ideas to make them even faster!

JRuby always pursued the goal of being a competent MRI replacement. If you have not followed what’s going on, JRuby changed version schemes when they switched from the 1.7 series to the 9.x series. You should read this blog post that explains it.

In summary, if you want MRI 2.2 compatibility you must use JRuby 9.0.x and if you want MRI 2.3 compatibility you must use the JRuby 9.1.x series. The most current release being 9.1.2.0. Anything prior to that you can refer to this table of versions from Heroku’s documentation.

There are important recommendations as well:

• Ideally you should use Rails 4.2. Try to be at least above 4.0, and you can turn on config.threadsafe! by default in the “config/environments/production.rb” file. To understand this subject, refer to Tenderlove’s excellent explanation.

• If you’re deploying to Heroku, don’t bother trying the free or 1X dyno, which only gives you 512Mb of RAM. While it is enough for most small Rails applications (even with 2 or 3 Puma concurrent workers), I found that even the smallest apps can easily go above that. So you must consider at least the 2X dyno. In any server configuration, always consider more than 1Gb of RAM.

Gems

There are several gems with C extensions that just won’t work. Some of them have drop-in replacements, some don’t. You should refer to their Wiki for a list of cases.

In my small sample application - which is nothing more than a content website backed by a Postgresql database, ActiveAdmin to manage content, RMagick + Paperclip (yes, it’s an old app) to handle image upload and resizing, there was not a lot to change. The important bits of my “Gemfile” end up looking like this:

source 'https://rubygems.org'

ruby '2.3.0', :engine => 'jruby', :engine_version => '9.1.1.0'
# ruby '2.3.1'

gem 'rails', '~> 4.2.6'

gem 'devise'
gem 'haml'
gem 'puma'
gem 'rack-attack'
gem 'rack-timeout'
gem 'rakismet'

# Database
gem 'pg', platforms: :ruby
gem 'activerecord-jdbcpostgresql-adapter', platforms: :jruby

# Cache
gem 'dalli'
gem "actionpack-action_caching", github: "rails/actionpack-action_caching"

# Admin
gem 'activeadmin', github: 'activeadmin'
gem 'active_skin'

# Assets
gem 'therubyracer', platforms: :ruby
gem 'therubyrhino', platforms: :jruby

gem 'asset_sync'
gem 'jquery-ui-rails'
gem 'sass-rails'
gem 'uglifier',     '>= 1.3.0'
gem 'coffee-rails', '~> 4.0.0'
gem 'compass-rails'
gem 'sprockets', '~>2.11.0'

# Image Processing
gem 'paperclip'
gem 'fog'
gem 'rmagick', platforms: :ruby
gem 'rmagick4j', platforms: :jruby

group :test do
  gem 'shoulda-matchers', require: false
end

group :test, :development do
  gem "sqlite3", platforms: :ruby
  gem "activerecord-jdbcsqlite3-adapter", platforms: :jruby

  # Pretty printed test output
  gem 'turn', require: false
  gem 'jasmine'

  gem 'pry-rails'

  gem 'rspec'
  gem 'rspec-rails'
  gem 'capybara'
  gem 'poltergeist'
  gem 'database_cleaner', '< 1.1.0'

  gem 'letter_opener'
  gem 'dotenv-rails'
end

group :production do
  gem 'rails_12factor'
  gem 'rack-cache', require: 'rack/cache'
  gem 'memcachier'
  gem 'newrelic_rpm'
end

Notice how I paired gems for the :ruby and :jruby platforms. After doing this change and bundle install everything, I ran my Rspec suite and - fortunately - they all passed on the first run without any further changes! Your mileage will vary depending on the complexity of your application, so have your tests ready.

Puma

In the case of Puma, the configuration is a bit trickier, mine looks like this:

web_concurrency = Integer(ENV['WEB_CONCURRENCY'] || 1)
if web_concurrency > 1
  workers web_concurrency
  preload_app!
end

threads_count = Integer(ENV['RAILS_MAX_THREADS'] || 5)
threads threads_count, threads_count

rackup      DefaultRackup
port        ENV['PORT']     || 3000
environment ENV['RACK_ENV'] || 'development'

on_worker_boot do
  # Worker specific setup for Rails 4.1+
  # See: https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server#on-worker-boot
  ActiveRecord::Base.establish_connection
end

It’s a bit different from what you might find in other documentation. The important part is to turn off workers and preload_app when loading over JRuby. Otherwise it will complain and crash. On my original MRI deploy I am using a small 1X dyno and I can leave WEB_CONCURRENCY=2 and RAILS_MAX_THREADS=5 but on the JRuby deploy I set it to WEB_CONCURRENCY=1 (to turn off workers) and RAILS_MAX_THREADS=16 (because I am assuming JRuby can handle more multithreading than MRI).

Another important bit: most people still assume that MRI can’t take advantage of native parallel threads at all because of the dreaded GIL (Global Interpreter Lock), but that’s only half true. MRI Ruby can parallelize threads on I/O waits. So, if a part of your app is waiting for the database to process and return rows, for example, another thread can take over and do something else, in parallel. There is some concurrency available, even if limited, so setting a small amount of threads for Puma might help a bit.

Do not forget to set the Pool size to at least the same number of RAILS_MAX_THREADS. You can either use the config/database.yml for Rails 4.1+ or add an initializer. Follow Heroku’s documentation on how to do so.

Initial Benchmarks

So, I was able to successfully deploy a secondary JRuby version of my original MRI-based Rails app.

The original app is still in a Hobby Dyno, sized at 512Mb of RAM. The secondary app needed a Standard 1X Dyno, sized at 1Gb of RAM.

The app itself is super simple and as I’m using caching - as you always should! -, the response time is very low, on the tens of milliseconds.

I tried to use the Boom tool to benchmark the requests. I did a lot of warming up on the JRuby version, running the benchmarks multiple times and even using Loader.io for added pressure.

I am running this test:

$ boom -n 200 -c 50 http://foo-my-site/

The MRI version performs like this:

Summary:
  Total:    16.4254 secs
  Slowest:  9.0785 secs
  Fastest:  0.8362 secs
  Average:  2.6551 secs
  Requests/sec: 12.1763
  Total data:   28837306 bytes
  Size/request: 144186 bytes

Status code distribution:
  [200] 200 responses

Response time histogram:
  0.836 [1] |
  1.660 [57]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  2.485 [57]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  3.309 [33]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  4.133 [22]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  4.957 [16]    |∎∎∎∎∎∎∎∎∎∎∎
  5.782 [6] |∎∎∎∎
  6.606 [3] |∎∎
  7.430 [1] |
  8.254 [3] |∎∎
  9.079 [1] |

Latency distribution:
  10% in 1.2391 secs
  25% in 1.5910 secs
  50% in 2.1974 secs
  75% in 3.4327 secs
  90% in 4.5580 secs
  95% in 5.6727 secs
  99% in 8.1567 secs

And the JRuby version performs like this:

Summary:
  Total:    15.5784 secs
  Slowest:  7.4106 secs
  Fastest:  0.5770 secs
  Average:  2.3224 secs
  Requests/sec: 12.8383
  Total data:   28848475 bytes
  Size/request: 144242 bytes

Status code distribution:
  [200] 200 responses

Response time histogram:
  0.577 [1] |
  1.260 [23]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  1.944 [62]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  2.627 [51]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  3.310 [24]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  3.994 [26]    |∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎
  4.677 [8] |∎∎∎∎∎
  5.361 [1] |
  6.044 [0] |
  6.727 [1] |
  7.411 [3] |∎

Latency distribution:
  10% in 1.1599 secs
  25% in 1.5154 secs
  50% in 2.0781 secs
  75% in 2.8909 secs
  90% in 3.7409 secs
  95% in 4.2556 secs
  99% in 6.7685 secs

In general, I’d say that they are around the same. As this is not a particularly CPU-intensive processing, and most of the time is spent going through the Rails stack and hitting Memcachier to pull back the same content all the time, maybe it’s only fair to expect similar results.

On the other hand, I’m not sure I’m using the tool in the best way possible. The log says something like this for every request:

source=rack-timeout id=c8ad5f0c-b5c1-47ec-b88b-3fc597ab01dc wait=29ms timeout=20000ms state=ready
Started GET "/" for XXX.35.10.XXX at 2016-06-03 18:54:34 +0000
Processing by HomeController#home as HTML
Read fragment views/radiant-XXXX-XXXXX.herokuapp.com/en (6.0ms)
Completed 200 OK in 8ms (ActiveRecord: 0.0ms)
source=rack-timeout id=c8ad5f0c-b5c1-47ec-b88b-3fc597ab01dc wait=29ms timeout=20000ms service=19ms state=completed
source=rack-timeout id=a5389dc4-9a1a-46b7-a1e5-53f334ca0941 wait=35ms timeout=20000ms state=ready

Started GET "/" for XXX.35.10.XXX at 2016-06-03 18:54:36 +0000
Processing by HomeController#home as HTML
Read fragment views/radiant-XXXX-XXXXX.herokuapp.com/en (6.0ms)
Completed 200 OK in 9ms (ActiveRecord: 0.0ms)
source=rack-timeout id=a5389dc4-9a1a-46b7-a1e5-53f334ca0941 wait=35ms timeout=20000ms service=21ms state=completed
at=info method=GET path="/" host=radiant-XXXX-XXXXX.herokuapp.com request_id=a5389dc4-9a1a-46b7-a1e5-53f334ca0941 fwd="XXX.35.10.XXX" dyno=web.1 connect=1ms service=38ms status=200 bytes=144608

The times reported by the Boom tool are much larger (2 seconds?) than the processing times in the logs (10ms?). Even considering some overhead for the router and so on, it’s still a big difference. I wonder if requests are being queued for too long because the app isn’t able to respond to more of the concurrent requests.

The amount of requests divided by the number of concurrent connections will bring the overall performance and throughput down if you increase it too much, I wasn’t able to go much above 14 rpm with this configuration, though.

If you have more experience with http benchmarking and you can spot something wrong I am doing here, please let me know in the comments section below.

Conclusion

JRuby continues to evolve, and you might benefit if you already have a large set of Dynos or large servers around. I wouldn’t recommend it for small to medium applications.

I’ve seen many orders of magnitude improvements in specific use cases (I believe it was a very high traffic API endpoint). This particular case I tested is probably not its sweet spot and changing from MRI to JRuby didn’t give me too much advantage, so in this case I would recommend sticking to MRI.

Startup time is still an issue. There is an entry in their Wiki giving some recommendations, but even in the Heroku deploy I ended up having R10 errors (Boot Timeout) every once in a while for this small app.

I didn’t try increasing the dynos to the Performance tier introduced last year. I would bet that JRuby would be better at those and more able to leverage the extra power of having from 2.5GB up to 14GB if you have really big traffic (on the order of thousands or tens of thousands of requests per minute). With MRI the recommendation would be using small-ish dynos (2X or at most Performance-M dynos) and scale horizontally. With JRuby you could have fewer dynos with larger sizes (Performance-L, for example). Again, depends on your case.

Don’t take the benchmarks above as “facts” to generalize everywhere, they are just there to give you a notion of a specific use case of mine. Your mileage will vary, so you must test it out yourself.

Another use case (that I did not test) is, instead of just “porting” an MRI app to JRuby, leveraging JRuby’s unique strengths as this post from Heroku explains, in the case of using JRuby with Ratpack, for example.

All in all, JRuby is still a great project to experiment with. MRI itself has come a long way as well and 2.3.1 is quite decent. Most of the time it’s down to your overall architecture choices, and the language is just one piece of the puzzle. If you didn’t try it yet, you definitely should. It “just works”.

In summary, most people are just exposing their contents through simple JSON API endpoints and consuming them from other microservices through simple HTTP GETs. The more microservices they add to the chain, the longer the last endpoint takes to return. There are many techniques to improve this situation, but I want to show just a simple one that can solve many common situations without too much hassle.

First of all, if you’re dealing with caching, never try to expire cache entries. The most important thing to learn about caching is how to generate proper cache keys. Do it right and most problems with caching are gone.

Second, if you’re using HTTP, try to use everything you can from it, instead of reinventing the wheel.

The “TL;DR” version is: make your APIs return proper ETAGs and handle “If-None-Match” headers properly, return the correct 304 status code instead of full blown 200 with full content body in the responses every time. And in the consumer end, cache the ETAG with the corresponding response body and use it from cache when you receive 304s. You will save at least expensive rendering time in the consumed end and slow bandwidth from the consumer end. In the end you should be able to at least be 100% faster, or more, with just a few tweaks.

The Example Applications

In a very very contrived example we could have a Rails API controller like this:

## 1st application
class Api::ProductsController < ApplicationController
  def index
    @products = Product.page (params[:page] || 1)
    render json: [num_pages: @products.num_pages, products: @products]
  end
  ...
end

For the purposes of this contrived post example, we load it up on localhost port 3001. Now, whenever we call “http://localhost:3001/api/products?page=1” the API server dumps something like this in the log:

Started GET "/api/products?page=1" for 127.0.0.1 at 2016-03-23 13:29:34 -0300
Processing by Api::ProductsController#index as */*
  Parameters: {"page"=>"1"}
   (0.3ms)  SELECT COUNT(*) FROM "products"
  Product Load (0.9ms)  SELECT  "products".* FROM "products" LIMIT 100 OFFSET 0
Completed 200 OK in 26ms (Views: 23.0ms | ActiveRecord: 1.2ms)

In summary, it’s taking around 26ms to send back a JSON with the first page of products of this application. Not too bad.

Then we can create another Rails API application that consumes this first one. Something also very contrived and stupid like this:

# 2nd application
class Api::ProductsController < ApplicationController
  def index
    # never, ever, hard code hostnames like this, use dotenv-rails or secrets.yml
    url = "http://localhost:3001/api/products?page=?" % (params[:page] || "1")
    json = Net::HTTP.get_response(URI(url)).body

    response.headers["Content-Type"] = "application/json"
    render plain: json
  end
  ...

We load this other app in localhost port 3000 and when we call “http://localhost:3000/api/products?page=1” the server dumps the following log:

Started GET "/api/products?page=1" for 127.0.0.1 at 2016-03-23 13:31:59 -0300
Processing by Api::ProductsController#index as HTML
  Parameters: {"page"=>"1"}
  Rendered text template (0.0ms)
Completed 200 OK in 51ms (Views: 7.1ms | ActiveRecord: 0.0ms)

Now, this second application is taking twice the time compared to the first one. We can assume that part of those 51ms are the 26ms of the first application.

The more APIs we add on top of each other, the more time the entire flow will take. 26ms for the first, another 25ms for the second, and so on.

There are many things we could do. But I’d argue that we should start simple: by actually using a bit more of the HTTP protocol.

Sending proper ETAGs and handling “If-None-Match”

In a nutshell, we can tag any HTTP response with an ETAG, an identifier for the content of the response. If the ETAG is the same, we can assume the content hasn’t changed. Web browsers receive the ETAGs and send them back if we choose to refresh the content as a “If-None-Match” header. When a web server receives this header it compares against the ETAG of the response and doesn’t send back any content, just a “304 Not Modified” HTTP header, which is much, much lighter and faster to transport back.

An ETAG can be as complicated as an entire SHA256 hexdigest of the entire response content or as simple as just the “updated_at” timestamp if this indicates that the record has changed (in a “show” controller action, for example). It must be a digest that represents the content and it must change if the content changes.

Rails has had support for ETAGs for a long time in the ActionController::ConditionalGet API.

In our contrived example, the 1st application on port 3001 fetches a page of ActiveRecord objects and sends back an array represented in JSON format. If we choose to digest the final content we would have to let ActionView do its job, but it is by far the most expensive operation so we want to avoid it.

One thing that we could do is just check the “updated_at” fields of all the records and see if they changed. If any one of them changed, we would need to re-render everything and send a new ETAG and a new response body. So the code could be like this:

class Api::ProductsController < ApplicationController
  layout false
  def index
    @products = Product.page (params[:page] || 1)

    if stale?(freshness @products)
      render json: [num_pages: @products.num_pages, products: @products]
    end
  end
  ...
  private

  def freshness(collection)
    dates = collection.map(&:updated_at).sort
    etag = dates.map(&:to_i).reduce(&:+)
    {etag: Digest::MD5.hexdigest(etag.to_s), last_modified: dates.last, public: true}
  end

Now, when we try to “curl -I http://localhost:3001/api/products?page=1” we will see the following headers:

HTTP/1.1 200 OK 
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Etag: "ccf372c24cd259d0943fa3dc99830b10"
Last-Modified: Wed, 23 Mar 2016 16:25:53 GMT
Content-Type: application/json; charset=utf-8
Cache-Control: public
X-Request-Id: 601f22bc-72a9-4960-97cb-c30a0b56dbf4
X-Runtime: 0.053529
Server: WEBrick/1.3.1 (Ruby/2.3.0/2015-12-25)
Date: Wed, 23 Mar 2016 17:00:13 GMT
Content-Length: 0
Connection: Keep-Alive

Great! We have an ETAG that uniquely represents this page of products. Now we can go one step further and add the following:

# Gemfile
gem 'dalli'
gem 'rack-cache'

# config/environments/[development|production].rb
...
config.cache_store = :dalli_store
client = Dalli::Client.new
config.action_dispatch.rack_cache = {
  :metastore    => client,
  :entitystore  => client
}
config.static_cache_control = "public, max-age=2592000"
...

This configuration is assuming that we have Memcached installed and running in the same localhost machine (our development environment), but in production you can follow this good documentation from Heroku.

Now, our 1st application has an internal HTTP cache, with the same role as something more advanced such as Varnish in front of it. It will cache all HTTP 200 responses from the application, together with the ETAGs. Whenever a new call comes for the same URI, it will check the cache first, and if the application sends back the same ETAG, it will send the content back from the cache.

So if we call the above “curl” command multiple times, we will see this from the Rails server log:

Started GET "/api/products?page=1" for 127.0.0.1 at 2016-03-23 14:05:16 -0300
Processing by Api::ProductsController#index as */*
  Parameters: {"page"=>"1"}
  Product Load (0.8ms)  SELECT  "products".* FROM "products" LIMIT 100 OFFSET 0
  Couldn't find template for digesting: products/index
   (1.0ms)  SELECT COUNT(*) FROM "products"
Completed 200 OK in 31ms (Views: 16.3ms | ActiveRecord: 1.8ms)
cache: [HEAD /api/products?page=1] miss, store

Notice the last line: it says that it tried to find the returning ETAG in the cache and it “missed”, so it “stored” the new content. Now, if we run the same “curl” command again, we will see this:

Started GET "/api/products?page=1" for 127.0.0.1 at 2016-03-23 14:05:59 -0300
Processing by Api::ProductsController#index as */*
  Parameters: {"page"=>"1"}
  Product Load (0.5ms)  SELECT  "products".* FROM "products" LIMIT 100 OFFSET 0
  Couldn't find template for digesting: products/index
Completed 304 Not Modified in 12ms (ActiveRecord: 0.5ms)
cache: [HEAD /api/products?page=1] stale, valid, store

The simple curl command is not sending the “If-None-Match” header, so it expects to receive the full response body. But because we have Rack Cache it is adding the “If-None-Match” ETAG digest from the cache to the request before hitting the Rails app. The Rails app now compares the received “If-None-Match” digest through the “stale?” methods with the ETAG it just computed and because they match, it just sends an empty body response with the status code of 304. Rack Cache receives the 304 and fetches the cached JSON from Memcached and changes the HTTP response from the 304 to a normal 200 with the full body, which is what Curl can receive.

Because we are skipping the expensive ActionView rendering, the response time went from the previous 26ms to around 12ms: we are now twice as fast!

Consuming APIs with ETAGs

But we can go one step further. If we change nothing about the 2nd application, it will keep receiving just HTTP 200 with full body responses from Rack Cache of the 1st application. Let’s see the code again

# 2nd application
class Api::ProductsController < ApplicationController
  def index
    # never, ever, hard code hostnames like this, use dotenv-rails or secrets.yml
    url = "http://localhost:3001/api/products?page=?" % (params[:page] || "1")
    json = Net::HTTP.get_response(URI(url)).body

    response.headers["Content-Type"] = "application/json"
    render plain: json
  end
  ...

We can do better. How about the following:

# 2nd application - upgrade!
class Api::ProductsController < ApplicationController
  def index
    page = params[:page] || "1"
    url = "http://localhost:3001/api/products?page=?" % page
    # 1 - fetch the ETAG for the URL
    etag = Rails.cache.fetch(url)
    # 2 - fetch from external API or fetch from internal cache
    json = fetch_with_etag(url, etag)

    response.headers["Content-Type"] = "application/json"
    render plain: json
  end
  ...
  private

  def fetch_with_etag(url, etag)
    uri = URI(url)

    req = Net::HTTP::Get.new(uri)
    # 3 - add the important If-None-Match header
    req['If-None-Match'] = etag if etag

    res = Net::HTTP.start(uri.hostname, uri.port) {|http|
      http.request(req)
    }

    etag = res['ETAG']
    etag = etag[1..-2] if etag.present?
    if res.is_a?(Net::HTTPNotModified)
      # 4 - if got a 304, then we already have the content in the internal cache
      Rails.cache.read(etag)
    elsif res.is_a?(Net::HTTPSuccess)
      # 5 - if we got a 200 it's new content to store in internal cache before returning
      Rails.cache.write(url, etag)
      Rails.cache.write(etag, res.body)
      res.body
    else
      "{}"
    end
  end
end

I know, feels overwhelming, but it’s actually quite simple. Let’s go over it step-by-step:

1. First we see if we already have an ETAG associated with the URL we want to fetch (be aware of query parameters!)

2. Now we call the separated “fetch_with_etag” method

3. This is all boilerplate “Net::HTTP” setup, but the important piece is that we add the “If-None-Match” header if we found an ETAG for the URL in the cache.

4. After we make the external call we can have 2 responses. The first being the very very short, body-less, header-only, “304 Not Modified”. In this case, it means that we already have the full content in the internal cache and it is still valid, so we use it.

5. In case we receive the normal HTTP “200” status code, we either didn’t send any ETAG or the one we sent was invalidated and a new ETAG and content body were returned, so we must update them in our internal cache before exiting.

Now, the first time we call “curl http://localhost:3000/api/products?page=1” for the 2nd application endpoint we will see this log:

Started GET "/api/products?page=1" for 127.0.0.1 at 2016-03-23 14:14:05 -0300
Processing by Api::ProductsController#index as */*
  Parameters: {"page"=>"1"}
  Rendered text template (0.0ms)
Completed 200 OK in 62ms (Views: 5.6ms | ActiveRecord: 0.0ms)

Caches are cold, it is taking the same “around 50ms” like we had before, in this case, it’s more like 62ms.

Just to recap, this call to the 2nd application made it call the 1st application API, which shows the following in its log:

Started GET "/api/products?page=?" for 127.0.0.1 at 2016-03-23 14:14:05 -0300
Processing by Api::ProductsController#index as */*
  Parameters: {"page"=>"?"}
/"32b82ebbd99854ea2ca0d49ff4a7c07c
  Product Load (0.9ms)  SELECT  "products".* FROM "products" LIMIT 100 OFFSET 0
  Couldn't find template for digesting: products/index
   (1.2ms)  SELECT COUNT(*) FROM "products"
Completed 200 OK in 37ms (Views: 21.5ms | ActiveRecord: 2.2ms)
cache: [GET /api/products?page=?] miss, store

Cache miss, new content stored!

Now, we call “curl” against the same URL for the 2nd application again and we now see what we wanted in the log:

Started GET "/api/products?page=1" for 127.0.0.1 at 2016-03-23 14:14:10 -0300
Processing by Api::ProductsController#index as */*
  Parameters: {"page"=>"1"}
  Rendered text template (0.0ms)
Completed 200 OK in 24ms (Views: 0.3ms | ActiveRecord: 0.0ms)

Down from 62ms to 24ms!! And in the 1st application log we see:

Started GET "/api/products?page=?" for 127.0.0.1 at 2016-03-23 14:14:10 -0300
Processing by Api::ProductsController#index as */*
  Parameters: {"page"=>"?"}
"ccf372c24cd259d0943fa3dc99830b10", ccf372c24cd259d0943fa3dc99830b10
  Product Load (1.2ms)  SELECT  "products".* FROM "products" LIMIT 100 OFFSET 0
  Couldn't find template for digesting: products/index
Completed 304 Not Modified in 12ms (ActiveRecord: 1.2ms)
cache: [GET /api/products?page=?] stale, valid, store

A cache hit! Content is stale and valid, so return just 304, the 2nd application acknowledges and fetches the still valid content from its own cache and returns to Curl.

Conclusion

If you remove ETAGs from the 1st application, the 2nd one will not break and vice-versa, because it’s optional. If “ETAG” and “If-None-Match” headers are present in received HTTP response, we can use them, otherwise they will work as before.

If the 2nd application is itself another API you should also add ETAGs for it, and so on. In this example we didn’t, just because I wanted to simplify the scenario. But instead of being just a simple one-to-one proxy, it could be one of those “porcelain” APIs that fetch data from several other smaller microservices, compile them down into a single structure and return it. You should create ETAGs that could be the returning ETAGs from all the other microservices digested together in a single ETAG, for example. Because you’re just receiving headers and fetching their content from an internal cache, it’s quite cheap. Something like this pseudo-code:

def index
  url1 = "http://somehost1.foo/some_endpoint/1"
  url2 = "http://somehost1.foo/some_endpoint/2"
  etag1 = etag_from(url1)
  etag2 = etag_from(url2)
  etag = Digest::MD5.hexdigest(etag1 + etag2)
  if stale?(etag: etag, public: true)
    body1 = JSON.parse fetch_from(url1)
    body2 = JSON.parse fetch_from(url2)
    result = do_some_processing(body1, body2)
    render json: result.to_json
  end
end

Another thing: you can add any vanilla HTTP Cache between your microservices, to add authorization, security, or just plain extra caching, it’s just HTTP with proper headers. But the more you exchange “304” between your services, the less processing and the less bandwidth you’re spending. It should be noticeably efficient in most cases. But again, it’s not always cheap or trivial to generate the cache keys/ETAGs to begin with, so this is the point to take more care.

And if you’re creating heavy Javascript apps that also consume those APIs, I “believe” the Ajax calls properly cache HTTP content and send back the correct “If-None-Match” and in case they receive 304s, your application should get the normal “success” triggers. I didn’t test this when I was writing this post but I think this is the case indeed. So you should automatically get better performance in your front-end application for free if you add proper ETAGs in your APIs.

This is particularly useful for APIs that return data that doesn’t change too often. If it changes every second, or every minute, you should not see too much gains. But if it’s something like this example: products lists that only change once every day or every week, or ZIP code lists, or Previous Orders in an e-commerce. Any data that changes infrequently is a good candidate. And the larger the dataset, the larger the benefits you will see (if it’s a megabyte long listing, for example). As usual, this is also no Silver Bullet, but in this case it is not so much work to add ETAGs and there are near-zero side-effects, so why not?

ETAG is just one of many other HTTP features you should be using, CORS is another one (Research Rack Cors).

If you’re from Brazil, you should watch Nando Vieira’s entire course on the broad subject of Rails Caching.

To be honest, I’m not sure how effective this technique can actually be in all kinds of scenarios so I am very interested in hearing your feedback in case you use something like this in your applications.

Comparisons are always unfair. There is no such thing as a “fair” synthetic benchmark comparison, the author is always biased towards some targeted result. It’s the old pseudo-science case of having a conclusion and picking and choosing data that backs those conclusions. There are just too many different variables. People think it’s fair when you run it in the same machine against 2 “similar” kinds of applications, but it is not. Please don’t trust me on this as well, do your own tests.

All that being said, let’s have some fun, shall we?

The Obligatory “Hello World” warm-up

For this very short post, I just created a Node.js + Express “hello world,” I will point to its root endpoint, Express, rendering a super simple HTML template with one title and one paragraph.

For Elixir, I just bootstrapped a bare-bone Phoenix project and added one extra endpoint called “/teste” in the Router. This endpoint will call the PageController, then the “teste” function, and render an EEX template with the same title and paragraph as in the Express example.

Simple enough. Phoenix does more than Express, but this is not supposed to be a fair trial anyway. I chose Siege as the testing tool for no particular reason. You can pick the testing tool you like the most. I am running this over my 2013 Macbook Pro with 8 cores and 16GB of RAM, so this benchmark will never max out my machine.

The first test is a simple run of 200 concurrent connections (the number of CPUs I have) firing just 8 requests each for 1,600. First, the Node.js + Express results:

The first run already broke a few connections, but the 2nd run picked up and finished all 1,600 requests. And this is the Phoenix results:

As you can see, Node.js has the upper hand in terms of the total time spent. One single Node.js process can only run one single real OS thread. Therefore it had to use just a single CPU core (although I had 7 other cores to spare). On the other hand, Elixir can reach all 8 cores of my machine, each running a Scheduler in a single real OS thread. So, if this test was CPU bound, it should have run 8 times faster than the Node.js version. As the test is largely a case of I/O bound operations, the clever async construction of Node.js does the job just fine.

This is not an impressive test by any stretch of the imagination. But we’re just warming up.

Oh, and by the way, notice how Phoenix logs show the request processing times in MICROseconds instead of milliseconds!

The Real Fun

Now comes the real fun. In this second run, I added a blocking “sleep” call to both projects, so each request will sleep for one full second, and this is not absurd. Many programmers will use poor code that blocks for that time, process too much data from the database, render templates that are too complex, and so on. Never trust a programmer to follow best practices all the time.

Then, I fire up Siege with ten concurrent connections and just one request each, for starters.

This is why in my previous article “Why Elixir?” (http://www.akitaonrails.com/en/2015/12/01/o-obrigatorio-por-que-elixir-visao-pessoal), I repeated many times how “rudimentary” a Reactor pattern-based solution is. It is super easy to block a single-threaded event loop.

If you didn’t know that already, how does Node.js work? In summary, it is a simple infinite loop. When a Javascript function runs, it blocks that event loop. The function has to explicitly yield control back to the loop for another function to have a chance to run. I/O calls take time and just sit back idly waiting for a response, so it can yield control back and wait for a callback to continue running, which is why you end up with the dreaded “callback pyramid hell”.

Now, with what I explained in all my previous articles, you may already know how Elixir + Phoenix will perform:

As expected, this is a walk in the park for Phoenix. It doesn’t have a rudimentary single-thread loop waiting for the running functions to yield control back willingly. The Schedulers can forcefully suspend running coroutines/processes if they think they are taking too much time (the 2,000 reductions count and priority configurations), so every running process has a fair share of resources to run.

Because of that, I can keep increasing the number of requests and concurrent connections, and it’s still fast.

In Node.js, if a function takes 1 second to run, it blocks the loop. When it finally returns, the next 1-second function can run. That’s why if I have ten requests taking 1 second each to run, the entire process will linearly take 10 seconds!

Which obviously does not scale! If you “do it right,” you can scale. But why bother?

“Node” done right

As a side note, I find it ironic that “Node” is called “Node.” I would assume that connecting multiple Nodes that communicate with each other should be easy. And as a matter of fact, it is not.

If I had spun up 5 Node processes, instead of 10 seconds, everything would take 2 seconds as five requests would block the 5 Node processes for 1 second, and when returned, the next five requests would block again. This is similar to what we need to do with Ruby or Python, which have the dreaded big Global Interpreter Locks (GIL) that, in reality, can only run one blocking computation at a time. (Ruby with Eventmachine and Python with Tornado or Twisted are similar to Node.js implementation of a reactor event loop).

Elixir can do much better in terms of actually coordinating different nodes, and it is the Erlang underpinnings that allow highly distributed systems such as ejabberd or RabbitMQ to do their thing as efficiently as they can.

Check out how simple it is for one Elixir Node to notice the presence of other Elixir nodes and make them send and receive messages between each other:

Yep, it is this simple. We have been using Remote Procedure Calls (RPC) for decades; this is not new. Erlang has implemented this for years, and it is built-in and available for easy usage out-of-the-box.

On their websites, ejabberd calls itself a “Robust, Scalable, and Extensible XMPP Server,” and RabbitMQ calls itself “Robust messaging for applications.” Now we know they deserve the labels “Robust” and “Scalable.”

So, we are struggling to do things that have been polished and ready for years. Elixir is the key to unlocking all this Erlang goodness right now. Let’s just use it and stop shrugging.

In the mid-section of the article I described how I was still puzzled about what would be the best way to handle several HTTP requests to an unstable external source where I can’t control timeout or other network problems.

A big Manga such as Naruto or Bleach has dozens of chapters with dozens of pages each, accounting for thousands of necessary HTTP requests. Elixir/Erlang allow me to fire up as many parallel HTTP requests as I want. But doing that makes the HTTP requests time out very quickly (it’s like doing a distributed denial of service attack against MangaReader).

By trial and error I found that firing up fewer than 100 HTTP requests at once allows me to finish. I capped it at 80 to be sure, but it really depends on your environment.

Then I had to manually chunk my list of pages into groups of 80 elements and process them in parallel, finally reducing the resulting lists back into a larger list to pass through to the next steps in the Workflow. The code gets convoluted like this:

def images_sources(pages_list) do
  pages_list
    |> chunk(@maximum_fetches)
    |> Enum.reduce([], fn pages_chunk, acc ->
      result = pages_chunk
        |> Enum.map(&(Task.async(fn -> Page.image(&1) end)))
        |> Enum.map(&(Task.await(&1, @http_timeout)))
        |> Enum.map(fn {:ok, image} -> image end)
      acc ++ result
    end)
end

Now I was able to reimplement this aspect and the same code now looks like this:

def images_sources(pages_list) do
  pages_list
    |> Enum.map(&Worker.page_image/1)
    |> Enum.map(&Task.await(&1, @await_timeout_ms))
    |> Enum.map(fn {:ok, image} -> image end)
end

Wow! Now this is a big improvement and it’s way more obvious what it’s doing.

Best of all: downloading a Manga the size of Akira (around 2,200 pages long) took less than 50 seconds. And this is not because Elixir is super fast, it’s because MangaReader can’t keep up if I extend the Pool size. It’s hitting at a constant rate of 50 concurrent connections!

This makes my 4-core machine, sitting on a 40 Mbps connection, use less than 30%-40% of CPU and no more than around 3.5 Mbps. If MangaReader could keep up we could easily fetch all pages 2 or 3 times faster without breaking a sweat.

It was fast with the previous strategy, but I guess it got at least twice as fast as a bonus now. But how did I accomplish that?

Open Telecom Platform

In the previous article I also said that I didn’t want to dive into “OTP and GenServers” just yet. But if you’re a beginner like me you probably didn’t understand what this means.

OTP is what makes Erlang (and consequently, Elixir) different from pretty much every other language platform but, maybe, Java.

Many new languages today let you do many tasks in parallel through convoluted Reactor patterns (Node.js, EventMachine/Ruby, Tornado/Twisted/Python, etc) or through (cleaner) green threads (Scala, Go).

But none of this matters. It’s not difficult to launch millions of lightweight processes, but it is not trivial to actually CONTROL them all. It doesn’t matter how fast you can exhaust your hardware if you can’t control it. You end up with millions of dumb minions wreaking havoc without an adult to coordinate them.

Erlang solved this problem decades ago through the development of OTP, initially called Open Systems, within Ericsson in 1995. By itself OTP is a subject that can easily fill an entire fat book and you will still not be able to call yourself an expert.

Just so I don’t get too boring here:

• Start with this very brief summary;
• Then read Elixir’s tutorial on Supervisors and the page on Processes first if you haven’t already;
• Finally, you can go further with The Elixir & OTP Guidebook from Manning Publications.

Now, below is my personal point of view. As I am a beginner myself, let me know in the comments section below if I got some concept wrong.

OTP is a collection of technologies and frameworks. The part that interests us the most is to understand that this is a sophisticated collection of patterns to achieve the Nirvana of highly reliable, highly scalable, distributed systems. You know? That thing every new platform promises you but fails to actually deliver.

For our very simple intents and purposes, let’s pick up where I said before: it’s trivial to fire up millions of small processes. We call them “workers”. OTP provides the means to control them: Supervisors. And then it also provides the concept of Supervisor Trees (Supervisors that supervise other Supervisors). This is the gist of it.

Supervisors are responsible for starting up the workers and also for recovering from exceptions coming from the workers (which is why in Erlang/Elixir we don’t do ugly try/catch stuff: let the error be raised and caught by the Supervisor). Then we can configure the Supervisor to deal with a faulty worker by, for example, restarting it.

We already touched on this OTP stuff before. An Elixir Task is just a high-level abstraction. It internally starts its own supervisor to watch over asynchronous tasks.

There are so many subjects and details that it’s difficult to even get started. One concept that is important to know is about state. There is no global state! (Yay, no JavaScript globals nightmare.) Each function has its own state and that’s it. There is no concept of an “object” that holds state and then methods that can modify that state.

But there is the concept of Erlang processes. Now, a process does have state, it’s a lightweight piece of state that exists only at runtime. To execute a function in a separate, parallel process, you can just do:

iex> spawn fn -> 1 + 2 end
#PID<0.43.0>

Unlike an object, a process does not have a set of methods that access its inner “this” or “self” state. Instead each process has a mailbox. When you start (or “spawn” in Erlang lingo) a new process, it returns a pid (process ID). You can now send messages to the process through its pid. Each process has a mailbox and you can choose to respond to incoming messages and send responses back to the pid that sent the message. This is how you can send a message to the IEx console and receive the messages in its mailbox:

iex> send self(), {:hello, "world"}
{:hello, "world"}
iex> receive do
...>   {:hello, msg} -> msg
...>   {:world, msg} -> "won't match"
...> end
"world"

In essence, it’s almost like an “object” that holds state. Each process has its own garbage collector, so when it dies it’s individually collected. And each process is isolated from other processes; they don’t bleed state out, which makes them much easier to reason about.

The Getting Started page on Processes from the Elixir website shows examples of what I just explained and I recommend you follow it thoroughly.

In summary, a process can hold internal state by blocking indefinitely while waiting for an incoming message in its mailbox and then recursing into itself! This is a mind-blowing concept at first.

But a simple process is just too damn weak. This is where you get OTP’s GenServer, which is a much more accomplished process. OTP exposes Behaviours for you to implement in order to add your own code but it takes care of the dirty infrastructure stuff so you don’t have to.

Deferring the Heavy Workflow calls to a Worker

All that said, we know that in the Workflow we implemented before, we have trouble with the Page.image/1 and Workflow.download_image/2 functions. This is why we made them asynchronous processes and we wait for batches of 80 calls every time.

Now, let’s start by moving this logic away to a GenServer Worker, for example, in the ex_manga_downloadr/pool_management/worker.ex file:

defmodule PoolManagement.Worker do
  use GenServer
  use ExMangaDownloadr.MangaReader

  @timeout_ms 1_000_000
  @transaction_timeout_ms 1_000_000 # larger just to be safe

  def start_link(_) do
    GenServer.start_link(__MODULE__, nil, [])
  end

  def handle_call({:page_image, page_link}, _from, state) do
    {:reply, Page.image(page_link), state}
  end

  def handle_call({:page_download_image, image_data, directory}, _from, state) do
    {:reply, Page.download_image(image_data, directory), state}
  end
end

I first moved the Workflow.download_image/2 to Page.download_image/2 just for consistency’s sake. But this is a GenServer in a nutshell. We have some setup in the start_link/1 function and then we have to implement handle_call/3 functions to handle each kind of argument it might receive. We separate them through pattern matching the arguments.

As a convention, we can add public functions that are just prettier versions that call each handle_call/3:

  def page_image(page_link) do
    Task.async fn ->
      :poolboy.transaction :worker_pool, fn(server) ->
        GenServer.call(server, {:page_image, page_link}, @timeout_ms)
      end, @transaction_timeout_ms
    end
  end

  def page_download_image(image_data, directory) do
    Task.async fn ->
      :poolboy.transaction :worker_pool, fn(server) ->
        GenServer.call(server, {:page_download_image, image_data, directory}, @timeout_ms)
      end, @transaction_timeout_ms
    end
  end

But we are not just calling the previous handle_call/3 functions. First there is the Task.async/1 we were already using in the Workflow functions to make the parallel batches. But inside the Task calls there is this other strange thing: :poolboy.

Controlling Pools of Processes through Poolboy

The entire OTP ordeal I wrote here was just an introduction so I could show off Poolboy.

Repeating myself again: it’s trivial to fire up millions of processes. OTP is how we control failures to those processes. But there is another problem: the computation within each process may be so heavy we can either bring down the machine or, in our case, do a Distributed Denial of Service (DDoS) against the poor MangaReader website.

My initial idea was to just do parallel requests in batches. But the logic is convoluted.

Instead, we can use a process pool! It queues up our requests for new processes. Whenever a process finishes, it is returned to the pool and a new computation can take over the available process. This is how pools work (you probably have an intuition of how it works from traditional database connection pools). Pools and queues are useful software constructs to deal with limited resources.

By doing this we can remove the chunking of the large list into batches and process every element of the large list in parallel at once, repeating again the initial version:

pages_list
  |> chunk(@maximum_fetches)
  |> Enum.reduce([], fn pages_chunk, acc ->
    result = pages_chunk
      |> Enum.map(&(Task.async(fn -> Page.image(&1) end)))
      |> Enum.map(&(Task.await(&1, @http_timeout)))
      |> Enum.map(fn {:ok, image} -> image end)
    acc ++ result
  end)

Now, removing the chunking and reducing logic:

pages_list
  |> Enum.map(&(Task.async(fn -> Page.image(&1) end)))
  |> Enum.map(&(Task.await(&1, @http_timeout)))
  |> Enum.map(fn {:ok, image} -> image end)

And finally, replacing the direct Task.async/1 call with the GenServer worker we just implemented above:

pages_list
  |> Enum.map(&Worker.page_image/1)
  |> Enum.map(&Task.await(&1, @await_timeout_ms))
  |> Enum.map(fn {:ok, image} -> image end)

Now, Poolboy will require a Supervisor that monitors our Worker. Let’s put it under ex_manga_downloadr/pool_management/supervisor.ex:

defmodule PoolManagement.Supervisor do
  use Supervisor

  def start_link do
    Supervisor.start_link(__MODULE__, [])
  end

  def init([]) do
    pool_options = [
      name: {:local, :worker_pool},
      worker_module: PoolManagement.Worker,
      size: 50,
      max_overflow: 0
    ]

    children = [
      :poolboy.child_spec(:worker_pool, pool_options, [])
    ]

    supervise(children, strategy: :one_for_one)
  end
end