Loading theme preference
Ilia Bukin
← Back to blog

colibri: the pure-C engine that streams 744B+ MoE models off your NVMe

Sep 14, 2026
inference
moe
local-llm
c
opensource

v1.11.0 of colibri landed yesterday, and it is a good excuse to look at what this project has become. The tagline is "tiny engine, immense model": a pure-C inference engine with zero runtime dependencies that streams Mixture-of-Experts weights from disk, so frontier-sized models run on hardware you already own. The new release adds a ninth model family, and on GitHub the repo is sitting at roughly 30.6k stars after picking up 868 stars today.

Don't fit it, place it

The core bet is a fact about MoE sparsity. A 744B model like GLM-5.2 only activates about 40B parameters per token, and only around 11 GB of those are the routed experts that change from token to token. So the model doesn't need to fit in fast memory; it needs to be placed across whatever memory hierarchy the machine has.

In colibri's terms: the dense trunk (attention, shared experts, embeddings, roughly 17B parameters) stays resident in RAM at int4, around 9.9 GB. The 19,456 routed experts (75 MoE layers × 256, plus the MTP head, about 19 MB each at int4) live on disk (~370 GB) and are streamed on demand through a per-layer LRU cache, a learned pinned hot-store, and an optional VRAM tier. The project describes the algorithm as "a JIT, but for weights": watch which experts the workload actually routes to, and stage those into the faster tiers exactly when they're needed.

That framing extends further. VRAM, RAM, and NVMe are treated as one placement hierarchy, and the README states a hard guarantee: placement only ever changes speed, never semantics. Insufficient fast memory makes generation slower; it does not quietly change precision or router decisions.

Inside the per-token path

Every layer of every token walks the same five steps: route, union, place, overlap, learn. Most of the cleverness is about not waiting on disk:

  • Batch-union reads. Multiple positions that hit the same expert read it once, not once per position.
  • One-layer-ahead prefetch. A router-lookahead thread preloads the next layer's experts; the README measures routing as 71.6% predictable one layer ahead.
  • O_DIRECT. Bypassing the page cache measured +34% decode on a Blackwell/Windows box, but it's drive-dependent: the docs say QLC or DRAM-less drives can make it neutral or negative, so measure on your own hardware.
  • Dual-SSD mirroring. If you have a second drive, a byte-identical copy is read from both in parallel, weighted by measured bandwidth; a 9 GB/s + 3 GB/s pair reads experts ~33% faster than the fast drive alone.
  • Compressed KV state. MLA stores 576 floats per token instead of 32,768, a 57× reduction, and persists it across restarts, so conversations reopen warm with zero re-prefill.
  • Speculative decoding that can be switched off. The native MTP head drafts 2.2 to 2.8 tokens per forward when it pays, and the engine ships with hard-won defaults: the MTP head must be int8 (int4 heads collapse to 0 to 4% acceptance), and draft and verify must compute the same function.

There's also a local cluster mode: a coordinator keeps generation, routing, and KV state on one machine while disk-backed expert workers on other Macs execute the routed FFNs, with each layer's batch-union sent as one persistent TCP request.

What v1.11.0 adds

The headline is a ninth engine: DeepSeek V4.1 Flash (552B), with no conversion step. The released checkpoint is read natively, fp8 dense with fp4 experts, streaming from an SSD on a CPU box. The release notes are specific about what that buys: batched expert reads took one measured turn from 78.7 s to 25.1 s (0.305 to 0.957 tok/s), and a five-turn chat session runs at 1.14 to 1.58 tok/s. Vision, tool calling, and the DSpark draft head are all wired in.

The same release adds an opt-in Metal path (COLI_METAL=1) that moves GLM-5.3-Flash's routed experts onto Apple Silicon GPUs, measured on an M4 Max at 0.469 to 0.621 tok/s at 32 tokens, up to 2.15 tok/s with the expert cache tuned. Less glamorous but telling: CI now builds against musl and compiles the environment tests on Windows, the two holes that had let bugs ship.

The honest numbers

What makes the project worth following is that the numbers are published alongside their caveats, on a benchmarks page built from community runs:

  • 6× RTX 5090, full residency: 5.8 to 6.8 tok/s decode, TTFT ~13 s.
  • 128 GB CPU-only desktop: ~1.8 tok/s warm.
  • Single RTX 5070 Ti laptop-class box: 1.07 tok/s.
  • The original dev box (WSL2, 12 cores, 25 GB RAM): 0.05 to 0.1 tok/s cold, the honest floor the project started from.
  • An Azure EPYC with 504 GB and four NVMe in RAID0: 2.83 tok/s rotating median, 6.91 tok/s warmed.

Quality is measured rather than assumed: the int4 container scores 62.5% mean acc_norm on hellaswag/arc/mmlu (0-shot), and an fp16-vs-int4 A/B puts the pure quantization cost at -8.2pp. The docs are equally explicit about what doesn't work. "No SLA on speed" is a design position, two of the ideas built for v1.11.0 (ways of hiding expert reads behind matmuls) measured worse and were removed with the numbers written up, and the project asks contributors to publish negative results. A controlled failure is worth more to it than an unexplained fast number.

Getting started

You need two things: the program (a few hundred KB) and the model. Prebuilt release archives cover Linux, macOS, and Windows, no compiler needed; only Python 3 is required, for the launcher and the optional API gateway. Building from source needs gcc or clang with OpenMP, then ./setup.sh. The engine itself is one C file (c/colibri.c) plus small headers: no BLAS, no Python at runtime, no GPU required.

The reference model is GLM-5.2, whose converted int4 container (grouped scales, int8 MTP head) runs to roughly 370 to 390 GB on disk. Put it on the fastest drive you have. Then:

COLI_MODEL=/nvme/glm52_i4 ./coli chat     # TUI
./coli web --model /nvme/glm52_i4         # API + dashboard
./coli serve --model /nvme/glm52_i4       # headless API

coli reads the model's config.json, picks the matching one of the nine engine binaries, and renders that family's chat template, so the command line never changes between models. The same flag three-liners run everything from OLMoE (7B, ~7 GB) up to Kimi K3 (2.8T, ~1.6 TB).

The caveats, and why it matters anyway

Colibri is not fast in the way you want. Decode is disk-bound, so your NVMe rather than your GPU sets the rate, and on a slow drive you wait. The RAM cap, not the GPU, is the binding constraint on small boxes. It's a research platform with no SLA, and some of its own headline features (MTP speculation, O_DIRECT) can lose on the wrong hardware. None of that is hidden; it's in the README, next to the numbers.

What matters is the direction. The project launched on July 10 and hit 453 points on Hacker News within hours (per this independent writeup), and nine model families later the argument is harder to dismiss: a 744B or even 2.8T model does not require hyperscaler-class hardware to run privately. It requires an engine that treats the model as data to be staged across VRAM, RAM, and NVMe rather than as resident state to be held. That's a bet on disk bandwidth replacing GPU scarcity, and colibri is publishing the measurements to back it up.