JustVugg / colibri

Tiny engine, immense model. Run GLM-5.2 (744B-parameter MoE) on a consumer machine with ~25 GB of RAM — in pure C, with zero dependencies, by streaming experts from disk.

$ ./coli chat
  🐦 colibrì v1.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU
  ✓ ready in 32s · resident 9.9 GB
  › ciao!
  ◆ Ciao! 😊 Come posso aiutarti oggi?

The idea

A 744B Mixture-of-Experts model activates only ~40B parameters per token — and only ~11 GB of those change from token to token (the routed experts). So:

• the dense part (attention, shared experts, embeddings — ~17B params) stays resident in RAM at int4 (~9.9 GB);
• the 21,504 routed experts (75 MoE layers × 256 experts + the MTP head, ~19 MB each at int4) live on disk (~370 GB) and are streamed on demand, with a per-layer LRU cache, an optional pinned hot-store, and the OS page cache as a free L2.

The engine is a single C file (c/glm.c, ~2,400 lines) plus small headers. No BLAS, no Python at runtime, no GPU required (an opt-in CUDA tier for pinned experts exists — see below).

What's implemented

• Faithful GLM-5.2 (glm_moe_dsa) forward — validated token-exact against a transformers oracle (teacher-forcing 32/32, greedy 20/20 on a tiny-random model with the real architecture).
• MLA attention (q/kv-LoRA, interleaved partial RoPE) with compressed KV-cache: 576 floats/token instead of 32,768 (57× smaller — GLM-5.2 has 64 heads and no GQA).
• DeepSeek-V3-style sigmoid router (noaux_tc, routed_scaling_factor), shared expert, first-3-dense layers.
• Native MTP speculative decoding — GLM-5.2's own multi-token-prediction head (layer 78) drafts tokens that the main model verifies in one batched forward. The head must be int8 (the converter does this by default): at int4 draft acceptance collapses to 0–4% and speculation never engages; at int8 it's 39–59% acceptance, 2.2–2.8 tokens/forward (community-measured, #8). Lossless in exact arithmetic — but not byte-identical to non-speculative greedy in practice (#100). This isn't MTP-specific: colibrì's quantized integer kernels are shape-dependent, so any batched (S>1) or GPU forward rounds slightly differently from the single-token path, and int4 GLM-5.2 sits close enough to argmax ties that such a rounding change can flip a token. MTP, the CUDA expert tier, and batched prefill are three different ways to trip the same sensitivity (community-confirmed in #100: swapping only the kernel family forks greedy output on 3/5 prompts, with zero speculation). Every emitted token is still the argmax of a valid forward — the continuation stays correct — it just isn't the same stream. For byte-exact reproducibility: DRAFT=0 (no speculation), plus IDOT=0 COLI_CUDA=0 if you also want kernel-family/GPU independence. Under sampling, rejection sampling keeps the distribution correct. Honest caveat from the same measurement: on a cold cache each verified draft routes to extra experts (~660 → ~1100 expert-loads/token), so speculation can be a net time loss until the cache/pin warms up.
• Grammar-forced speculative drafts (GRAMMAR=file.gbnf, #48) — on constrained-output workloads (JSON/NDJSON, function calling, structured extraction) the grammar itself is a third draft source: wherever it admits exactly one legal byte (braces, quotes, key names, enum bodies), that forced span is tokenized and injected as pre-accepted drafts with ~1.0 acceptance — no draft head, no lookup table, and it engages even with the int4 MTP head from #8. It never constrains sampling: forced spans are verified in the same batch-union forward as any draft, so a wrong or out-of-sync grammar cannot change the output — worst case is rejected drafts, and an adaptive guard turns the source off below 50% acceptance. Byte-level GBNF subset (literals, char classes, | ( ) ? * +, comments); GRAMMAR_DRAFT=n caps the forced span per forward (default 24). Composes with DRAFT/MTP, which fill the free-text gaps between forced spans.
• True sampling — temperature + nucleus, defaults tuned for int4 reality (0.7 / 0.90; the official 1.0 / 0.95 samples quantization noise from the tail).
• Integer-dot kernels (Q8_0-style int8 activations, AVX2 maddubs): int8 matmuls 1.4–2.5× faster (119 GFLOP/s measured), int4 1.8× in batch — routing decided per shape by measurement (int4 single-row stays f32: it measured slower).
• MLA weight absorption (DeepSeek trick) for decode: no per-token k/v reconstruction — the query absorbs kv_b, context is projected after attention. Validated exact: TF 32/32 and generation 20/20 with absorption forced everywhere.
• Async expert readahead: while one block of experts is being multiplied, the kernel is already reading the next (WILLNEED).
• Quantization kernels: int8 / packed int4 / packed int2, per-row scales, AVX2, dequant-on-use. Packing validated bit-identical to the int8 container.
• DSA sparse attention — GLM-5.2's lightning indexer, faithful to the reference glm_moe_dsa modeling: per-layer top-2048 causal key selection (full/shared indexer layers), auto-detected from the out-idx-* weights (--indexer converter mode, ~189 MB extracted from the FP8 repo). Validated exact: forcing the selection to keep every key reproduces dense attention token-for-token. DSA=0 disables, DSA_TOPK overrides.
• KV-cache persistence — conversations reopen warm across engine restarts: serve mode appends the compressed MLA KV to .coli_kv after every turn (~182 KB/token, crash-safe) and resumes it at startup with zero re-prefill. Validated byte-identical to an uninterrupted session. KVSAVE=0 disables.
• Router-lookahead prefetch (PILOT=1, experimental) — the next layer's routing is 71.6% predictable from the current layer's post-attention state (measured); a dedicated I/O thread prefetches those experts while the current layer computes.
• Batch-union MoE: in prefill (and MTP verification), each unique expert of the batch is read once and applied to every position that routes to it.
• Byte-level BPE tokenizer in C (GPT-2-style with Unicode-property regex, 320k merges).
• RAM safety: the expert cache is auto-sized from MemAvailable at startup — an honest peak projection (working set, KV, MTP row, reconstruction buffers) so the kernel OOM-killer never fires.
• Offline FP8→int4 converter (c/tools/convert_fp8_to_int4.py): downloads one shard at a time (~5 GB), dequants (128×128 block scales), requantizes to the engine's container, deletes the shard — the 756 GB FP8 checkpoint never needs to exist on disk at once. Resumable.

Honest numbers (WSL2, 12 cores, 25 GB RAM, NVMe via VHDX)

metric	value
model on disk (int4 container)	~370 GB
resident RAM (dense, int4)	9.9 GB
load time	~30 s
peak RSS during chat	~20 GB (auto-capped)
cold decode cost	~11 GB disk reads/token (75 layers × 8 experts)
disk ceiling (this dev box's drive)	~1 GB/s → ~0.05–0.1 tok/s cold
MTP speculation (int8 head)	2.2–2.8 tok/forward measured (#8)

This is not fast. It is a 744B frontier-class model answering correctly on a machine that costs less than one H100 fan. Warm cache, pinned hot experts and MTP push the useful-response latency down considerably; the physics of the disk does the rest.

SSD note

Cold starts are heavy on random reads (~11 GB/token), but reads don't meaningfully wear an SSD — colibrì's streaming is read-only. The real concerns under heavy use are (1) swap traffic if the system runs out of RAM (writes do wear the drive — keep a sane --ram