JustVugg / colibri

Website · Discord · English · 简体中文 · 繁體中文 · Italiano

Tiny engine, immense model. Run frontier MoE models — 744B to 2.8T parameters — on consumer and heterogeneous hardware, in pure C with zero engine dependencies, by treating storage, RAM, and VRAM as a single inference hierarchy (AI memory multitiering).

Six families run today: GLM-5.2 (744B), Inkling (975B), Kimi K3 (2.8T), DeepSeek V4 Flash (284B), Qwen3.6 (35B-A3B) and OLMoE (7B) — one C file each, the same coli chat / coli serve / coli web front end. Full roster ↓

Colibrì is an inference engine you can run today, and an open research platform. Its primary goal is to pursue inference-side performance across the entire software/hardware boundary — model formats, memory hierarchy, storage I/O, placement, scheduling, kernels, speculation, and CPU/GPU overlap — so large models depend less on scarce hardware and cost less to run.

Colibrì treats VRAM, RAM, and storage as a single multitier hierarchy, and it is deliberately a place to test aggressive systems ideas — so there is no SLA on speed, and a hard guarantee on semantics: experiments must earn their place through reproducible end-to-end measurements, and the default policy never silently changes model precision or router semantics. Insufficient fast memory may reduce speed; it must not quietly redefine the model.

$ ./coli chat
  🐦 colibri v1.8.0 — GLM-5.2 · 744B MoE · int4 · streaming CPU
  ✓ ready in 32s · resident 9.9 GB
  › ciao!
  ◆ Ciao! 😊 Come posso aiutarti oggi?

See it running

The web dashboard (./coli web): a 744B model at 4 tok/s, TTFT 1.6 s, disk 0 — full expert residency on 6× RTX 5090, with live token metrics, the per-turn time breakdown, the VRAM/RAM/disk tier bar and the live mini-brain in the corner.

The Brain page: all 19,456 experts as a living cortex — colour is the storage tier, brightness is routing heat, and every expert routed in a turn flashes white. Hovering shows the expert's measured topic affinity.

The Atlas page: the measured expert atlas as a 3-D galaxy — 13,260 characterised experts, 1,041 replicated specialists clustering by topic (poetry, law, Chinese, SQL…). Position is measured routing affinity, not a learned embedding. Drag to spin.

The research mission

With Colibrì, private frontier model access is not limited by availability of hyperscaler-class hardware.

With its multitiering features Colibrì removes proprietary hardware dependencies aggressively optimizing functional inference engine pipelines.

Our operational mission includes changing how weights are represented and moved, deciding what lives in VRAM, RAM, or storage, overlapping heterogeneous compute, reducing launch and synchronization overhead, exploiting sparsity and reuse, and testing new decoding algorithms. Nothing is protected merely because it is conventional; nothing is adopted merely because a microbenchmark looks fast. The deciding result is end-to-end inference on real machines, with correctness and quality measured alongside throughput, latency, memory, and cost.

The practical consequence is accessibility: run a 744B-parameter model on hardware you already own, watch every expert fire in real time, and change the code that does it. Not renting intelligence behind an API — holding it: probing it, measuring it, improving it. The engine is deliberately small enough that the next useful optimization can come from anyone willing to measure it.

Core techniques and measured findings

• One hierarchy, not limited by tier capacity. VRAM, RAM, and NVMe are placement tiers for the same weights; limited fast memory changes speed, not model semantics.
• A JIT for weights. Measured routing heat drives a per-layer LRU, a learned pinned hot-store, and one-layer-ahead prefetch instead of loading every expert. It wins on repeatable workloads; history can overfit, and lookahead can lose on some hosts, so both remain measurable policies rather than promises.
• I/O is part of the engine. Batched expert unions, overlapped reads and compute, O_DIRECT, and weighted dual-SSD striping attack the streaming path rather than pretending storage latency is free. O_DIRECT is drive-dependent, and dual-SSD still needs broader end-to-end community A/Bs.
• Heterogeneous execution. CPU, CUDA, Metal, NUMA memory, and partial or full expert residency share one runtime and can be combined according to the machine; the profitable combination depends on compute, bandwidth, residency, and workload.
• Compressed state without a different model. Token-exact forward validation, 57× smaller MLA KV state, persistent warm conversations, and faithful DSA keep optimization tied to correctness. These are memory, latency, and correctness properties — not a blanket throughput claim.
• Speculation that must earn its keep. Native MTP and grammar-forced drafts are measured end to end and can be disabled when acceptance does not repay verification.

Open hypotheses, experiments, and how to help

Colibrì treats an optimization as a hypothesis until a controlled end-to-end A/B shows otherwise. These are the main questions now:

hypothesis	evidence so far	experiment still needed
Routing history can place experts better than plain LRU	learned pins improve repeated workloads, but can overfit a prompt	held-out, cross-session A/Bs across coding, chat, multilingual, and long-context workloads
Multiple SSDs can turn independent bandwidth into decode speed	weighted mirror/split routing is implemented and validated; the bandwidth model is sound	cold-cache one-drive vs two-drive GLM-5.2 runs on real, independent controllers
A hardware-aware planner can approach each machine's best configuration automatically	RAM/VRAM budgets and several backends are detected today	compare the generated plan with a controlled parameter sweep across laptops, workstations, NUMA hosts, and multi-GPU systems
Lossless or quality-bounded representations can reduce weight movement enough to matter	format and quantization ablations exist, with correctness/quality gates	reproduce quality, bytes moved, latency, and cost per useful token together — not compression ratio alone
Routing-aware speculation can pay before near-full residency	MTP and grammar drafts work, but MTP has also measured a 32% loss around 85% expert hit	map the break-even surface across acceptance, expert hit rate, batch union, and draft depth
CPU/GPU over