Pure C Engine Runs Trillion-Parameter Models on Consumer Hardware 🔗
Colibrì streams experts from disk to treat storage, RAM, and VRAM as a unified inference hierarchy
JustVugg/colibri is a pure C inference engine that runs frontier mixture-of-experts (MoE) models — from 744B to 2.8T parameters — on existing consumer and heterogeneous hardware without dependencies.
By treating storage, RAM, and VRAM as a single multitier memory hierarchy, it streams experts from disk on demand, enabling massive models to run where GPU memory alone would fail. The engine consists of one C file per model family (GLM-5.2, Inkling, Kimi K3, DeepSeek V4 Flash, OLMoE), sharing a common coli CLI for chat, serve, and web frontends. Its web dashboard visualizes expert routing in real time: the Brain page shows all 19,456 experts as a living cortex, with color indicating storage tier and brightness reflecting routing heat; the Atlas page maps expert affinity as a 3-D galaxy of characterized specialists. Recent v1.6.2 release patches six memory-safety flaws reachable via malicious model files or input, validating all fixes at the trust boundary without altering behavior on well-formed models. The engine prioritizes semantic correctness over speed guarantees — insufficient fast memory reduces performance but never silently changes model precision or router semantics.
The catch: Colibrì makes no SLA on speed; throughput depends entirely on storage I/O and memory hierarchy efficiency, making it unsuitable for latency-sensitive applications despite enabling massive model access.
Why this leads today JustVugg/colibri enables running large Mixture-of-Experts models on standard hardware using only C and no external dependencies, making advanced AI accessible without cloud or specialized infrastructure.
- Researchers testing trillion-parameter models on multi-GPU workstations
- Developers serving large MoE models without specialized AI accelerators
- Educators demonstrating expert routing and memory hierarchy dynamics live
Source: JustVugg/colibri — based on the README and release notes.