Running 744B-Parameter AI Models on Consumer Hardware Without GPU 🔗
Pure C engine streams Mixture-of-Experts layers from disk, enabling massive language models on modest RAM setups
Why this leads today JustVugg/colibri demonstrates that a 744B Mixture-of-Experts model can run on a 25GB RAM consumer machine using only pure C and disk-streamed experts, eliminating the need for GPU clusters and enabling frontier AI development on everyday hardware.
A new open-source project named Colibrì is turning heads in the AI infrastructure space by demonstrating how to run GLM-5.2, a 744-billion-parameter Mixture-of-Experts (MoE) model, on a consumer machine with just 25GB of RAM — no GPU required, no Python dependencies, and a single C file at its core. The project, hosted at JustVugg/colibri, achieves this by treating system RAM, VRAM, and storage as a unified memory hierarchy, streaming only the necessary experts from disk during inference while keeping the dense model components resident in memory.
At the heart of Colibrì’s innovation is its recognition that although GLM-5.2 contains 744B parameters, only a fraction — roughly 40B — are activated per token, with just ~11GB of those changing dynamically as routed experts shift. The engine keeps the dense portion (attention layers, shared experts, embeddings) in RAM at int4 precision (~9.9GB), while the 19,456 routed experts — each ~19MB at int4 — reside on disk (~370GB total) and are loaded on demand via an LRU caching mechanism. This approach leverages the OS page cache as a free secondary tier, with optional CUDA support for pinning frequently used experts to GPU memory.
The runtime, implemented in a single C source file (c/glm.c) with minimal headers, avoids external libraries like BLAS or frameworks such as PyTorch. It delivers token-exact forward passes validated against the Hugging Face transformers library, ensuring numerical fidelity. A built-in web dashboard (./coli web) provides real-time visualization of expert routing, hardware utilization, and storage tiers, showing the model operating at 4+ tokens per second end-to-end on high-end consumer GPUs — though CPU-only operation remains viable, albeit slower.
Colibrì reframes the accessibility of frontier-scale models: instead of requiring datacenter-grade hardware, it enables researchers, indie developers, and educators to experiment with state-of-the-art MoE architectures using existing laptops or workstations. By eliminating runtime dependencies and optimizing for disk-backed expert streaming, it lowers the barrier to probing massive models without cloud costs or complex DevOps overhead.
The catch: The current implementation prioritizes correctness and flexibility over peak throughput, with performance heavily dependent on disk I/O speed — meaning sustained generation rates on HDDs or slow SSDs may fall significantly below the 4+ tok/s demonstrated on premium NVMe setups, and expert streaming latency remains an open optimization challenge for real-time interactive use.
- Researchers testing massive MoE models on personal workstations
- Educators demonstrating LLM inference mechanics without cloud reliance
- Developers prototyping AI applications avoiding GPU dependency and vendor lock-in
Source: JustVugg/colibri — based on the project README.