Running a 35B MoE Model on Your Laptop by Streaming Experts from SSD

You've probably looked at the memory requirements for a 35B-parameter model and quietly closed the tab. Consumer hardware has a hard ceiling, and dense model weights blow right past it. The usual answer is "buy more RAM" or "rent a GPU." edge0 takes a different route: it treats your SSD as an extension of memory and streams only the expert weights the model actually needs, when it needs them.

What It Does

edge0 is an open-source streaming MoE inference framework. Instead of holding every expert weight in memory, it offloads experts to storage and loads them on demand. Peak memory is bounded by the active expert set rather than the total parameter count, which is what makes large sparse-MoE models viable on hardware that couldn't otherwise hold them.

The framework rests on three mechanisms. SSD expert offload handles the streaming itself. A prerouter is a trained head that predicts expert routing one step ahead, so expert loads overlap with the forward pass instead of stalling it. And Recover-LoRA keeps the int4 base frozen while training LoRA adapters via distillation from an FP teacher, recovering most of the quantization loss at 4-bit. The adapters are stored separately from the base and can be switched on or off.

As of the November 2026 release, the project ships a unified inference engine called edge0-infer, built on GGML. One engine and one native C++ API run the same model across macOS, iOS, Android, and Windows, with a CLI, an OpenAI-compatible local server, and chat app examples for desktop, Android, and iOS. This replaced the earlier Python framework and per-platform engines. The first release supports edge0-35b, with edge0-8b planned for a later release.

Why It's Cool

• The memory ceiling moves in the right direction. Most quantization work tries to shrink the model. edge0 sidesteps that framing entirely — peak memory tracks the active expert set, not the parameter count. That's a fundamentally different constraint, and it's the one that actually matters on a laptop or phone.

• The prerouter is the clever bit. Streaming weights from disk is slow if you do it naively — the forward pass sits idle waiting for data. By training a head to predict routing one step ahead, edge0 overlaps the load with computation. The README claims up to +59% decode throughput, and notes the gain grows with storage latency, model size, and routed width K. That last part is honest and useful: it tells you where this approach pays off most.

• Recover-LoRA addresses the obvious follow-up question. If you're running int4, you're losing quality. Training adapters by distillation from the FP teacher to recover most of that loss at 4-bit is a practical answer, and keeping adapters separate and switchable is a nice design touch.

• One engine, four platforms. Native C++ on GGML across macOS, iOS, Android, and Windows means you're not maintaining four separate inference paths. The OpenAI-compatible local server also means existing tooling can talk to it without changes.

• There's a paper. The technical report, "The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction," is on arXiv if you want the details behind the throughput numbers.

How to Try It

1. Head to the repository: github.com/edge0-ai/edge0. The README links the edge0-infer engine docs at edge0-infer/README.md — start there for build and run instructions.
2. Grab a model. Both tiers are on Hugging Face and ModelScope: Edge0-35B-A3B-preview and Edge0-8B-A1B-preview. Note that the current edge0-infer release supports edge0-35b; the 8B tier is slated for a later release.
3. Build the engine and use the CLI or spin up the OpenAI-compatible local server. Chat app examples are included for desktop, Android, and iOS if you want a reference integration.
4. If you want the older Python framework or the per-platform engines, they're preserved at the pre-edge0-infer tag.
5. The project is Apache 2.0 licensed, and there's a Discord linked from the README if you hit snags.

Final Thoughts

edge0 is aimed at a specific kind of developer: someone who wants to run a large sparse-MoE model locally and is willing to accept the tradeoffs that come with SSD streaming — storage latency, the active-set bound, and quantized weights with LoRA recovery. It's not a drop-in replacement for a GPU cluster, and the README doesn't pretend otherwise. What it does offer is a credible, published, multi-platform path to running models that would otherwise be out of reach on consumer hardware. If that's a constraint you've been bumping into, the paper and the engine are both worth your time.

Follow @githubprojects for more developer tools and open source projects.