opensourceprojects.dev

A broadsheet for software that doesn't ask for your email

A 125B model on a 12 GB GPU, running locally
GitHub RepoViews5

Project Description

View on GitHub

Running a 125B Model on a 12 GB GPU: Inside Linger's Local Agent

You've probably noticed that running a genuinely capable language model locally usually means one of two things: buying a GPU that costs more than your car, or settling for a smaller model that stumbles on anything complex. Linger is an open-source project that takes a different approach—it's a personal agent built around a 125B model that runs on everyday Macs and PCs, including machines with just 12 GB of VRAM.

What It Does

Linger describes itself as an open-source personal agent with a "Claude-class brain" running locally on your PC. The core pitch is straightforward: a 125B model (specifically Qwen3.8-Flash-Next) running on consumer hardware, with zero token fees for local inference and your data staying on your device.

The technical trick behind this is PolyStrata, an MoE (mixture-of-experts) inference engine the project builds on. Instead of requiring the entire model to fit on your GPU, PolyStrata keeps the busiest experts on the GPU and offloads the rest to RAM or SSD. For each token, a router picks a few experts and sends it to wherever they live. Experts that get picked often migrate up to the GPU; ones sitting on the SSD get read into RAM when needed. The README cites a concrete result: 45.5 tokens per second on a 12 GB RTX 3060 with 64 GB of RAM.

Beyond the inference engine, Linger pitches itself as a full agent—one that remembers what matters, proactively gets things done, and connects to your apps. The demo walks through a scenario where mail, chats, and meetings become shared memory, an agent team gets assembled on request, and a browser-using agent waits for approval before sending an email. The project also references two supporting components, PolyStrata and Strata, both linked from the README.

Why It's Cool

  • The memory math is the real story. A 2-bit version of this model is 39 GB. On a GPU-only setup, that means you need 39 GB or more of VRAM—more than an RTX 5090 has. PolyStrata's tiered placement (GPU, then RAM, then SSD) is what makes a 12 GB card viable. That's a meaningful shift in what "local AI" can mean for people without datacenter hardware.

  • No token fees, no cloud bill. Running the model on your own machine means local inference costs nothing per token. If you've ever watched an API bill climb while building an agent that makes lots of calls, that's an appealing property.

  • Data stays on your device. The README is explicit about this, and it follows naturally from local inference. For anyone handling sensitive mail, chats, or meeting notes, that's a real practical benefit rather than just a talking point.

  • It's an agent, not just a model runner. The demo isn't about chat completions—it's about shared memory across apps, assembling agent teams, and approval gates before actions like sending email. The "Grok Bot experience on your PC" framing suggests the project cares about the full agent loop, not just inference speed.

  • The hardware requirements are stated honestly. The README specifies 12 GB of GPU memory plus 64 GB of RAM for the 125B model, and it doesn't pretend otherwise. That kind of specificity is refreshing.

How to Try It

The README you're working from is truncated, so I can't give you exact install commands—but here's how to get oriented:

  1. Check your hardware first. The headline configuration is a 12 GB GPU with 64 GB of RAM. Make sure your machine is in the ballpark before diving in.

  2. Read the docs. The project maintains documentation at docs.heylinger.com, which is the right place to start for setup instructions.

  3. Look at the inference details. There's a dedicated doc at docs/local-inference.md covering how the local model setup works, referenced directly from the feature section.

  4. Grab the repo. Head to github.com/simplexai-labs/linger and follow the setup instructions there. The project also has a Discord if you hit snags.

  5. Watch the demo. The README links a 28.5-second video (assets/linger-story.mp4) that shows the full agent flow end to end—worth a look before you commit to setup.

Final Thoughts

Linger is aimed at people who want a capable local agent without cloud dependence or per-token costs, and who have reasonably modern consumer hardware—a 12 GB GPU and 64 GB of RAM isn't nothing, but it's far more accessible than the VRAM a 125B model would otherwise demand. The tiered expert placement is the kind of engineering that makes local AI genuinely practical rather than aspirational. If you've been waiting for local models to catch up to the quality you get from hosted APIs, this is one of the more interesting attempts to close that gap—and it's worth checking whether your machine can run it.


Follow @githubprojects for more developer tools and open source projects.

Back to Projects
Project ID: lingerLast updated: November 19, 2026 at 07:14 PM