llm-d: Production Inference Orchestration That Actually Works at Scale
You've got a model server running—maybe vLLM, maybe SGLang—and it handles a single request just fine. But the moment you put real traffic on it, with real users doing multi-turn conversations, sharing prefix caches, and expecting consistent latency, things fall apart. You start tuning, adding load balancers, writing custom scheduling logic, and before you know it you're maintaining infrastructure instead of shipping features. llm-d is an open-source distributed inference serving stack that sits above your model server and handles all that orchestration complexity, so you don't have to.
What It Does
llm-d is a CNCF sandbox project founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA. It's built specifically for production deployments on Kubernetes. Think of it as the control plane for your inference infrastructure—it manages routing, caching, scaling, and scheduling so your model servers can focus on what they do best: running models on accelerators.
The project organizes its capabilities around four core themes. Intelligent routing uses prefix-cache-aware and load-aware balancing to direct requests efficiently, with experimental predicted-latency scheduling that aims to decrease latency and increase throughput. Advanced KV-cache management increases the effective working set size for multi-turn requests by offloading to CPU or disk and maintaining precise global indexing of cache state. For serving large models like DeepSeek-R1 or GPT-OSS, llm-d supports prefill/decode disaggregation and wide expert-parallelism over fast accelerator interconnects. And for operational stability, it provides intelligent flow control for multi-tenant serving and SLO-aware autoscaling based on real-time inference signals. There's also an experimental batch processing system with OpenAI-compatible Batch APIs for offline inference.
The project provides what they call "well-lit path guides"—benchmarked recipes and Helm charts that encode best practices from production deployments. The idea is to eliminate the heavy tuning work that typically goes into deploying generative AI inference on modern accelerators.
Why It's Cool
What makes llm-d interesting isn't that it runs models faster—model servers already do that. It's that it solves the problems that appear when you try to serve those models to actual users at scale.
The performance numbers from production deployments are worth paying attention to. A Tesla/Red Hat benchmark showed 3x higher output throughput and 2x faster time-to-first-token with prefix-cache-aware routing compared to round-robin on Llama 3.1 70B. Google demonstrated a 40% reduction in both TTFT and inter-token latency with predicted-latency scheduling on NVIDIA GPUs. AWS showed up to 70% higher tokens per second with prefill/decode disaggregation on GPT-OSS. These aren't synthetic benchmarks—they're validated results from organizations that actually deploy this stuff.
The KV-cache management is particularly clever for multi-turn conversations. By offloading to CPU or disk and maintaining global indexing, you effectively increase the number of concurrent conversations you can support without running out of accelerator memory. That's a real practical win for chat applications and assistants.
The project's lineage matters too. Having Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA as founders means the architecture was designed by people who've dealt with production inference at massive scale. And being a CNCF sandbox project means it's following the Kubernetes-native patterns that enterprises actually use.
How to Try It
The project is at version 0.7 and available on GitHub. Start with the well-lit path guides in the documentation:
git clone https://github.com/llm-d/llm-d.git
cd llm-d
Then follow the deployment guides at llm-d.ai/docs/guides for your specific accelerator and model combination. The guides include Helm charts and benchmarked recipes, so you're not guessing at configuration.
If you run into issues or want to see what others are doing, join the Slack community at llm-d.ai/slack. The documentation site at llm-d.ai has the full details on routing, caching, and scaling configurations.
The repository is at github.com/llm-d/llm-d and licensed under Apache 2.0.
Final Thoughts
llm-d is for teams that have outgrown a single model server and need real orchestration for production inference. If you're running a chatbot, an API service, or any application where users expect consistent latency and high throughput across multiple turns, this project addresses the problems that appear at that scale. It's still at 0.7, so expect rough edges, but the architecture is solid and the performance results are backed by serious deployments. For anyone building production inference infrastructure on Kubernetes, this is worth a close look.
Follow @githubprojects for more developer tools and open source projects.