SGLang: Day-Zero Support for New Models and Faster Inference on Modern Hardware

You've probably been there: a new open model drops, everyone's talking about it, and you want to try it out—only to discover that serving it properly means waiting weeks for the inference stack to catch up. SGLang is a project that tries to close that gap, combining a fast serving runtime with a habit of shipping support for new models the day they're released.

What It Does

SGLang is an inference engine for large language models and other generative workloads. It's distributed through PyPI, so you can pull it in like any other Python package, and it's designed to serve models efficiently across a range of hardware—including GPUs and, as of recent work, TPUs via an SGLang-Jax backend.

The project's headline pattern is what it calls "day-0 support" for new open models. When a notable model launches, SGLang aims to have it running on the same day. The README's news section reads like a timeline of this: day-0 support for Kimi K3, DeepSeek-V4, Mistral Large 3, MiniMax M2, and a long list of others. Beyond text LLMs, the project has branched into other modalities too—there's an SGLang Diffusion component for video and image generation, and support for TTS models like Higgs Audio v3.

Speculative decoding is another area the project invests in, with recent work on DFlash and what it calls "Spec V2," a next-generation approach to speeding up generation.

Why It's Cool

• Day-0 support is a real workflow win. If you're evaluating new models as they come out, the bottleneck is often the serving layer, not the model itself. Having a runtime that commits to same-day support means you can actually test things when they're relevant—not a month later when the hype has moved on.

• The hardware story is broader than you'd expect. Most inference engines are CUDA-first and stop there. SGLang's work with Google and RadixArk to bring full features to TPUs, plus the earlier native TPU support through SGLang-Jax, means you're not locked into a single vendor's stack.

• The performance numbers are concrete and specific. The README points to a benchmark showing 25x inference performance on NVIDIA GB300 NVL72. There's also a writeup on deploying DeepSeek on GB200 NVL72 that reports 3.8x prefill and 4.8x decode throughput improvements. These are the kind of numbers you can actually reason about, tied to named hardware and named models.

• It's not just text. The Diffusion work for video and image generation, plus audio model support, suggests the project is positioning itself as a general serving layer for generative workloads rather than an LLM-only tool. That matters if you're building something that mixes modalities.

• The project moves fast and documents it. Between blog posts, a public roadmap, weekly dev meetings, and Slack, there's a visible development cadence. You can see what's coming and what's already landed, which is useful when you're deciding whether to build on it.

How to Try It

The simplest path is to install from PyPI:

pip install sglang

From there, the README points to the documentation for actual serving setup—that's where you'll find the specifics for launching models, configuring hardware, and tuning performance. The website and the blog are worth bookmarking if you want to follow along with new model support as it lands.

If you want to dig into the source, contribute, or just see how it's built, the repository is at github.com/sgl-project/sglang. There's also a roadmap, a Slack community, and a weekly dev meeting if you'd rather talk to people directly. For learning material, the project maintains a separate learning materials repo with slides.

If you're working with a specific new model, check the news section of the README first—there's a decent chance there's already a blog post or PR covering it.

Final Thoughts

SGLang is most useful if you're serving models in production or doing serious evaluation work, especially if you're juggling multiple hardware targets or want to test new releases as they appear. The day-0 support pattern and the breadth of hardware coverage are the real differentiators here; there are plenty of inference engines, but few that treat "new model support" as a first-class commitment. If you're just running a single model on a single machine for a side project, it might be more than you need—but if you're at the point where the serving layer is the thing slowing you down, it's worth a look.

Follow @githubprojects for more developer tools and open source projects.