opensourceprojects.dev

A broadsheet for software that doesn't ask for your email

An open LLM inference engine for AWS Trainium, built on NKI megakernels
GitHub RepoViews7

Project Description

View on GitHub

Serving LLMs on AWS Trainium Without the Usual Headaches

If you've looked at AWS Trainium for LLM inference, you've probably run into the same wall everyone does: the hardware is compelling on paper, but the software stack is a different story. Getting a model to actually serve well on Neuron cores—with real throughput, not just a demo—takes a lot of work. TrnRT is an open inference engine that tries to close that gap, and it does it with an approach worth paying attention to.

What It Does

TrnRT is an open inference engine for large language models on AWS Trainium. It serves an OpenAI-compatible API from a single trn2.48xlarge, which means if you've already got tooling that speaks the OpenAI chat and completions format, you don't need to rewrite your client code.

The interesting architectural choice is how it handles decode. Instead of launching many small kernels, TrnRT runs decode as NKI megakernels—roughly one fused kernel per layer half. That bundles KDA attention, the fused DSA layer, mHC, and the MoE FFN into fewer, larger launches. It also supports a 1M-token context via sparse attention (top 2,048) and linear attention with fixed-size state, and keeps a prefix cache built as a radix tree over FP8 KV pages, with a host-memory tier for evicted pages and state checkpoints. Each box runs two engines, each tensor-parallel over 32 logical NeuronCores, behind a session-affine router.

The stack is Python 3.13, built against Neuron SDK 2.32 (neuronx-cc 2.27), and targets Trainium2 hardware. GLM-5.3-Flash is the first supported model, with more open models stated as on the way.

Why It's Cool

The megakernel approach is the real story here. Fusing decode into roughly one kernel per layer half is a deliberate bet that kernel launch overhead and memory movement are where you lose time on this hardware. The README's numbers back that up—a batch-8 decode step drops from 32.2 ms to 24 ms with NKI megakernels. That's not a rounding error.

It's honest about the economics. The README quotes 3.57M tokens per minute on one trn2.48xlarge at batch 12 with AgentX, and puts cost at $0.040 per million tokens on spot pricing, or $0.167 on a capacity block. Those are the numbers that actually decide whether a deployment makes sense, and they're right there up front.

Offline compile is a quiet win. Every graph is built on a CPU host, and the serving box loads them and compiles nothing by default. Anyone who's waited on a Neuron compile while a production box sits idle will appreciate why that matters—it separates the slow, fiddly part from the part that needs to stay up.

Two engines per box with a session-affine router is a practical design. You get the full 32 logical NeuronCores per engine, and routing sessions to the right engine keeps prefix cache locality intact rather than scattering requests.

The prefix cache has a host-memory tier. Evicted KV pages and state checkpoints don't just vanish—they spill to host memory. Combined with fixed-size state for the linear attention path, that's a coherent story for long-context serving rather than a bolted-on cache.

One caveat worth stating plainly: this is early. A single supported model and a truncated quickstart in the README mean you're adopting something in motion, not a mature drop-in.

How to Try It

You'll need Docker, a trn2.48xlarge to serve on, and any x86_64 host to compile on. If you're new to Trainium, the repo points you to docs/AWS.md first.

  1. Clone the repository:
git clone https://github.com/RunanywhereAI/trnrt.git && cd trnrt
  1. Build the image (Neuron SDK 2.32, Python 3.13):
docker build -f docker/Dockerfile \
  --build-arg TRNRT_VERSION=$(python3 -c 'import trnrt; print(trnrt.__version__)') \
  --build-arg TRNRT_REVISION=$(git rev-parse HEAD) \
  .

Note that the README's quickstart is truncated past this point, so you'll want to check the repo directly for the remaining build arguments and the serving steps. The docs folder is the place to go next—docs/README.md for general documentation, docs/MEGAKERNELS.md for the kernel design, and docs/RESULTS.md for benchmarks. There's also a linked blog post covering how they built it and the AgentX results.

Find it at github.com/runanywhereai/trnrt. It's Apache 2.0 licensed.

Final Thoughts

TrnRT is best suited to teams already committed to Trainium2 and willing to work close to the metal—people who care about decode latency and per-token cost more than about broad model coverage today. If you need a dozen models supported out of the box, this isn't there yet. But if you're running GLM-5.3-Flash-shaped workloads on Neuron and want an open, inspectable engine rather than a black box, the megakernel design and the offline compile story make it worth a serious look. The "more models on the way" line is the one to watch.


Follow @githubprojects for more developer tools and open source projects.

Back to Projects
Project ID: trnrtLast updated: November 18, 2026 at 06:15 AM