LLMRouter: One Library to Train, Evaluate, and Deploy Your LLM Router
You've got access to half a dozen language models, and every query that comes in could go to any of them. Some are cheap and fast. Some are expensive and smart. Picking the right one by hand doesn't scale, and rolling your own routing logic from scratch is a project in itself. LLMRouter is a library that treats that selection problem as a first-class engineering task, giving you the infrastructure to build, compare, and ship routers instead of hand-tuning heuristics.
What It Does
LLMRouter is an intelligent routing system for LLM inference. Its job is to look at an incoming query and dynamically pick the most suitable model for it, balancing task complexity against cost and performance requirements. The project frames routing as a unified sequential decision process that spans single-turn, multi-turn, and personalized scenarios, so the same framework covers a lot of ground.
The library ships with support for over 16 routing models, organized into five categories: single-round routers, multi-round routers, multimodal routers, agentic routers, and personalized routers. Within those categories you'll find a wide spread of strategies, including KNN, SVM, MLP, matrix factorization, Elo rating, graph-based routing, BERT-based routing, hybrid probabilistic methods, and transformed-score routers.
Around the models themselves, LLMRouter provides a unified CLI for training, inference, and interactive chat with a Gradio-based UI, plus a complete data generation pipeline that builds training data from 11 benchmark datasets with automatic API calling and evaluation. It's Python 3.10, MIT-licensed, and the repo is open to pull requests.
Why It's Cool
-
The model zoo is genuinely broad. Sixteen-plus routers across five categories is a lot of ground for one library. Instead of committing to a single approach, you can compare KNN against matrix factorization against a BERT-based router on the same data and pick what actually works for your workload. That kind of head-to-head comparison is usually the hard part of building routing infrastructure, and here it's built in.
-
The data pipeline is the unglamorous part that matters. Anyone can write a router function. Generating the supervision to train one is where projects stall. LLMRouter includes a full pipeline for producing training data from 11 benchmark datasets, with automatic API calling and evaluation. That's the difference between a demo and something you can actually iterate on.
-
It comes with a benchmark. The project introduces xRouteBench, a benchmark covering generic LLM, memory-augmented, vision, time-series, and personalized routing, with automated supervision construction and joint evaluation of response quality and inference cost. Evaluating routers on accuracy alone is misleading if you ignore what they cost you, so scoring both together is the right call.
-
The reported numbers are concrete. The project's experiments show learned routers outperforming the strongest fixed-model baseline by 14.6% relatively. Whether that holds for your traffic is an open question, but it's a specific, checkable claim rather than a vague promise.
-
There's a CLI and a UI. Training, inference, and interactive chat all run through a command-line interface, with a Gradio-based UI for chatting. You don't have to write glue code just to see whether a router behaves sensibly.
How to Try It
- Head to the repository and clone it:
git clone https://github.com/ulab-uiuc/LLMRouter
cd LLMRouter
-
The project targets Python 3.10, so make sure that's your environment before installing dependencies.
-
Check the online documentation for setup and configuration details, since router training typically needs API credentials for the models you're routing between.
-
Once installed, the unified CLI is your entry point for training a router, running inference, or launching the interactive Gradio chat UI.
-
If you want to benchmark rather than build, the xRouteBench dataset is available on Hugging Face, and you can compare your routers against the results described in the paper.
The repo also has a Slack workspace and a WeChat group if you'd rather ask questions than read docs. Links to both are in the README.
Final Thoughts
LLMRouter is aimed at people who are past the "should I route at all?" question and into "which router, and how do I know it's working?" That's a narrower audience than a general-purpose LLM toolkit, but for that audience the combination of 16+ routing models, a data generation pipeline, and a paired quality-and-cost benchmark is a lot of scaffolding you won't have to build yourself. If you're running multiple models in production and guessing at which one to send queries to, this is worth an afternoon.
Follow @githubprojects for more developer tools and open source projects.