FastChat: An Open Platform for Training, Serving, and Evaluating LLM Chatbots

If you've ever tried to spin up a chatbot with a large language model, you know the pain: training code here, serving infrastructure there, evaluation tools scattered somewhere else. FastChat aims to solve that fragmentation by bundling the whole lifecycle—training, serving, and evaluating LLM-based chatbots—into a single open platform.

What It Does

FastChat is an open platform for training, serving, and evaluating large language model based chatbots. It provides two core pieces of functionality: the training and evaluation code for state-of-the-art models (like Vicuna and MT-Bench), and a distributed multi-model serving system with a web UI and OpenAI-compatible RESTful APIs.

The project's scale is worth noting. FastChat powers Chatbot Arena (lmarena.ai), which has served over 10 million chat requests for more than 70 LLMs. That same Arena has collected over 1.5 million human votes from side-by-side LLM battles, which feed into an online LLM Elo leaderboard. So this isn't a weekend experiment—it's infrastructure that's been stress-tested at real volume.

Why It's Cool

• It covers the full lifecycle in one repo. Most projects pick a lane—either serving or evaluation or fine-tuning. FastChat handles all three, which means you don't have to glue together three different codebases just to get from a model checkpoint to a working chatbot with benchmarks.

• The serving system is distributed and multi-model. You're not limited to running one model at a time, and the architecture is built to scale across machines. That's a meaningful difference from the single-process demos you'll find in a lot of LLM tutorials.

• OpenAI-compatible APIs are a practical win. If you've already built tooling against OpenAI's REST format, you can point it at FastChat instead. That lowers the switching cost considerably—you don't have to rewrite your client code just to self-host.

• The evaluation story is backed by real data. MT-Bench is a challenging multi-turn question set for evaluating chatbots, and the Chatbot Arena leaderboard is compiled from over 1.5 million human votes. Evaluation is often the weakest link in open-source LLM projects, and FastChat treats it as a first-class concern.

• The model weights are accessible. Vicuna weights are based on Llama 2 and can be downloaded automatically from Hugging Face repos when you run the CLI—they land in a .cache folder in your home directory. There's a 16K context version too, though it requires transformers>=4.31.

• There's a real dataset behind it. LMSYS-Chat-1M is a large-scale real-world LLM conversation dataset, and Chatbot Arena Conversations contains 33k conversations with human preferences. If you're doing research or fine-tuning, these are genuinely useful resources.

How to Try It

Getting started is straightforward. You've got two install paths—pip or from source.

Method 1: With pip

pip3 install "fschat[model_worker,webui]"

Method 2: From source

git clone https://github.com/lm-sys/FastChat.git
cd FastChat

If you're on a Mac, you'll need to install Rust and CMake first:

brew install rust cmake

Then install the package:

pip3 install --upgrade pip
pip3 install -e ".[model_worker,webui]"

Once that's done, you can start chatting with Vicuna directly from the command line. The weights download automatically from Hugging Face:

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5

There's also a 16K context variant if you need longer conversations:

python3 -m fastchat.serve.cli --model-path lmsys/vicuna-7b-v1.5-16k

The README covers serving with a web GUI, the API layer, evaluation, and fine-tuning in separate sections, so you can dig into whichever part of the pipeline matters most to you. Out-of-memory handling is discussed in the CLI inference section if you're running on constrained hardware.

The full repository is at github.com/lm-sys/FastChat.

Final Thoughts

FastChat is best suited for developers and researchers who need more than a toy demo—people who want to serve models at some scale, evaluate them rigorously, or fine-tune them without stitching together a half-dozen unrelated tools. The fact that it powers Chatbot Arena gives it a credibility that's hard to fake. If you're working with open LLMs and want a single platform that handles the messy middle between "I have a model" and "I have a deployed, evaluated chatbot," this is worth a serious look.