Real-Time Whisper Transcription That Actually Keeps Up With You

You've probably tried running OpenAI's Whisper on a live audio stream and watched it lag behind reality by a good thirty seconds. The model is great, but it wasn't built for streaming. WhisperLive is a project from Collabora that wraps Whisper in a client-server architecture so you can get transcription that's nearly live—from your microphone or from pre-recorded files.

What It Does

WhisperLive is a real-time transcription application built on OpenAI's Whisper model. It converts speech to text through a server that handles the model inference and a client that sends audio and receives transcription results. You can point it at live microphone input or feed it audio files, and it supports both a WebSocket-based protocol and an OpenAI-compatible REST interface.

The server supports three backends: faster_whisper, tensorrt, and openvino. That means you're not locked into a single inference path—you can pick the one that matches your hardware and latency requirements. The project includes options for limiting concurrent clients and maximum connection time, which matters if you're running this as a shared service rather than a local tool.

Why It's Cool

• Word-level timestamps. Beyond just transcribing speech, WhisperLive can give you timestamps for individual words. That's genuinely useful if you're building anything that needs to sync text to audio—subtitles, searchable transcripts, or highlight extraction.

• Speaker diarization. The project supports identifying who said what. For meeting transcription or interview workflows, this is the difference between a wall of text and something you can actually navigate.

• Custom vocabulary and hotwords. Whisper sometimes struggles with domain-specific terms—product names, acronyms, technical jargon. WhisperLive lets you supply custom vocabulary so the model biases toward the words you actually care about. Small feature, big practical difference.

• Batch inference and raw PCM input. You're not limited to one audio format or one request at a time. Batch inference helps with throughput, and raw PCM input means you can pipe audio from just about anywhere without converting it first.

• A manual streaming client. If you want to handle audio chunking yourself instead of relying on the built-in client behavior, there's a streaming client for that. It's a nice escape hatch for people with unusual audio pipelines.

• Browser extensions and Docker support. There's a browser extension component and a Docker setup for the server. The Docker path is especially relevant if you're trying to run the TensorRT backend, which the README recommends doing via Docker rather than a bare-metal install.

The OpenAI REST interface is a smart addition too. If you already have tooling that speaks the OpenAI API, you can point it at a WhisperLive server running with --enable_rest and skip writing a custom client entirely.

How to Try It

The setup is straightforward if you're comfortable with Python environments. Here's the short version:

1. Install PortAudio, which is needed for microphone input via PyAudio. The project includes a setup script:

bash scripts/setup.sh

On Debian/Ubuntu this installs portaudio19-dev, on Fedora portaudio-devel, and on macOS it uses Homebrew.

1. Create a Python 3.12 virtual environment:

python3.12 -m venv whisper_env
source whisper_env/bin/activate

1. Install the package from pip:

pip install whisper-live

1. Start the server. Here's a typical invocation using the Faster Whisper backend:

python3 run_server.py --port 9090 \
                      --backend faster_whisper \
                      --max_clients 4 \
                      --max_connection_time 600

1. If you want the OpenAI-compatible REST interface, add the relevant flags:

python3 run_server.py --port 9090 --backend faster_whisper --max_clients 4 --max_connection_time 600 --enable_rest --cors-origins="http://localhost:8080,http://127.0.0.1:8080"

Then run the client against a file:

python3 client_openai.py $AUDIO_FILE

If you're using the TensorRT backend, the README points you to a separate TensorRT setup guide and recommends the Docker path. You'll need to build your TensorRT engines before starting the server.

Full details, including the custom model and cache directory options, are in the repository.

Final Thoughts

WhisperLive is a solid option if you need streaming transcription and you're willing to run your own server. The three-backend support gives you room to optimize for your hardware, and the advanced features—word timestamps, diarization, hotwords—cover the gaps that make raw Whisper output hard to use in production. It's not a turnkey SaaS product, and the TensorRT path has some setup overhead, but for developers who want control over their transcription pipeline, it's worth a look.