WhisperPlus: Whisper, But with Quantization, MLX, and Diarization Built In

If you've worked with OpenAI's Whisper for speech-to-text, you know it's powerful but can be a bit of a resource hog. Running it efficiently on a laptop or in production often means dealing with quantization, hardware acceleration, or speaker diarization separately. That's where WhisperPlus comes in.

It's a lightweight wrapper around Whisper that takes care of three common pain points: model quantization (to shrink memory usage), MLX acceleration (for Apple Silicon), and speaker diarization (to know who said what). No need to stitch together different libraries or tweak configs.

What It Does

WhisperPlus is a Python package that gives you a unified interface to transcribe audio with Whisper models, but with several improvements out of the box:

• Quantization: Automatically applies quantization (like 8-bit or 4-bit) to reduce model size and speed up inference, especially useful on consumer GPUs or even CPUs.
• MLX Support: If you're on an Apple Silicon Mac (M1/M2/M3), it uses MLX to run Whisper models efficiently. This means near-real-time transcription without breaking a sweat.
• Diarization: It integrates speaker diarization (via pyannote-audio or similar) so you can get transcripts with speaker labels. Great for meeting notes or multi-speaker recordings.
• Simple API: You can call WhisperPlus(audio_path) and get back a transcript with timestamps, segments, and speaker IDs.

Under the hood, it's not reinventing the wheel. It's a smart orchestration layer that pulls in Whisper (via faster-whisper for quantization), MLX for Apple chips, and a diarization model. It also caches models and handles batching.

Why It's Cool

The most impressive part is how much it simplifies a previously messy workflow. Normally, to get quantized Whisper with diarization on an M2 Mac, you'd need to:

1. Install faster-whisper or whisper.cpp for quantized models.
2. Set up MLX separately and rig it to Whisper.
3. Run a diarization pipeline (often with Docker) and align the results.

WhisperPlus collapses that into a single pip install and a few lines of code. Plus, it handles model downloading and caching automatically.

For developers building voice assistants, meeting bots, or transcription APIs, this saves hours of boilerplate. And because it uses faster-whisper (which is CTranslate2-based), it's already faster than the original Whisper.

How to Try It

Install it directly from PyPI:

pip install whisper-plus

Basic usage:

from whisper_plus import WhisperPlus

whisper = WhisperPlus()
result = whisper.transcribe("meeting.mp3")
print(result["segments"])  # has text, start/end times, speaker if diarization enabled

For diarization, you'll need to install the diarization dependencies separately:

pip install whisper-plus[diarization]

Then enable it:

whisper = WhisperPlus(diarize=True)

You can also control quantization level (e.g., quantize="int8") and device placement. Check the GitHub README for full options.

Final Thoughts

WhisperPlus isn't trying to be a new transcription engine. It's a sensible wrapper that takes a messy, multi-step process and makes it a one-liner. If you've ever fought with Whisper quantization or speaker diarization, you'll appreciate how much friction it removes.

It's still early days — diarization quality depends on the underlying model — but for a quick setup that just works, this is a solid tool. Worth a try if you need accurate, speaker-aware transcripts without the infrastructure headache.

Follow us at @githubprojects for more developer tools and open source finds.