Real-Time Voice Conversion Hits 90ms Latency with RVC and ASIO

If you've ever messed with voice conversion models, you know the pain of waiting 2-3 seconds for a sentence to process before you hear the output. That's fine for offline edits, but it kills any chance of live performance or realtime gameplay streaming. The Retrieval-based Voice Conversion (RVC) project just changed that.

A recent update brings real-time voice conversion down to about 90 milliseconds of latency when used with ASIO drivers. That's fast enough to feel immediate. You speak, and the converted voice comes out almost simultaneously. For a model that uses retrieval-based features rather than pure end-to-end synthesis, this is a significant milestone.

What It Does

RVC is a voice conversion framework. You feed it audio of someone speaking or singing, and it converts the voice characteristics to match a target speaker. It's not text-to-speech. It's voice-to-voice. The input stays the same words and melody, but the timbre and accent shift to sound like someone else.

The repo is the RVC WebUI, which wraps the core model in a Gradio interface. You get training scripts, inference, and now realtime mode, all in one place.

The realtime feature is what's new here. Using ASIO (the audio driver standard that bypasses Windows' default audio stack) the pipeline achieves around 90ms round-trip latency. That includes capturing your mic input, processing it through the model, and playing back the converted result.

Why It's Cool

The clever part is how they got there. Instead of forcing the full model to run in realtime, the pipeline is split. It uses a chunked processing approach where audio is buffered in small slices, processed as fast as possible, and queued for playback. ASIO helps because it gives low-level access to audio buffers with minimal overhead, avoiding the 100-200ms of latency that typical DirectSound or WASAPI paths add.

Use cases are obvious but still exciting:

• Live streaming with a voice changer that doesn't feel laggy
• Voice acting with rapid back-and-forth dialogue
• Singing conversion while monitoring your own voice in real time
• Games or VR chat where you want a character voice without the "robotic delay"

The retrieval-based approach is also worth noting. Unlike some models that generate audio from spectrograms, RVC retrieves relevant features from a pre-built database of the target voice. This means it generally needs fewer training samples and produces less warbling on sustained vowels. It's not perfect, but it's much more stable than older realtime voice conversion attempts.

How to Try It

You have two paths here.

Option A: Run the WebUI locally

Clone the repo and follow the README. You'll need a GPU with at least 4GB VRAM for decent performance. The install script handles dependencies including PyTorch, torchaudio, and the Gradio app. Then you load a model, either train your own using the provided scripts or download pretrained weights from the HuggingFace hub, and enable the realtime checkbox in the UI.

The key for the 90ms latency is selecting ASIO as your audio backend. On Windows, you'll likely need to install a driver like ASIO4ALL if you don't have dedicated hardware. On Linux, you can use JACK with the same low-latency benefits.

Option B: Use a hosted demo

There are public Gradio demos linked in the repo description. Be aware that network latency will add 50-100ms on top of the processing time, so you won't get the same near-realtime feel over the web. It's fine for testing, but for actual realtime use, you want to run it locally.

Final Thoughts

I've seen a lot of "realtime voice conversion" claims that turn out to be 500ms or worse. This is the first time I've tried something where the delay feels genuinely live. The 90ms number is the processing time, not including your mic's own buffer, but even with that, it's vastly better than anything I've tested before.

For developers, the interesting part is that this opens up voice conversion for any interactive application. Think about Discord bots, game mods, or even accessibility tools that need instant voice feedback. The repo is well structured with clear separation between the model core and the UI, so you can extract the inference logic and embed it in your own apps.

One caveat: the realtime mode is still a bit CPU and GPU hungry. You'll want a decent machine, and you may need to lower the audio quality or model settings to hit the 90ms target consistently on mid-range hardware.

If you've been waiting for voice conversion that feels like a real-time effect pedal, this is worth a weekend experiment. The code is approachable, the community is active, and the latency finally makes it usable for more than just offline projects.

Found this useful? Follow @githubprojects for more dev tools and open-source finds.