Translate Any Video and Clone the Voices Inside It

You've got a video in one language, an audience in another, and a deadline that doesn't care about either. The manual route—transcribe, translate, re-record, re-sync—is a weekend you'll never get back. pyVideoTrans is an open-source project that tries to collapse that entire pipeline into a single automated workflow, and it's worth a look if you deal with multilingual content.

What It Does

pyVideoTrans is a video translation, audio transcription, AI dubbing, and subtitle translation tool. It's built around a complete workflow: speech recognition (ASR) picks up what's being said, the text gets translated, speech synthesis (TTS) generates new audio, and the result is synthesized back into the video with audio-video synchronization. The whole thing runs as a one-click pipeline if you want it to.

It's written in Python (3.10 or higher) and runs on Windows, macOS, and Linux. You can deploy it locally and offline, or lean on a range of mainstream online APIs for the individual stages. It ships with both a WebUI for browser-based access and a CLI for headless operation—so it works whether you're clicking through a desktop app or wiring it into a server for batch jobs.

Why It's Cool

There's a lot packed in here, but a few things stand out:

• Multi-role dubbing is the real differentiator. Most translation tools treat a video as one continuous stream of speech. pyVideoTrans supports speaker diarization, which means it can distinguish between different speakers and assign a different AI dubbing voice to each one. If you've ever watched a translated interview where everyone sounds like the same robot, you know why this matters.

• Voice cloning with real models. It integrates F5-TTS, CosyVoice, and GPT-SoVITS for zero-shot voice cloning. The idea is that the dubbed audio can retain some character of the original speaker rather than sounding like a generic voice reading a script.

• You're not locked into one provider. The model support is broad. For ASR you've got Faster-Whisper (local), OpenAI Whisper, Alibaba Qwen, ByteDance Volcano, Azure, and Google. For translation, DeepSeek, ChatGPT, Claude, Gemini, MiniMax, Ollama (local), and Alibaba Bailian. For TTS, Edge-TTS (free), OpenAI, Azure, Minimaxi, ChatTTS, and ChatterBox. You can mix and match—use a free local model for one stage and a cloud API for another.

• It doesn't force full automation on you. This might be the most practical part. The workflow supports pausing and manual proofreading at each stage—recognition, translation, and dubbing. Automatic pipelines are great until they're subtly wrong, and having checkpoints means you can fix things before they compound.

• There's a utility toolkit. Beyond the main workflow, it includes auxiliary tools like vocal separation, video/subtitle merging, audio-video alignment, and transcript matching. These are the little jobs that usually send you hunting for five different scripts.

How to Try It

The fastest path is the pre-packaged Windows build, which needs no Python setup.

1. Download the latest release from the releases page.
2. Extract it to a path without Chinese characters or spaces (for example, D:\pyVideoTrans).
3. Double-click sp.exe inside the folder.

A couple of notes from the README: don't run it directly from inside the compressed archive, and if you want GPU acceleration, you'll need CUDA 12.8 and cuDNN 9.11 installed.

If you're on macOS or Linux—or you just prefer working from source—the project recommends using uv for package management. You'll want Python 3.10 and FFmpeg installed and available in your environment variables. On macOS, that starts with:

brew install libsndfile git [email protected]

The full setup instructions live in the repository and the project documentation, and there's an online Q&A forum if you get stuck.

Final Thoughts

pyVideoTrans is best suited to people who regularly handle multilingual video and want more control than a hosted service gives them—especially anyone who needs offline operation, local models, or the ability to proofread between stages. The breadth of supported providers means you can start free and scale up as needed. It's GPL v3 licensed and actively documented, so if the workflow matches your problem, it's a reasonable place to start.