Real-Time Video-to-Video Translation on a Single RTX 4090

You've probably seen those AI video demos where someone's face gets swapped into an Elon Musk clip or a webcam feed turns into a living painting. The catch is usually the hardware—these things tend to need a cluster of GPUs or a render farm to run in anything approaching real time. StreamV2V, a research project from a team at UC Berkeley and UT Austin, takes a different approach: it performs real-time video-to-video translation on a single RTX 4090.

What It Does

StreamV2V is a framework for streaming video-to-video translation. It takes a video input—either a file or a live camera feed—and applies transformations to it frame by frame, in real time. The research behind it is called "Looking Backward: Streaming Video-to-Video Translation with Feature Banks," and it was accepted at ICLR 2025.

The project builds on StreamDiffusion, an existing pipeline-level solution for real-time interactive generation, and incorporates LCM-LoRA for faster inference. It uses Stable Diffusion 1.5 as its base model, along with a variety of LoRAs sourced from CIVITAI. The "feature banks" in the paper title refer to the mechanism that helps maintain temporal consistency across frames—essentially, the system looks backward at previous frames to keep the output stable rather than flickering or drifting.

Out of the box, StreamV2V supports face swapping (the README mentions swapping to Elon Musk or Will Smith as examples) and video stylization (Claymation and doodle art are the examples given). It also works for a slightly different use case: continuous text-to-image generation, where instead of feeding it video, you feed it text prompts and it generates a continuously evolving stream of images.

Why It's Cool

• It runs on hardware you might actually own. A single RTX 4090 is expensive, sure, but it's a consumer card. You don't need an A100 or a multi-GPU setup. For anyone who's wanted to experiment with real-time video transformation but assumed the barrier to entry was too high, this changes the math.

• Real-time is the whole point. There's a meaningful difference between a tool that processes a video clip and one that transforms a live camera feed as you're looking at it. The project includes a dedicated demo for camera input, which suggests the authors intend for you to actually use this interactively, not just run it on pre-recorded files.

• The temporal consistency problem gets real attention. Frame-by-frame video processing tends to produce output that shimmers or morphs unpredictably between frames. The feature bank approach—looking backward at prior frames—is a practical way to address that. It's the kind of detail that separates a fun demo from something you'd actually want to use.

• It doubles as a continuous text-to-image tool. The README notes that StreamV2V can integrate with text-to-image applications, and compared to per-image StreamDiffusion, it generates images continuously from text, producing smoother transitions. That's a nice bit of flexibility—you're not locked into video input just because the project is framed around video-to-video.

• It's honest about its lineage. The acknowledgements section is upfront that the code is copied and adapted from StreamDiffusion, and credits the LoRAs and base model used. That kind of transparency makes it easier to understand what you're actually getting and where the novel contributions lie.

How to Try It

The README points to separate guides for installation and each demo, so the first step is to follow the installation guide.

Once that's done, you have a few paths depending on what you want to do:

1. Video-to-video translation: Follow the getting started instructions in the vid2vid directory. This is where you'll reproduce the face swap and stylization results.

2. Real-time camera demo: If you want to try it with a live feed, the demo with camera guide walks you through it.

3. Continuous text-to-image: For the txt2img use case, see the demo continuous txt2img guide.

There's also a Hugging Face Space if you want to try it without setting anything up locally.

One thing to note: StreamV2V is licensed under a UT Austin Research License, so if you're thinking about commercial use, you'll want to read the license terms carefully.

The full repository is at github.com/jeff-liangf/streamv2v.

Final Thoughts

StreamV2V is a research project, and it shows—the documentation is split across multiple README files, and you'll need to be comfortable with the command line and Python environments to get it running. But if you have an RTX 4090 and an interest in real-time video transformation, it's one of the more accessible entry points into a task that's usually reserved for well-funded labs. The combination of real-time performance, temporal consistency work, and the bonus text-to-image mode makes it worth a look. If you've been waiting for video-to-video translation to become something you can actually play with on your own hardware, this is a good place to start.

Follow @githubprojects for more developer tools and open source projects.