Turn a Short Video Into 48 Frames of 4D: Meet SV4D 2.0
You've got a 12-frame video of a moving object, and you want to see it from angles the camera never captured. Normally, that means either a complex multi-camera rig or a pile of separate models that don't play well together. Stability AI's SV4D 2.0 takes a different approach: it's a single diffusion model that takes those 12 frames and generates 48 novel-view frames—12 frames from each of 4 camera angles—in one pass. It's part of the Generative Models repository, and it's designed to make video-to-4D generation feel a lot more practical.
What It Does
SV4D 2.0 is a video-to-4D diffusion model. You feed it a short input video—ideally 12 frames at 576x576 resolution, showing a moving object against a white background—and it outputs 48 frames: the original 12 frames rendered from 4 distinct camera views. The result is a coherent novel-view video you can orbit around, which is the raw material for 4D asset generation.
The architecture builds on Stability AI's previous work but with some notable changes. Unlike the original SV4D, this version doesn't rely on a reference multi-view of the first frame generated by SV3D. That means it handles self-occlusions more robustly—when parts of the object disappear and reappear as it moves, the model doesn't fall apart the way it might when conditioned on a single static reference.
For longer outputs, the model works autoregressively. It generates 12 frames at a time, then uses the previous generation as conditioning views for the next batch. So you can keep chaining generations to produce longer novel-view videos than the base 48-frame output.
There's also an 8-view variant that generates 5 frames × 8 views at a time, similar to the original SV4D. Both models share the same inference scripts, so switching between them is just a matter of pointing to a different checkpoint file.
Why It's Cool
It's a single model doing heavy lifting. Most 4D generation pipelines stitch together separate components—one model for geometry, another for texture, another for motion. SV4D 2.0 is a unified diffusion model that handles the whole problem at once. That's simpler to run and easier to reason about.
The autoregressive approach is pragmatic. Generating 48 frames in one shot is memory-intensive and hard to stabilize. By generating 12 frames at a time and feeding results back as conditioning, the model keeps quality consistent without requiring a massive GPU budget. It's a clever engineering trade-off that makes the tool more accessible.
It generalizes better to real-world video. The README notes that SV4D 2.0 handles real-world videos much better than its predecessor. That's a big deal because synthetic training data often doesn't translate well to messy, real-world footage. If you're working with actual camera captures rather than clean renders, this matters a lot.
The practical workflow is well thought out. The README includes concrete advice for handling background noise: use rembg for plain backgrounds, or segment the foreground with SAM2 or Clipdrop for messier footage. There's also a low-VRAM mode (--encoding_t=1, --decoding_t=1) and a resolution drop to 512, which means you don't need an A100 just to try it out.
It's research-grade but approachable. The project links to an arXiv paper and a project page, so you can dig into the methodology if you want. But you can also just download the checkpoint, run one command, and get results. That's a nice balance for open-source AI projects.
How to Try It
Getting started is straightforward. First, clone the repository and set up a virtual environment:
git clone https://github.com/stability-ai/generative-models
cd generative-models
python3.10 -m venv .generativemodels
source .generativemodels/bin/activate
pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
Then download the model checkpoint from HuggingFace:
huggingface-cli download stabilityai/sv4d2.0 sv4d2.safetensors --local-dir checkpoints
Now you can run inference on a sample video:
python scripts/sampling/simple_video_sample_4d2.py --input_path assets/sv4d_videos/camel.gif --output_folder outputs
The --input_path argument is flexible—it accepts a .gif or .mp4 file, a folder of image frames (.jpg, .jpeg, .png), or a file pattern. You can adjust num_steps (default 50) to trade quality for speed, and elevations_deg to specify camera elevations relative to the input view.
If your input has a plain background, you can enable --remove_bg=True to use rembg for automatic background removal. On low-VRAM GPUs, try --encoding_t=1 and --decoding_t=1, or lower the resolution with --img_size=512.
For the 8-view model, swap the checkpoint and run the same script:
huggingface-cli download stabilityai/sv4d2.0 sv4d2_8views.safetensors --local-dir checkpoints
python scripts/sampling/simple_video_sample_4d2.py --model_path checkpoints/sv4d2_8views.safetensors --input_path assets/sv4d_videos/chest.gif --output_folder outputs
Full details are in the repository README.
Final Thoughts
SV4D 2.0 is a solid step forward for practical 4D generation. The single-model approach, the autoregressive design, and the attention to real-world robustness make it a genuinely useful tool for researchers and developers working on novel-view synthesis. It's not magic—you'll still want clean input footage for best results—but the fact that you can run it on a modest GPU and get coherent multi-view video from a short clip is impressive. If you're exploring 4D asset generation or just curious about where video diffusion is headed, this is worth a weekend experiment.
Follow @githubprojects for more developer tools and open source projects.