When Zero-Shot Stereo Matching Needs to Happen in Real Time

You've probably run into this wall before: you need stereo depth estimation that generalizes to new environments without fine-tuning, but the models that actually work zero-shot are too slow for real-time use. The fast ones? They need expensive per-domain tuning to be usable. So you're stuck choosing between robust-but-slow or fast-but-brittle. Fast-FoundationStereo, a new project from NVIDIA Research and accepted to CVPR 2026, tries to break that trade-off by giving you zero-shot generalization at frame rates that actually matter for real-time applications.

What It Does

Fast-FoundationStereo is a family of stereo matching architectures that perform depth estimation from pairs of images without needing any training on your specific domain. It's the follow-up to FoundationStereo, which prioritized accuracy over speed. This version flips the priority: it targets real-time performance while keeping the zero-shot generalization that makes foundation models useful.

The core insight is a divide-and-conquer acceleration strategy with three components. First, knowledge distillation compresses the hybrid backbone from the original FoundationStereo into a single, more efficient student model. Second, blockwise neural architecture search automatically discovers optimal cost filtering designs within given latency budgets—the paper claims this reduces search complexity exponentially. Third, structured pruning removes redundancy in the iterative refinement module.

The team also built an automatic pseudo-labeling pipeline that generated 1.4 million in-the-wild stereo pairs to supplement synthetic training data. That's important because synthetic data alone doesn't always transfer well to real-world conditions, and manual labeling at that scale isn't practical.

Technically, the implementation uses PyTorch, and the README shows support for TensorRT acceleration as well. The models can run on a GPU like a 3090 with varying trade-offs between speed and accuracy depending on which checkpoint you choose and how many refinement iterations you run.

Why It's Cool

The most interesting thing here is that they've made the speed-accuracy trade-off explicit and tunable rather than forcing you into one fixed configuration.

• You get a family of models, not just one. The README lists checkpoints like 23-36-37, 20-26-39, and 20-30-48, each with different runtime profiles. The fastest hits 14 milliseconds with TensorRT at 640x480 resolution. The slowest is still 23.4 milliseconds. For context, that's comfortably above 30 FPS for the slowest configuration.

• The valid_iters parameter is your speed dial. You can drop from 8 refinement iterations to 4 and shave off noticeable time with the same checkpoint. The peak memory stays the same, so you're not trading VRAM for speed—just accepting slightly lower accuracy.

• They're honest about the trade-off. The README explicitly says models are sorted from slowest to fastest with accuracy descending. No pretending the fastest model is the best. You pick based on your latency budget.

• The problem it solves is genuinely painful. If you've ever tried to deploy stereo depth in robotics, AR/VR, or autonomous systems, you know that domain-specific fine-tuning is a time sink. A model that works zero-shot at real-time rates removes that entire workflow step.

• Docker support is there. The setup gives you both a Docker option and a pip install path. That's thoughtful for teams that need reproducible environments.

How to Try It

The repository is straightforward to get running. You have two environment options.

For Docker:

docker build --network host -t ffs -f docker/dockerfile .
bash docker/run_container.sh

For pip with a conda environment:

conda create -n ffs python=3.12 && conda activate ffs
pip install torch==2.6.0 torchvision==0.21.0 xformers --index-url https://download.pytorch.org/whl/cu124
pip install -r requirements.txt

You'll need to download the model weights from Google Drive and place them in a weights/ folder. Then run the demo:

python scripts/run_demo.py --model_dir weights/23-36-37/model_best_bp2_serialize.pth --left_file demo_data/left.png --right_file demo_data/right.png --intrinsic_file demo_data/K.txt --out_dir output

The repository provides sample data in demo_data/ to test with. If you want to experiment with different speed-accuracy trade-offs, you can swap in different checkpoints or adjust the valid_iters parameter.

Final Thoughts

Fast-FoundationStereo is best for anyone building real-time applications that need stereo depth without the overhead of per-environment fine-tuning. If you're working on robotics, autonomous navigation, or AR/VR and you've been frustrated by the gap between research-grade zero-shot models and production-ready frame rates, this is worth a close look. The 10x speedup over FoundationStereo is real, and the zero-shot accuracy being "comparable" rather than identical is an honest trade-off that the project lets you navigate yourself. Check out the repository at github.com/NVlabs/Fast-FoundationStereo if you want to see if it fits your latency budget.

Follow @githubprojects for more developer tools and open source projects.