Tracking Your Face for Vtubing Without Breaking a Sweat

If you’ve ever tried to build a real-time avatar animation system, you know the pain. Most facial tracking libraries are either too heavy for a webcam, require a dedicated GPU, or were built for academic benchmarks instead of practical use. OpenSeeFace flips that script.

It’s a lightweight, cross-platform facial tracking library built specifically for avatar animation. Not for hitting a score on a dataset, but for driving a 2D or 3D character in real time with minimal latency. And it runs on a CPU. Yes, a CPU.

What It Does

OpenSeeFace is a pure Python/C++ toolkit that tracks 68 facial landmarks and outputs blend shape coefficients (visemes and expressions) in real time. It uses a custom neural network model trained for speed over raw accuracy. The output is designed to feed directly into avatar rigs, especially for VRChat, VTubing, or custom game engines.

Key outputs include head position, rotation, eye gaze, mouth opening, and eyebrow movement. It can also handle multiple faces in one frame if you’re feeling ambitious.

Why It’s Cool

The headline feature is efficiency. OpenSeeFace can run at 60 FPS on a modern CPU without a discrete GPU. That’s a huge deal for anyone building a live avatar tool on a laptop or a machine without a powerful graphics card.

It also has a few clever touches:

• Automatic calibration that adapts to your face and lighting without manual tweaking.
• Low latency – it’s optimized for streaming, not batch processing.
• Simple output format – JSON or raw head pose angles, straight from the camera. No need to parse a dense research output.
• Cross-platform – works on Linux, Windows, and macOS.

Developers use it for VRChat face tracking, animated web overlays, or even as a building block for their own custom avatar apps. It’s also a solid alternative to expensive hardware-based solutions like the Leap Motion or Vive Trackers.

How to Try It

Getting started is straightforward. Clone the repo and run the demo:

git clone https://github.com/emilianavt/OpenSeeFace.git
cd OpenSeeFace/facelandmarker
python3 facetracker.py

You’ll need Python 3.7+ and a few dependencies (onnxruntime, numpy, opencv-python). If you have a webcam, you’re ready. The tracker will launch a window showing the landmarks in real time.

For integration, check the model/ directory for the pre-trained ONNX model and the facelandmarker/ folder for the core inference code. There’s also a sample Unity integration in the repository if you want to plug it into a game engine.

Final Thoughts

OpenSeeFace is one of those tools that feels like a hidden gem. No fluff, no heavy research wrapper, just something that works for the job it was designed for. If you’re building any kind of avatar tool, or just curious about lightweight face tracking, give it a spin. It’s a great example of what happens when you optimize for the real use case instead of a leaderboard.

Originally spotted on Twitter: @githubprojects