Hear the Person You're Looking At: Target Speaker Audio with Just a Glance

You're at a crowded party, and someone across the room is saying something important—but the noise makes it impossible to focus on their voice. You wish you could just look at them and suddenly hear only them. That's exactly the problem this project tackles.

Look Once to Hear is an open-source system that lets you select a target speaker by simply looking at them for a few seconds. The code, released with the paper "Look Once to Hear: Target Speech Hearing with Noisy Examples" (which won a best paper honorable mention at CHI 2024), implements an intelligent hearable system where your gaze becomes the control mechanism for audio focus.

What It Does

The system works by capturing a brief visual reference of the person you want to hear—that "look once" moment—and using it to isolate their voice from a mixture of sounds. The repository provides everything you need to train and evaluate the model yourself, including data preparation scripts, training configuration, and evaluation tools.

The technical approach is grounded in realistic audio simulation. Training data combines clean speech, background sounds, head-related transfer functions (HRTFs), and binaural room impulse responses (BRIRs). These are mixed synthetically using the Scaper toolkit, which generates audio mixtures on-the-fly during training and evaluation from .jams specification files. This means the model learns to separate voices under varied acoustic conditions rather than just clean, isolated speech.

The setup is straightforward if you're familiar with Python and conda. You create an environment, install dependencies, and you're ready to go. The repository also provides pre-built datasets you can download, so you don't have to reconstruct everything from scratch if you just want to run a training experiment.

Why It's Cool

What makes this project stand out is how it bridges two normally separate domains: computer vision and audio processing. You're not just building a speech separator—you're building one that understands human intention through gaze.

• The interaction model is novel. Most target speaker systems require you to provide an audio sample or enroll a voiceprint ahead of time. Here, you just look at the person. That's a fundamentally different user experience, and it feels like something from a sci-fi interface.

• It's honest about real-world conditions. The paper title mentions "noisy examples," and the training pipeline reflects that. By using HRTFs and BRIRs, the model learns how sound actually behaves in physical spaces—how it bounces off walls, how it arrives at each ear differently. That's the kind of detail that separates a demo from something that could work in practice.

• The codebase is structured for experimentation. You can train with a fraction of the dataset using the --frac flag to test things quickly, resume partial runs, and evaluate on your own speech mixtures. It's designed for researchers who want to iterate, not just reproduce.

• It's a complete pipeline, not a toy. From data preparation to training to evaluation, the repository covers the full workflow. The self-contained datasets mean you can get started without hunting down multiple external resources.

How to Try It

If you want to explore this project, start by cloning the repository and setting up the environment:

git clone https://github.com/vb000/lookoncetohear
cd lookoncetohear
conda create -n ts-hear python=3.9
conda activate ts-hear
pip install -r requirements.txt

Next, you'll need data. The README points to a downloadable MixLibriSpeech.tar file that contains the source .jams specifications used for training. Extract it to the data/ directory, and you can kick off a training run:

python -m src.trainer --config <configs/tsh.json> --run_dir <runs/tsh>

Want to test with fewer batches first? Add the --frac flag to use only a fraction of the training and validation batches:

python -m src.trainer --config <configs/tsh.json> --run_dir <runs/tsh> --frac 0.05

If a run gets interrupted, you can resume it with the same command without the --frac flag. For evaluation, there's a separate script:

python -m src.ts_hear_test

The full data preparation instructions live in data/README.md if you want to rebuild datasets from scratch, but the provided download should be enough to get you started.

Final Thoughts

This is a research codebase, so don't expect a polished consumer app. But if you're working on audio source separation, hearable technology, or multimodal interaction systems, this is a solid reference implementation with a genuinely interesting idea at its core. The gaze-based interaction model is the kind of concept that could reshape how we think about assistive listening devices, and having the code open source means you can build on it rather than just read about it. Start with the provided datasets, tweak the training config, and see what happens when you point the system at your own audio mixtures.