Sound Localization by Self-Supervised Time Delay Estimation

TL;DR

This paper introduces a self-supervised learning approach for sound localization by estimating interaural time delays, leveraging contrastive learning techniques and multimodal cues to achieve competitive results without labeled data.

Contribution

It adapts contrastive random walk for self-supervised sound correspondence learning and integrates visual cues for improved multi-speaker localization.

Findings

Performs on par with supervised methods on internet recordings.

Effective multimodal localization of specific speakers in mixtures.

Self-supervised approach reduces reliance on labeled data.

Abstract

Sounds reach one microphone in a stereo pair sooner than the other, resulting in an interaural time delay that conveys their directions. Estimating a sound's time delay requires finding correspondences between the signals recorded by each microphone. We propose to learn these correspondences through self-supervision, drawing on recent techniques from visual tracking. We adapt the contrastive random walk of Jabri et al. to learn a cycle-consistent representation from unlabeled stereo sounds, resulting in a model that performs on par with supervised methods on "in the wild" internet recordings. We also propose a multimodal contrastive learning model that solves a visually-guided localization task: estimating the time delay for a particular person in a multi-speaker mixture, given a visual representation of their face. Project site: https://ificl.github.io/stereocrw/