wav2pos: Sound Source Localization using Masked Autoencoders

TL;DR

This paper introduces wav2pos, a novel masked autoencoder-based method for 3D sound source localization that effectively handles variable microphone setups and missing data, demonstrating competitive results in indoor environments.

Contribution

Proposes a flexible, set-to-set regression approach using masked autoencoders for accurate 3D sound localization with arbitrary microphone configurations.

Findings

Achieves accurate localization in simulated and real-world recordings.

Handles missing microphone data effectively.

Performs competitively against classical and learning-based methods.

Abstract

We present a novel approach to the 3D sound source localization task for distributed ad-hoc microphone arrays by formulating it as a set-to-set regression problem. By training a multi-modal masked autoencoder model that operates on audio recordings and microphone coordinates, we show that such a formulation allows for accurate localization of the sound source, by reconstructing coordinates masked in the input. Our approach is flexible in the sense that a single model can be used with an arbitrary number of microphones, even when a subset of audio recordings and microphone coordinates are missing. We test our method on simulated and real-world recordings of music and speech in indoor environments, and demonstrate competitive performance compared to both classical and other learning based localization methods.