GRAM: Spatial general-purpose audio representations for real-world environments

TL;DR

GRAM is a novel spatial audio model that learns representations from multi-channel recordings, excelling in real-world environments and tasks like sound localization, outperforming existing models with less training data.

Contribution

The paper introduces GRAM, a multi-channel masked autoencoder for spatial audio, and provides standardized benchmarks demonstrating its superior performance in real-world acoustic tasks.

Findings

Outperforms state-of-the-art models on NatHEAR and HEAR benchmarks.

Achieves high localization accuracy in simulated environments.

Generalizes effectively to real-world recordings in RealSELD.

Abstract

Audio foundation models learn general-purpose audio representations that facilitate a wide range of downstream tasks. While the performance of these models has greatly increased for conventional single-channel, dry audio clips, their success in real-world acoustic environments with reverberation and noise is limited. Furthermore, most audio foundation models ignore the spatial dimension of real-world acoustic environments, ruling out tasks involving sound localization. To address these limitations, we propose GRAM: a general-purpose real-world audio model that employs a multi-channel masked autoencoder to efficiently learn spatial audio representations. We evaluated GRAM and other audio foundation models in a standardized manner on high-quality simulations of naturalistic, spatial acoustic environments as well as recordings of real-world environments and release these two complementary benchmark task suites: NatHEAR and RealSELD. Our results demonstrate that GRAM outperforms all state-of-the-art self-supervised audio foundation models on NatHEAR and the clean, single-channel version HEAR, while using only a fraction of the training data. GRAM also shows state-of-the-art localization performance in simulated environments and generalizes efficiently to real-world recordings in RealSELD. Taken together, GRAM presents a significant advance toward robust spatial audio foundation models for real-world environments.