Towards Generalisable Audio Representations for Audio-Visual Navigation

TL;DR

This paper introduces a contrastive learning approach to improve the generalisation of audio-visual navigation agents to unheard sounds by regularising the audio encoder and using data augmentation, leading to significant performance gains.

Contribution

It proposes a novel contrastive learning method with data augmentation for better generalisation in audio-visual navigation, applicable to existing frameworks.

Findings

13.4% increase in SPL on Replica

12.2% increase in SPL on MP3D

Enhanced generalisation to unseen sounds

Abstract

In audio-visual navigation (AVN), an intelligent agent needs to navigate to a constantly sound-making object in complex 3D environments based on its audio and visual perceptions. While existing methods attempt to improve the navigation performance with preciously designed path planning or intricate task settings, none has improved the model generalisation on unheard sounds with task settings unchanged. We thus propose a contrastive learning-based method to tackle this challenge by regularising the audio encoder, where the sound-agnostic goal-driven latent representations can be learnt from various audio signals of different classes. In addition, we consider two data augmentation strategies to enrich the training sounds. We demonstrate that our designs can be easily equipped to existing AVN frameworks to obtain an immediate performance gain (13.4%$\uparrow$ in SPL on Replica and 12.2%$\uparrow$ in SPL on MP3D). Our project is available at https://AV-GeN.github.io/.