Semantic Audio-Visual Navigation

TL;DR

This paper introduces semantic audio-visual navigation, where agents locate objects based on sporadic, semantically meaningful sounds, using a transformer model with multimodal memory to improve navigation performance.

Contribution

The paper presents a new semantic AudioGoal task and a transformer-based model with multimodal memory, expanding sound simulations with semantically grounded sounds for improved navigation.

Findings

Model outperforms existing methods significantly.

Effective integration of semantic, acoustic, and visual cues.

Enhanced simulation environment with semantically grounded sounds.

Abstract

Recent work on audio-visual navigation assumes a constantly-sounding target and restricts the role of audio to signaling the target's position. We introduce semantic audio-visual navigation, where objects in the environment make sounds consistent with their semantic meaning (e.g., toilet flushing, door creaking) and acoustic events are sporadic or short in duration. We propose a transformer-based model to tackle this new semantic AudioGoal task, incorporating an inferred goal descriptor that captures both spatial and semantic properties of the target. Our model's persistent multimodal memory enables it to reach the goal even long after the acoustic event stops. In support of the new task, we also expand the SoundSpaces audio simulations to provide semantically grounded sounds for an array of objects in Matterport3D. Our method strongly outperforms existing audio-visual navigation methods by learning to associate semantic, acoustic, and visual cues.