Multi-goal Audio-visual Navigation using Sound Direction Map

TL;DR

This paper introduces a new multi-goal audio-visual navigation framework that combines visual and auditory cues for complex indoor navigation tasks involving multiple sound sources, and proposes a sound direction map method to improve localization.

Contribution

It defines the multi-goal audio-visual navigation task, analyzes its difficulty, and proposes the sound direction map method to enhance source localization and navigation performance.

Findings

SDM significantly improves baseline navigation methods

Multi-goal audio-visual navigation is more challenging due to source separation

Experimental results validate SDM's effectiveness across various goals

Abstract

Over the past few years, there has been a great deal of research on navigation tasks in indoor environments using deep reinforcement learning agents. Most of these tasks use only visual information in the form of first-person images to navigate to a single goal. More recently, tasks that simultaneously use visual and auditory information to navigate to the sound source and even navigation tasks with multiple goals instead of one have been proposed. However, there has been no proposal for a generalized navigation task combining these two types of tasks and using both visual and auditory information in a situation where multiple sound sources are goals. In this paper, we propose a new framework for this generalized task: multi-goal audio-visual navigation. We first define the task in detail, and then we investigate the difficulty of the multi-goal audio-visual navigation task relative to the current navigation tasks by conducting experiments in various situations. The research shows that multi-goal audio-visual navigation has the difficulty of the implicit need to separate the sources of sound. Next, to mitigate the difficulties in this new task, we propose a method named sound direction map (SDM), which dynamically localizes multiple sound sources in a learning-based manner while making use of past memories. Experimental results show that the use of SDM significantly improves the performance of multiple baseline methods, regardless of the number of goals.