AcousticFusion: Fusing Sound Source Localization to Visual SLAM in Dynamic Environments

TL;DR

This paper introduces AcousticFusion, a low-resource audio-visual fusion method that enhances SLAM in dynamic environments by integrating sound source direction, effectively removing dynamic obstacles and improving localization stability.

Contribution

It presents a novel fusion approach that combines sound source localization with visual SLAM, reducing computational costs and handling dynamic obstacles more efficiently.

Findings

Stable self-localization in dynamic environments

Low computational resource usage

Effective removal of dynamic obstacles

Abstract

Dynamic objects in the environment, such as people and other agents, lead to challenges for existing simultaneous localization and mapping (SLAM) approaches. To deal with dynamic environments, computer vision researchers usually apply some learning-based object detectors to remove these dynamic objects. However, these object detectors are computationally too expensive for mobile robot on-board processing. In practical applications, these objects output noisy sounds that can be effectively detected by on-board sound source localization. The directional information of the sound source object can be efficiently obtained by direction of sound arrival (DoA) estimation, but depth estimation is difficult. Therefore, in this paper, we propose a novel audio-visual fusion approach that fuses sound source direction into the RGB-D image and thus removes the effect of dynamic obstacles on the multi-robot SLAM system. Experimental results of multi-robot SLAM in different dynamic environments show that the proposed method uses very small computational resources to obtain very stable self-localization results.