Multimodal Aggregation Approach for Memory Vision-Voice Indoor Navigation with Meta-Learning

TL;DR

This paper introduces MVV-IN, a novel indoor navigation model that combines vision and voice, utilizing self-attention and meta-learning to improve environment understanding and adaptability in robots.

Contribution

The paper proposes a new multimodal indoor navigation approach integrating self-attention and meta-learning, enhancing robot adaptability and environment comprehension.

Findings

Outperforms state-of-the-art baselines in indoor navigation tasks.

Utilizes self-attention to focus on key visual areas.

Employs meta-learning for better scene adaptation.

Abstract

Vision and voice are two vital keys for agents' interaction and learning. In this paper, we present a novel indoor navigation model called Memory Vision-Voice Indoor Navigation (MVV-IN), which receives voice commands and analyzes multimodal information of visual observation in order to enhance robots' environment understanding. We make use of single RGB images taken by a first-view monocular camera. We also apply a self-attention mechanism to keep the agent focusing on key areas. Memory is important for the agent to avoid repeating certain tasks unnecessarily and in order for it to adapt adequately to new scenes, therefore, we make use of meta-learning. We have experimented with various functional features extracted from visual observation. Comparative experiments prove that our methods outperform state-of-the-art baselines.