Autonomous Navigation in Complex Environments with Deep Multimodal Fusion Network

TL;DR

This paper introduces NMFNet, a multimodal fusion network that combines laser, RGB, and point cloud data to enhance autonomous navigation in complex, obstacle-rich environments, demonstrating superior performance in simulation and real-world tests.

Contribution

The paper presents a novel multimodal fusion network specifically designed for complex environment navigation, integrating three visual modalities for improved robustness and real-time operation.

Findings

NMFNet outperforms existing methods in complex environment navigation.

Using multiple modalities is crucial for effective autonomous navigation.

The network successfully operates in both simulated and real-world scenarios.

Abstract

Autonomous navigation in complex environments is a crucial task in time-sensitive scenarios such as disaster response or search and rescue. However, complex environments pose significant challenges for autonomous platforms to navigate due to their challenging properties: constrained narrow passages, unstable pathway with debris and obstacles, or irregular geological structures and poor lighting conditions. In this work, we propose a multimodal fusion approach to address the problem of autonomous navigation in complex environments such as collapsed cites, or natural caves. We first simulate the complex environments in a physics-based simulation engine and collect a large-scale dataset for training. We then propose a Navigation Multimodal Fusion Network (NMFNet) which has three branches to effectively handle three visual modalities: laser, RGB images, and point cloud data. The extensively experimental results show that our NMFNet outperforms recent state of the art by a fair margin while achieving real-time performance. We further show that the use of multiple modalities is essential for autonomous navigation in complex environments. Finally, we successfully deploy our network to both simulated and real mobile robots.