A Multi-modal Fusion Network for Terrain Perception Based on Illumination Aware

TL;DR

This paper introduces an illumination-aware multi-modal fusion network for autonomous vehicle terrain perception, improving accuracy under varying lighting conditions by dynamically adjusting sensor data fusion based on illumination features.

Contribution

The paper presents a novel illumination-aware fusion network that adaptively weights sensor modalities for robust terrain perception in changing lighting environments.

Findings

Outperforms state-of-the-art methods in diverse lighting conditions.

Effectively estimates illumination features for dynamic sensor fusion.

Enhances perception accuracy by integrating illumination-aware optimization.

Abstract

Road terrains play a crucial role in ensuring the driving safety of autonomous vehicles (AVs). However, existing sensors of AVs, including cameras and Lidars, are susceptible to variations in lighting and weather conditions, making it challenging to achieve real-time perception of road conditions. In this paper, we propose an illumination-aware multi-modal fusion network (IMF), which leverages both exteroceptive and proprioceptive perception and optimizes the fusion process based on illumination features. We introduce an illumination-perception sub-network to accurately estimate illumination features. Moreover, we design a multi-modal fusion network which is able to dynamically adjust weights of different modalities according to illumination features. We enhance the optimization process by pre-training of the illumination-perception sub-network and incorporating illumination loss as one of the training constraints. Extensive experiments demonstrate that the IMF shows a superior performance compared to state-of-the-art methods. The comparison results with single modality perception methods highlight the comprehensive advantages of multi-modal fusion in accurately perceiving road terrains under varying lighting conditions. Our dataset is available at: https://github.com/lindawang2016/IMF.