Continual Road-Scene Semantic Segmentation via Feature-Aligned Symmetric Multi-Modal Network

TL;DR

This paper proposes a symmetric, feature-aligned multimodal segmentation network that remains effective even when one modality is missing, suitable for real-world autonomous driving scenarios with sensor failures.

Contribution

It introduces a symmetric information-sharing scheme and a class-incremental continual learning method for robust multimodal segmentation.

Findings

Effective even with missing modalities

Achieves high performance on SemanticKITTI

Suitable for safety-critical autonomous driving

Abstract

State-of-the-art multimodal semantic segmentation strategies combining LiDAR and color data are usually designed on top of asymmetric information-sharing schemes and assume that both modalities are always available. This strong assumption may not hold in real-world scenarios, where sensors are prone to failure or can face adverse conditions that make the acquired information unreliable. This problem is exacerbated when continual learning scenarios are considered since they have stringent data reliability constraints. In this work, we re-frame the task of multimodal semantic segmentation by enforcing a tightly coupled feature representation and a symmetric information-sharing scheme, which allows our approach to work even when one of the input modalities is missing. We also introduce an ad-hoc class-incremental continual learning scheme, proving our approach's effectiveness and reliability even in safety-critical settings, such as autonomous driving. We evaluate our approach on the SemanticKITTI dataset, achieving impressive performances.