Boosting Cross-spectral Unsupervised Domain Adaptation for Thermal Semantic Segmentation

TL;DR

This paper introduces a novel cross-spectral unsupervised domain adaptation method for thermal image semantic segmentation, leveraging mutual learning and self-supervised prototypical loss to improve performance in adverse conditions.

Contribution

The paper proposes a masked mutual learning strategy and a prototypical self-supervised loss to enhance thermal segmentation in domain adaptation tasks, especially at night.

Findings

Outperforms previous UDA methods in thermal segmentation

Achieves comparable results to supervised methods

Effective in low-light nighttime scenarios

Abstract

In autonomous driving, thermal image semantic segmentation has emerged as a critical research area, owing to its ability to provide robust scene understanding under adverse visual conditions. In particular, unsupervised domain adaptation (UDA) for thermal image segmentation can be an efficient solution to address the lack of labeled thermal datasets. Nevertheless, since these methods do not effectively utilize the complementary information between RGB and thermal images, they significantly decrease performance during domain adaptation. In this paper, we present a comprehensive study on cross-spectral UDA for thermal image semantic segmentation. We first propose a novel masked mutual learning strategy that promotes complementary information exchange by selectively transferring results between each spectral model while masking out uncertain regions. Additionally, we introduce a novel prototypical self-supervised loss designed to enhance the performance of the thermal segmentation model in nighttime scenarios. This approach addresses the limitations of RGB pre-trained networks, which cannot effectively transfer knowledge under low illumination due to the inherent constraints of RGB sensors. In experiments, our method achieves higher performance over previous UDA methods and comparable performance to state-of-the-art supervised methods.