Understanding Dark Scenes by Contrasting Multi-Modal Observations

TL;DR

This paper proposes a supervised multi-modal contrastive learning method to improve dark scene understanding by enhancing semantic discriminability in multi-modal feature spaces, outperforming previous approaches.

Contribution

It introduces a novel contrastive learning framework that jointly performs cross-modal and intra-modal contrast to better utilize multi-modal data for dark scene analysis.

Findings

Achieves state-of-the-art performance on dark scene understanding tasks.

Effectively enhances semantic discriminability in multi-modal feature spaces.

Demonstrates robustness across diverse lighting conditions and modalities.

Abstract

Understanding dark scenes based on multi-modal image data is challenging, as both the visible and auxiliary modalities provide limited semantic information for the task. Previous methods focus on fusing the two modalities but neglect the correlations among semantic classes when minimizing losses to align pixels with labels, resulting in inaccurate class predictions. To address these issues, we introduce a supervised multi-modal contrastive learning approach to increase the semantic discriminability of the learned multi-modal feature spaces by jointly performing cross-modal and intra-modal contrast under the supervision of the class correlations. The cross-modal contrast encourages same-class embeddings from across the two modalities to be closer and pushes different-class ones apart. The intra-modal contrast forces same-class or different-class embeddings within each modality to be together or apart. We validate our approach on a variety of tasks that cover diverse light conditions and image modalities. Experiments show that our approach can effectively enhance dark scene understanding based on multi-modal images with limited semantics by shaping semantic-discriminative feature spaces. Comparisons with previous methods demonstrate our state-of-the-art performance. Code and pretrained models are available at https://github.com/palmdong/SMMCL.