CMDA: Cross-Modality Domain Adaptation for Nighttime Semantic Segmentation

TL;DR

This paper introduces a novel unsupervised framework called CMDA that leverages multi-modality data from images and event cameras to improve nighttime semantic segmentation, especially under low-light conditions.

Contribution

The paper proposes the first cross-modality domain adaptation method for nighttime segmentation using images and event data, along with a new dataset for this task.

Findings

Effective in leveraging event data for nighttime segmentation

Improves boundary and structural detail capture in low-light conditions

Validated on both public and new datasets

Abstract

Most nighttime semantic segmentation studies are based on domain adaptation approaches and image input. However, limited by the low dynamic range of conventional cameras, images fail to capture structural details and boundary information in low-light conditions. Event cameras, as a new form of vision sensors, are complementary to conventional cameras with their high dynamic range. To this end, we propose a novel unsupervised Cross-Modality Domain Adaptation (CMDA) framework to leverage multi-modality (Images and Events) information for nighttime semantic segmentation, with only labels on daytime images. In CMDA, we design the Image Motion-Extractor to extract motion information and the Image Content-Extractor to extract content information from images, in order to bridge the gap between different modalities (Images to Events) and domains (Day to Night). Besides, we introduce the first image-event nighttime semantic segmentation dataset. Extensive experiments on both the public image dataset and the proposed image-event dataset demonstrate the effectiveness of our proposed approach. We open-source our code, models, and dataset at https://github.com/XiaRho/CMDA.