Disentangle then Parse:Night-time Semantic Segmentation with Illumination Disentanglement

TL;DR

This paper introduces a novel night-time semantic segmentation method called Disentangle then Parse (DTP), which explicitly separates lighting and reflectance to improve accuracy in challenging lighting conditions.

Contribution

The paper proposes a new paradigm that disentangles illumination and reflectance for better night-time segmentation, and introduces the Semantic-Oriented Disentanglement and Illumination-Aware Parser components.

Findings

DTP significantly outperforms state-of-the-art methods on night-time segmentation tasks.

DTP can enhance existing day-time segmentation models for night-time use with minimal additional parameters.

The approach effectively handles complex lighting variations in night-time scenes.

Abstract

Most prior semantic segmentation methods have been developed for day-time scenes, while typically underperforming in night-time scenes due to insufficient and complicated lighting conditions. In this work, we tackle this challenge by proposing a novel night-time semantic segmentation paradigm, i.e., disentangle then parse (DTP). DTP explicitly disentangles night-time images into light-invariant reflectance and light-specific illumination components and then recognizes semantics based on their adaptive fusion. Concretely, the proposed DTP comprises two key components: 1) Instead of processing lighting-entangled features as in prior works, our Semantic-Oriented Disentanglement (SOD) framework enables the extraction of reflectance component without being impeded by lighting, allowing the network to consistently recognize the semantics under cover of varying and complicated lighting conditions. 2) Based on the observation that the illumination component can serve as a cue for some semantically confused regions, we further introduce an Illumination-Aware Parser (IAParser) to explicitly learn the correlation between semantics and lighting, and aggregate the illumination features to yield more precise predictions. Extensive experiments on the night-time segmentation task with various settings demonstrate that DTP significantly outperforms state-of-the-art methods. Furthermore, with negligible additional parameters, DTP can be directly used to benefit existing day-time methods for night-time segmentation.