Mutual-GAN: Towards Unsupervised Cross-Weather Adaptation with Mutual Information Constraint

TL;DR

Mutual-GAN introduces a mutual information constraint to improve unsupervised cross-weather image translation, significantly enhancing semantic segmentation accuracy in adverse weather conditions for autonomous driving videos.

Contribution

This paper presents a novel Mutual-GAN with mutual information constraint for unsupervised cross-weather adaptation, addressing object preservation during image translation.

Findings

Improves semantic segmentation accuracy under adverse weather conditions.

Produces visually plausible translated images.

Enhances robustness of autonomous driving perception systems.

Abstract

Convolutional neural network (CNN) have proven its success for semantic segmentation, which is a core task of emerging industrial applications such as autonomous driving. However, most progress in semantic segmentation of urban scenes is reported on standard scenarios, i.e., daytime scenes with favorable illumination conditions. In practical applications, the outdoor weather and illumination are changeable, e.g., cloudy and nighttime, which results in a significant drop of semantic segmentation accuracy of CNN only trained with daytime data. In this paper, we propose a novel generative adversarial network (namely Mutual-GAN) to alleviate the accuracy decline when daytime-trained neural network is applied to videos captured under adverse weather conditions. The proposed Mutual-GAN adopts mutual information constraint to preserve image-objects during cross-weather adaptation, which is an unsolved problem for most unsupervised image-to-image translation approaches (e.g., CycleGAN). The proposed Mutual-GAN is evaluated on two publicly available driving video datasets (i.e., CamVid and SYNTHIA). The experimental results demonstrate that our Mutual-GAN can yield visually plausible translated images and significantly improve the semantic segmentation accuracy of daytime-trained deep learning network while processing videos under challenging weathers.