Cooperative Semantic Segmentation and Image Restoration in Adverse Environmental Conditions

TL;DR

This paper introduces a unified deep-learning framework that simultaneously improves semantic segmentation accuracy and image restoration quality in adverse environmental conditions by leveraging semantic information and exemplar guidance.

Contribution

It presents a novel holistic approach combining Semantically-Guided Adaptation and Exemplar-Guided Synthesis for joint segmentation and restoration in challenging conditions.

Findings

Enhanced segmentation accuracy in degraded environments.

Improved perceptual quality of restored images.

Better structural similarity compared to baseline methods.

Abstract

Most state-of-the-art semantic segmentation approaches only achieve high accuracy in good conditions. In practically-common but less-discussed adverse environmental conditions, their performance can decrease enormously. Existing studies usually cast the handling of segmentation in adverse conditions as a separate post-processing step after signal restoration, making the segmentation performance largely depend on the quality of restoration. In this paper, we propose a novel deep-learning framework to tackle semantic segmentation and image restoration in adverse environmental conditions in a holistic manner. The proposed approach contains two components: Semantically-Guided Adaptation, which exploits semantic information from degraded images to refine the segmentation; and Exemplar-Guided Synthesis, which restores images from semantic label maps given degraded exemplars as the guidance. Our method cooperatively leverages the complementarity and interdependence of low-level restoration and high-level segmentation in adverse environmental conditions. Extensive experiments on various datasets demonstrate that our approach can not only improve the accuracy of semantic segmentation with degradation cues, but also boost the perceptual quality and structural similarity of image restoration with semantic guidance.