Model Adaptation with Synthetic and Real Data for Semantic Dense Foggy Scene Understanding

TL;DR

This paper introduces CMAda, a curriculum-based method for adapting semantic segmentation models from synthetic to real dense fog, along with new fog simulation, a density estimator, and a foggy dataset, improving scene understanding in foggy conditions.

Contribution

The paper presents a novel curriculum adaptation approach, a new fog simulation technique, a fog density estimator, and a comprehensive real foggy dataset for semantic scene understanding.

Findings

CMAda significantly improves segmentation performance in dense fog.

Synthetic fog simulation outperforms existing methods.

The Foggy Zurich dataset provides valuable real foggy images with annotations.

Abstract

This work addresses the problem of semantic scene understanding under dense fog. Although considerable progress has been made in semantic scene understanding, it is mainly related to clear-weather scenes. Extending recognition methods to adverse weather conditions such as fog is crucial for outdoor applications. In this paper, we propose a novel method, named Curriculum Model Adaptation (CMAda), which gradually adapts a semantic segmentation model from light synthetic fog to dense real fog in multiple steps, using both synthetic and real foggy data. In addition, we present three other main stand-alone contributions: 1) a novel method to add synthetic fog to real, clear-weather scenes using semantic input; 2) a new fog density estimator; 3) the Foggy Zurich dataset comprising $3808$ real foggy images, with pixel-level semantic annotations for $16$ images with dense fog. Our experiments show that 1) our fog simulation slightly outperforms a state-of-the-art competing simulation with respect to the task of semantic foggy scene understanding (SFSU); 2) CMAda improves the performance of state-of-the-art models for SFSU significantly by leveraging unlabeled real foggy data. The datasets and code are publicly available.