Cityscape-Adverse: Benchmarking Robustness of Semantic Segmentation with Realistic Scene Modifications via Diffusion-Based Image Editing

TL;DR

This paper introduces Cityscape-Adverse, a benchmark using diffusion-based image editing to simulate adverse conditions for evaluating and improving the robustness of semantic segmentation models in realistic scenes.

Contribution

It presents a novel benchmark that creates realistic adverse scene modifications and evaluates model robustness, highlighting the benefits of training on synthetic data for resilience.

Findings

Transformer models are more robust than CNNs under adverse conditions.

Models trained on Cityscape-Adverse perform better on unseen domains.

Extreme weather and lighting cause significant performance drops.

Abstract

Recent advancements in generative AI, particularly diffusion-based image editing, have enabled the transformation of images into highly realistic scenes using only text instructions. This technology offers significant potential for generating diverse synthetic datasets to evaluate model robustness. In this paper, we introduce Cityscape-Adverse, a benchmark that employs diffusion-based image editing to simulate eight adverse conditions, including variations in weather, lighting, and seasons, while preserving the original semantic labels. We evaluate the reliability of diffusion-based models in generating realistic scene modifications and assess the performance of state-of-the-art CNN and Transformer-based semantic segmentation models under these challenging conditions. Additionally, we analyze which modifications have the greatest impact on model performance and explore how training on synthetic datasets can improve robustness in real-world adverse scenarios. Our results demonstrate that all tested models, particularly CNN-based architectures, experienced significant performance degradation under extreme conditions, while Transformer-based models exhibited greater resilience. We verify that models trained on Cityscape-Adverse show significantly enhanced resilience when applied to unseen domains. Code and datasets will be released at https://github.com/naufalso/cityscape-adverse.