Do We Need Perfect Data? Leveraging Noise for Domain Generalized Segmentation

TL;DR

FLEX-Seg introduces a novel framework that leverages inherent misalignment in diffusion-based synthetic data to improve domain generalization in semantic segmentation, outperforming state-of-the-art methods.

Contribution

The paper proposes FLEX-Seg, a new approach that exploits misalignment in synthetic data for robust learning, incorporating adaptive prototypes, uncertainty emphasis, and hardness-aware sampling.

Findings

Achieves 2.44% and 2.63% mIoU improvements on ACDC and Dark Zurich datasets.

Demonstrates robustness across five real-world datasets.

Validates that handling imperfect synthetic data enhances domain generalization.

Abstract

Domain generalization in semantic segmentation faces challenges from domain shifts, particularly under adverse conditions. While diffusion-based data generation methods show promise, they introduce inherent misalignment between generated images and semantic masks. This paper presents FLEX-Seg (FLexible Edge eXploitation for Segmentation), a framework that transforms this limitation into an opportunity for robust learning. FLEX-Seg comprises three key components: (1) Granular Adaptive Prototypes that captures boundary characteristics across multiple scales, (2) Uncertainty Boundary Emphasis that dynamically adjusts learning emphasis based on prediction entropy, and (3) Hardness-Aware Sampling that progressively focuses on challenging examples. By leveraging inherent misalignment rather than enforcing strict alignment, FLEX-Seg learns robust representations while capturing rich stylistic variations. Experiments across five real-world datasets demonstrate consistent improvements over state-of-the-art methods, achieving 2.44% and 2.63% mIoU gains on ACDC and Dark Zurich. Our findings validate that adaptive strategies for handling imperfect synthetic data lead to superior domain generalization. Code is available at https://github.com/VisualScienceLab-KHU/FLEX-Seg.