Unlocking Robust Semantic Segmentation Performance via Label-only Elastic Deformations against Implicit Label Noise

TL;DR

This paper proposes NSegment+, a novel data augmentation method that applies elastic deformations solely to segmentation labels to improve robustness against subtle, implicit label noise in semantic segmentation tasks.

Contribution

The paper introduces a new augmentation framework that decouples image and label transformations, effectively addressing realistic label imperfections and enhancing model robustness.

Findings

Achieves up to +3.39 mIoU improvement on PASCAL VOC.

Consistently improves performance across multiple datasets.

Enhances robustness against implicit label noise.

Abstract

While previous studies on image segmentation focus on handling severe (or explicit) label noise, real-world datasets also exhibit subtle (or implicit) label imperfections. These arise from inherent challenges, such as ambiguous object boundaries and annotator variability. Although not explicitly present, such mild and latent noise can still impair model performance. Typical data augmentation methods, which apply identical transformations to the image and its label, risk amplifying these subtle imperfections and limiting the model's generalization capacity. In this paper, we introduce NSegment+, a novel augmentation framework that decouples image and label transformations to address such realistic noise for semantic segmentation. By introducing controlled elastic deformations only to segmentation labels while preserving the original images, our method encourages models to focus on learning robust representations of object structures despite minor label inconsistencies. Extensive experiments demonstrate that NSegment+ consistently improves performance, achieving mIoU gains of up to +2.29, +2.38, +1.75, and +3.39 in average on Vaihingen, LoveDA, Cityscapes, and PASCAL VOC, respectively-even without bells and whistles, highlighting the importance of addressing implicit label noise. These gains can be further amplified when combined with other training tricks, including CutMix and Label Smoothing.