Imperfect Segmentation Labels: How Much Do They Matter?

TL;DR

This study investigates how different types and degrees of label errors in training data affect the performance of various semantic segmentation models, highlighting the robustness of U-Net to boundary errors.

Contribution

It provides a large-scale analysis of the impact of label imperfections on segmentation models, revealing architecture-specific robustness to boundary-localized errors.

Findings

U-Net is more robust to boundary errors than SegNet and FCN32.

All models are robust to non-boundary errors.

Boundary errors significantly degrade model performance.

Abstract

Labeled datasets for semantic segmentation are imperfect, especially in medical imaging where borders are often subtle or ill-defined. Little work has been done to analyze the effect that label errors have on the performance of segmentation methodologies. Here we present a large-scale study of model performance in the presence of varying types and degrees of error in training data. We trained U-Net, SegNet, and FCN32 several times for liver segmentation with 10 different modes of ground-truth perturbation. Our results show that for each architecture, performance steadily declines with boundary-localized errors, however, U-Net was significantly more robust to jagged boundary errors than the other architectures. We also found that each architecture was very robust to non-boundary-localized errors, suggesting that boundary-localized errors are fundamentally different and more challenging problem than random label errors in a classification setting.