Optimizing against Infeasible Inclusions from Data for Semantic Segmentation through Morphology

TL;DR

This paper introduces InSeIn, a method that extracts spatial class inclusion constraints from training data and enforces them during training to reduce infeasible segmentations in semantic segmentation models, improving their accuracy.

Contribution

The paper proposes a novel, lightweight, plug-and-play approach that explicitly models and enforces spatial class inclusion constraints to improve semantic segmentation.

Findings

Significant performance improvements on ADE20K, Cityscapes, and ACDC datasets.

Reduction in infeasible class inclusions in segmentation outputs.

Enhanced model robustness to domain shifts.

Abstract

State-of-the-art semantic segmentation models are typically optimized in a data-driven fashion, minimizing solely per-pixel or per-segment classification objectives on their training data. This purely data-driven paradigm often leads to absurd segmentations, especially when the domain of input images is shifted from the one encountered during training. For instance, state-of-the-art models may assign the label "road" to a segment that is included by another segment that is respectively labeled as "sky". However, the ground truth of the existing dataset at hand dictates that such inclusion is not feasible. Our method, Infeasible Semantic Inclusions (InSeIn), first extracts explicit inclusion constraints that govern spatial class relations from the semantic segmentation training set at hand in an offline, data-driven fashion, and then enforces a morphological yet differentiable loss that penalizes violations of these constraints during training to promote prediction feasibility. InSeIn is a light-weight plug-and-play method, constitutes a novel step towards minimizing infeasible semantic inclusions in the predictions of learned segmentation models, and yields consistent and significant performance improvements over diverse state-of-the-art networks across the ADE20K, Cityscapes, and ACDC datasets. https://github.com/SHAMIK-97/InSeIn