Learning Ordinality in Semantic Segmentation

TL;DR

This paper introduces methods to incorporate ordinal relationships between classes in semantic segmentation, improving accuracy and generalization without increasing inference time.

Contribution

It proposes novel regularization terms and metrics for spatial ordinal segmentation that explicitly model inter-class dependencies in the image space.

Findings

Achieves up to 15.7% relative improvement in Dice coefficient

Enhances ordinal metric performance and generalization

No additional inference time required

Abstract

Semantic segmentation consists of predicting a semantic label for each image pixel. While existing deep learning approaches achieve high accuracy, they often overlook the ordinal relationships between classes, which can provide critical domain knowledge (e.g., the pupil lies within the iris, and lane markings are part of the road). This paper introduces novel methods for spatial ordinal segmentation that explicitly incorporate these inter-class dependencies. By treating each pixel as part of a structured image space rather than as an independent observation, we propose two regularization terms and a new metric to enforce ordinal consistency between neighboring pixels. Two loss regularization terms and one metric are proposed for structural ordinal segmentation, which penalizes predictions of non-ordinal adjacent classes. Five biomedical datasets and multiple configurations of autonomous driving datasets demonstrate the efficacy of the proposed methods. Our approach achieves improvements in ordinal metrics and enhances generalization, with up to a 15.7% relative increase in the Dice coefficient. Importantly, these benefits come without additional inference time costs. This work highlights the significance of spatial ordinal relationships in semantic segmentation and provides a foundation for further exploration in structured image representations.