Multi-Rater Calibrated Segmentation Models

TL;DR

This paper introduces a novel ordinal learning approach to improve the calibration of medical image segmentation models by leveraging inter-rater agreement as ordered information.

Contribution

It reformulates multi-rater supervision as an ordinal learning problem, enhancing probabilistic calibration without sacrificing segmentation accuracy.

Findings

Ordinal-aware training significantly improves calibration across multiple benchmarks.

The approach maintains high segmentation accuracy while better reflecting annotation uncertainty.

Calibration metrics show consistent improvement over existing methods.

Abstract

Objective: Accurate probability estimates are essential for the safe deployment of medical image segmentation models in clinical decision-making. However, modern deep segmentation networks are often poorly calibrated, a problem exacerbated when multiple expert annotations exhibit substantial disagreement. While inter-rater variability is typically treated as noise, it provides valuable information about intrinsic annotation ambiguity that must be reflected in model confidence. Methods: We improve the probabilistic calibration of medical image segmentation models by reformulating multi-rater supervision as an ordinal learning problem. Voxel-wise annotator agreement is treated as an ordered target, linking predictive confidence to the empirical variability in training data. This formulation allows the use of ordinal-aware scoring rules, such as the Ranked Probability Score ordinal loss, combined with a standard binary objective to preserve discriminative performance. Results: We evaluated the proposed approach across four public segmentation benchmarks spanning ophthalmology, histopathology, and thoracic imaging. Calibration was assessed using a multi-rater extension of expected calibration error. Results consistently show that ordinal-aware training yields substantially improved calibration with respect to inter-rater agreement without degrading segmentation accuracy. Conclusions: Treating multi-rater annotations as ordered information provides a principled and architecture-agnostic route to more reliable probabilistic segmentation models.