On Calibrating Semantic Segmentation Models: Analyses and An Algorithm

TL;DR

This paper systematically studies semantic segmentation calibration, identifies key factors affecting it, and proposes a simple, effective selective scaling method that outperforms existing approaches across various benchmarks.

Contribution

It introduces a novel selective scaling approach for calibration, emphasizing misprediction smoothing, and provides comprehensive analysis and comparison with existing methods.

Findings

Selective scaling improves calibration accuracy.

Prediction correctness significantly impacts calibration.

The method outperforms existing calibration techniques across benchmarks.

Abstract

We study the problem of semantic segmentation calibration. Lots of solutions have been proposed to approach model miscalibration of confidence in image classification. However, to date, confidence calibration research on semantic segmentation is still limited. We provide a systematic study on the calibration of semantic segmentation models and propose a simple yet effective approach. First, we find that model capacity, crop size, multi-scale testing, and prediction correctness have impact on calibration. Among them, prediction correctness, especially misprediction, is more important to miscalibration due to over-confidence. Next, we propose a simple, unifying, and effective approach, namely selective scaling, by separating correct/incorrect prediction for scaling and more focusing on misprediction logit smoothing. Then, we study popular existing calibration methods and compare them with selective scaling on semantic segmentation calibration. We conduct extensive experiments with a variety of benchmarks on both in-domain and domain-shift calibration and show that selective scaling consistently outperforms other methods.