Calibrating Where It Matters: Constrained Temperature Scaling

TL;DR

This paper proposes a modified temperature scaling method for calibrating convolutional classifiers, focusing on regions of the probability space that influence decision-making, demonstrated on dermoscopy image classification.

Contribution

It introduces a calibration approach that emphasizes decision-critical regions, improving calibration effectiveness in practical diagnostic scenarios.

Findings

Enhanced calibration in decision-critical regions

Improved decision-making accuracy in dermoscopy classification

Focus on cost-sensitive calibration regions

Abstract

We consider calibration of convolutional classifiers for diagnostic decision making. Clinical decision makers can use calibrated classifiers to minimise expected costs given their own cost function. Such functions are usually unknown at training time. If minimising expected costs is the primary aim, algorithms should focus on tuning calibration in regions of probability simplex likely to effect decisions. We give an example, modifying temperature scaling calibration, and demonstrate improved calibration where it matters using convnets trained to classify dermoscopy images.