Calibration and Discrimination Optimization Using Clusters of Learned Representation

TL;DR

This paper presents a flexible calibration pipeline that uses clusters of learned representations and ensembles of calibration functions to significantly improve model calibration and discrimination, crucial for high-stakes decision-making.

Contribution

It introduces a novel clustering-based calibration method that enhances calibration scores and provides a new matching metric for optimizing both discrimination and calibration.

Findings

Calibration score improved from 82.28% to 100%.

The method adapts to various representations and calibration techniques.

Enhanced model reliability for critical decision-making tasks.

Abstract

Machine learning models are essential for decision-making and risk assessment, requiring highly reliable predictions in terms of both discrimination and calibration. While calibration often receives less attention, it is crucial for critical decisions, such as those in clinical predictions. We introduce a novel calibration pipeline that leverages an ensemble of calibration functions trained on clusters of learned representations of the input samples to enhance overall calibration. This approach not only improves the calibration score of various methods from 82.28% up to 100% but also introduces a unique matching metric that ensures model selection optimizes both discrimination and calibration. Our generic scheme adapts to any underlying representation, clustering, calibration methods and metric, offering flexibility and superior performance across commonly used calibration methods.