Mitigating the Multiplicity Burden: The Role of Calibration in Reducing Predictive Multiplicity of Classifiers

TL;DR

This paper investigates how calibration techniques can reduce predictive multiplicity in classifiers, especially for minority classes, thereby enhancing fairness and stability in high-stakes decision-making environments.

Contribution

It demonstrates that post-hoc calibration methods like Platt Scaling and Isotonic Regression effectively lower predictive multiplicity and improve model consensus.

Findings

Calibration reduces predictive multiplicity in the Rashomon set.

Minority class observations experience higher multiplicity burden.

Post-hoc calibration methods decrease prediction obscurity.

Abstract

As machine learning models are increasingly deployed in high-stakes environments, ensuring both probabilistic reliability and prediction stability has become critical. This paper examines the interplay between classification calibration and predictive multiplicity - the phenomenon in which multiple near-optimal models within the Rashomon set yield conflicting credit outcomes for the same applicant. Using nine diverse credit risk benchmark datasets, we investigate whether predictive multiplicity concentrates in regions of low predictive confidence and how post-hoc calibration can mitigate algorithmic arbitrariness. Our empirical analysis reveals that minority class observations bear a disproportionate multiplicity burden, as confirmed by significant disparities in predictive multiplicity and prediction confidence. Furthermore, our empirical comparisons indicate that applying post-hoc calibration methods - specifically Platt Scaling, Isotonic Regression, and Temperature Scaling - is associated with lower obscurity across the Rashomon set. Among the tested techniques, Platt Scaling and Isotonic Regression provide the most robust reduction in predictive multiplicity. These findings suggest that calibration can function as a consensus-enforcing layer and may support procedural fairness by mitigating predictive multiplicity.