Heterogeneous Calibration: A post-hoc model-agnostic framework for improved generalization

TL;DR

This paper proposes a post-hoc, model-agnostic heterogeneous calibration framework that improves AUC performance by calibrating model outputs within data partitions identified through tree-based algorithms, especially for overconfident models.

Contribution

It introduces heterogeneous calibration as a novel post-hoc method that optimally calibrates data partitions to enhance AUC, with theoretical backing and practical testing on deep neural networks.

Findings

Heterogeneous calibration improves AUC across multiple datasets.

Partition-based calibration outperforms uniform calibration methods.

Framework shows promise for integration with advanced partitioning and calibration techniques.

Abstract

We introduce the notion of heterogeneous calibration that applies a post-hoc model-agnostic transformation to model outputs for improving AUC performance on binary classification tasks. We consider overconfident models, whose performance is significantly better on training vs test data and give intuition onto why they might under-utilize moderately effective simple patterns in the data. We refer to these simple patterns as heterogeneous partitions of the feature space and show theoretically that perfectly calibrating each partition separately optimizes AUC. This gives a general paradigm of heterogeneous calibration as a post-hoc procedure by which heterogeneous partitions of the feature space are identified through tree-based algorithms and post-hoc calibration techniques are applied to each partition to improve AUC. While the theoretical optimality of this framework holds for any model, we focus on deep neural networks (DNNs) and test the simplest instantiation of this paradigm on a variety of open-source datasets. Experiments demonstrate the effectiveness of this framework and the future potential for applying higher-performing partitioning schemes along with more effective calibration techniques.