Approximate Conditional Coverage & Calibration via Neural Model Approximations

TL;DR

This paper introduces a novel calibration method for deep classifiers using KNN-based approximations and Mondrian Conformal Predictors, achieving well-calibrated, robust prediction sets in NLP tasks with distribution shifts.

Contribution

It develops a data-driven partitioning approach with a new Inductive Venn Predictor for improved calibration and robustness in deep model uncertainty quantification.

Findings

Achieves well-calibrated prediction sets in NLP classification tasks.

Demonstrates robustness to data partition changes and covariate shifts.

Outperforms recent conformal prediction methods on challenging datasets.

Abstract

A typical desideratum for quantifying the uncertainty from a classification model as a prediction set is class-conditional singleton set calibration. That is, such sets should map to the output of well-calibrated selective classifiers, matching the observed frequencies of similar instances. Recent works proposing adaptive and localized conformal p-values for deep networks do not guarantee this behavior, nor do they achieve it empirically. Instead, we use the strong signals for prediction reliability from KNN-based approximations of Transformer networks to construct data-driven partitions for Mondrian Conformal Predictors, which are treated as weak selective classifiers that are then calibrated via a new Inductive Venn Predictor, the Venn-ADMIT Predictor. The resulting selective classifiers are well-calibrated, in a conservative but practically useful sense for a given threshold. They are inherently robust to changes in the proportions of the data partitions, and straightforward conservative heuristics provide additional robustness to covariate shifts. We compare and contrast to the quantities produced by recent Conformal Predictors on several representative and challenging natural language processing classification tasks, including class-imbalanced and distribution-shifted settings.