Conformal Prediction Meets Long-tail Classification

TL;DR

This paper introduces Tail-Aware Conformal Prediction (TACP) and its extension sTACP to improve uncertainty quantification in long-tail classification, ensuring more balanced coverage across head and tail classes.

Contribution

The paper proposes TACP and sTACP methods that leverage long-tail structure to reduce coverage imbalance, with theoretical guarantees and empirical validation.

Findings

TACP reduces head-tail coverage gap compared to standard CP.

sTACP further balances coverage across classes via reweighting.

Experiments show improved coverage on long-tail benchmarks.

Abstract

Conformal Prediction (CP) is a popular method for uncertainty quantification that converts a pretrained model's point prediction into a prediction set, with the set size reflecting the model's confidence. Although existing CP methods are guaranteed to achieve marginal coverage, they often exhibit imbalanced coverage across classes under long-tail label distributions, tending to over cover the head classes at the expense of under covering the remaining tail classes. This under coverage is particularly concerning, as it undermines the reliability of the prediction sets for minority classes, even with coverage ensured on average. In this paper, we propose the Tail-Aware Conformal Prediction (TACP) method to mitigate the under coverage of the tail classes by utilizing the long-tail structure and narrowing the head-tail coverage gap. Theoretical analysis shows that it consistently achieves a smaller head-tail coverage gap than standard methods. To further improve coverage balance across all classes, we introduce an extension of TACP: soft TACP (sTACP) via a reweighting mechanism. The proposed framework can be combined with various non-conformity scores, and experiments on multiple long-tail benchmark datasets demonstrate the effectiveness of our methods.