Conformal Prediction for Deep Classifier via Label Ranking
Jianguo Huang, Huajun Xi, Linjun Zhang, Huaxiu Yao, Yue Qiu, Hongxin, Wei
TL;DR
This paper introduces SAPS, a novel conformal prediction method that uses only the maximum softmax probability to produce smaller, more accurate prediction sets with reliable uncertainty estimates for deep classifiers.
Contribution
The paper proposes SAPS, a new algorithm that improves conformal prediction by reducing dependence on probability calibration, resulting in more compact and reliable prediction sets.
Findings
SAPS produces smaller prediction sets than existing methods.
SAPS enhances the conditional coverage rate of prediction sets.
SAPS effectively communicates instance-wise uncertainty.
Abstract
Conformal prediction is a statistical framework that generates prediction sets containing ground-truth labels with a desired coverage guarantee. The predicted probabilities produced by machine learning models are generally miscalibrated, leading to large prediction sets in conformal prediction. To address this issue, we propose a novel algorithm named (SAPS), which discards all the probability values except for the maximum softmax probability. The key idea behind SAPS is to minimize the dependence of the non-conformity score on the probability values while retaining the uncertainty information. In this manner, SAPS can produce compact prediction sets and communicate instance-wise uncertainty. Extensive experiments validate that SAPS not only lessens the prediction sets but also broadly enhances the conditional coverage rate of prediction sets.
Peer Reviews
Decision·ICML 2024 Poster
The paper is easy to read. The idea and the solution are both clearly articulated and the results are convincing. There are theoretical justifications.
See questions.
1. The authors propose a novel non-conformity measure in classification problem, and theoretically show it always dominates APS in the size of prediction sets if $\hat{\pi} = \pi$. 2. The authors conduct the experiments on three different datasets. They propose a novel metric ESCV to evaluate the performance of methods.
1. The theoretical contribution of this paper seems limited. Proposition 1 represents a common property of any non-conformity measure. Moreover, the condition in Proposition 2, $\hat{\pi} = \pi$, is challenging to satisfy in practice. As for another condition $\lambda \geq 1 - \frac{1}{K}$, note that $\lambda$ used in experiments is searched in the range of 0.001 to 0.5, which conflicts with this condition. Figure 4a shows that when $\lambda$ exceeds 0.2, the set size increases with $\lambda$.
* The authors empirically showed that using all probabilities is not necessary in APS. * The authors empirically showed that, under different network architectures, the proposed method returns more efficient prediction sets compared to APS and RAPS on three datasets.
* I have a reservation about the claimed contribution of higher adaption, i.e., the adaption is not that convincing: For the example of Figure 3(b), now that both SAPS and RAPS achieve the same coverage, why should we require a larger prediction set for difficult observations? In general, the smaller the better. RAPS gives more efficient predictions on those observations with higher difficulty, while the proposed SAPS only gives efficient predictions on a shorter interval of difficulty. * Even
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMachine Learning and Data Classification · Anomaly Detection Techniques and Applications · Neural Networks and Applications
MethodsSoftmax