Semi-Supervised Conformal Prediction With Unlabeled Nonconformity Score

TL;DR

This paper extends conformal prediction to semi-supervised settings by introducing SemiCP, which leverages unlabeled data through an unlabeled nonconformity score, significantly improving coverage stability with limited labeled data.

Contribution

The paper proposes SemiCP, a novel semi-supervised conformal prediction method using an unlabeled nonconformity score, reducing coverage gap with limited labeled data.

Findings

SemiCP reduces coverage gap by up to 77% on benchmarks.

Theoretical guarantee of convergence rate $ ext{O}(1/\sqrt{N})$ for coverage gap.

Effective in scenarios with very few labeled samples.

Abstract

Conformal prediction (CP) is a powerful framework for uncertainty quantification, generating prediction sets with coverage guarantees. Split conformal prediction relies on labeled data in the calibration procedure. However, the labeled data is often limited in real-world scenarios, leading to unstable coverage performance in different runs. To address this issue, we extend CP to the semi-supervised setting and propose SemiCP, a new paradigm that leverages both labeled and unlabeled data for calibration. To achieve this, we introduce an unlabeled nonconformity score, Nearest Neighbor Matching (NNM) score. Specifically, NNM estimates the nonconformity scores of unlabeled samples using their most similar pseudo-labeled counterparts during calibration, while maintaining the original scores for labeled data. Theoretically, we demonstrate that the average coverage gap (i.e., the absolute difference between the empirical marginal coverage and the target coverage) of SemiCP can decrease significantly at a rate $\mathcal{O}(1/\sqrt{N})$ and converge to an error term, where $N$ is the number of unlabeled data. Extensive experiments validate the effectiveness of SemiCP under limited labeled data, reducing the average coverage gap by up to 77% on common benchmarks with 4000 unlabeled examples, when there are only 20 labeled examples.