Adapting Prediction Sets to Distribution Shifts Without Labels

TL;DR

This paper introduces ECP and EACP, two methods that adapt conformal prediction sets to distribution shifts using only unlabeled data, significantly improving their robustness without requiring labels.

Contribution

The paper proposes two novel methods, ECP and EACP, for improving conformal prediction under distribution shifts using unlabeled data, without needing ground truth labels.

Findings

ECP and EACP outperform existing baselines.

Methods nearly match fully supervised performance.

Consistent improvements across large-scale datasets.

Abstract

Recently there has been a surge of interest to deploy confidence set predictions rather than point predictions in machine learning. Unfortunately, the effectiveness of such prediction sets is frequently impaired by distribution shifts in practice, and the challenge is often compounded by the lack of ground truth labels at test time. Focusing on a standard set-valued prediction framework called conformal prediction (CP), this paper studies how to improve its practical performance using only unlabeled data from the shifted test domain. This is achieved by two new methods called ECP and EACP, whose main idea is to adjust the score function in CP according to its base model's own uncertainty evaluation. Through extensive experiments on a number of large-scale datasets and neural network architectures, we show that our methods provide consistent improvement over existing baselines and nearly match the performance of fully supervised methods.