Efficient Conformal Prediction under Data Heterogeneity

TL;DR

This paper introduces an efficient conformal prediction method that provides valid uncertainty quantification for non-exchangeable data, especially in federated learning scenarios with data heterogeneity.

Contribution

It presents a novel approach to conformal prediction that is computationally feasible and guarantees validity under general non-exchangeable data distributions, with applications to federated learning.

Findings

Valid confidence sets for non-exchangeable data

Personalized prediction sets in federated learning

Demonstrated effectiveness on real-world datasets

Abstract

Conformal Prediction (CP) stands out as a robust framework for uncertainty quantification, which is crucial for ensuring the reliability of predictions. However, common CP methods heavily rely on data exchangeability, a condition often violated in practice. Existing approaches for tackling non-exchangeability lead to methods that are not computable beyond the simplest examples. This work introduces a new efficient approach to CP that produces provably valid confidence sets for fairly general non-exchangeable data distributions. We illustrate the general theory with applications to the challenging setting of federated learning under data heterogeneity between agents. Our method allows constructing provably valid personalized prediction sets for agents in a fully federated way. The effectiveness of the proposed method is demonstrated in a series of experiments on real-world datasets.