Uncertainty Quantification With Multiple Sources

TL;DR

This paper extends weighted conformal prediction to multi-source settings with varying covariate distributions, proposing aggregation and pooling strategies with theoretical guarantees and empirical validation.

Contribution

It introduces two novel methods for multi-source uncertainty quantification under covariate shift, with theoretical analysis and experimental validation.

Findings

Proposed merge-based aggregation improves prediction sets across sources.

Data pooling with reweighting enhances uncertainty quantification in multi-source data.

Theoretical guarantees support the effectiveness of the methods.

Abstract

Weighted conformal prediction (WCP) has been commonly used to quantify prediction uncertainty under covariate shift. However, the effectiveness of WCP relies heavily on the degree of overlap between the training and test covariate distributions. This challenge is exacerbated in multi-source settings with varying covariate distributions, where direct application of WCP can be impractical. In this paper, we address the multi-source setup by leveraging WCP under the assumption of a shared conditional distribution. We investigate two extensions of WCP: (i) a merge-based aggregation of source-specific weighted conformal prediction sets, and (ii) a data-pooling strategy that jointly reweights samples across all sources. Theoretical guarantees are provided for the proposed approaches, and experiments are conducted based on a synthetic regression task and a multi-domain image classification benchmark to validate our proposed methods.