Optimal Aggregation of Prediction Intervals under Unsupervised Domain Shift

TL;DR

This paper introduces methods for aggregating prediction intervals to maintain coverage and minimize width under unsupervised domain shift, supported by theoretical guarantees and practical evaluations.

Contribution

It proposes novel, computationally efficient aggregation techniques for prediction intervals under domain shift, with theoretical analysis and real-world validation.

Findings

Method achieves minimal interval width with adequate coverage

Theoretical guarantees ensure reliability under domain shift

Effective on real-world datasets

Abstract

As machine learning models are increasingly deployed in dynamic environments, it becomes paramount to assess and quantify uncertainties associated with distribution shifts. A distribution shift occurs when the underlying data-generating process changes, leading to a deviation in the model's performance. The prediction interval, which captures the range of likely outcomes for a given prediction, serves as a crucial tool for characterizing uncertainties induced by their underlying distribution. In this paper, we propose methodologies for aggregating prediction intervals to obtain one with minimal width and adequate coverage on the target domain under unsupervised domain shift, under which we have labeled samples from a related source domain and unlabeled covariates from the target domain. Our analysis encompasses scenarios where the source and the target domain are related via i) a bounded density ratio, and ii) a measure-preserving transformation. Our proposed methodologies are computationally efficient and easy to implement. Beyond illustrating the performance of our method through real-world datasets, we also delve into the theoretical details. This includes establishing rigorous theoretical guarantees, coupled with finite sample bounds, regarding the coverage and width of our prediction intervals. Our approach excels in practical applications and is underpinned by a solid theoretical framework, ensuring its reliability and effectiveness across diverse contexts.