Information-theoretic Analysis of Test Data Sensitivity in Uncertainty

TL;DR

This paper introduces an information-theoretic framework to analyze how predictive uncertainty in Bayesian inference depends on the similarity between test and training data, extending to Bayesian meta-learning.

Contribution

It proposes a novel decomposition method to quantify test data sensitivity in Bayesian uncertainty, including for meta-learning scenarios.

Findings

Defines uncertainty sensitivity using information-theoretic measures

Extends analysis to Bayesian meta-learning and task sensitivities

Provides a new perspective on the relationship between data similarity and uncertainty

Abstract

Bayesian inference is often utilized for uncertainty quantification tasks. A recent analysis by Xu and Raginsky 2022 rigorously decomposed the predictive uncertainty in Bayesian inference into two uncertainties, called aleatoric and epistemic uncertainties, which represent the inherent randomness in the data-generating process and the variability due to insufficient data, respectively. They analyzed those uncertainties in an information-theoretic way, assuming that the model is well-specified and treating the model's parameters as latent variables. However, the existing information-theoretic analysis of uncertainty cannot explain the widely believed property of uncertainty, known as the sensitivity between the test and training data. It implies that when test data are similar to training data in some sense, the epistemic uncertainty should become small. In this work, we study such uncertainty sensitivity using our novel decomposition method for the predictive uncertainty. Our analysis successfully defines such sensitivity using information-theoretic quantities. Furthermore, we extend the existing analysis of Bayesian meta-learning and show the novel sensitivities among tasks for the first time.