Uncertainty Quantification and Data Efficiency in AI: An Information-Theoretic Perspective

TL;DR

This paper reviews information-theoretic methods for quantifying epistemic uncertainty and improving data efficiency in AI systems, especially in data-scarce applications like healthcare and robotics.

Contribution

It provides a comprehensive survey of formal uncertainty quantification techniques and data augmentation strategies from an information-theoretic perspective.

Findings

Generalized Bayesian frameworks characterize epistemic uncertainty.

Information-theoretic bounds relate data quantity to predictive uncertainty.

Finite-sample guarantees like conformal prediction enhance reliability.

Abstract

In context-specific applications such as robotics, telecommunications, and healthcare, artificial intelligence systems often face the challenge of limited training data. This scarcity introduces epistemic uncertainty, i.e., reducible uncertainty stemming from incomplete knowledge of the underlying data distribution, which fundamentally limits predictive performance. This review paper examines formal methodologies that address data-limited regimes through two complementary approaches: quantifying epistemic uncertainty and mitigating data scarcity via synthetic data augmentation. We begin by reviewing generalized Bayesian learning frameworks that characterize epistemic uncertainty through generalized posteriors in the model parameter space, as well as ``post-Bayes'' learning frameworks. We continue by presenting information-theoretic generalization bounds that formalize the relationship between training data quantity and predictive uncertainty, providing a theoretical justification for generalized Bayesian learning. Moving beyond methods with asymptotic statistical validity, we survey uncertainty quantification methods that provide finite-sample statistical guarantees, including conformal prediction and conformal risk control. Finally, we examine recent advances in data efficiency by combining limited labeled data with abundant model predictions or synthetic data. Throughout, we take an information-theoretic perspective, highlighting the role of information measures in quantifying the impact of data scarcity.