Epistemic Errors of Imperfect Multitask Learners When Distributions Shift

TL;DR

This paper introduces a framework for understanding and bounding epistemic errors in uncertainty-aware multitask learners under distribution shift, providing insights into error attribution and bounds in various settings.

Contribution

It offers a new definition of epistemic error and a decompositional bound applicable to imperfect multitask learning with distribution shifts.

Findings

Provides a general epistemic error bound for multitask learners.

Specializes bounds for Bayesian transfer learning.

Analyzes errors under distribution shifts within epsilon neighborhoods.

Abstract

Uncertainty-aware machine learners, such as Bayesian neural networks, output a quantification of uncertainty instead of a point prediction. We provide uncertainty-aware learners with a principled framework to characterize, and identify ways to eliminate, errors that arise from reducible (epistemic) uncertainty. We introduce a principled definition of epistemic error, and provide a decompositional epistemic error bound which operates in the very general setting of imperfect multitask learning under distribution shift. In this setting, the training (source) data may arise from multiple tasks, the test (target) data may differ systematically from the source data tasks, and/or the learner may not arrive at an accurate characterization of the source data. Our bound separately attributes epistemic errors to each of multiple aspects of the learning procedure and environment. As corollaries of the general result, we provide epistemic error bounds specialized to the settings of Bayesian transfer learning and distribution shift within $\epsilon$-neighborhoods.