Representative, Informative, and De-Amplifying: Requirements for Robust Bayesian Active Learning under Model Misspecification

TL;DR

This paper analyzes the impact of model misspecification on Bayesian experimental design, introduces a new acquisition function to mitigate error amplification, and demonstrates improved performance over existing methods.

Contribution

It provides a mathematical analysis of generalization error under model misspecification and proposes R-IDeA, a novel acquisition function for robust Bayesian active learning.

Findings

The analysis reveals error (de-)amplification as a key factor in generalization error.

The R-IDeA acquisition function outperforms existing methods in experiments.

Including representativeness and de-amplification terms improves robustness.

Abstract

In many science and industry settings, a central challenge is designing experiments under time and budget constraints. Bayesian Optimal Experimental Design (BOED) is a paradigm to pick maximally informative designs that has been widely applied to such problems. During training, BOED selects inputs according to a pre-determined acquisition criterion to target informativeness. During testing, the model learned during training encounters a naturally occurring distribution of test samples. This leads to an instance of covariate shift, where the train and test samples are drawn from different distributions (the training samples are not representative of the test distribution). Prior work has shown that in the presence of model misspecification, covariate shift amplifies generalization error. Our first contribution is to provide a mathematical analysis of generalization error in the presence of model misspecification, revealing that, beyond covariate shift, generalization error is also driven by a previously unidentified phenomenon we term error (de-)amplification. We then develop a new acquisition function that mitigates the effects of model misspecification by including terms for representativeness, informativeness, and de-amplification (R-IDeA). Our experimental results demonstrate that the proposed method performs better than methods that target only informativeness, only representativeness, or both.