Variational Sequential Optimal Experimental Design using Reinforcement Learning

TL;DR

This paper introduces vsOED, a reinforcement learning-based Bayesian method for sequential experimental design that maximizes information gain efficiently across diverse applications.

Contribution

The paper develops a novel variational sequential optimal experimental design framework using reinforcement learning with flexible criteria and posterior approximations.

Findings

vsOED achieves superior sample efficiency over existing methods.

It accommodates nuisance parameters, implicit likelihoods, and multiple models.

Demonstrated effectiveness across engineering and science applications.

Abstract

We present variational sequential optimal experimental design (vsOED), a novel method for optimally designing a finite sequence of experiments within a Bayesian framework with information-theoretic criteria. vsOED employs a one-point reward formulation with variational posterior approximations, providing a provable lower bound to the expected information gain. Numerical methods are developed following an actor-critic reinforcement learning approach, including derivation and estimation of variational and policy gradients to optimize the design policy, and posterior approximation using Gaussian mixture models and normalizing flows. vsOED accommodates nuisance parameters, implicit likelihoods, and multiple candidate models, while supporting flexible design criteria that can target designs for model discrimination, parameter inference, goal-oriented prediction, and their weighted combinations. We demonstrate vsOED across various engineering and science applications, illustrating its superior sample efficiency compared to existing sequential experimental design algorithms.