Active Exploration via Autoregressive Generation of Missing Data

TL;DR

This paper introduces a novel approach to uncertainty quantification and exploration in online decision-making by using autoregressive generative models to predict missing outcomes, enabling more effective exploration and decision strategies.

Contribution

It proposes viewing uncertainty as missing future outcomes, leveraging autoregressive models for prediction and exploration, and demonstrates theoretical and empirical benefits in meta-bandit problems.

Findings

Successful reduction from offline prediction to online decision-making

Effective exploration in a news recommendation task using text features

Theoretical guarantees for uncertainty quantification

Abstract

We pose uncertainty quantification and exploration in online decision-making as a problem of training and generation from an autoregressive sequence model, an area experiencing rapid innovation. Our approach rests on viewing uncertainty as arising from missing future outcomes that would be revealed through appropriate action choices, rather than from unobservable latent parameters of the environment. This reformulation aligns naturally with modern machine learning capabilities: we can i) train generative models through next-outcome prediction rather than fit explicit priors, ii) assess uncertainty through autoregressive generation rather than parameter sampling, and iii) adapt to new information through in-context learning rather than explicit posterior updating. To showcase these ideas, we formulate a challenging meta-bandit problem where effective performance requires leveraging unstructured prior information (like text features) while exploring judiciously to resolve key remaining uncertainties. We validate our approach through both theory and experiments. Our theory establishes a reduction, showing success at offline next-outcome prediction translates to reliable online uncertainty quantification and decision-making, even with strategically collected data. Semi-synthetic experiments show our insights bear out in a news-article recommendation task, where article text can be leveraged to minimize exploration.