Metareasoning in uncertain environments: a meta-BAMDP framework

TL;DR

This paper introduces a meta-BAMDP framework for metareasoning in uncertain environments, extending traditional models to handle unknown reward and transition distributions, and applies it to Bernoulli bandit tasks with novel theoretical insights.

Contribution

It generalizes metareasoning models by proposing the meta-BAMDP framework for unknown environments and introduces two theorems that improve problem tractability and approximation quality.

Findings

The framework applies to Bernoulli bandit tasks.

Theorems enhance problem tractability and robustness.

Provides testable predictions for human exploration behavior.

Abstract

\textit{Reasoning} may be viewed as an algorithm $P$ that makes a choice of an action $a^* \in \mathcal{A}$, aiming to optimize some outcome. However, executing $P$ itself bears costs (time, energy, limited capacity, etc.) and needs to be considered alongside explicit utility obtained by making the choice in the underlying decision problem. Finding the right $P$ can itself be framed as an optimization problem over the space of reasoning processes $P$, generally referred to as \textit{metareasoning}. Conventionally, human metareasoning models assume that the agent knows the transition and reward distributions of the underlying MDP. This paper generalizes such models by proposing a meta Bayes-Adaptive MDP (meta-BAMDP) framework to handle metareasoning in environments with unknown reward/transition distributions, which encompasses a far larger and more realistic set of planning problems that humans and AI systems face. As a first step, we apply the framework to Bernoulli bandit tasks. Owing to the meta problem's complexity, our solutions are necessarily approximate. However, we introduce two novel theorems that significantly enhance the tractability of the problem, enabling stronger approximations that are robust within a range of assumptions grounded in realistic human decision-making scenarios. These results offer a resource-rational perspective and a normative framework for understanding human exploration under cognitive constraints, as well as providing experimentally testable predictions about human behavior in Bernoulli Bandit tasks.