Selecting Belief-State Approximations in Simulators with Latent States

TL;DR

This paper addresses the challenge of selecting belief-state approximations in simulators with latent states, proposing new algorithms and analyses for effective belief-state sampling under limited access, with implications for planning and calibration.

Contribution

It introduces a reduction of the belief-state selection problem to a conditional distribution-selection task and develops algorithms with theoretical guarantees for sampling-only access scenarios.

Findings

Observation-based selection may fail under Single-Reset rollouts.

Latent state-based selection provides guarantees under Repeated-Reset.

The paper discusses the impact of distribution shift and sampling policies.

Abstract

State resetting is a fundamental but often overlooked capability of simulators. It supports sample-based planning by allowing resets to previously encountered simulation states, and enables calibration of simulators using real data by resetting to states observed in real-system traces. While often taken for granted, state resetting in complex simulators can be nontrivial: when the simulator comes with latent variables (states), state resetting requires sampling from the posterior over the latent state given the observable history, a.k.a. the belief state (Silver and Veness, 2010). While exact sampling is often infeasible, many approximate belief-state samplers can be constructed, raising the question of how to select among them using only sampling access to the simulator. In this paper, we show that this problem reduces to a general conditional distribution-selection task and develop a new algorithm and analysis under sampling-only access. Building on this reduction, the belief-state selection problem admits two different formulations: latent state-based selection, which directly targets the conditional distribution of the latent state, and observation-based selection, which targets the induced distribution over the observation. Interestingly, these formulations differ in how their guarantees interact with the downstream roll-out methods: perhaps surprisingly, observation-based selection may fail under the most natural roll-out method (which we call Single-Reset) but enjoys guarantees under the less conventional alternative (which we call Repeated-Reset). Together with discussion on issues such as distribution shift and the choice of sampling policies, our paper reveals a rich landscape of algorithmic choices, theoretical nuances, and open questions, in this seemingly simple problem.