LLM-Guided Probabilistic Program Induction for POMDP Model Estimation

TL;DR

This paper introduces a novel approach that leverages large language models to guide the induction of probabilistic programs for estimating POMDP models, improving over traditional methods in complex decision-making under uncertainty.

Contribution

It presents a new method combining LLMs with probabilistic programming to learn POMDP models, addressing limitations of existing approaches.

Findings

LLM-guided models outperform tabular POMDP learning.

The approach is effective on toy, simulated, and real robotics domains.

Guided probabilistic program induction enhances model accuracy.

Abstract

Partially Observable Markov Decision Processes (POMDPs) model decision making under uncertainty. While there are many approaches to approximately solving POMDPs, we aim to address the problem of learning such models. In particular, we are interested in a subclass of POMDPs wherein the components of the model, including the observation function, reward function, transition function, and initial state distribution function, can be modeled as low-complexity probabilistic graphical models in the form of a short probabilistic program. Our strategy to learn these programs uses an LLM as a prior, generating candidate probabilistic programs that are then tested against the empirical distribution and adjusted through feedback. We experiment on a number of classical toy POMDP problems, simulated MiniGrid domains, and two real mobile-base robotics search domains involving partial observability. Our results show that using an LLM to guide in the construction of a low-complexity POMDP model can be more effective than tabular POMDP learning, behavior cloning, or direct LLM planning.