Online Bayesian Goal Inference for Boundedly-Rational Planning Agents

TL;DR

This paper introduces an online Bayesian goal inference method for agents with bounded rationality, capable of inferring goals from optimal and sub-optimal actions, including failures, using probabilistic models and a sequential Monte Carlo algorithm.

Contribution

It presents a novel architecture modeling boundedly-rational agents as probabilistic programs and introduces SIPS, an efficient inference algorithm for goal inference from complex action sequences.

Findings

Outperforms Bayesian inverse reinforcement learning baselines.

Accurately infers goals from sub-optimal trajectories involving failures.

Generalizes across domains with sparse rewards and compositional structure.

Abstract

People routinely infer the goals of others by observing their actions over time. Remarkably, we can do so even when those actions lead to failure, enabling us to assist others when we detect that they might not achieve their goals. How might we endow machines with similar capabilities? Here we present an architecture capable of inferring an agent's goals online from both optimal and non-optimal sequences of actions. Our architecture models agents as boundedly-rational planners that interleave search with execution by replanning, thereby accounting for sub-optimal behavior. These models are specified as probabilistic programs, allowing us to represent and perform efficient Bayesian inference over an agent's goals and internal planning processes. To perform such inference, we develop Sequential Inverse Plan Search (SIPS), a sequential Monte Carlo algorithm that exploits the online replanning assumption of these models, limiting computation by incrementally extending inferred plans as new actions are observed. We present experiments showing that this modeling and inference architecture outperforms Bayesian inverse reinforcement learning baselines, accurately inferring goals from both optimal and non-optimal trajectories involving failure and back-tracking, while generalizing across domains with compositional structure and sparse rewards.