Belief-State Query Policies for User-Aligned POMDPs
Daniel Bramblett, Siddharth Srivastava

TL;DR
This paper introduces a new framework using belief-state query policies for planning in partially observable environments, ensuring user preferences are met while providing algorithms with guaranteed convergence.
Contribution
It presents the first formal analysis of user constraints in belief-state policies and develops algorithms for optimal, user-aligned planning in gPOMDPs.
Findings
Algorithms converge to optimal user-aligned behavior.
Parameterized BSQ policies are computationally feasible.
The expected cost function is piecewise constant with a finite search space.
Abstract
Planning in real-world settings often entails addressing partial observability while aligning with users' requirements. We present a novel framework for expressing users' constraints and preferences about agent behavior in a partially observable setting using parameterized belief-state query (BSQ) policies in the setting of goal-oriented partially observable Markov decision processes (gPOMDPs). We present the first formal analysis of such constraints and prove that while the expected cost function of a parameterized BSQ policy w.r.t its parameters is not convex, it is piecewise constant and yields an implicit discrete parameter search space that is finite for finite horizons. This theoretical result leads to novel algorithms that optimize gPOMDP agent behavior with guaranteed user alignment. Analysis proves that our algorithms converge to the optimal user-aligned behavior in the limit.…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
Taxonomy
TopicsLogic, Reasoning, and Knowledge · AI-based Problem Solving and Planning · Constraint Satisfaction and Optimization
