Optimizing Expectation with Guarantees in POMDPs (Technical Report)

TL;DR

This paper introduces a new approach for POMDPs that guarantees a minimum payoff for all outcomes while optimizing the expected payoff, addressing safety and performance in uncertain decision-making.

Contribution

It proposes the first practical method for guaranteed payoff optimization in POMDPs, balancing safety constraints with expected reward maximization.

Findings

Method effectively ensures minimum payoff guarantees.

Approach outperforms existing threshold-based methods.

Evaluations on standard benchmarks demonstrate practical viability.

Abstract

A standard objective in partially-observable Markov decision processes (POMDPs) is to find a policy that maximizes the expected discounted-sum payoff. However, such policies may still permit unlikely but highly undesirable outcomes, which is problematic especially in safety-critical applications. Recently, there has been a surge of interest in POMDPs where the goal is to maximize the probability to ensure that the payoff is at least a given threshold, but these approaches do not consider any optimization beyond satisfying this threshold constraint. In this work we go beyond both the "expectation" and "threshold" approaches and consider a "guaranteed payoff optimization (GPO)" problem for POMDPs, where we are given a threshold $t$ and the objective is to find a policy $\sigma$ such that a) each possible outcome of $\sigma$ yields a discounted-sum payoff of at least $t$, and b) the expected discounted-sum payoff of $\sigma$ is optimal (or near-optimal) among all policies satisfying a). We present a practical approach to tackle the GPO problem and evaluate it on standard POMDP benchmarks.