Simplification of Risk Averse POMDPs with Performance Guarantees

TL;DR

This paper introduces a framework to simplify risk-averse POMDPs by using cheaper belief-MDP models, providing bounds and performance guarantees for the CVaR-based value function, thus enabling faster decision-making in uncertain domains.

Contribution

The work develops a general bounding method for CVaR in POMDPs and applies it to simplify belief-MDP models with theoretical performance guarantees.

Findings

Bounds for CVaR using distribution bounds

Efficient approximation of CVaR in POMDPs

Performance guarantees for simplified models

Abstract

Risk averse decision making under uncertainty in partially observable domains is a fundamental problem in AI and essential for reliable autonomous agents. In our case, the problem is modeled using partially observable Markov decision processes (POMDPs), when the value function is the conditional value at risk (CVaR) of the return. Calculating an optimal solution for POMDPs is computationally intractable in general. In this work we develop a simplification framework to speedup the evaluation of the value function, while providing performance guarantees. We consider as simplification a computationally cheaper belief-MDP transition model, that can correspond, e.g., to cheaper observation or transition models. Our contributions include general bounds for CVaR that allow bounding the CVaR of a random variable X, using a random variable Y, by assuming bounds between their cumulative distributions. We then derive bounds for the CVaR value function in a POMDP setting, and show how to bound the value function using the computationally cheaper belief-MDP transition model and without accessing the computationally expensive model in real-time. Then, we provide theoretical performance guarantees for the estimated bounds. Our results apply for a general simplification of a belief-MDP transition model and support simplification of both the observation and state transition models simultaneously.