Beyond Confidence Regions: Tight Bayesian Ambiguity Sets for Robust MDPs

TL;DR

This paper introduces a Bayesian approach to constructing ambiguity sets in robust MDPs, replacing traditional confidence regions to achieve less conservative and more effective policies with provable robustness.

Contribution

It proposes a novel Bayesian inference-based method for optimizing ambiguity sets in RMDPs, improving over confidence region approaches.

Findings

Theoretical proof of safety for the proposed method.

Empirical results show improved robustness and policy performance.

Method effectively incorporates prior knowledge into ambiguity set construction.

Abstract

Robust MDPs (RMDPs) can be used to compute policies with provable worst-case guarantees in reinforcement learning. The quality and robustness of an RMDP solution are determined by the ambiguity set---the set of plausible transition probabilities---which is usually constructed as a multi-dimensional confidence region. Existing methods construct ambiguity sets as confidence regions using concentration inequalities which leads to overly conservative solutions. This paper proposes a new paradigm that can achieve better solutions with the same robustness guarantees without using confidence regions as ambiguity sets. To incorporate prior knowledge, our algorithms optimize the size and position of ambiguity sets using Bayesian inference. Our theoretical analysis shows the safety of the proposed method, and the empirical results demonstrate its practical promise.