Robust Phi-Divergence MDPs

TL;DR

This paper introduces a new efficient solution framework for robust MDPs with phi-divergence ambiguity sets, enabling faster computation compared to existing methods and enhancing decision-making under uncertainty.

Contribution

It develops a decomposition-based solution method for robust MDPs with s-rectangular ambiguity sets leveraging phi-divergence structure, improving computational efficiency.

Findings

Faster solution times than commercial solvers.

Effective decomposition of robust Bellman updates.

Applicable to practical decision-making under uncertainty.

Abstract

In recent years, robust Markov decision processes (MDPs) have emerged as a prominent modeling framework for dynamic decision problems affected by uncertainty. In contrast to classical MDPs, which only account for stochasticity by modeling the dynamics through a stochastic process with a known transition kernel, robust MDPs additionally account for ambiguity by optimizing in view of the most adverse transition kernel from a prescribed ambiguity set. In this paper, we develop a novel solution framework for robust MDPs with s-rectangular ambiguity sets that decomposes the problem into a sequence of robust Bellman updates and simplex projections. Exploiting the rich structure present in the simplex projections corresponding to phi-divergence ambiguity sets, we show that the associated s-rectangular robust MDPs can be solved substantially faster than with state-of-the-art commercial solvers as well as a recent first-order solution scheme, thus rendering them attractive alternatives to classical MDPs in practical applications.