Dual Formulation for Non-Rectangular Lp Robust Markov Decision Processes

TL;DR

This paper introduces a dual formulation for non-rectangular Lp robust Markov decision processes, enabling efficient policy evaluation and revealing adversarial strategies, thus advancing robust decision-making under complex uncertainties.

Contribution

It develops a novel dual formulation for non-rectangular Lp RMDPs, facilitating the first robust policy evaluation algorithms for these complex models.

Findings

Our approach outperforms brute-force methods in empirical tests.

The structural decomposition simplifies analysis of non-rectangular uncertainty sets.

The dual formulation provides new insights into adversarial strategies in RMDPs.

Abstract

We study robust Markov decision processes (RMDPs) with non-rectangular uncertainty sets, which capture interdependencies across states unlike traditional rectangular models. While non-rectangular robust policy evaluation is generally NP-hard, even in approximation, we identify a powerful class of $L_p$-bounded uncertainty sets that avoid these complexity barriers due to their structural simplicity. We further show that this class can be decomposed into infinitely many \texttt{sa}-rectangular $L_p$-bounded sets and leverage its structural properties to derive a novel dual formulation for $L_p$ RMDPs. This formulation provides key insights into the adversary's strategy and enables the development of the first robust policy evaluation algorithms for non-rectangular RMDPs. Empirical results demonstrate that our approach significantly outperforms brute-force methods, establishing a promising foundation for future investigation into non-rectangular robust MDPs.