An Efficient Solution to s-Rectangular Robust Markov Decision Processes

TL;DR

This paper introduces a fast and efficient robust value iteration method for s-rectangular robust MDPs, deriving the optimal Bellman operator and unveiling novel threshold policies with proportional action probabilities.

Contribution

It provides the first efficient algorithm for s-rectangular robust MDPs with a derivation of the optimal Bellman operator and explicit form of optimal policies.

Findings

Significantly faster computation than existing methods

Explicit form of optimal robust policies

Efficient robust value iteration algorithm

Abstract

We present an efficient robust value iteration for \texttt{s}-rectangular robust Markov Decision Processes (MDPs) with a time complexity comparable to standard (non-robust) MDPs which is significantly faster than any existing method. We do so by deriving the optimal robust Bellman operator in concrete forms using our $L_p$ water filling lemma. We unveil the exact form of the optimal policies, which turn out to be novel threshold policies with the probability of playing an action proportional to its advantage.