Efficient Solution and Learning of Robust Factored MDPs

TL;DR

This paper introduces new methods for efficiently solving and learning robust factored MDPs by exploiting state-space structure, leading to improved sample efficiency and performance guarantees.

Contribution

It presents novel reformulations of factored robust MDPs into linear programs and methods for learning factored models directly, enhancing computational tractability.

Findings

Factored structure improves sample efficiency in robust policy learning.

Reformulating into linear programs makes solving robust factored MDPs tractable.

Experimental results outperform state-of-the-art methods in robustness and efficiency.

Abstract

Robust Markov decision processes (r-MDPs) extend MDPs by explicitly modelling epistemic uncertainty about transition dynamics. Learning r-MDPs from interactions with an unknown environment enables the synthesis of robust policies with provable (PAC) guarantees on performance, but this can require a large number of sample interactions. We propose novel methods for solving and learning r-MDPs based on factored state-space representations that leverage the independence between model uncertainty across system components. Although policy synthesis for factored r-MDPs leads to hard, non-convex optimisation problems, we show how to reformulate these into tractable linear programs. Building on these, we also propose methods to learn factored model representations directly. Our experimental results show that exploiting factored structure can yield dimensional gains in sample efficiency, producing more effective robust policies with tighter performance guarantees than state-of-the-art methods.