Robust Markov Decision Processes without Model Estimation

TL;DR

This paper introduces a model-free, sample-efficient approach to robust Markov Decision Processes that eliminates the need for large memory and an oracle, enabling practical application and theoretical guarantees.

Contribution

It transforms robust MDPs into an alternative form suitable for stochastic gradient methods, removing the need for an oracle and reducing memory requirements.

Findings

Proposes a new formulation for robust MDPs compatible with stochastic gradient methods.

Develops a sample-efficient, model-free algorithm for robust MDPs.

Numerical experiments confirm the effectiveness of the proposed approach.

Abstract

Robust Markov Decision Processes (MDPs) are receiving much attention in learning a robust policy which is less sensitive to environment changes. There are an increasing number of works analyzing sample-efficiency of robust MDPs. However, there are two major barriers to applying robust MDPs in practice. First, most works study robust MDPs in a model-based regime, where the transition probability needs to be estimated and requires a large amount of memories $\mathcal{O}(|\mathcal{S}|^2|\mathcal{A}|)$. Second, prior work typically assumes a strong oracle to obtain the optimal solution as an intermediate step to solve robust MDPs. However, in practice, such an oracle does not exist usually. To remove the oracle, we transform the original robust MDPs into an alternative form, which allows us to use stochastic gradient methods to solve the robust MDPs. Moreover, we prove the alternative form still plays a similar role as the original form. With this new formulation, we devise a sample-efficient algorithm to solve the robust MDPs in a model-free regime, which does not require an oracle and trades off a lower storage requirement $\mathcal{O}(|\mathcal{S}||\mathcal{A}|)$ with being able to generate samples from a generative model or Markovian chain. Finally, we validate our theoretical findings via numerical experiments, showing the efficiency with the alternative form of robust MDPs.