A Single-Loop Robust Policy Gradient Method for Robust Markov Decision Processes

TL;DR

This paper introduces a novel single-loop robust policy gradient method for solving robust Markov Decision Processes, providing global optimality guarantees and demonstrating faster, more robust convergence in experiments.

Contribution

It presents the first single-loop policy gradient algorithm with global optimality guarantees for RMDPs, addressing a gap in existing methods.

Findings

Faster convergence compared to nested-loop methods

Robust performance in dynamic environments

Theoretical convergence guarantees

Abstract

Robust Markov Decision Processes (RMDPs) have recently been recognized as a valuable and promising approach to discovering a policy with creditable performance, particularly in the presence of a dynamic environment and estimation errors in the transition matrix due to limited data. Despite extensive exploration of dynamic programming algorithms for solving RMDPs, there has been a notable upswing in interest in developing efficient algorithms using the policy gradient method. In this paper, we propose the first single-loop robust policy gradient (SRPG) method with the global optimality guarantee for solving RMDPs through its minimax formulation. Moreover, we complement the convergence analysis of the nonconvex-nonconcave min-max optimization problem with the objective function's gradient dominance property, which is not explored in the prior literature. Numerical experiments validate the efficacy of SRPG, demonstrating its faster and more robust convergence behavior compared to its nested-loop counterpart.