Riemannian Proximal Policy Optimization

TL;DR

This paper introduces a Riemannian proximal policy optimization algorithm for Markov decision processes, leveraging Gaussian mixture models and Wasserstein distance bounds, with proven convergence and promising experimental results.

Contribution

It presents a novel Riemannian optimization framework for policy learning in MDPs using GMMs, with convergence guarantees and policy improvement bounds.

Findings

Algorithm demonstrates guaranteed convergence.

Experimental results show improved policy optimization.

Bounds on policy improvement are derived using Wasserstein distance.

Abstract

In this paper, We propose a general Riemannian proximal optimization algorithm with guaranteed convergence to solve Markov decision process (MDP) problems. To model policy functions in MDP, we employ Gaussian mixture model (GMM) and formulate it as a nonconvex optimization problem in the Riemannian space of positive semidefinite matrices. For two given policy functions, we also provide its lower bound on policy improvement by using bounds derived from the Wasserstein distance of GMMs. Preliminary experiments show the efficacy of our proposed Riemannian proximal policy optimization algorithm.