Provably Convergent Policy Optimization via Metric-aware Trust Region Methods

TL;DR

This paper introduces Wasserstein and Sinkhorn trust region methods for policy optimization in reinforcement learning, providing theoretical guarantees of convergence and demonstrating improved performance and robustness over existing methods.

Contribution

It proposes novel Wasserstein and Sinkhorn trust region methods that directly optimize policies and offers theoretical convergence guarantees and empirical validation.

Findings

WPO guarantees monotonic performance improvement.

SPO converges faster and is more sample-efficient.

Both methods outperform state-of-the-art policy gradient algorithms.

Abstract

Trust-region methods based on Kullback-Leibler divergence are pervasively used to stabilize policy optimization in reinforcement learning. In this paper, we exploit more flexible metrics and examine two natural extensions of policy optimization with Wasserstein and Sinkhorn trust regions, namely Wasserstein policy optimization (WPO) and Sinkhorn policy optimization (SPO). Instead of restricting the policy to a parametric distribution class, we directly optimize the policy distribution and derive their closed-form policy updates based on the Lagrangian duality. Theoretically, we show that WPO guarantees a monotonic performance improvement, and SPO provably converges to WPO as the entropic regularizer diminishes. Moreover, we prove that with a decaying Lagrangian multiplier to the trust region constraint, both methods converge to global optimality. Experiments across tabular domains, robotic locomotion, and continuous control tasks further demonstrate the performance improvement of both approaches, more robustness of WPO to sample insufficiency, and faster convergence of SPO, over state-of-art policy gradient methods.