Efficient Wasserstein Natural Gradients for Reinforcement Learning

TL;DR

This paper introduces a Wasserstein natural gradient method for reinforcement learning that improves optimization efficiency and performance by leveraging Wasserstein geometry and divergence penalties.

Contribution

It presents a novel Wasserstein natural gradient approach for RL, enhancing optimization speed and effectiveness over existing methods.

Findings

Reduced computational cost in RL training

Improved policy performance on challenging tasks

Effective use of Wasserstein geometry in optimization

Abstract

A novel optimization approach is proposed for application to policy gradient methods and evolution strategies for reinforcement learning (RL). The procedure uses a computationally efficient Wasserstein natural gradient (WNG) descent that takes advantage of the geometry induced by a Wasserstein penalty to speed optimization. This method follows the recent theme in RL of including a divergence penalty in the objective to establish a trust region. Experiments on challenging tasks demonstrate improvements in both computational cost and performance over advanced baselines.