Boosting Trust Region Policy Optimization by Normalizing Flows Policy

TL;DR

This paper introduces a normalizing flows policy to enhance trust region policy optimization, enabling better exploration and avoiding local optima, especially in high-dimensional complex tasks.

Contribution

It presents a novel integration of normalizing flows into trust region policy search, improving exploration and performance in complex, high-dimensional environments.

Findings

Normalizing flows policy enables better exploration.

Significant performance improvements on high-dimensional tasks.

Enhanced avoidance of local optima.

Abstract

We propose to improve trust region policy search with normalizing flows policy. We illustrate that when the trust region is constructed by KL divergence constraints, normalizing flows policy generates samples far from the 'center' of the previous policy iterate, which potentially enables better exploration and helps avoid bad local optima. Through extensive comparisons, we show that the normalizing flows policy significantly improves upon baseline architectures especially on high-dimensional tasks with complex dynamics.