NFPO: Stabilized Policy Optimization of Normalizing Flow for Robotic Policy Learning

TL;DR

This paper introduces NFPO, a stabilized normalizing flow-based policy optimization method for robotic reinforcement learning, addressing training instability and enabling multi-modal policy modeling with robust real-world transfer.

Contribution

It proposes a novel stabilized training approach for normalizing flow policies in RL, improving stability and performance in robotic tasks.

Findings

NFPO achieves robust performance across multiple simulation environments.

NFPO successfully transfers to real-world robotic tasks.

The method outperforms traditional Gaussian policies in multi-modal modeling.

Abstract

Deep Reinforcement Learning (DRL) has experienced significant advancements in recent years and has been widely used in many fields. In DRL-based robotic policy learning, however, current de facto policy parameterization is still multivariate Gaussian (with diagonal covariance matrix), which lacks the ability to model multi-modal distribution. In this work, we explore the adoption of a modern network architecture, i.e. Normalizing Flow (NF) as the policy parameterization for its ability of multi-modal modeling, closed form of log probability and low computation and memory overhead. However, naively training NF in online Reinforcement Learning (RL) usually leads to training instability. We provide a detailed analysis for this phenomenon and successfully address it via simple but effective technique. With extensive experiments in multiple simulation environments, we show our method, NFPO could obtain robust and strong performance in widely used robotic learning tasks and successfully transfer into real-world robots.