Maximum Entropy Reinforcement Learning via Energy-Based Normalizing Flow

TL;DR

This paper introduces a novel MaxEnt RL framework using Energy-Based Normalizing Flows that unifies policy evaluation and improvement into a single process, enabling efficient training and multi-modal action modeling.

Contribution

The paper proposes a new MaxEnt RL approach with Energy-Based Normalizing Flows that simplifies training and supports complex action distributions, outperforming existing methods.

Findings

Achieves superior performance on MuJoCo benchmarks

Supports multi-modal action distributions

Enables direct calculation of soft value functions

Abstract

Existing Maximum-Entropy (MaxEnt) Reinforcement Learning (RL) methods for continuous action spaces are typically formulated based on actor-critic frameworks and optimized through alternating steps of policy evaluation and policy improvement. In the policy evaluation steps, the critic is updated to capture the soft Q-function. In the policy improvement steps, the actor is adjusted in accordance with the updated soft Q-function. In this paper, we introduce a new MaxEnt RL framework modeled using Energy-Based Normalizing Flows (EBFlow). This framework integrates the policy evaluation steps and the policy improvement steps, resulting in a single objective training process. Our method enables the calculation of the soft value function used in the policy evaluation target without Monte Carlo approximation. Moreover, this design supports the modeling of multi-modal action distributions while facilitating efficient action sampling. To evaluate the performance of our method, we conducted experiments on the MuJoCo benchmark suite and a number of high-dimensional robotic tasks simulated by Omniverse Isaac Gym. The evaluation results demonstrate that our method achieves superior performance compared to widely-adopted representative baselines.