S$^2$AC: Energy-Based Reinforcement Learning with Stein Soft Actor Critic

TL;DR

This paper introduces S$^2$AC, a novel energy-based reinforcement learning algorithm that efficiently learns expressive stochastic policies using Stein variational methods, improving performance over existing MaxEnt RL algorithms.

Contribution

S$^2$AC derives a closed-form entropy expression for Stein variational policies, enabling efficient and expressive MaxEnt RL without high computational costs.

Findings

S$^2$AC outperforms SAC and SQL in multi-goal environments.

S$^2$AC achieves better results on MuJoCo benchmarks.

The entropy formula simplifies policy evaluation in energy-based RL.

Abstract

Learning expressive stochastic policies instead of deterministic ones has been proposed to achieve better stability, sample complexity, and robustness. Notably, in Maximum Entropy Reinforcement Learning (MaxEnt RL), the policy is modeled as an expressive Energy-Based Model (EBM) over the Q-values. However, this formulation requires the estimation of the entropy of such EBMs, which is an open problem. To address this, previous MaxEnt RL methods either implicitly estimate the entropy, resulting in high computational complexity and variance (SQL), or follow a variational inference procedure that fits simplified actor distributions (e.g., Gaussian) for tractability (SAC). We propose Stein Soft Actor-Critic (S$^2$AC), a MaxEnt RL algorithm that learns expressive policies without compromising efficiency. Specifically, S$^2$AC uses parameterized Stein Variational Gradient Descent (SVGD) as the underlying policy. We derive a closed-form expression of the entropy of such policies. Our formula is computationally efficient and only depends on first-order derivatives and vector products. Empirical results show that S$^2$AC yields more optimal solutions to the MaxEnt objective than SQL and SAC in the multi-goal environment, and outperforms SAC and SQL on the MuJoCo benchmark. Our code is available at: https://github.com/SafaMessaoud/S2AC-Energy-Based-RL-with-Stein-Soft-Actor-Critic