Bidirectional Soft Actor-Critic: Leveraging Forward and Reverse KL Divergence for Efficient Reinforcement Learning

TL;DR

This paper introduces Bidirectional SAC, a reinforcement learning algorithm that combines forward and reverse KL divergences to improve policy optimization, resulting in better performance and sample efficiency in continuous control tasks.

Contribution

It proposes a novel bidirectional approach that explicitly leverages forward KL for initialization and reverse KL for refinement, enhancing stability and efficiency.

Findings

Achieves up to 30% higher episodic rewards

Outperforms standard SAC and baselines

Improves sample efficiency significantly

Abstract

The Soft Actor-Critic (SAC) algorithm, a state-of-the-art method in maximum entropy reinforcement learning, traditionally relies on minimizing reverse Kullback-Leibler (KL) divergence for policy updates. However, this approach leads to an intractable optimal projection policy, necessitating gradient-based approximations that can suffer from instability and poor sample efficiency. This paper investigates the alternative use of forward KL divergence within SAC. We demonstrate that for Gaussian policies, forward KL divergence yields an explicit optimal projection policy -- corresponding to the mean and variance of the target Boltzmann distribution's action marginals. Building on the distinct advantages of both KL directions, we propose Bidirectional SAC, an algorithm that first initializes the policy using the explicit forward KL projection and then refines it by optimizing the reverse KL divergence. Comprehensive experiments on continuous control benchmarks show that Bidirectional SAC significantly outperforms standard SAC and other baselines, achieving up to a $30\%$ increase in episodic rewards, alongside enhanced sample efficiency.