Soft Actor-Critic Algorithms and Applications

TL;DR

The paper introduces Soft Actor-Critic (SAC), an off-policy deep reinforcement learning algorithm based on maximum entropy principles, which improves training stability, sample efficiency, and robustness, making it suitable for real-world robotics applications.

Contribution

The paper extends SAC with modifications for faster training and hyperparameter stability, including automatic temperature tuning, demonstrating state-of-the-art performance on benchmarks and real-world tasks.

Findings

SAC outperforms prior methods in sample efficiency and asymptotic performance.

SAC demonstrates high stability across different random seeds.

SAC is effective in real-world robotics tasks like locomotion and manipulation.

Abstract

Model-free deep reinforcement learning (RL) algorithms have been successfully applied to a range of challenging sequential decision making and control tasks. However, these methods typically suffer from two major challenges: high sample complexity and brittleness to hyperparameters. Both of these challenges limit the applicability of such methods to real-world domains. In this paper, we describe Soft Actor-Critic (SAC), our recently introduced off-policy actor-critic algorithm based on the maximum entropy RL framework. In this framework, the actor aims to simultaneously maximize expected return and entropy. That is, to succeed at the task while acting as randomly as possible. We extend SAC to incorporate a number of modifications that accelerate training and improve stability with respect to the hyperparameters, including a constrained formulation that automatically tunes the temperature hyperparameter. We systematically evaluate SAC on a range of benchmark tasks, as well as real-world challenging tasks such as locomotion for a quadrupedal robot and robotic manipulation with a dexterous hand. With these improvements, SAC achieves state-of-the-art performance, outperforming prior on-policy and off-policy methods in sample-efficiency and asymptotic performance. Furthermore, we demonstrate that, in contrast to other off-policy algorithms, our approach is very stable, achieving similar performance across different random seeds. These results suggest that SAC is a promising candidate for learning in real-world robotics tasks.