Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning

TL;DR

This paper systematically evaluates over 60 off-policy RL agents with various regularizations across multiple tasks, revealing that well-regularized simple agents like Soft Actor-Critic can outperform more complex methods in finding better policies.

Contribution

It provides a comprehensive empirical analysis of regularization techniques in off-policy RL, highlighting their effects on overestimation, overfitting, and plasticity across diverse tasks.

Findings

Certain regularization combinations are consistently effective across tasks.

A well-regularized Soft Actor-Critic outperforms complex algorithms in policy quality.

Regularization improves sample efficiency and policy robustness.

Abstract

Recent advancements in off-policy Reinforcement Learning (RL) have significantly improved sample efficiency, primarily due to the incorporation of various forms of regularization that enable more gradient update steps than traditional agents. However, many of these techniques have been tested in limited settings, often on tasks from single simulation benchmarks and against well-known algorithms rather than a range of regularization approaches. This limits our understanding of the specific mechanisms driving RL improvements. To address this, we implemented over 60 different off-policy agents, each integrating established regularization techniques from recent state-of-the-art algorithms. We tested these agents across 14 diverse tasks from 2 simulation benchmarks, measuring training metrics related to overestimation, overfitting, and plasticity loss -- issues that motivate the examined regularization techniques. Our findings reveal that while the effectiveness of a specific regularization setup varies with the task, certain combinations consistently demonstrate robust and superior performance. Notably, a simple Soft Actor-Critic agent, appropriately regularized, reliably finds a better-performing policy within the training regime, which previously was achieved mainly through model-based approaches.