State Entropy Regularization for Robust Reinforcement Learning

TL;DR

This paper investigates how state entropy regularization enhances robustness in reinforcement learning, especially against structured perturbations, providing theoretical guarantees and practical insights into its advantages over policy entropy.

Contribution

It offers the first theoretical analysis of state entropy regularization's robustness benefits and compares it with policy entropy, highlighting its effectiveness against correlated perturbations.

Findings

State entropy improves robustness to structured perturbations.

Theoretical guarantees are provided for reward and transition uncertainties.

Robustness benefits are sensitive to the number of rollouts.

Abstract

State entropy regularization has empirically shown better exploration and sample complexity in reinforcement learning (RL). However, its theoretical guarantees have not been studied. In this paper, we show that state entropy regularization improves robustness to structured and spatially correlated perturbations. These types of variation are common in transfer learning but often overlooked by standard robust RL methods, which typically focus on small, uncorrelated changes. We provide a comprehensive characterization of these robustness properties, including formal guarantees under reward and transition uncertainty, as well as settings where the method performs poorly. Much of our analysis contrasts state entropy with the widely used policy entropy regularization, highlighting their different benefits. Finally, from a practical standpoint, we illustrate that compared with policy entropy, the robustness advantages of state entropy are more sensitive to the number of rollouts used for policy evaluation.