DSAC-C: Constrained Maximum Entropy for Robust Discrete Soft-Actor Critic

TL;DR

This paper introduces DSAC-C, an extension of the Soft Actor-Critic algorithm that incorporates statistical constraints from a surrogate critic, enhancing robustness and performance in low-data and out-of-distribution scenarios.

Contribution

The paper proposes a novel constrained maximum entropy approach for discrete SAC, improving robustness and performance through additional statistical constraints derived from a surrogate critic.

Findings

Enhanced robustness against domain shifts

Improved performance in low-data regimes

Effective in out-of-distribution Atari experiments

Abstract

We present a novel extension to the family of Soft Actor-Critic (SAC) algorithms. We argue that based on the Maximum Entropy Principle, discrete SAC can be further improved via additional statistical constraints derived from a surrogate critic policy. Furthermore, our findings suggests that these constraints provide an added robustness against potential domain shifts, which are essential for safe deployment of reinforcement learning agents in the real-world. We provide theoretical analysis and show empirical results on low data regimes for both in-distribution and out-of-distribution variants of Atari 2600 games.