Rethinking Soft Actor-Critic in High-Dimensional Action Spaces: The Cost of Ignoring Distribution Shift

TL;DR

This paper analyzes the distribution shift caused by the tanh transformation in Soft Actor-Critic algorithms, revealing its impact on performance in high-dimensional tasks and proposing improvements to address this issue.

Contribution

It provides a theoretical derivation of the action distribution's PDF after tanh transformation and demonstrates how accounting for this shift improves SAC's effectiveness.

Findings

Distribution shift distorts the Gaussian action distribution.

Correcting for the shift improves cumulative rewards.

Enhanced performance in high-dimensional control tasks.

Abstract

Soft Actor-Critic algorithm is widely recognized for its robust performance across a range of deep reinforcement learning tasks, where it leverages the tanh transformation to constrain actions within bounded limits. However, this transformation induces a distribution shift, distorting the original Gaussian action distribution and potentially leading the policy to select suboptimal actions, particularly in high-dimensional action spaces. In this paper, we conduct a comprehensive theoretical and empirical analysis of this distribution shift, deriving the precise probability density function (PDF) for actions following the tanh transformation to clarify the misalignment introduced between the transformed distribution's mode and the intended action output. We substantiate these theoretical insights through extensive experiments on high-dimensional tasks within the HumanoidBench benchmark. Our findings indicate that accounting for this distribution shift substantially enhances SAC's performance, resulting in notable improvements in cumulative rewards, sample efficiency, and reliability across tasks. These results underscore a critical consideration for SAC and similar algorithms: addressing transformation-induced distribution shifts is essential to optimizing policy effectiveness in high-dimensional deep reinforcement learning environments, thereby expanding the robustness and applicability of SAC in complex control tasks.