Stochastic Actor-Critic: Mitigating Overestimation via Temporal Aleatoric Uncertainty

TL;DR

This paper introduces STAC, a reinforcement learning algorithm that mitigates overestimation by modeling temporal aleatoric uncertainty with a distributional critic and dropout, leading to more stable and risk-averse policies.

Contribution

STAC leverages temporal aleatoric uncertainty with a single distributional critic and dropout, offering a novel approach to reduce overestimation without relying on epistemic uncertainty.

Findings

Pessimism from distributional critic alone reduces overestimation.

Dropout improves training stability and performance.

STAC achieves better computational efficiency.

Abstract

Off-policy actor-critic methods in reinforcement learning train a critic with temporal-difference updates and use it as a learning signal for the policy (actor). This design typically achieves higher sample efficiency than purely on-policy methods. However, critic networks tend to overestimate value estimates systematically. This is often addressed by introducing a pessimistic bias based on uncertainty estimates. Current methods employ ensembling to quantify the critic's epistemic uncertainty-uncertainty due to limited data and model ambiguity-to scale pessimistic updates. In this work, we propose a new algorithm called Stochastic Actor-Critic (STAC) that incorporates temporal (one-step) aleatoric uncertainty-uncertainty arising from stochastic transitions, rewards, and policy-induced variability in Bellman targets-to scale pessimistic bias in temporal-difference updates, rather than relying on epistemic uncertainty. STAC uses a single distributional critic network to model the temporal return uncertainty, and applies dropout to both the critic and actor networks for regularization. Our results show that pessimism based on a distributional critic alone suffices to mitigate overestimation, and naturally leads to risk-averse behavior in stochastic environments. Introducing dropout further improves training stability and performance by means of regularization. With this design, STAC achieves improved computational efficiency using a single distributional critic network.