Improving Actor-Critic Training with Steerable Action-Value Approximation Errors

TL;DR

This paper introduces USAC, a flexible framework for actor-critic reinforcement learning that dynamically balances optimism and pessimism in value estimates, improving performance in continuous control tasks.

Contribution

USAC allows independent control of optimism and pessimism in actor-critic algorithms, enabling adaptive exploration strategies based on critic uncertainty.

Findings

USAC outperforms state-of-the-art algorithms in various control tasks.

Adjusting optimism and pessimism impacts learning stability and performance.

Dynamic adaptation of exploration improves policy refinement.

Abstract

Off-policy actor-critic algorithms have shown strong potential in deep reinforcement learning for continuous control tasks. Their success primarily comes from leveraging pessimistic state-action value function updates, which reduce function approximation errors and stabilize learning. However, excessive pessimism can limit exploration, preventing the agent from effectively refining its policies. Conversely, optimism can encourage exploration but may lead to high-risk behaviors and unstable learning if not carefully managed. To address this trade-off, we propose Utility Soft Actor-Critic (USAC), a novel framework that allows independent, interpretable control of pessimism and optimism for both the actor and the critic. USAC dynamically adapts its exploration strategy based on the uncertainty of critics using a utility function, enabling a task-specific balance between optimism and pessimism. This approach goes beyond binary choices of pessimism or optimism, making the method both theoretically meaningful and practically feasible. Experiments across a variety of continuous control tasks show that adjusting the degree of pessimism or optimism significantly impacts performance. When configured appropriately, USAC consistently outperforms state-of-the-art algorithms, demonstrating its practical utility and feasibility.