A Theoretical Justification for Asymmetric Actor-Critic Algorithms

TL;DR

This paper provides a theoretical justification for asymmetric actor-critic algorithms in reinforcement learning, demonstrating how the asymmetric critic reduces errors and improves learning in partially observable environments with linear function approximators.

Contribution

It offers the first finite-time convergence analysis for asymmetric actor-critic algorithms, explaining their benefits in partially observable settings.

Findings

Asymmetric critic reduces aliasing errors in the agent state.

Finite-time bounds demonstrate improved convergence properties.

Theoretical insights support the use of asymmetric algorithms in practice.

Abstract

In reinforcement learning for partially observable environments, many successful algorithms have been developed within the asymmetric learning paradigm. This paradigm leverages additional state information available at training time for faster learning. Although the proposed learning objectives are usually theoretically sound, these methods still lack a precise theoretical justification for their potential benefits. We propose such a justification for asymmetric actor-critic algorithms with linear function approximators by adapting a finite-time convergence analysis to this setting. The resulting finite-time bound reveals that the asymmetric critic eliminates error terms arising from aliasing in the agent state.