Compatible Gradient Approximations for Actor-Critic Algorithms

TL;DR

This paper presents a novel actor-critic algorithm that uses zeroth-order gradient estimation to improve compatibility and performance in deterministic policy gradient methods for continuous control tasks.

Contribution

It introduces a zeroth-order approximation technique for action-value gradients, addressing compatibility issues in deterministic policy gradients.

Findings

Algorithm matches or exceeds state-of-the-art performance

Effectively addresses gradient approximation inaccuracies

Demonstrates robustness in continuous control environments

Abstract

Deterministic policy gradient algorithms are foundational for actor-critic methods in controlling continuous systems, yet they often encounter inaccuracies due to their dependence on the derivative of the critic's value estimates with respect to input actions. This reliance requires precise action-value gradient computations, a task that proves challenging under function approximation. We introduce an actor-critic algorithm that bypasses the need for such precision by employing a zeroth-order approximation of the action-value gradient through two-point stochastic gradient estimation within the action space. This approach provably and effectively addresses compatibility issues inherent in deterministic policy gradient schemes. Empirical results further demonstrate that our algorithm not only matches but frequently exceeds the performance of current state-of-the-art methods by a substantial extent.