Quasi-Newton Compatible Actor-Critic for Deterministic Policies

TL;DR

This paper introduces a second-order deterministic actor-critic method that leverages curvature information for faster convergence in reinforcement learning, applicable to any differentiable policy class.

Contribution

It develops a quadratic critic with compatible function approximation and a quasi-Newton actor update, enhancing convergence speed over first-order methods.

Findings

Achieves faster convergence than first-order methods.

Demonstrates improved performance in numerical experiments.

Applicable to any differentiable policy class.

Abstract

In this paper, we propose a second-order deterministic actor-critic framework in reinforcement learning that extends the classical deterministic policy gradient method to exploit curvature information of the performance function. Building on the concept of compatible function approximation for the critic, we introduce a quadratic critic that simultaneously preserves the true policy gradient and an approximation of the performance Hessian. A least-squares temporal difference learning scheme is then developed to estimate the quadratic critic parameters efficiently. This construction enables a quasi-Newton actor update using information learned by the critic, yielding faster convergence compared to first-order methods. The proposed approach is general and applicable to any differentiable policy class. Numerical examples demonstrate that the method achieves improved convergence and performance over standard deterministic actor-critic baselines.