Stabilizing the Q-Gradient Field for Policy Smoothness in Actor-Critic

TL;DR

This paper introduces a critic-centric regularization method called PAVE that stabilizes the Q-gradient field to produce smoother, more robust policies in actor-critic algorithms without altering the actor directly.

Contribution

The work provides a theoretical foundation linking policy smoothness to the critic’s differential geometry and proposes PAVE, a novel regularization framework focusing on the critic to enhance policy stability.

Findings

PAVE reduces Q-gradient volatility effectively.

PAVE achieves smooth policies comparable to existing methods.

PAVE maintains competitive task performance without actor modifications.

Abstract

Policies learned via continuous actor-critic methods often exhibit erratic, high-frequency oscillations, making them unsuitable for physical deployment. Current approaches attempt to enforce smoothness by directly regularizing the policy's output. We argue that this approach treats the symptom rather than the cause. In this work, we theoretically establish that policy non-smoothness is fundamentally governed by the differential geometry of the critic. By applying implicit differentiation to the actor-critic objective, we prove that the sensitivity of the optimal policy is bounded by the ratio of the Q-function's mixed-partial derivative (noise sensitivity) to its action-space curvature (signal distinctness). To empirically validate this theoretical insight, we introduce PAVE (Policy-Aware Value-field Equalization), a critic-centric regularization framework that treats the critic as a scalar field and stabilizes its induced action-gradient field. PAVE rectifies the learning signal by minimizing the Q-gradient volatility while preserving local curvature. Experimental results demonstrate that PAVE achieves smoothness and robustness comparable to policy-side smoothness regularization methods, while maintaining competitive task performance, without modifying the actor.