A Variance-Reduced Cubic-Regularized Newton for Policy Optimization

TL;DR

This paper introduces VR-CR-PN, a novel second-order policy optimization algorithm in reinforcement learning that uses variance reduction and cubic regularization to improve sample complexity without relying on importance sampling.

Contribution

It is the first to combine Hessian-based variance reduction with second-order policy optimization, achieving horizon-independent sample complexity and improved theoretical guarantees.

Findings

Achieves $ ilde{O}( ext{ extcolor{red}{ ext{epsilon}}}^{-3})$ sample complexity for second-order stationary points.

Introduces a Hessian estimator with a horizon-independent upper bound.

Outperforms previous methods with $ ilde{O}( ext{ extcolor{red}{ ext{epsilon}}}^{-3.5})$ complexity.

Abstract

In this paper, we study a second-order approach to policy optimization in reinforcement learning. Existing second-order methods often suffer from suboptimal sample complexity or rely on unrealistic assumptions about importance sampling. To overcome these limitations, we propose VR-CR-PN, a variance-reduced cubic-regularized policy Newton algorithm. To the best of our knowledge, this is the first algorithm that integrates Hessian-aided variance reduction with second-order policy optimization, effectively addressing the distribution shift problem and achieving best-known sample complexity under general nonconvex conditions but without the need for importance sampling. We theoretically establish that VR-CR-PN achieves a sample complexity of $\tilde{\mathcal{O}}(\epsilon^{-3})$ to reach an $\epsilon$-second-order stationary point, significantly improving upon the previous best result of $\tilde{\mathcal{O}}(\epsilon^{-3.5})$ under comparable assumptions. As an additional contribution, we introduce a novel Hessian estimator for the expected return function, which admits a uniform upper bound independent of the horizon length $H$, allowing the algorithm to achieve horizon-independent sample complexity.