Policy Newton Algorithm in Reproducing Kernel Hilbert Space

TL;DR

This paper introduces Policy Newton in RKHS, a novel second-order optimization method for RL policies in Reproducing Kernel Hilbert Spaces, enabling faster convergence and better performance than existing first-order methods.

Contribution

It develops the first second-order optimization framework for RKHS-based RL policies, transforming an infinite-dimensional problem into a finite-dimensional one with theoretical guarantees.

Findings

Achieves quadratic convergence to a local optimum.

Demonstrates superior convergence speed over first-order methods.

Attains higher episodic rewards on benchmark tasks.

Abstract

Reinforcement learning (RL) policies represented in Reproducing Kernel Hilbert Spaces (RKHS) offer powerful representational capabilities. While second-order optimization methods like Newton's method demonstrate faster convergence than first-order approaches, current RKHS-based policy optimization remains constrained to first-order techniques. This limitation stems primarily from the intractability of explicitly computing and inverting the infinite-dimensional Hessian operator in RKHS. We introduce Policy Newton in RKHS, the first second-order optimization framework specifically designed for RL policies represented in RKHS. Our approach circumvents direct computation of the inverse Hessian operator by optimizing a cubic regularized auxiliary objective function. Crucially, we leverage the Representer Theorem to transform this infinite-dimensional optimization into an equivalent, computationally tractable finite-dimensional problem whose dimensionality scales with the trajectory data volume. We establish theoretical guarantees proving convergence to a local optimum with a local quadratic convergence rate. Empirical evaluations on a toy financial asset allocation problem validate these theoretical properties, while experiments on standard RL benchmarks demonstrate that Policy Newton in RKHS achieves superior convergence speed and higher episodic rewards compared to established first-order RKHS approaches and parametric second-order methods. Our work bridges a critical gap between non-parametric policy representations and second-order optimization methods in reinforcement learning.