Rates of Convergence in Certain Native Spaces of Approximations used in Reinforcement Learning

TL;DR

This paper derives explicit convergence rates and error bounds for value function approximations in native RKHS spaces used in reinforcement learning, improving understanding of approximation quality in policy iteration.

Contribution

It introduces new geometric convergence bounds for value function approximations in native RKHS spaces, refining classical results in reinforcement learning.

Findings

Explicit error bounds in terms of power functions

Geometric convergence rates established

Refinement of classical approximation results

Abstract

This paper studies convergence rates for some value function approximations that arise in a collection of reproducing kernel Hilbert spaces (RKHS) $H(\Omega)$. By casting an optimal control problem in a specific class of native spaces, strong rates of convergence are derived for the operator equation that enables offline approximations that appear in policy iteration. Explicit upper bounds on error in value function and controller approximations are derived in terms of power function $\mathcal{P}_{H,N}$ for the space of finite dimensional approximants $H_N$ in the native space $H(\Omega)$. These bounds are geometric in nature and refine some well-known, now classical results concerning convergence of approximations of value functions.