Convergence Rates of Online Critic Value Function Approximation in Native Spaces

TL;DR

This paper analyzes the convergence rates of online critic value function approximation within reproducing kernel Hilbert spaces, providing explicit performance bounds based on kernel properties and basis placement.

Contribution

It introduces a general framework for analyzing online critic convergence in RKHSs, deriving explicit bounds related to basis functions and center placement.

Findings

Performance bounds depend on kernel and basis function properties.

Explicit relationship between critic accuracy and center placement.

Bounds facilitate basis selection and error assessment.

Abstract

In this paper, the evolution equation that defines the online critic for the approximation of the optimal value function is cast in a general class of reproducing kernel Hilbert spaces (RKHSs). Exploiting some core tools of RKHS theory, this formulation allows deriving explicit bounds on the performance of the critic in terms of the kernel and definition of the RKHS, the number of basis functions, and the location of centers used to define scattered bases. The performance of the critic is precisely measured in terms of the power function of the scattered basis used in approximations, and it can be used either in an a priori evaluation of potential bases or in an a posteriori assessments of value function error for basis enrichment or pruning. The most concise bounds in the paper describe explicitly how the critic performance depends on the placement of centers, as measured by their fill distance in a subset that contains the trajectory of the critic.