A Measure-Theoretic Finite-Sample Theory for Adaptive-Data Fitted Q-Iteration

TL;DR

This paper develops a measure-theoretic finite-sample framework for fitted Q-iteration in reinforcement learning, addressing theoretical gaps and providing performance bounds in continuous spaces.

Contribution

It introduces a unified measure-theoretic approach to analyze FQI with finite-sample guarantees and online regret bounds in general measurable spaces.

Findings

Finite-sample performance bounds for FQI on general spaces.

Sequential Rademacher complexity controls Bellman-regression generalization.

First cumulative online regret guarantee for FQI in continuous spaces.

Abstract

While reinforcement learning (RL) promises to revolutionize the control of complex nonlinear robotic systems, a profound gap persists between the heuristic success of model-free off-policy deep RL and the underlying theory, which remains largely confined to tabular or linearizable settings. We identify the cause of this gap as an emergent isolation of three traditions: (i) measure-theoretic MDP foundations on general spaces limit their analysis to exact dynamic programming and ignore all error sources of a learning process; (ii) deterministic error propagation analysis addresses the approximation error via concentrability coefficients without a finite-sample analysis of the estimation error; and (iii) PAC generalization bounds characterize the estimation errors of simplified topologies. We bridge these traditions with a unified theoretical framework for fitted Q-iteration (FQI) on general measurable Borel spaces. Our main result provides a finite-sample, adaptive-data performance bound by chaining measure-theoretic probability with Bellman-operator contraction in Banach spaces. We prove that sequential Rademacher complexity controls Bellman-regression generalization under policy-dependent data collection. We further extend this analysis to provide the first cumulative, pathwise online regret guarantee for FQI in continuous spaces. These results lay the necessary foundations for the formal analysis of many modern deep RL algorithms.