Agnostic Reinforcement Learning: Foundations and Algorithms

TL;DR

This paper provides a rigorous theoretical analysis of reinforcement learning with function approximation in large state spaces, focusing on agnostic policy learning without assuming optimal policy within the class.

Contribution

It introduces a comprehensive framework for agnostic policy learning, designing new algorithms with guarantees and establishing fundamental performance bounds.

Findings

New algorithms with theoretical guarantees

Fundamental performance bounds characterized

Revealed limitations and capabilities of agnostic RL

Abstract

Reinforcement Learning (RL) has demonstrated tremendous empirical success across numerous challenging domains. However, we lack a strong theoretical understanding of the statistical complexity of RL in environments with large state spaces, where function approximation is required for sample-efficient learning. This thesis addresses this gap by rigorously examining the statistical complexity of RL with function approximation from a learning theoretic perspective. Departing from a long history of prior work, we consider the weakest form of function approximation, called agnostic policy learning, in which the learner seeks to find the best policy in a given class $\Pi$, with no guarantee that $\Pi$ contains an optimal policy for the underlying task. We systematically explore agnostic policy learning along three key axes: environment access -- how a learner collects data from the environment; coverage conditions -- intrinsic properties of the underlying MDP measuring the expansiveness of state-occupancy measures for policies in the class $\Pi$, and representational conditions -- structural assumptions on the class $\Pi$ itself. Within this comprehensive framework, we (1) design new learning algorithms with theoretical guarantees and (2) characterize fundamental performance bounds of any algorithm. Our results reveal significant statistical separations that highlight the power and limitations of agnostic policy learning.