The Sample Complexity of Online Reinforcement Learning: A Multi-model Perspective

TL;DR

This paper analyzes the sample complexity of online reinforcement learning for nonlinear dynamical systems with continuous spaces, providing regret bounds for various model classes and highlighting practical algorithm features.

Contribution

It introduces a unified analysis of sample complexity for diverse nonlinear dynamical models, extending previous results to more general settings.

Findings

Policy regret of O(N ε^2 + d_u ln(m(ε))/ε^2) in general models

Regret of O(√(d_u N p)) for parametrized models like neural networks

Algorithms are simple, incorporate prior knowledge, and have benign transients

Abstract

We study the sample complexity of online reinforcement learning in the general \hzyrev{non-episodic} setting of nonlinear dynamical systems with continuous state and action spaces. Our analysis accommodates a large class of dynamical systems ranging from a finite set of nonlinear candidate models to models with bounded and Lipschitz continuous dynamics, to systems that are parametrized by a compact and real-valued set of parameters. In the most general setting, our algorithm achieves a policy regret of $\mathcal{O}(N \epsilon^2 + d_\mathrm{u}\mathrm{ln}(m(\epsilon))/\epsilon^2)$, where $N$ is the time horizon, $\epsilon$ is a user-specified discretization width, $d_\mathrm{u}$ the input dimension, and $m(\epsilon)$ measures the complexity of the function class under consideration via its packing number. In the special case where the dynamics are parametrized by a compact and real-valued set of parameters (such as neural networks, transformers, etc.), we prove a policy regret of $\mathcal{O}(\sqrt{d_\mathrm{u}N p})$, where $p$ denotes the number of parameters, recovering earlier sample-complexity results that were derived for linear time-invariant dynamical systems. While this article focuses on characterizing sample complexity, the proposed algorithms are likely to be useful in practice, due to their simplicity, their ability to incorporate prior knowledge, and their benign transient behaviors.