Statistical and Algorithmic Foundations of Reinforcement Learning

TL;DR

This paper reviews recent theoretical and algorithmic advances in reinforcement learning, focusing on sample efficiency, computational challenges, and various RL scenarios using Markov Decision Processes.

Contribution

It synthesizes key developments in RL theory and algorithms, connecting classical ideas with new approaches across multiple RL settings.

Findings

Highlights the importance of sample complexity and computational efficiency in RL.

Discusses lower bounds and theoretical limits of RL algorithms.

Examines different RL scenarios including offline, online, and robust RL.

Abstract

As a paradigm for sequential decision making in unknown environments, reinforcement learning (RL) has received a flurry of attention in recent years. However, the explosion of model complexity in emerging applications and the presence of nonconvexity exacerbate the challenge of achieving efficient RL in sample-starved situations, where data collection is expensive, time-consuming, or even high-stakes (e.g., in clinical trials, autonomous systems, and online advertising). How to understand and enhance the sample and computational efficacies of RL algorithms is thus of great interest. In this tutorial, we aim to introduce several important algorithmic and theoretical developments in RL, highlighting the connections between new ideas and classical topics. Employing Markov Decision Processes as the central mathematical model, we cover several distinctive RL scenarios (i.e., RL with a simulator, online RL, offline RL, robust RL, and RL with human feedback), and present several mainstream RL approaches (i.e., model-based approach, value-based approach, and policy optimization). Our discussions gravitate around the issues of sample complexity, computational efficiency, as well as algorithm-dependent and information-theoretic lower bounds from a non-asymptotic viewpoint.