Statistical Inference in Reinforcement Learning: A Selective Survey

TL;DR

This paper reviews statistical inference methods in reinforcement learning, emphasizing their importance and encouraging broader adoption in the field to enhance understanding and application.

Contribution

It provides a selective survey of statistical inferential tools for RL, including hypothesis testing and confidence intervals, highlighting their relevance and potential.

Findings

Highlights the recent integration of statistical inference in RL research

Emphasizes the importance of classical statistical tools in RL applications

Encourages cross-disciplinary collaboration between statistics and machine learning

Abstract

Reinforcement learning (RL) is concerned with how intelligence agents take actions in a given environment to maximize the cumulative reward they receive. In healthcare, applying RL algorithms could assist patients in improving their health status. In ride-sharing platforms, applying RL algorithms could increase drivers' income and customer satisfaction. For large language models, applying RL algorithms could align their outputs with human preferences. Over the past decade, RL has been arguably one of the most vibrant research frontiers in machine learning. Nevertheless, statistics as a field, as opposed to computer science, has only recently begun to engage with RL both in depth and in breadth. This chapter presents a selective review of statistical inferential tools for RL, covering both hypothesis testing and confidence interval construction. Our goal is to highlight the value of statistical inference in RL for both the statistics and machine learning communities, and to promote the broader application of classical statistical inference tools in this vibrant area of research.