Toward Optimal Statistical Inference in Noisy Linear Quadratic Reinforcement Learning over a Finite Horizon

TL;DR

This paper develops a statistical inference framework for noisy linear quadratic reinforcement learning, providing asymptotic and non-asymptotic confidence intervals for policies and losses, applicable in online and offline settings.

Contribution

It introduces a novel online bootstrapping method for confidence intervals in LQ RL, with proven distributional consistency and rate of approximation, advancing uncertainty quantification in reinforcement learning.

Findings

Confidence intervals achieve asymptotic and non-asymptotic validity.

Quantiles of the distribution can be approximated at a rate of n^{-1/4}.

Method is effective in numerical experiments across various systems.

Abstract

Recent developments in Reinforcement learning have significantly enhanced sequential decision-making in uncertain environments. Despite their strong performance guarantees, most existing work has focused primarily on improving the operational accuracy of learned control policies and the convergence rates of learning algorithms, with comparatively little attention to uncertainty quantification and statistical inference. Yet, these aspects are essential for assessing the reliability and variability of control policies, especially in high-stakes applications. In this paper, we study statistical inference for the policy gradient (PG) method for noisy Linear Quadratic Reinforcement learning (LQ RL) over a finite time horizon, where linear dynamics with both known and unknown drift parameters are controlled subject to a quadratic cost. We establish the theoretical foundations for statistical inference in LQ RL, deriving exact asymptotics for both the PG estimators and the corresponding objective loss. Furthermore, we introduce a principled inference framework that leverages online bootstrapping to construct confidence intervals for both the learned optimal policy and the corresponding objective losses. The method updates the PG estimates along with a set of randomly perturbed PG estimates as new observations arrive. We prove that the proposed bootstrapping procedure is distributionally consistent and that the resulting confidence intervals achieve both asymptotic and non-asymptotic validity. Notably, our results imply that the quantiles of the exact distribution can be approximated at a rate of $n^{-1/4}$, where $n$ is the number of samples used during the procedure. The proposed procedure is easy to implement and applicable to both offline and fully online settings. Numerical experiments illustrate the effectiveness of our approach across a range of noisy linear dynamical systems.