Least-Squares Temporal Difference Learning for the Linear Quadratic Regulator

TL;DR

This paper provides the first finite-time analysis of the sample complexity for Least-Squares Temporal Difference learning applied to the Linear Quadratic Regulator, advancing understanding of RL in continuous control tasks.

Contribution

It introduces a finite-sample analysis of LSTD for LQR, including a new eigenvalue concentration result for empirical covariance matrices in stochastic processes.

Findings

Finite-time bounds for LSTD in LQR

Eigenvalue concentration characterization for stochastic processes

Experimental validation of theoretical results

Abstract

Reinforcement learning (RL) has been successfully used to solve many continuous control tasks. Despite its impressive results however, fundamental questions regarding the sample complexity of RL on continuous problems remain open. We study the performance of RL in this setting by considering the behavior of the Least-Squares Temporal Difference (LSTD) estimator on the classic Linear Quadratic Regulator (LQR) problem from optimal control. We give the first finite-time analysis of the number of samples needed to estimate the value function for a fixed static state-feedback policy to within $\varepsilon$-relative error. In the process of deriving our result, we give a general characterization for when the minimum eigenvalue of the empirical covariance matrix formed along the sample path of a fast-mixing stochastic process concentrates above zero, extending a result by Koltchinskii and Mendelson in the independent covariates setting. Finally, we provide experimental evidence indicating that our analysis correctly captures the qualitative behavior of LSTD on several LQR instances.