Analyzing the Impact of Computation in Adaptive Dynamic Programming for Stochastic LQR Problem

TL;DR

This paper investigates how the sampling period and computational errors in adaptive dynamic programming affect control performance in stochastic LQR problems, revealing a linear convergence rate and validating findings with a sensorimotor task.

Contribution

It establishes the impact of sampling period on ADP convergence and control quality, linking computational errors to convergence behavior in stochastic LQR.

Findings

Convergence rate is O(h) with respect to sampling period h.

Sampling period significantly influences control performance.

Theoretical results validated by sensorimotor control experiment.

Abstract

Adaptive dynamic programming (ADP) for stochastic linear quadratic regulation (LQR) demands the precise computation of stochastic integrals during policy iteration (PI). In a fully model-free problem setting, this computation can only be approximated by state samples collected at discrete time points using computational methods such as the canonical Euler-Maruyama method. Our research reveals a critical phenomenon: the sampling period can significantly impact control performance. This impact is due to the fact that computational errors introduced in each step of PI can significantly affect the algorithm's convergence behavior, which in turn influences the resulting control policy. We draw a parallel between PI and Newton's method applied to the Ricatti equation to elucidate how the computation impacts control. In this light, the computational error in each PI step manifests itself as an extra error term in each step of Newton's method, with its upper bound proportional to the computational error. Furthermore, we demonstrate that the convergence rate for ADP in stochastic LQR problems using the Euler-Maruyama method is O(h), with h being the sampling period. A sensorimotor control task finally validates these theoretical findings.