Convergence of Dynamic Programming on the Semidefinite Cone

TL;DR

This paper provides new convergence analysis for dynamic programming methods applied to the linear quadratic regulator problem, demonstrating exponential convergence of Q-value and policy iteration under mild assumptions.

Contribution

It introduces simple bounds on errors and proves exponential convergence of Q-value iteration and policy iteration for LQR problems, extending existing theoretical understanding.

Findings

Q-value iteration converges exponentially to the optimal solution.

Policy iteration always converges exponentially fast.

Global asymptotic convergence is established.

Abstract

The goal of this paper is to investigate new and simple convergence analysis of dynamic programming for linear quadratic regulator problem of discrete-time linear time-invariant systems. In particular, bounds on errors are given in terms of both matrix inequalities and matrix norm. Under a mild assumption on the initial parameter, we prove that the Q-value iteration exponentially converges to the optimal solution. Moreover, a global asymptotic convergence is also presented. These results are then extended to the policy iteration. We prove that in contrast to the Q-value iteration, the policy iteration always converges exponentially fast. An example is given to illustrate the results.