Distributed Q-Learning for Stochastic LQ Control with Unknown Uncertainty

TL;DR

This paper introduces a distributed Q-learning algorithm for stochastic linear quadratic control systems with unknown parameters, ensuring convergence to the optimal controller over an infinite horizon.

Contribution

It presents a novel distributed stochastic approximation method to solve Riccati equations in uncertain stochastic control systems, with proven convergence guarantees.

Findings

The algorithm converges asymptotically to the optimal solution.

The method effectively handles unknown system uncertainties.

Numerical simulations confirm the convergence and stability.

Abstract

This paper studies a discrete-time stochastic control problem with linear quadratic criteria over an infinite-time horizon. We focus on a class of control systems whose system matrices are associated with random parameters involving unknown statistical properties. In particular, we design a distributed Q-learning algorithm to tackle the Riccati equation and derive the optimal controller stabilizing the system. The key technique is that we convert the problem of solving the Riccati equation into deriving the zero point of a matrix equation and devise a distributed stochastic approximation method to compute the estimates of the zero point. The convergence analysis proves that the distributed Q-learning algorithm converges to the correct value eventually. A numerical example sheds light on that the distributed Q-learning algorithm converges asymptotically.