Distributed Q-Learning for Dynamically Decoupled Systems

TL;DR

This paper introduces a distributed Q-learning algorithm for large-scale networked systems with decoupled dynamics, enabling data-driven control design that converges to optimal LQR controllers without requiring detailed models.

Contribution

It presents a novel distributed Q-learning method tailored for dynamically decoupled systems, ensuring convergence to optimal controllers based solely on observed data.

Findings

The algorithm converges to the optimal LQR controller for each subsystem.

The method effectively handles systems with complex interaction structures.

Verification through an example demonstrates practical applicability.

Abstract

Control of large-scale networked systems often necessitates the availability of complex models for the interactions amongst the agents. However in many applications, building accurate models of agents or interactions amongst them might be infeasible or computationally prohibitive due to the curse of dimensionality or the complexity of these interactions. In the meantime, data-guided control methods can circumvent model complexity by directly synthesizing the controller from the observed data. In this paper, we propose a distributed Q-learning algorithm to design a feedback mechanism based on a given underlying graph structure parameterizing the agents' interaction network. We assume that the distributed nature of the system arises from the cost function of the corresponding control problem and show that for the specific case of identical dynamically decoupled systems, the learned controller converges to the optimal Linear Quadratic Regulator (LQR) controller for each subsystem. We provide a convergence analysis and verify the result with an example.