Model-Free Optimal Control of Linear Multi-Agent Systems via Decomposition and Hierarchical Approximation

TL;DR

This paper introduces a hierarchical, decomposition-based approach for model-free optimal control of large-scale multi-agent systems, reducing computational complexity by solving smaller LQR problems and a coupling least squares problem.

Contribution

It proposes a novel graph clustering-based decomposition method for large-scale LQR design, enabling model-free reinforcement learning solutions with reduced complexity.

Findings

Hierarchical controllers improve scalability for large multi-agent systems.

Decomposition reduces the size of Riccati equations to be solved.

Numerical simulations demonstrate the effectiveness of the approach.

Abstract

Designing the optimal linear quadratic regulator (LQR) for a large-scale multi-agent system (MAS) is time-consuming since it involves solving a large-size matrix Riccati equation. The situation is further exasperated when the design needs to be done in a model-free way using schemes such as reinforcement learning (RL). To reduce this computational complexity, we decompose the large-scale LQR design problem into multiple smaller-size LQR design problems. We consider the objective function to be specified over an undirected graph, and cast the decomposition as a graph clustering problem. The graph is decomposed into two parts, one consisting of independent clusters of connected components, and the other containing edges that connect different clusters. Accordingly, the resulting controller has a hierarchical structure, consisting of two components. The first component optimizes the performance of each independent cluster by solving the smaller-size LQR design problem in a model-free way using an RL algorithm. The second component accounts for the objective coupling different clusters, which is achieved by solving a least squares problem in one shot. Although suboptimal, the hierarchical controller adheres to a particular structure as specified by inter-agent couplings in the objective function and by the decomposition strategy. Mathematical formulations are established to find a decomposition that minimizes the number of required communication links or reduces the optimality gap. Numerical simulations are provided to highlight the pros and cons of the proposed designs.