Hierarchical Control of Multi-Agent Systems using Online Reinforcement Learning

TL;DR

This paper introduces a hierarchical reinforcement learning approach for multi-agent systems, enabling efficient controller design by decoupling objectives and leveraging online measurements for faster learning.

Contribution

It presents a novel hierarchical RL method that decouples group-level and centroid control objectives, reducing learning time for multi-agent controllers.

Findings

Decoupled Riccati equations enable parallel controller learning.

Hierarchical approach reduces overall learning time.

Effective control of heterogeneous multi-agent systems achieved.

Abstract

We propose a new reinforcement learning based approach to designing hierarchical linear quadratic regulator (LQR) controllers for heterogeneous linear multi-agent systems with unknown state-space models and separated control objectives. The separation arises from grouping the agents into multiple non-overlapping groups, and defining the control goal as two distinct objectives. The first objective aims to minimize a group-wise block-decentralized LQR function that models group-level mission. The second objective, on the other hand, tries to minimize an LQR function between the average states (centroids) of the groups. Exploiting this separation, we redefine the weighting matrices of the LQR functions in a way that they allow us to decouple their respective algebraic Riccati equations. Thereafter, we develop a reinforcement learning strategy that uses online measurements of the agent states and the average states to learn the respective controllers based on the approximate Riccati equations. Since the first controller is block-decentralized and, therefore, can be learned in parallel, while the second controller is reduced-dimensional due to averaging, the overall design enjoys a significantly reduced learning time compared to centralized reinforcement learning.