Scaling up Stability: Reinforcement Learning for Distributed Control of Networked Systems in the Space of Stabilizing Policies

TL;DR

This paper presents a novel reinforcement learning approach for distributed control of networked systems using GNN-embedded policies that ensure stability and transferability across different network sizes and topologies.

Contribution

The authors introduce a GNN-based policy parameterization embedded in a Youla-like framework that guarantees stability by design and demonstrates robustness and transferability in multi-agent tasks.

Findings

Policies trained on small networks transfer to larger ones.

Achieves higher returns and lower variance than existing MARL methods.

Guarantees network-level stability by design.

Abstract

We study distributed control of networked systems through reinforcement learning, where neural policies must be simultaneously scalable, expressive and stabilizing. We introduce a policy parameterization that embeds Graph Neural Networks (GNNs) into a Youla-like magnitude-direction parameterization, yielding distributed stochastic controllers that guarantee network-level closed-loop stability by design. The magnitude is implemented as a stable operator consisting of a GNN acting on disturbance feedback, while the direction is a GNN acting on local observations. We prove robustness of the closed loop to perturbations in both the graph topology and model parameters, and show how to integrate our parameterization with Proximal Policy Optimization. Experiments on a multi-agent navigation task show that policies trained on small networks transfer directly to larger ones and unseen network topologies, achieve higher returns and lower variance than a state-of-the-art MARL baseline while preserving stability.