An Encoded Corrective Double Deep Q-Networks for Multi-Agent Control Systems

TL;DR

This paper introduces a distributed encoded corrective double actor-critic framework for multi-agent control, explicitly modeling communication delays and noise to improve policy synthesis.

Contribution

It presents a novel message-passing mechanism that refines global state information over time, enhancing multi-agent control under communication uncertainties.

Findings

Effective in multiple test cases

Outperforms various baseline methods

Numerical regret analysis supports effectiveness

Abstract

This paper studies the synthesis of control policies for heterogeneous and interconnected multi-agent systems that collaborate through data exchange over a communication network to minimize a collective cost. We propose a distributed encoded corrective double actor-critic framework that integrates a novel message-passing mechanism. Existing methods assume noise-free and delay-free access to the global or partial states and overlook the fact that the global states, though noisy and delayed, can be progressively reconstructed and refined over time. In contrast, this work explicitly models communication sampling asynchrony, delay, and link noise based on the network configuration. The proposed message-passing mechanism characterizes timing and information flow to refine and time shift global state information, which is then used to incrementally correct the Q-networks. The double Q-network design mitigates overestimation bias, while the shared encoder coupling the actor-critic networks captures inter-agent dependencies. We evaluate our approach in multiple test cases, demonstrate its effectiveness over various baselines, and provide a numerical regret analysis.