A Communication-Efficient Decentralized Actor-Critic Algorithm

TL;DR

This paper introduces a decentralized actor-critic reinforcement learning algorithm that reduces communication among agents through local updates, with proven convergence and practical validation in cooperative control tasks.

Contribution

The paper proposes a novel communication-efficient decentralized actor-critic algorithm with finite-time convergence analysis and neural network approximation considerations.

Findings

Achieves $ ilde{O}(rac{1}{ au \, ext{epsilon}^3})$ sample complexity.

Communication complexity is reduced to $ ilde{O}(rac{1}{ au \, ext{epsilon}})$.

Numerical experiments validate theoretical results in cooperative control.

Abstract

In this paper, we study the problem of reinforcement learning in multi-agent systems where communication among agents is limited. We develop a decentralized actor-critic learning framework in which each agent performs several local updates of its policy and value function, where the latter is approximated by a multi-layer neural network, before exchanging information with its neighbors. This local training strategy substantially reduces the communication burden while maintaining coordination across the network. We establish finite-time convergence analysis for the algorithm under Markov-sampling. Specifically, to attain the $\varepsilon$-accurate stationary point, the sample complexity is of order $\mathcal{O}(\varepsilon^{-3})$ and the communication complexity is of order $\mathcal{O}(\varepsilon^{-1}\tau^{-1})$, where tau denotes the number of local training steps. We also show how the final error bound depends on the neural network's approximation quality. Numerical experiments in a cooperative control setting illustrate and validate the theoretical findings.