Communication-Efficient Consensus Mechanism for Federated Reinforcement Learning

TL;DR

This paper proposes a communication-efficient consensus mechanism for federated reinforcement learning that balances convergence performance with reduced communication overhead, backed by theoretical guarantees and simulations.

Contribution

It introduces a consensus-based optimization scheme with convergence guarantees to improve federated reinforcement learning efficiency.

Findings

Enhanced training efficiency and stability in IRL with FL.

Reduced communication overhead through the proposed consensus scheme.

Theoretical convergence guarantees for the new method.

Abstract

The paper considers independent reinforcement learning (IRL) for multi-agent decision-making process in the paradigm of federated learning (FL). We show that FL can clearly improve the policy performance of IRL in terms of training efficiency and stability. However, since the policy parameters are trained locally and aggregated iteratively through a central server in FL, frequent information exchange incurs a large amount of communication overheads. To reach a good balance between improving the model's convergence performance and reducing the required communication and computation overheads, this paper proposes a system utility function and develops a consensus-based optimization scheme on top of the periodic averaging method, which introduces the consensus algorithm into FL for the exchange of a model's local gradients. This paper also provides novel convergence guarantees for the developed method, and demonstrates its superior effectiveness and efficiency in improving the system utility value through theoretical analyses and numerical simulation results.