Momentum-Based Federated Reinforcement Learning with Interaction and Communication Efficiency

TL;DR

This paper presents MFPO, a federated reinforcement learning algorithm that improves interaction and communication efficiency using momentum, importance sampling, and server adjustments, achieving near-optimal complexities and better performance.

Contribution

The paper introduces MFPO, a novel FRL algorithm that reduces interaction and communication costs while maintaining high performance, with theoretical complexity guarantees and extensive experimental validation.

Findings

MFPO achieves linear speedup with the number of agents.

MFPO attains the best known communication complexity for first-order FL algorithms.

Experiments show significant performance improvements over existing methods.

Abstract

Federated Reinforcement Learning (FRL) has garnered increasing attention recently. However, due to the intrinsic spatio-temporal non-stationarity of data distributions, the current approaches typically suffer from high interaction and communication costs. In this paper, we introduce a new FRL algorithm, named $\texttt{MFPO}$, that utilizes momentum, importance sampling, and additional server-side adjustment to control the shift of stochastic policy gradients and enhance the efficiency of data utilization. We prove that by proper selection of momentum parameters and interaction frequency, $\texttt{MFPO}$ can achieve $\tilde{\mathcal{O}}(H N^{-1}\epsilon^{-3/2})$ and $\tilde{\mathcal{O}}(\epsilon^{-1})$ interaction and communication complexities ($N$ represents the number of agents), where the interaction complexity achieves linear speedup with the number of agents, and the communication complexity aligns the best achievable of existing first-order FL algorithms. Extensive experiments corroborate the substantial performance gains of $\texttt{MFPO}$ over existing methods on a suite of complex and high-dimensional benchmarks.