Single-Loop Federated Actor-Critic across Heterogeneous Environments

TL;DR

This paper introduces SFAC, a federated actor-critic algorithm for heterogeneous environments, providing theoretical convergence guarantees and demonstrating improved sample efficiency through federated learning.

Contribution

The paper presents the first theoretical analysis of federated actor-critic algorithms in heterogeneous environments, including convergence bounds and sample complexity analysis.

Findings

Convergence error asymptotically approaches a near-stationary point.

Sample complexity benefits from linear speed-up with more agents.

Numerical experiments confirm the effectiveness of SFAC.

Abstract

Federated reinforcement learning (FRL) has emerged as a promising paradigm, enabling multiple agents to collaborate and learn a shared policy adaptable across heterogeneous environments. Among the various reinforcement learning (RL) algorithms, the actor-critic (AC) algorithm stands out for its low variance and high sample efficiency. However, little to nothing is known theoretically about AC in a federated manner, especially each agent interacts with a potentially different environment. The lack of such results is attributed to various technical challenges: a two-level structure illustrating the coupling effect between the actor and the critic, heterogeneous environments, Markovian sampling and multiple local updates. In response, we study \textit{Single-loop Federated Actor Critic} (SFAC) where agents perform actor-critic learning in a two-level federated manner while interacting with heterogeneous environments. We then provide bounds on the convergence error of SFAC. The results show that the convergence error asymptotically converges to a near-stationary point, with the extent proportional to environment heterogeneity. Moreover, the sample complexity exhibits a linear speed-up through the federation of agents. We evaluate the performance of SFAC through numerical experiments using common RL benchmarks, which demonstrate its effectiveness.