Federated UCBVI: Communication-Efficient Federated Regret Minimization with Heterogeneous Agents

TL;DR

This paper introduces Fed-UCBVI, a communication-efficient federated reinforcement learning algorithm that achieves near-optimal regret bounds and scales linearly with the number of agents, even with heterogeneous environments.

Contribution

We propose Fed-UCBVI, a novel federated UCB-based algorithm with theoretical regret guarantees and minimal communication overhead in heterogeneous multi-agent settings.

Findings

Regret scales as  ext{O}(\u007B ext{sqrt}(H^3 | ext{S}| | ext{A}| T / M))

Achieves minimax lower bounds in single-agent case

Linearly scales with the number of agents in multi-agent setting

Abstract

In this paper, we present the Federated Upper Confidence Bound Value Iteration algorithm ($\texttt{Fed-UCBVI}$), a novel extension of the $\texttt{UCBVI}$ algorithm (Azar et al., 2017) tailored for the federated learning framework. We prove that the regret of $\texttt{Fed-UCBVI}$ scales as $\tilde{\mathcal{O}}(\sqrt{H^3 |\mathcal{S}| |\mathcal{A}| T / M})$, with a small additional term due to heterogeneity, where $|\mathcal{S}|$ is the number of states, $|\mathcal{A}|$ is the number of actions, $H$ is the episode length, $M$ is the number of agents, and $T$ is the number of episodes. Notably, in the single-agent setting, this upper bound matches the minimax lower bound up to polylogarithmic factors, while in the multi-agent scenario, $\texttt{Fed-UCBVI}$ has linear speed-up. To conduct our analysis, we introduce a new measure of heterogeneity, which may hold independent theoretical interest. Furthermore, we show that, unlike existing federated reinforcement learning approaches, $\texttt{Fed-UCBVI}$'s communication complexity only marginally increases with the number of agents.