Convergence Guarantees for Federated SARSA with Local Training and Heterogeneous Agents

TL;DR

This paper provides the first convergence analysis and complexity bounds for Federated SARSA with heterogeneous agents, demonstrating linear speed-up and supporting the theory with numerical experiments.

Contribution

It introduces a novel theoretical framework for Federated SARSA with heterogeneity, including a new multi-step error expansion and convergence guarantees.

Findings

FedSARSA converges despite heterogeneity in local data.

The method achieves linear speed-up with the number of agents.

Numerical results validate the theoretical analysis.

Abstract

We present a novel theoretical analysis of Federated SARSA (FedSARSA) with linear function approximation and local training. We establish convergence guarantees for FedSARSA in the presence of heterogeneity, both in local transitions and rewards, providing the first sample and communication complexity bounds in this setting. At the core of our analysis is a new, exact multi-step error expansion for single-agent SARSA, which is of independent interest. Our analysis precisely quantifies the impact of heterogeneity, demonstrating the convergence of FedSARSA with multiple local updates. Crucially, we show that FedSARSA achieves linear speed-up with respect to the number of agents, up to higher-order terms due to Markovian sampling. Numerical experiments support our theoretical findings.