Local Stochastic Approximation: A Unified View of Federated Learning and Distributed Multi-Task Reinforcement Learning Algorithms

TL;DR

This paper analyzes local stochastic approximation algorithms in federated and multi-task reinforcement learning, providing convergence rates with dependent data and demonstrating their applicability to various learning problems.

Contribution

It offers a unified analysis of stochastic approximation methods for dependent data in federated and multi-task reinforcement learning, including convergence rates.

Findings

Convergence rates are within a logarithmic factor of those with independent data.

Applicable to multi-task reinforcement learning and federated learning.

Provides finite-time performance guarantees for dependent data scenarios.

Abstract

Motivated by broad applications in reinforcement learning and federated learning, we study local stochastic approximation over a network of agents, where their goal is to find the root of an operator composed of the local operators at the agents. Our focus is to characterize the finite-time performance of this method when the data at each agent are generated from Markov processes, and hence they are dependent. In particular, we provide the convergence rates of local stochastic approximation for both constant and time-varying step sizes. Our results show that these rates are within a logarithmic factor of the ones under independent data. We then illustrate the applications of these results to different interesting problems in multi-task reinforcement learning and federated learning.