Analysis of regularized federated learning

TL;DR

This paper analyzes a federated learning algorithm called Loopless Local Gradient Descent, focusing on its convergence properties under non-convex and strongly convex conditions, with improvements allowing flexible step sizes.

Contribution

It introduces a flexible step size version of Loopless Local Gradient Descent and provides novel convergence analysis in non-convex and strongly convex settings.

Findings

Convergence rates established for non-convex functions satisfying Polyak-Łojasiewicz condition.

Necessary and sufficient conditions for convergence in strongly convex case.

Algorithm reduces communication costs through probabilistic control.

Abstract

Federated learning is an efficient machine learning tool for dealing with heterogeneous big data and privacy protection. Federated learning methods with regularization can control the level of communications between the central and local machines. Stochastic gradient descent is often used for implementing such methods on heterogeneous big data, to reduce the communication costs. In this paper, we consider such an algorithm called Loopless Local Gradient Descent which has advantages in reducing the expected communications by controlling a probability level. We improve the method by allowing flexible step sizes and carry out novel analysis for the convergence of the algorithm in a non-convex setting in addition to the standard strongly convex setting. In the non-convex setting, we derive rates of convergence when the smooth objective function satisfies a Polyak-{\L}ojasiewicz condition. When the objective function is strongly convex, a sufficient and necessary condition for the convergence in expectation is presented.