Accelerated Stochastic ExtraGradient: Mixing Hessian and Gradient Similarity to Reduce Communication in Distributed and Federated Learning

TL;DR

This paper introduces an accelerated stochastic method that leverages Hessian and gradient similarities to reduce communication costs in distributed and federated learning, while also considering privacy through noise addition.

Contribution

It combines Hessian and gradient similarity assumptions in a new method and analyzes the impact of added noise on convergence, addressing efficiency and privacy.

Findings

The proposed method reduces communication in distributed learning.

Adding noise affects convergence but maintains privacy.

Theoretical analysis is supported by experiments on real datasets.

Abstract

Modern realities and trends in learning require more and more generalization ability of models, which leads to an increase in both models and training sample size. It is already difficult to solve such tasks in a single device mode. This is the reason why distributed and federated learning approaches are becoming more popular every day. Distributed computing involves communication between devices, which requires solving two key problems: efficiency and privacy. One of the most well-known approaches to combat communication costs is to exploit the similarity of local data. Both Hessian similarity and homogeneous gradients have been studied in the literature, but separately. In this paper, we combine both of these assumptions in analyzing a new method that incorporates the ideas of using data similarity and clients sampling. Moreover, to address privacy concerns, we apply the technique of additional noise and analyze its impact on the convergence of the proposed method. The theory is confirmed by training on real datasets.