FLECS-CGD: A Federated Learning Second-Order Framework via Compression and Sketching with Compressed Gradient Differences

TL;DR

This paper introduces FLECS-CGD, an improved federated learning framework that employs compressed gradient differences to reduce communication costs, with proven convergence and practical benefits demonstrated through experiments.

Contribution

It extends the FLECS framework by incorporating compressed gradient differences, enabling stochastic optimization with convergence guarantees.

Findings

Reduced communication costs through gradient compression.

Proven convergence in strongly convex and nonconvex settings.

Experimental validation showing practical efficiency.

Abstract

In the recent paper FLECS (Agafonov et al, FLECS: A Federated Learning Second-Order Framework via Compression and Sketching), the second-order framework FLECS was proposed for the Federated Learning problem. This method utilize compression of sketched Hessians to make communication costs low. However, the main bottleneck of FLECS is gradient communication without compression. In this paper, we propose the modification of FLECS with compressed gradient differences, which we call FLECS-CGD (FLECS with Compressed Gradient Differences) and make it applicable for stochastic optimization. Convergence guarantees are provided in strongly convex and nonconvex cases. Experiments show the practical benefit of proposed approach.