FLeNS: Federated Learning with Enhanced Nesterov-Newton Sketch

TL;DR

FLeNS introduces a federated learning algorithm that combines Nesterov acceleration with Hessian sketching to achieve super-linear convergence while significantly reducing communication costs.

Contribution

It presents a novel federated optimization method that approximates second-order Newton updates efficiently using Hessian sketching combined with Nesterov's acceleration.

Findings

Achieves super-linear convergence in communication rounds.

Reduces communication overhead via Hessian sketching.

Outperforms existing methods in empirical evaluations.

Abstract

Federated learning faces a critical challenge in balancing communication efficiency with rapid convergence, especially for second-order methods. While Newton-type algorithms achieve linear convergence in communication rounds, transmitting full Hessian matrices is often impractical due to quadratic complexity. We introduce Federated Learning with Enhanced Nesterov-Newton Sketch (FLeNS), a novel method that harnesses both the acceleration capabilities of Nesterov's method and the dimensionality reduction benefits of Hessian sketching. FLeNS approximates the centralized Newton's method without relying on the exact Hessian, significantly reducing communication overhead. By combining Nesterov's acceleration with adaptive Hessian sketching, FLeNS preserves crucial second-order information while preserving the rapid convergence characteristics. Our theoretical analysis, grounded in statistical learning, demonstrates that FLeNS achieves super-linear convergence rates in communication rounds - a notable advancement in federated optimization. We provide rigorous convergence guarantees and characterize tradeoffs between acceleration, sketch size, and convergence speed. Extensive empirical evaluation validates our theoretical findings, showcasing FLeNS's state-of-the-art performance with reduced communication requirements, particularly in privacy-sensitive and edge-computing scenarios. The code is available at https://github.com/sunnyinAI/FLeNS