Accelerated Training of Federated Learning via Second-Order Methods

TL;DR

This paper investigates second-order optimization techniques in Federated Learning to address slow convergence and high communication costs, providing a comprehensive analysis of their performance and challenges.

Contribution

It offers a detailed categorization and comparison of second-order FL methods, highlighting their potential and key challenges for scalable federated optimization.

Findings

Second-order methods can accelerate FL training.

Hessian-based optimization shows promise but faces computational challenges.

Future work needed for scalable second-order FL algorithms.

Abstract

This paper explores second-order optimization methods in Federated Learning (FL), addressing the critical challenges of slow convergence and the excessive communication rounds required to achieve optimal performance from the global model. While existing surveys in FL primarily focus on challenges related to statistical and device label heterogeneity, as well as privacy and security concerns in first-order FL methods, less attention has been given to the issue of slow model training. This slow training often leads to the need for excessive communication rounds or increased communication costs, particularly when data across clients are highly heterogeneous. In this paper, we examine various FL methods that leverage second-order optimization to accelerate the training process. We provide a comprehensive categorization of state-of-the-art second-order FL methods and compare their performance based on convergence speed, computational cost, memory usage, transmission overhead, and generalization of the global model. Our findings show the potential of incorporating Hessian curvature through second-order optimization into FL and highlight key challenges, such as the efficient utilization of Hessian and its inverse in FL. This work lays the groundwork for future research aimed at developing scalable and efficient federated optimization methods for improving the training of the global model in FL.