Scalable and Resource-Efficient Second-Order Federated Learning via Over-the-Air Aggregation

TL;DR

This paper introduces a scalable second-order federated learning method that uses sparse Hessian estimates and over-the-air aggregation to reduce communication and energy costs for large models.

Contribution

It presents a novel second-order FL algorithm combining sparsity and over-the-air techniques to improve scalability and efficiency over existing methods.

Findings

Achieves over 67% reduction in communication resources and energy consumption.

Enables scalable second-order FL for large models.

Demonstrates superior efficiency compared to first and second-order baselines.

Abstract

Second-order federated learning (FL) algorithms offer faster convergence than their first-order counterparts by leveraging curvature information. However, they are hindered by high computational and storage costs, particularly for large-scale models. Furthermore, the communication overhead associated with large models and digital transmission exacerbates these challenges, causing communication bottlenecks. In this work, we propose a scalable second-order FL algorithm using a sparse Hessian estimate and leveraging over-the-air aggregation, making it feasible for larger models. Our simulation results demonstrate more than $67\%$ of communication resources and energy savings compared to other first and second-order baselines.