GP-FL: Model-Based Hessian Estimation for Second-Order Over-the-Air Federated Learning

TL;DR

This paper presents GP-FL, a novel second-order federated learning framework for wireless channels that estimates the global Hessian directly from noisy gradients using Gaussian process modeling, achieving faster convergence.

Contribution

Introduces a Gaussian process-based Hessian estimation method for over-the-air FL, enabling effective second-order optimization with noisy gradient data.

Findings

GP-FL achieves linear-quadratic convergence rate.

Outperforms classical first and second order FL methods.

Effective Hessian estimation from noisy gradients in wireless channels.

Abstract

Second-order methods are widely adopted to improve the convergence rate of learning algorithms. In federated learning (FL), these methods require the clients to share their local Hessian matrices with the parameter server (PS), which comes at a prohibitive communication cost. A classical solution to this issue is to approximate the global Hessian matrix from the first-order information. Unlike in idealized networks, this solution does not perform effectively in over-the-air FL settings, where the PS receives noisy versions of the local gradients. This paper introduces a novel second-order FL framework tailored for wireless channels. The pivotal innovation lies in the PS's capability to directly estimate the global Hessian matrix from the received noisy local gradients via a non-parametric method: the PS models the unknown Hessian matrix as a Gaussian process, and then uses the temporal relation between the gradients and Hessian along with the channel model to find a stochastic estimator for the global Hessian matrix. We refer to this method as Gaussian process-based Hessian modeling for wireless FL (GP-FL) and show that it exhibits a linear-quadratic convergence rate. Numerical experiments on various datasets demonstrate that GP-FL outperforms all classical baseline first and second order FL approaches.