pFedSOP : Accelerating Training Of Personalized Federated Learning Using Second-Order Optimization

TL;DR

This paper introduces pFedSOP, a second-order optimization method for personalized federated learning that accelerates training and reduces communication rounds by efficiently approximating the Hessian with Fisher Information Matrices.

Contribution

The paper proposes a novel second-order optimization approach for PFL that uses Fisher Information Matrices to approximate the Hessian, improving training speed and personalization performance.

Findings

pFedSOP reduces communication rounds compared to state-of-the-art methods.

It achieves faster convergence in heterogeneous data settings.

Outperforms existing algorithms on image classification datasets.

Abstract

Personalized Federated Learning (PFL) enables clients to collaboratively train personalized models tailored to their individual objectives, addressing the challenge of model generalization in traditional Federated Learning (FL) due to high data heterogeneity. However, existing PFL methods often require increased communication rounds to achieve the desired performance, primarily due to slow training caused by the use of first-order optimization, which has linear convergence. Additionally, many of these methods increase local computation because of the additional data fed into the model during the search for personalized local models. One promising solution to this slow training is second-order optimization, known for its quadratic convergence. However, employing it in PFL is challenging due to the Hessian matrix and its inverse. In this paper, we propose pFedSOP, which efficiently utilizes second-order optimization in PFL to accelerate the training of personalized models and enhance performance with fewer communication rounds. Our approach first computes a personalized local gradient update using the Gompertz function-based normalized angle between local and global gradient updates, incorporating client-specific global information. We then use a regularized Fisher Information Matrix (FIM), computed from this personalized gradient update, as an approximation of the Hessian to update the personalized models. This FIM-based second-order optimization speeds up training with fewer communication rounds by tackling the challenges with exact Hessian and avoids additional data being fed into the model during the search for personalized local models. Extensive experiments on heterogeneously partitioned image classification datasets with partial client participation demonstrate that pFedSOP outperforms state-of-the-art FL and PFL algorithms.