Personalized Federated Learning with Residual Fisher Information for Medical Image Segmentation

TL;DR

This paper introduces pFL-ResFIM, a personalized federated learning framework for medical image segmentation that uses a novel metric to adaptively personalize models at the parameter level, improving performance across heterogeneous clients.

Contribution

The paper proposes a new metric, Residual Fisher Information Matrix, and a spectral transfer strategy to enable client-adaptive personalization in federated learning for medical imaging.

Findings

pFL-ResFIM outperforms existing methods on public datasets.

The method effectively distinguishes domain-sensitive and domain-invariant parameters.

Personalized models improve segmentation accuracy across diverse clients.

Abstract

Federated learning enables multiple clients (institutions) to collaboratively train machine learning models without sharing their private data. To address the challenge of data heterogeneity across clients, personalized federated learning (pFL) aims to learn customized models for each client. In this work, we propose pFL-ResFIM, a novel pFL framework that achieves client-adaptive personalization at the parameter level. Specifically, we introduce a new metric, Residual Fisher Information Matrix (ResFIM), to quantify the sensitivity of model parameters to domain discrepancies. To estimate ResFIM for each client model under privacy constraints, we employ a spectral transfer strategy that generates simulated data reflecting the domain styles of different clients. Based on the estimated ResFIM, we partition model parameters into domain-sensitive and domain-invariant components. A personalized model for each client is then constructed by aggregating only the domain-invariant parameters on the server. Extensive experiments on public datasets demonstrate that pFL-ResFIM consistently outperforms state-of-the-art methods, validating its effectiveness.