Federated Learning with Partial Model Personalization

TL;DR

This paper analyzes two federated learning algorithms for training models with partial personalization, providing convergence guarantees and empirical evidence that partial personalization with alternating updates offers effective performance with fewer personal parameters.

Contribution

It offers the first convergence analysis of the alternating partial personalization algorithm in nonconvex federated learning and compares it with simultaneous updates.

Findings

Partial personalization achieves most benefits with fewer parameters.

Alternating update often slightly outperforms simultaneous update.

Empirical results on real datasets support theoretical insights.

Abstract

We consider two federated learning algorithms for training partially personalized models, where the shared and personal parameters are updated either simultaneously or alternately on the devices. Both algorithms have been proposed in the literature, but their convergence properties are not fully understood, especially for the alternating variant. We provide convergence analyses of both algorithms in the general nonconvex setting with partial participation and delineate the regime where one dominates the other. Our experiments on real-world image, text, and speech datasets demonstrate that (a) partial personalization can obtain most of the benefits of full model personalization with a small fraction of personal parameters, and, (b) the alternating update algorithm often outperforms the simultaneous update algorithm by a small but consistent margin.