Tensor Decomposition based Personalized Federated Learning

TL;DR

This paper introduces TDPFed, a personalized federated learning framework utilizing tensor decomposition to reduce communication costs and improve scalability for diverse data and large models.

Contribution

The paper proposes a novel tensorized local model and bi-level loss function for personalized federated learning, enhancing scalability and communication efficiency.

Findings

Achieves state-of-the-art performance in personalized FL tasks.

Reduces communication cost through tensorized model design.

Demonstrates effective convergence and scalability in experiments.

Abstract

Federated learning (FL) is a new distributed machine learning framework that can achieve reliably collaborative training without collecting users' private data. However, due to FL's frequent communication and average aggregation strategy, they experience challenges scaling to statistical diversity data and large-scale models. In this paper, we propose a personalized FL framework, named Tensor Decomposition based Personalized Federated learning (TDPFed), in which we design a novel tensorized local model with tensorized linear layers and convolutional layers to reduce the communication cost. TDPFed uses a bi-level loss function to decouple personalized model optimization from the global model learning by controlling the gap between the personalized model and the tensorized local model. Moreover, an effective distributed learning strategy and two different model aggregation strategies are well designed for the proposed TDPFed framework. Theoretical convergence analysis and thorough experiments demonstrate that our proposed TDPFed framework achieves state-of-the-art performance while reducing the communication cost.