A Survey on Efficient Federated Learning Methods for Foundation Model Training

TL;DR

This survey reviews federated learning techniques tailored for foundation models, emphasizing efficiency, privacy, and the potential of parameter-efficient fine-tuning in collaborative, privacy-preserving AI training.

Contribution

It introduces a new taxonomy for efficient federated learning with foundation models, focusing on computational and communication aspects, and discusses future research directions.

Findings

PEFT methods offer benefits for FL applications.

FL frameworks are increasingly compatible with foundation models.

Future research includes evaluating generative models and privacy-PEFT interplay.

Abstract

Federated Learning (FL) has become an established technique to facilitate privacy-preserving collaborative training across a multitude of clients. However, new approaches to FL often discuss their contributions involving small deep-learning models only and focus on training full models on clients. In the wake of Foundation Models (FM), the reality is different for many deep learning applications. Typically, FMs have already been pre-trained across a wide variety of tasks and can be fine-tuned to specific downstream tasks over significantly smaller datasets than required for full model training. However, access to such datasets is often challenging. By its design, FL can help to open data silos. With this survey, we introduce a novel taxonomy focused on computational and communication efficiency, the vital elements to make use of FMs in FL systems. We discuss the benefits and drawbacks of parameter-efficient fine-tuning (PEFT) for FL applications, elaborate on the readiness of FL frameworks to work with FMs, and provide future research opportunities on how to evaluate generative models in FL as well as the interplay of privacy and PEFT.