Federated Customization of Large Models: Approaches, Experiments, and Insights

TL;DR

This paper reviews federated customization techniques for large models, introduces federated prefix-tuning through experiments, and compares its performance and efficiency with other methods within the federated learning framework.

Contribution

It is the first to experimentally apply and evaluate federated prefix-tuning, demonstrating its feasibility and competitive performance compared to other methods.

Findings

Federated prefix-tuning achieves performance close to centralized methods.

Federated prefix-tuning demonstrates satisfactory efficiency and robustness.

Comparison shows federated prefix-tuning is competitive with other customization techniques.

Abstract

In this article, we explore federated customization of large models and highlight the key challenges it poses within the federated learning framework. We review several popular large model customization techniques, including full fine-tuning, efficient fine-tuning, prompt engineering, prefix-tuning, knowledge distillation, and retrieval-augmented generation. Then, we discuss how these techniques can be implemented within the federated learning framework. Moreover, we conduct experiments on federated prefix-tuning, which, to the best of our knowledge, is the first trial to apply prefix-tuning in the federated learning setting. The conducted experiments validate its feasibility with performance close to centralized approaches. Further comparison with three other federated customization methods demonstrated its competitive performance, satisfactory efficiency, and consistent robustness.