FATE-LLM: A Industrial Grade Federated Learning Framework for Large Language Models

TL;DR

FATE-LLM is a federated learning framework designed for large language models, enabling privacy-preserving, resource-efficient training across multiple enterprises while protecting intellectual property.

Contribution

The paper introduces FATE-LLM, a novel federated learning framework specifically tailored for large language models, addressing resource and data privacy challenges.

Findings

Supports federated training of LLMs across enterprises

Uses parameter-efficient fine-tuning for resource savings

Ensures data privacy and intellectual property protection

Abstract

Large Language Models (LLMs), such as ChatGPT, LLaMA, GLM, and PaLM, have exhibited remarkable performances across various tasks in recent years. However, LLMs face two main challenges in real-world applications. One challenge is that training LLMs consumes vast computing resources, preventing LLMs from being adopted by small and medium-sized enterprises with limited computing resources. Another is that training LLM requires a large amount of high-quality data, which are often scattered among enterprises. To address these challenges, we propose FATE-LLM, an industrial-grade federated learning framework for large language models. FATE-LLM (1) facilitates federated learning for large language models (coined FedLLM); (2) promotes efficient training of FedLLM using parameter-efficient fine-tuning methods; (3) protects the intellectual property of LLMs; (4) preserves data privacy during training and inference through privacy-preserving mechanisms. We release the code of FATE-LLM at https://github.com/FederatedAI/FATE-LLM to facilitate the research of FedLLM and enable a broad range of industrial applications.