FlowerTune: A Cross-Domain Benchmark for Federated Fine-Tuning of Large Language Models

TL;DR

This paper introduces FlowerTune, a comprehensive benchmark suite for evaluating federated fine-tuning of large language models across multiple domains, addressing data privacy and domain adaptation challenges.

Contribution

It presents the first benchmarking suite for federated fine-tuning of LLMs across diverse domains, including datasets, evaluation metrics, and a comparative analysis of 26 models.

Findings

Federated fine-tuning performance varies across models and domains.

Resource constraints significantly impact federated LLM training.

Domain-specific adaptation improves model effectiveness.

Abstract

Large Language Models (LLMs) have achieved state-of-the-art results across diverse domains, yet their development remains reliant on vast amounts of publicly available data, raising concerns about data scarcity and the lack of access to domain-specific, sensitive information. Federated Learning (FL) presents a compelling framework to address these challenges by enabling decentralized fine-tuning on pre-trained LLMs without sharing raw data. However, the compatibility and performance of pre-trained LLMs in FL settings remain largely under explored. We introduce the FlowerTune LLM Leaderboard, a first-of-its-kind benchmarking suite designed to evaluate federated fine-tuning of LLMs across four diverse domains: general NLP, finance, medical, and coding. Each domain includes federated instruction-tuning datasets and domain-specific evaluation metrics. Our results, obtained through a collaborative, open-source and community-driven approach, provide the first comprehensive comparison across 26 pre-trained LLMs with different aggregation and fine-tuning strategies under federated settings, offering actionable insights into model performance, resource constraints, and domain adaptation. This work lays the foundation for developing privacy-preserving, domain-specialized LLMs for real-world applications.