Optimizing Multilingual LLMs via Federated Learning: A Study of Client Language Composition

TL;DR

This paper investigates how client language diversity impacts federated learning of multilingual large language models, proposing a new early stopping method and analyzing effects on model quality, fairness, and training efficiency.

Contribution

We extended the FederatedScope-LLM framework for multilingual instruction-tuning and introduced Local Dynamic Early Stopping (LDES-FL) to improve training efficiency and sustainability.

Findings

Monolingual fine-tuning excels for single-language tasks.

Federated training benefits from increased multilinguality for fairness and lower-resource languages.

Client language composition significantly influences model performance, fairness, and training cost.

Abstract

Federated Learning (FL) of Large Language Models (LLMs) in multilingual environments presents significant challenges stemming from heterogeneous language distributions across clients and disparities in language resource availability. To address these challenges, we extended the FederatedScope-LLM framework to support multilingual instruction-tuning experiments with LLMs. We also introduced a novel client-specific early stopping mechanism, Local Dynamic Early Stopping (LDES-FL), which allows clients to pause and resume local training based on client-side validation performance, enhancing training efficiency and sustainability. Through a series of experiments, we studied how client language composition - from fully monolingual to increasingly multilingual clients - affects multilingual quality, fairness and training cost. Monolingual local fine-tuning remains the most effective for single-language specialization, whereas federated training is better suited to learning a single balanced multilingual model. In FL, increasing within-client multilinguality leads to stronger and fairer global models, narrows the gap to centralized multilingual fine-tuning, and yields the largest gains for lower-resource languages, albeit at the cost of more optimization steps. Overall, our results identify client language composition as a key design variable in multilingual FL, shaping performance, fairness and efficiency.