FDAPT: Federated Domain-adaptive Pre-training for Language Models

TL;DR

This paper introduces FDAPT, a federated approach to domain-adaptive pre-training of language models, demonstrating competitive performance with centralized methods and proposing an efficient variant, FFDAPT, that reduces computational costs.

Contribution

It provides the first comprehensive empirical evaluation of federated domain-adaptive pre-training and proposes a novel, more efficient algorithm, FFDAPT, for this task.

Findings

FDAPT maintains competitive downstream performance in IID and non-IID settings.

FFDAPT improves computational efficiency by 12.1% on average.

Performance fluctuations of FFDAPT are less than 1% compared to FDAPT.

Abstract

Foundation models (FMs) have shown prominent success in a wide range of tasks. Their applicability to specific domain-task pairings relies on the availability of, both, high-quality data and significant computational resources. These challenges are not new to the field and, indeed, Federated Learning (FL) has been shown to be a promising solution in similar setups. This paper tackles the specific case of Domain-Adaptive Pre-Training (DAPT), a key step in the application of FMs. We conduct the first comprehensive empirical study to evaluate the performance of Federated Domain-Adaptive Pre-Training (FDAPT). We demonstrate that FDAPT can maintain competitive downstream task performance to the centralized baseline in both IID and non-IID situations. Finally, we propose a novel algorithm, Frozen Federated Domain-Adaptive Pre-Training (FFDAPT). FFDAPT improves the computational efficiency by 12.1% on average and exhibits similar downstream task performance to vanilla FDAPT, with general performance fluctuations remaining less than 1%.