Is Biomedical Specialization Still Worth It? Insights from Domain-Adaptive Language Modelling with a New French Health Corpus

TL;DR

This paper evaluates the effectiveness of domain-adaptive pre-training for French biomedical language models, revealing limited benefits of DAPT but highlighting its potential in resource-constrained settings and the importance of model merging.

Contribution

It introduces a new French biomedical corpus, trains specialized LLMs, and provides insights into DAPT's viability and implementation challenges in non-English biomedical NLP.

Findings

DAPT's efficacy is questionable compared to prior studies.

Model merging after DAPT can mitigate generalization trade-offs.

In some cases, DAPT improves performance on targeted specialized tasks.

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities across diverse domains, yet their adaptation to specialized fields remains challenging, particularly for non-English languages. This study investigates domain-adaptive pre-training (DAPT) as a strategy for specializing small to mid-sized LLMs in the French biomedical domain through continued pre-training. We address two key research questions: the viability of specialized continued pre-training for domain adaptation and the relationship between domain-specific performance gains and general capability degradation. Our contributions include the release of a fully open-licensed French biomedical corpus suitable for commercial and open-source applications, the training and release of specialized French biomedical LLMs, and novel insights for DAPT implementation. Our methodology encompasses the collection and refinement of high-quality French biomedical texts, the exploration of causal language modeling approaches using DAPT, and conducting extensive comparative evaluations. Our results cast doubt on the efficacy of DAPT, in contrast to previous works, but we highlight its viability in smaller-scale, resource-constrained scenarios under the right conditions. Findings in this paper further suggest that model merging post-DAPT is essential to mitigate generalization trade-offs, and in some cases even improves performance on specialized tasks at which the DAPT was directed.