Low-Resource Dialect Adaptation of Large Language Models: A French Dialect Case-Study

TL;DR

This study demonstrates that cost-effective continual pre-training with parameter-efficient fine-tuning can adapt large language models to low-resource dialects like Quebec French, improving minority dialect benchmarks with minimal impact on high-resource language performance.

Contribution

The paper introduces a practical approach using LoRA and CPT to adapt LLMs to low-resource dialects, providing new benchmarks and releasing Quebec French LLMs for wider access.

Findings

CPT improves dialect benchmarks with only 1% parameter updates.

Corpus composition significantly affects adaptation success.

Releasing Quebec French LLMs enhances accessibility for minority language communities.

Abstract

Despite the widespread adoption of Large Language Models (LLMs), their strongest capabilities remain largely confined to a small number of high-resource languages for which there is abundant training data. Recently, continual pre-training (CPT) has emerged as a means to fine-tune these models to low-resource regional dialects. In this paper, we study the use of CPT for dialect learning under tight data and compute budgets. Using low-rank adaptation (LoRA) and compute-efficient continual pre-training, we adapt three LLMs to the Qu\'ebec French dialect using a very small dataset and benchmark them on the COLE suite. Our experiments demonstrate an improvement on the minority dialect benchmarks with minimal regression on the prestige language benchmarks with around 1% of model parameters updated. Analysis of the results demonstrate that gains are highly contingent on corpus composition. These findings indicate that CPT with parameter-efficient fine-tuning (PEFT) can narrow the dialect gap by providing cost-effective and sustainable language resource creation, expanding high-quality LLM access to minority linguistic communities. To support reproducibility and broaden access, we release the first Qu\'ebec French LLMs on Hugging Face.