Text Generation Models for Luxembourgish with Limited Data: A Balanced Multilingual Strategy

TL;DR

This paper introduces a multilingual text generation approach for Luxembourgish using limited data, leveraging German and French data to improve cross-lingual transfer, and presents LuxGen, a new benchmark for Luxembourgish language models.

Contribution

It proposes a novel T5-based multilingual model for Luxembourgish that enhances cross-lingual transfer and introduces the LuxGen benchmark for evaluation.

Findings

Multilingual training outperforms monolingual models for Luxembourgish.

The proposed model improves text generation quality in Luxembourgish.

LuxGen benchmark enables standardized evaluation for Luxembourgish models.

Abstract

This paper addresses the challenges in developing language models for less-represented languages, with a focus on Luxembourgish. Despite its active development, Luxembourgish faces a digital data scarcity, exacerbated by Luxembourg's multilingual context. We propose a novel text generation model based on the T5 architecture, combining limited Luxembourgish data with equal amounts, in terms of size and type, of German and French data. We hypothesise that a model trained on Luxembourgish, German, and French will improve the model's cross-lingual transfer learning capabilities and outperform monolingual and large multilingual models. To verify this, the study at hand explores whether multilingual or monolingual training is more beneficial for Luxembourgish language generation. For the evaluation, we introduce LuxGen, a text generation benchmark that is the first of its kind for Luxembourgish.