BERnaT: Basque Encoders for Representing Natural Textual Diversity

TL;DR

This paper emphasizes the importance of linguistic diversity in language models, introduces new Basque corpora and models capturing diverse language varieties, and demonstrates improved performance on varied NLU tasks.

Contribution

It constructs new Basque corpora and trains models on diverse language data, enhancing linguistic generalization and inclusivity in language modeling.

Findings

Models trained on diverse data outperform standard-only models.

Diverse training improves performance across all NLU tasks.

Standard benchmark accuracy remains unaffected.

Abstract

Language models depend on massive text corpora that are often filtered for quality, a process that can unintentionally exclude non-standard linguistic varieties, reduce model robustness and reinforce representational biases. In this paper, we argue that language models should aim to capture the full spectrum of language variation (dialectal, historical, informal, etc.) rather than relying solely on standardized text. Focusing on the Basque language, we construct new corpora combining standard, social media, and historical sources, and pre-train the BERnaT family of encoder-only models in three configurations: standard, diverse, and combined. We further propose an evaluation framework that separates Natural Language Understanding (NLU) tasks into standard and diverse subsets to assess linguistic generalization. Results show that models trained on both standard and diverse data consistently outperform those trained on standard corpora, improving performance across all task types without compromising standard benchmark accuracy. These findings highlight the importance of linguistic diversity in building inclusive, generalizable language models.