BabyLMs for isiXhosa: Data-Efficient Language Modelling in a Low-Resource Context

TL;DR

This paper demonstrates that data-efficient language models like BabyLMs can effectively learn low-resource languages such as isiXhosa, outperforming larger models in certain NLP tasks despite limited training data.

Contribution

It is the first to evaluate BabyLM architectures on isiXhosa, showing their potential for low-resource language modeling and highlighting data quality issues.

Findings

BabyLMs outperform vanilla models on POS tagging and NER.

BabyLMs sometimes outperform XLM-R in low-resource settings.

Data quality remains a critical challenge for low-resource language models.

Abstract

The BabyLM challenge called on participants to develop sample-efficient language models. Submissions were pretrained on a fixed English corpus, limited to the amount of words children are exposed to in development (<100m). The challenge produced new architectures for data-efficient language modelling, which outperformed models trained on trillions of words. This is promising for low-resource languages, where available corpora are limited to much less than 100m words. In this paper, we explore the potential of BabyLMs for low-resource languages, using the isiXhosa language as a case study. We pretrain two BabyLM architectures, ELC-BERT and MLSM, on an isiXhosa corpus. They outperform a vanilla pretrained model on POS tagging and NER, achieving notable gains (+3.2 F1) for the latter. In some instances, the BabyLMs even outperform XLM-R. Our findings show that data-efficient models are viable for low-resource languages, but highlight the continued importance, and lack of, high-quality pretraining data. Finally, we visually analyse how BabyLM architectures encode isiXhosa.