Can Character-based Language Models Improve Downstream Task Performance in Low-Resource and Noisy Language Scenarios?

TL;DR

This paper demonstrates that character-based language models trained on limited, noisy, low-resource language data can achieve competitive performance on downstream NLP tasks, offering a promising approach for underrepresented languages.

Contribution

It introduces a character-based language model approach for low-resource, high-variability languages and shows its effectiveness compared to larger pre-trained models.

Findings

Character-based models perform well with limited data.

Models trained on 99k sentences achieve near state-of-the-art results.

Effective on both North-African dialectal Arabic and noisy French data.

Abstract

Recent impressive improvements in NLP, largely based on the success of contextual neural language models, have been mostly demonstrated on at most a couple dozen high-resource languages. Building language models and, more generally, NLP systems for non-standardized and low-resource languages remains a challenging task. In this work, we focus on North-African colloquial dialectal Arabic written using an extension of the Latin script, called NArabizi, found mostly on social media and messaging communication. In this low-resource scenario with data displaying a high level of variability, we compare the downstream performance of a character-based language model on part-of-speech tagging and dependency parsing to that of monolingual and multilingual models. We show that a character-based model trained on only 99k sentences of NArabizi and fined-tuned on a small treebank of this language leads to performance close to those obtained with the same architecture pre-trained on large multilingual and monolingual models. Confirming these results a on much larger data set of noisy French user-generated content, we argue that such character-based language models can be an asset for NLP in low-resource and high language variability set-tings.