TunBERT: Pretrained Contextualized Text Representation for Tunisian Dialect

TL;DR

This paper introduces TunBERT, a Transformer-based language model trained on noisy web data for Tunisian dialect, achieving state-of-the-art results in sentiment analysis, dialect identification, and reading comprehension.

Contribution

It is the first to train a monolingual Transformer model for Tunisian dialect using web crawled data, demonstrating competitive performance with smaller datasets.

Findings

Noisy web data is more effective than structured data for dialect modeling.

Small web datasets can achieve performance comparable to larger datasets.

TunBERT outperforms previous models in all evaluated tasks.

Abstract

Pretrained contextualized text representation models learn an effective representation of a natural language to make it machine understandable. After the breakthrough of the attention mechanism, a new generation of pretrained models have been proposed achieving good performances since the introduction of the Transformer. Bidirectional Encoder Representations from Transformers (BERT) has become the state-of-the-art model for language understanding. Despite their success, most of the available models have been trained on Indo-European languages however similar research for under-represented languages and dialects remains sparse. In this paper, we investigate the feasibility of training monolingual Transformer-based language models for under represented languages, with a specific focus on the Tunisian dialect. We evaluate our language model on sentiment analysis task, dialect identification task and reading comprehension question-answering task. We show that the use of noisy web crawled data instead of structured data (Wikipedia, articles, etc.) is more convenient for such non-standardized language. Moreover, results indicate that a relatively small web crawled dataset leads to performances that are as good as those obtained using larger datasets. Finally, our best performing TunBERT model reaches or improves the state-of-the-art in all three downstream tasks. We release the TunBERT pretrained model and the datasets used for fine-tuning.