Pre-Training BERT on Arabic Tweets: Practical Considerations

TL;DR

This paper explores pretraining BERT models on Arabic tweets, emphasizing data diversity and linguistic segmentation, resulting in models that achieve state-of-the-art performance on Arabic NLP tasks.

Contribution

It introduces multiple BERT models trained on diverse Arabic social media data, demonstrating the importance of data variety and linguistic segmentation for effective NLP.

Findings

Data diversity is crucial for model performance.

Linguistically aware segmentation improves results.

More data or training steps do not always lead to better models.

Abstract

Pretraining Bidirectional Encoder Representations from Transformers (BERT) for downstream NLP tasks is a non-trival task. We pretrained 5 BERT models that differ in the size of their training sets, mixture of formal and informal Arabic, and linguistic preprocessing. All are intended to support Arabic dialects and social media. The experiments highlight the centrality of data diversity and the efficacy of linguistically aware segmentation. They also highlight that more data or more training step do not necessitate better models. Our new models achieve new state-of-the-art results on several downstream tasks. The resulting models are released to the community under the name QARiB.