BERTweetFR : Domain Adaptation of Pre-Trained Language Models for French Tweets

TL;DR

BERTweetFR is a large-scale French tweet language model that outperforms previous models on Twitter NLP tasks, with newly created datasets and public availability to advance French tweet analysis.

Contribution

Introduction of BERTweetFR, the first large-scale French tweet language model, and creation of annotated datasets for offensiveness detection and NER.

Findings

BERTweetFR outperforms previous French language models on Twitter tasks.

Created and annotated a new dataset for offensiveness detection in French.

Model is publicly available for future research.

Abstract

We introduce BERTweetFR, the first large-scale pre-trained language model for French tweets. Our model is initialized using the general-domain French language model CamemBERT which follows the base architecture of RoBERTa. Experiments show that BERTweetFR outperforms all previous general-domain French language models on two downstream Twitter NLP tasks of offensiveness identification and named entity recognition. The dataset used in the offensiveness detection task is first created and annotated by our team, filling in the gap of such analytic datasets in French. We make our model publicly available in the transformers library with the aim of promoting future research in analytic tasks for French tweets.