TweetBERT: A Pretrained Language Representation Model for Twitter Text Analysis

TL;DR

This paper introduces TweetBERT, a domain-specific language model trained on Twitter data, which significantly outperforms traditional BERT models in Twitter text analysis tasks, demonstrating the importance of domain adaptation.

Contribution

The paper presents two TweetBERT models trained on millions of tweets, the first domain-specific language models for Twitter text analysis, outperforming general BERT models.

Findings

TweetBERT models outperform traditional BERT by over 7% on Twitter datasets.

Training on Twitter data improves performance across multiple datasets.

Extensive evaluation on 31 datasets validates the effectiveness of domain-specific training.

Abstract

Twitter is a well-known microblogging social site where users express their views and opinions in real-time. As a result, tweets tend to contain valuable information. With the advancements of deep learning in the domain of natural language processing, extracting meaningful information from tweets has become a growing interest among natural language researchers. Applying existing language representation models to extract information from Twitter does not often produce good results. Moreover, there is no existing language representation models for text analysis specific to the social media domain. Hence, in this article, we introduce two TweetBERT models, which are domain specific language presentation models, pre-trained on millions of tweets. We show that the TweetBERT models significantly outperform the traditional BERT models in Twitter text mining tasks by more than 7% on each Twitter dataset. We also provide an extensive analysis by evaluating seven BERT models on 31 different datasets. Our results validate our hypothesis that continuously training language models on twitter corpus help performance with Twitter.