Named Entity Recognition in Twitter: A Dataset and Analysis on Short-Term Temporal Shifts

TL;DR

This paper introduces TweetNER7, a new Twitter NER dataset covering diverse temporal trends, and analyzes how language models' NER performance degrades over time, proposing fine-tuning and self-labeling strategies to mitigate this issue.

Contribution

The paper presents a novel Twitter NER dataset with temporal annotations and analyzes temporal effects on model performance, offering strategies to adapt models over time.

Findings

NER performance degrades over short-term periods

Fine-tuning improves temporal robustness of models

Self-labeling can compensate for lack of recent annotations

Abstract

Recent progress in language model pre-training has led to important improvements in Named Entity Recognition (NER). Nonetheless, this progress has been mainly tested in well-formatted documents such as news, Wikipedia, or scientific articles. In social media the landscape is different, in which it adds another layer of complexity due to its noisy and dynamic nature. In this paper, we focus on NER in Twitter, one of the largest social media platforms, and construct a new NER dataset, TweetNER7, which contains seven entity types annotated over 11,382 tweets from September 2019 to August 2021. The dataset was constructed by carefully distributing the tweets over time and taking representative trends as a basis. Along with the dataset, we provide a set of language model baselines and perform an analysis on the language model performance on the task, especially analyzing the impact of different time periods. In particular, we focus on three important temporal aspects in our analysis: short-term degradation of NER models over time, strategies to fine-tune a language model over different periods, and self-labeling as an alternative to lack of recently-labeled data. TweetNER7 is released publicly (https://huggingface.co/datasets/tner/tweetner7) along with the models fine-tuned on it.