A Systematic Analysis on the Temporal Generalization of Language Models in Social Media

TL;DR

This paper systematically evaluates how language models trained on social media data perform over time, revealing consistent performance drops in entity-focused tasks and showing that continued pre-training does not enhance temporal robustness.

Contribution

It introduces a unified evaluation scheme for assessing language models under temporal shifts on social media tasks, highlighting their limitations and the ineffectiveness of continuous pre-training.

Findings

Performance drops under temporal shift are consistent for entity-focused tasks.

No significant performance decline observed in topic and sentiment classification.

Continuous pre-training does not improve models' temporal adaptability.

Abstract

In machine learning, temporal shifts occur when there are differences between training and test splits in terms of time. For streaming data such as news or social media, models are commonly trained on a fixed corpus from a certain period of time, and they can become obsolete due to the dynamism and evolving nature of online content. This paper focuses on temporal shifts in social media and, in particular, Twitter. We propose a unified evaluation scheme to assess the performance of language models (LMs) under temporal shift on standard social media tasks. LMs are tested on five diverse social media NLP tasks under different temporal settings, which revealed two important findings: (i) the decrease in performance under temporal shift is consistent across different models for entity-focused tasks such as named entity recognition or disambiguation, and hate speech detection, but not significant in the other tasks analysed (i.e., topic and sentiment classification); and (ii) continuous pre-training on the test period does not improve the temporal adaptability of LMs.