TimeLMs: Diachronic Language Models from Twitter

TL;DR

TimeLMs are specialized diachronic Twitter language models that leverage continual learning to better handle evolving language, trends, and out-of-distribution data, improving robustness and adaptability in social media NLP tasks.

Contribution

This paper introduces TimeLMs, a novel approach using continual learning for diachronic Twitter language modeling, addressing the gap of temporal dynamics in NLP.

Findings

Continual learning enhances Twitter language models' ability to handle future data.

TimeLMs perform competitively with standard benchmarks.

Qualitative analyses reveal how models adapt to trends and concept drift.

Abstract

Despite its importance, the time variable has been largely neglected in the NLP and language model literature. In this paper, we present TimeLMs, a set of language models specialized on diachronic Twitter data. We show that a continual learning strategy contributes to enhancing Twitter-based language models' capacity to deal with future and out-of-distribution tweets, while making them competitive with standardized and more monolithic benchmarks. We also perform a number of qualitative analyses showing how they cope with trends and peaks in activity involving specific named entities or concept drift.