Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media

TL;DR

This study explores how adapting BERT to temporal changes in social media language over three years affects downstream document classification, revealing that temporal adaptation benefits upstream tasks but has limited impact on classification performance.

Contribution

Introduces a temporal social media corpus and evaluates the effects of temporal adaptation on BERT's downstream classification, highlighting when it is beneficial.

Findings

Temporal adaptation improves upstream masked language modeling.

Time-specific models perform better on past data.

Temporal adaptation does not significantly enhance downstream classification.

Abstract

Language use differs between domains and even within a domain, language use changes over time. For pre-trained language models like BERT, domain adaptation through continued pre-training has been shown to improve performance on in-domain downstream tasks. In this article, we investigate whether temporal adaptation can bring additional benefits. For this purpose, we introduce a corpus of social media comments sampled over three years. It contains unlabelled data for adaptation and evaluation on an upstream masked language modelling task as well as labelled data for fine-tuning and evaluation on a downstream document classification task. We find that temporality matters for both tasks: temporal adaptation improves upstream and temporal fine-tuning downstream task performance. Time-specific models generally perform better on past than on future test sets, which matches evidence on the bursty usage of topical words. However, adapting BERT to time and domain does not improve performance on the downstream task over only adapting to domain. Token-level analysis shows that temporal adaptation captures event-driven changes in language use in the downstream task, but not those changes that are actually relevant to task performance. Based on our findings, we discuss when temporal adaptation may be more effective.