Evolving Domain Adaptation of Pretrained Language Models for Text Classification

TL;DR

This paper evaluates evolving domain adaptation strategies for pre-trained language models in text classification, emphasizing incremental self-training to handle temporal domain shifts effectively.

Contribution

It introduces and benchmarks an incremental self-training method for adapting PLMs to evolving domains, demonstrating its superiority over traditional techniques.

Findings

Incremental self-training outperforms traditional domain adaptation methods.

Evolving domain adaptation significantly improves PLM performance over time.

Continual updating of PLMs is crucial for real-world, time-sensitive applications.

Abstract

Adapting pre-trained language models (PLMs) for time-series text classification amidst evolving domain shifts (EDS) is critical for maintaining accuracy in applications like stance detection. This study benchmarks the effectiveness of evolving domain adaptation (EDA) strategies, notably self-training, domain-adversarial training, and domain-adaptive pretraining, with a focus on an incremental self-training method. Our analysis across various datasets reveals that this incremental method excels at adapting PLMs to EDS, outperforming traditional domain adaptation techniques. These findings highlight the importance of continually updating PLMs to ensure their effectiveness in real-world applications, paving the way for future research into PLM robustness against the natural temporal evolution of language.