Is It Worth the (Environmental) Cost? Limited Evidence for Temporal Adaptation via Continuous Training

TL;DR

This study evaluates whether continuous training of language models for social media data improves performance over time, finding that it often does not justify the environmental costs involved.

Contribution

The paper provides empirical evidence questioning the benefits of temporal adaptation in language models and highlights the need for suitable benchmarks and sustainable practices.

Findings

Pretrained models outperform temporally adapted models in social media tasks.

Continuous training does not significantly improve downstream task performance.

Current benchmarks are insufficient for evaluating temporal adaptation.

Abstract

Language is constantly changing and evolving, leaving language models to become quickly outdated. Consequently, we should continuously update our models with new data to expose them to new events and facts. However, that requires additional computing, which means new carbon emissions. Do any measurable benefits justify this cost? This paper looks for empirical evidence to support continuous training. We reproduce existing benchmarks and extend them to include additional time periods, models, and tasks. Our results show that the downstream task performance of temporally adapted English models for social media data do not improve over time. Pretrained models without temporal adaptation are actually significantly more effective and efficient. However, we also note a lack of suitable temporal benchmarks. Our findings invite a critical reflection on when and how to temporally adapt language models, accounting for sustainability.