Building for Tomorrow: Assessing the Temporal Persistence of Text Classifiers

TL;DR

This paper investigates how well text classifiers perform over time, proposing evaluation methods to predict their temporal stability and analyzing factors affecting their longevity across diverse datasets.

Contribution

It introduces an evaluation setup for assessing the temporal persistence of text classifiers and provides empirical insights into factors influencing model longevity.

Findings

Model performance declines as temporal gap increases.

Dataset characteristics can predict model stability.

Longitudinal experiments reveal varying persistence across models.

Abstract

Performance of text classification models tends to drop over time due to changes in data, which limits the lifetime of a pretrained model. Therefore an ability to predict a model's ability to persist over time can help design models that can be effectively used over a longer period of time. In this paper, we provide a thorough discussion into the problem, establish an evaluation setup for the task. We look at this problem from a practical perspective by assessing the ability of a wide range of language models and classification algorithms to persist over time, as well as how dataset characteristics can help predict the temporal stability of different models. We perform longitudinal classification experiments on three datasets spanning between 6 and 19 years, and involving diverse tasks and types of data. By splitting the longitudinal datasets into years, we perform a comprehensive set of experiments by training and testing across data that are different numbers of years apart from each other, both in the past and in the future. This enables a gradual investigation into the impact of the temporal gap between training and test sets on the classification performance, as well as measuring the extent of the persistence over time.