Boosting classification reliability of NLP transformer models in the long run

TL;DR

This study evaluates methods for maintaining NLP transformer model performance over time in dynamic environments, emphasizing the importance of continuous fine-tuning and annotation to sustain classification reliability.

Contribution

It compares different fine-tuning strategies for BERT in long-term classification tasks and demonstrates the effectiveness of using all available unlabeled data for model updates.

Findings

Using all available unlabeled comments improves model performance.

Random sampling from new data is more effective than focusing only on unseen words.

Regular annotation and fine-tuning are necessary to prevent performance degradation.

Abstract

Transformer-based machine learning models have become an essential tool for many natural language processing (NLP) tasks since the introduction of the method. A common objective of these projects is to classify text data. Classification models are often extended to a different topic and/or time period. In these situations, deciding how long a classification is suitable for and when it is worth re-training our model is difficult. This paper compares different approaches to fine-tune a BERT model for a long-running classification task. We use data from different periods to fine-tune our original BERT model, and we also measure how a second round of annotation could boost the classification quality. Our corpus contains over 8 million comments on COVID-19 vaccination in Hungary posted between September 2020 and December 2021. Our results show that the best solution is using all available unlabeled comments to fine-tune a model. It is not advisable to focus only on comments containing words that our model has not encountered before; a more efficient solution is randomly sample comments from the new period. Fine-tuning does not prevent the model from losing performance but merely slows it down. In a rapidly changing linguistic environment, it is not possible to maintain model performance without regularly annotating new text.