Large Language Models in the Task of Automatic Validation of Text Classifier Predictions

TL;DR

This paper explores using large language models to automatically validate text classifier predictions, aiming to reduce reliance on human annotators and lower costs in model training and updating processes.

Contribution

It introduces novel methods leveraging LLMs to replace human validation in text classification, addressing cost and scalability issues in ongoing model maintenance.

Findings

LLMs can effectively validate classifier predictions

Proposed methods reduce the need for human annotation

Improves efficiency in incremental learning pipelines

Abstract

Machine learning models for text classification are trained to predict a class for a given text. To do this, training and validation samples must be prepared: a set of texts is collected, and each text is assigned a class. These classes are usually assigned by human annotators with different expertise levels, depending on the specific classification task. Collecting such samples from scratch is labor-intensive because it requires finding specialists and compensating them for their work; moreover, the number of available specialists is limited, and their productivity is constrained by human factors. While it may not be too resource-intensive to collect samples once, the ongoing need to retrain models (especially in incremental learning pipelines) to address data drift (also called model drift) makes the data collection process crucial and costly over the model's entire lifecycle. This paper proposes several approaches to replace human annotators with Large Language Models (LLMs) to test classifier predictions for correctness, helping ensure model quality and support high-quality incremental learning.