Enhancing Text Classification through LLM-Driven Active Learning and Human Annotation

TL;DR

This paper presents a novel active learning framework that combines human annotators and LLMs like GPT-3.5 to reduce annotation costs while maintaining or improving text classification accuracy across multiple datasets.

Contribution

It introduces an integrated methodology that balances human and LLM annotations based on uncertainty, enhancing cost efficiency and performance in text classification.

Findings

Significant reduction in annotation costs.

Maintained or improved classification accuracy.

Effective integration of human and LLM annotations.

Abstract

In the context of text classification, the financial burden of annotation exercises for creating training data is a critical issue. Active learning techniques, particularly those rooted in uncertainty sampling, offer a cost-effective solution by pinpointing the most instructive samples for manual annotation. Similarly, Large Language Models (LLMs) such as GPT-3.5 provide an alternative for automated annotation but come with concerns regarding their reliability. This study introduces a novel methodology that integrates human annotators and LLMs within an Active Learning framework. We conducted evaluations on three public datasets. IMDB for sentiment analysis, a Fake News dataset for authenticity discernment, and a Movie Genres dataset for multi-label classification.The proposed framework integrates human annotation with the output of LLMs, depending on the model uncertainty levels. This strategy achieves an optimal balance between cost efficiency and classification performance. The empirical results show a substantial decrease in the costs associated with data annotation while either maintaining or improving model accuracy.