LLM-as-classifier: Semi-Supervised, Iterative Framework for Hierarchical Text Classification using Large Language Models

TL;DR

This paper presents a semi-supervised, iterative framework leveraging Large Language Models for hierarchical text classification, addressing industry challenges of robustness, scalability, and adaptability in real-world applications.

Contribution

It introduces a novel human-in-the-loop, semi-supervised approach that combines prompt refinement and hierarchical expansion for scalable, adaptable classification systems using LLMs.

Findings

Effective in reducing resource requirements compared to fine-tuning

Improves classification accuracy with iterative prompt refinement

Supports continuous monitoring and adaptation in production

Abstract

The advent of Large Language Models (LLMs) has provided unprecedented capabilities for analyzing unstructured text data. However, deploying these models as reliable, robust, and scalable classifiers in production environments presents significant methodological challenges. Standard fine-tuning approaches can be resource-intensive and often struggle with the dynamic nature of real-world data distributions, which is common in the industry. In this paper, we propose a comprehensive, semi-supervised framework that leverages the zero- and few-shot capabilities of LLMs for building hierarchical text classifiers as a framework for a solution to these industry-wide challenges. Our methodology emphasizes an iterative, human-in-the-loop process that begins with domain knowledge elicitation and progresses through prompt refinement, hierarchical expansion, and multi-faceted validation. We introduce techniques for assessing and mitigating sequence-based biases and outline a protocol for continuous monitoring and adaptation. This framework is designed to bridge the gap between the raw power of LLMs and the practical need for accurate, interpretable, and maintainable classification systems in industry applications.