Hierarchical Text Classification Using Black Box Large Language Models

TL;DR

This paper investigates using black box large language models with different prompting strategies for hierarchical text classification, demonstrating their potential to outperform traditional models in deep hierarchies while highlighting cost-performance trade-offs.

Contribution

It introduces and evaluates three novel prompting strategies for LLMs in hierarchical text classification, showing their effectiveness and cost implications compared to traditional methods.

Findings

Few-shot learning improves accuracy over zero-shot.

Deep hierarchies benefit more from LLMs than shallow ones.

Prompt strategy choice affects both accuracy and API cost.

Abstract

Hierarchical Text Classification (HTC) aims to assign texts to structured label hierarchies; however, it faces challenges due to data scarcity and model complexity. This study explores the feasibility of using black box Large Language Models (LLMs) accessed via APIs for HTC, as an alternative to traditional machine learning methods that require extensive labeled data and computational resources. We evaluate three prompting strategies -- Direct Leaf Label Prediction (DL), Direct Hierarchical Label Prediction (DH), and Top-down Multi-step Hierarchical Label Prediction (TMH) -- in both zero-shot and few-shot settings, comparing the accuracy and cost-effectiveness of these strategies. Experiments on two datasets show that a few-shot setting consistently improves classification accuracy compared to a zero-shot setting. While a traditional machine learning model achieves high accuracy on a dataset with a shallow hierarchy, LLMs, especially DH strategy, tend to outperform the machine learning model on a dataset with a deeper hierarchy. API costs increase significantly due to the higher input tokens required for deeper label hierarchies on DH strategy. These results emphasize the trade-off between accuracy improvement and the computational cost of prompt strategy. These findings highlight the potential of black box LLMs for HTC while underscoring the need to carefully select a prompt strategy to balance performance and cost.