Majority Rules: LLM Ensemble is a Winning Approach for Content Categorization

TL;DR

This paper presents an ensemble framework for large language models that significantly improves content categorization accuracy and robustness, approaching human-level performance by combining multiple models to overcome individual weaknesses.

Contribution

The study introduces a formal ensemble method for LLMs in text categorization, demonstrating substantial performance gains and robustness improvements over single models.

Findings

eLLM improves F1-score by up to 65% over single models

eLLM achieves near human-expert-level accuracy

Ensemble approach reduces reliance on human labeling

Abstract

This study introduces an ensemble framework for unstructured text categorization using large language models (LLMs). By integrating multiple models, the ensemble large language model (eLLM) framework addresses common weaknesses of individual systems, including inconsistency, hallucination, category inflation, and misclassification. The eLLM approach yields a substantial performance improvement of up to 65\% in F1-score over the strongest single model. We formalize the ensemble process through a mathematical model of collective decision-making and establish principled aggregation criteria. Using the Interactive Advertising Bureau (IAB) hierarchical taxonomy, we evaluate ten state-of-the-art LLMs under identical zero-shot conditions on a human-annotated corpus of 8{,}660 samples. Results show that individual models plateau in performance due to the compression of semantically rich text into sparse categorical representations, while eLLM improves both robustness and accuracy. With a diverse consortium of models, eLLM achieves near human-expert-level performance, offering a scalable and reliable solution for taxonomy-based classification that may significantly reduce dependence on human expert labeling.