Reliable Decision Support with LLMs: A Framework for Evaluating Consistency in Binary Text Classification Applications

TL;DR

This paper presents a framework for evaluating the consistency and reliability of large language models in binary text classification, with a case study on financial news sentiment analysis across multiple models.

Contribution

It introduces a psychometrically inspired framework for assessing LLM reliability, including sample size determination and response validity metrics, demonstrated through a comprehensive case study.

Findings

Models showed high intra-rater consistency with 90-98% agreement.

Smaller models outperformed larger ones in sentiment classification accuracy.

Models performed at chance level in predicting actual market movements.

Abstract

This study introduces a framework for evaluating consistency in large language model (LLM) binary text classification, addressing the lack of established reliability assessment methods. Adapting psychometric principles, we determine sample size requirements, develop metrics for invalid responses, and evaluate intra- and inter-rater reliability. Our case study examines financial news sentiment classification across 14 LLMs (including claude-3-7-sonnet, gpt-4o, deepseek-r1, gemma3, llama3.2, phi4, and command-r-plus), with five replicates per model on 1,350 articles. Models demonstrated high intra-rater consistency, achieving perfect agreement on 90-98% of examples, with minimal differences between expensive and economical models from the same families. When validated against StockNewsAPI labels, models achieved strong performance (accuracy 0.76-0.88), with smaller models like gemma3:1B, llama3.2:3B, and claude-3-5-haiku outperforming larger counterparts. All models performed at chance when predicting actual market movements, indicating task constraints rather than model limitations. Our framework provides systematic guidance for LLM selection, sample size planning, and reliability assessment, enabling organizations to optimize resources for classification tasks.