Effective Proxy for Human Labeling: Ensemble Disagreement Scores in Large Language Models for Industrial NLP

TL;DR

This paper shows that ensemble disagreement scores can effectively estimate LLM performance in industrial NLP tasks, reducing the need for costly human labeling and outperforming other proxy methods across multiple languages and domains.

Contribution

It introduces ensemble disagreement scores as a reliable proxy for human labeling in assessing LLM performance in real-world industrial NLP applications.

Findings

Disagreement scores closely match human error measurements with MAE as low as 0.4%.

Disagreement scores outperform silver labels by an average of 13.8%.

Effective across multiple languages and domains.

Abstract

Large language models (LLMs) have demonstrated significant capability to generalize across a large number of NLP tasks. For industry applications, it is imperative to assess the performance of the LLM on unlabeled production data from time to time to validate for a real-world setting. Human labeling to assess model error requires considerable expense and time delay. Here we demonstrate that ensemble disagreement scores work well as a proxy for human labeling for language models in zero-shot, few-shot, and fine-tuned settings, per our evaluation on keyphrase extraction (KPE) task. We measure fidelity of the results by comparing to true error measured from human labeled ground truth. We contrast with the alternative of using another LLM as a source of machine labels, or silver labels. Results across various languages and domains show disagreement scores provide a better estimation of model performance with mean average error (MAE) as low as 0.4% and on average 13.8% better than using silver labels.