Metric Ensembles For Hallucination Detection

TL;DR

This paper evaluates various unsupervised metrics for detecting hallucinations in abstractive summaries, demonstrating that ensemble methods, especially those combining LLM-based metrics, outperform individual metrics and previous state-of-the-art approaches.

Contribution

It introduces an ensemble approach for hallucination detection that leverages LLM-based metrics, improving accuracy over existing methods and establishing a new state-of-the-art.

Findings

LLM-based metrics outperform other unsupervised metrics

Ensemble methods improve hallucination detection scores

Ensemble of uncorrelated metrics enhances performance

Abstract

Abstractive text summarization has garnered increased interest as of late, in part due to the proliferation of large language models (LLMs). One of the most pressing problems related to generation of abstractive summaries is the need to reduce "hallucinations," information that was not included in the document being summarized, and which may be wholly incorrect. Due to this need, a wide array of metrics estimating consistency with the text being summarized have been proposed. We examine in particular a suite of unsupervised metrics for summary consistency, and measure their correlations with each other and with human evaluation scores in the wiki_bio_gpt3_hallucination dataset. We then compare these evaluations to models made from a simple linear ensemble of these metrics. We find that LLM-based methods outperform other unsupervised metrics for hallucination detection. We also find that ensemble methods can improve these scores even further, provided that the metrics in the ensemble have sufficiently similar and uncorrelated error rates. Finally, we present an ensemble method for LLM-based evaluations that we show improves over this previous SOTA.