Real-World Summarization: When Evaluation Reaches Its Limits

TL;DR

This paper investigates the effectiveness of various evaluation metrics for hotel highlight summaries generated by language models, revealing that simple word overlap metrics often outperform complex methods and that LLMs are unreliable as evaluators.

Contribution

It provides a comprehensive comparison of evaluation methods for factual summarization, highlighting the limitations of LLM-based evaluation and the robustness of simple metrics in real-world settings.

Findings

Word overlap metrics correlate well with human judgments (Spearman 0.63)

LLMs are unreliable as evaluators due to under- or over-annotation

Incorrect information in summaries poses significant risks in real-world applications

Abstract

We examine evaluation of faithfulness to input data in the context of hotel highlights: brief LLM-generated summaries that capture unique features of accommodations. Through human evaluation campaigns involving categorical error assessment and span-level annotation, we compare traditional metrics, trainable methods, and LLM-as-a-judge approaches. Our findings reveal that simpler metrics like word overlap correlate surprisingly well with human judgments (Spearman correlation rank of 0.63), often outperforming more complex methods when applied to out-of-domain data. We further demonstrate that while LLMs can generate high-quality highlights, they prove unreliable for evaluation as they tend to severely under- or over-annotate. Our analysis of real-world business impacts shows incorrect and non-checkable information pose the greatest risks. We also highlight challenges in crowdsourced evaluations.