Optimizing the role of human evaluation in LLM-based spoken document summarization systems

TL;DR

This paper proposes a structured human evaluation framework for LLM-based spoken document summarization, emphasizing robustness, replicability, and trustworthiness, with practical case studies from industry.

Contribution

It introduces a tailored evaluation paradigm for AI-generated spoken document summaries, integrating social science methodologies and detailed guidelines.

Findings

Human evaluation methods improve robustness and trustworthiness.

Case studies demonstrate practical implementation at industry scale.

Guidelines enhance replicability of evaluation studies.

Abstract

The emergence of powerful LLMs has led to a paradigm shift in abstractive summarization of spoken documents. The properties that make LLMs so valuable for this task -- creativity, ability to produce fluent speech, and ability to abstract information from large corpora -- also present new challenges to evaluating their content. Quick, cost-effective automatic evaluations such as ROUGE and BERTScore offer promise, but do not yet show competitive performance when compared to human evaluations. We draw on methodologies from the social sciences to propose an evaluation paradigm for spoken document summarization explicitly tailored for generative AI content. We provide detailed evaluation criteria and best practices guidelines to ensure robustness in the experimental design, replicability, and trustworthiness of human evaluation studies. We additionally include two case studies that show how these human-in-the-loop evaluation methods have been implemented at a major U.S. technology company.