Can We Trust the Performance Evaluation of Uncertainty Estimation Methods in Text Summarization?

TL;DR

This paper introduces a comprehensive benchmark for evaluating uncertainty estimation methods in text summarization, highlighting the importance of multiple metrics and diverse methods for reliable assessment.

Contribution

It presents a new benchmark with 31 metrics across four dimensions, evaluating 14 uncertainty methods on multiple datasets with human annotations, addressing evaluation reliability issues.

Findings

Multiple uncorrelated metrics are crucial for reliable evaluation.

Diverse uncertainty estimation methods vary significantly in performance.

Benchmark results guide better evaluation practices in text summarization.

Abstract

Text summarization, a key natural language generation (NLG) task, is vital in various domains. However, the high cost of inaccurate summaries in risk-critical applications, particularly those involving human-in-the-loop decision-making, raises concerns about the reliability of uncertainty estimation on text summarization (UE-TS) evaluation methods. This concern stems from the dependency of uncertainty model metrics on diverse and potentially conflicting NLG metrics. To address this issue, we introduce a comprehensive UE-TS benchmark incorporating 31 NLG metrics across four dimensions. The benchmark evaluates the uncertainty estimation capabilities of two large language models and one pre-trained language model on three datasets, with human-annotation analysis incorporated where applicable. We also assess the performance of 14 common uncertainty estimation methods within this benchmark. Our findings emphasize the importance of considering multiple uncorrelated NLG metrics and diverse uncertainty estimation methods to ensure reliable and efficient evaluation of UE-TS techniques. Our code and data are available https://github.com/he159ok/Benchmark-of-Uncertainty-Estimation-Methods-in-Text-Summarization.