Is Summary Useful or Not? An Extrinsic Human Evaluation of Text Summaries on Downstream Tasks

TL;DR

This paper evaluates the usefulness of text summaries in downstream tasks using extrinsic human evaluation, revealing that summaries are more beneficial for overall judgment tasks than for question answering, and highlighting limitations of intrinsic metrics.

Contribution

It introduces a novel extrinsic evaluation framework for summaries across multiple downstream tasks and compares the effectiveness of different summarization models and automatic metrics.

Findings

Summaries improve performance in overall judgment tasks.

Fine-tuned models produce more consistent usefulness across tasks.

Intrinsic automatic metrics are effective for question answering but less so for other tasks.

Abstract

Research on automated text summarization relies heavily on human and automatic evaluation. While recent work on human evaluation mainly adopted intrinsic evaluation methods, judging the generic quality of text summaries, e.g. informativeness and coherence, our work focuses on evaluating the usefulness of text summaries with extrinsic methods. We carefully design three different downstream tasks for extrinsic human evaluation of summaries, i.e., question answering, text classification and text similarity assessment. We carry out experiments using system rankings and user behavior data to evaluate the performance of different summarization models. We find summaries are particularly useful in tasks that rely on an overall judgment of the text, while being less effective for question answering tasks. The results show that summaries generated by fine-tuned models lead to higher consistency in usefulness across all three tasks, as rankings of fine-tuned summarization systems are close across downstream tasks according to the proposed extrinsic metrics. Summaries generated by models in the zero-shot setting, however, are found to be biased towards the text classification and similarity assessment tasks, due to its general and less detailed summary style. We further evaluate the correlation of 14 intrinsic automatic metrics with human criteria and show that intrinsic automatic metrics perform well in evaluating the usefulness of summaries in the question-answering task, but are less effective in the other two tasks. This highlights the limitations of relying solely on intrinsic automatic metrics in evaluating the performance and usefulness of summaries.