QuestEval: Summarization Asks for Fact-based Evaluation

TL;DR

QuestEval introduces a reference-free, question-answering based summarization evaluation metric that significantly improves correlation with human judgments across multiple quality dimensions.

Contribution

It presents a unified framework for summarization evaluation that outperforms traditional metrics like ROUGE without needing reference summaries.

Findings

QuestEval correlates better with human judgments than ROUGE.

It improves evaluation across consistency, coherence, fluency, and relevance.

The method does not require ground-truth reference summaries.

Abstract

Summarization evaluation remains an open research problem: current metrics such as ROUGE are known to be limited and to correlate poorly with human judgments. To alleviate this issue, recent work has proposed evaluation metrics which rely on question answering models to assess whether a summary contains all the relevant information in its source document. Though promising, the proposed approaches have so far failed to correlate better than ROUGE with human judgments. In this paper, we extend previous approaches and propose a unified framework, named QuestEval. In contrast to established metrics such as ROUGE or BERTScore, QuestEval does not require any ground-truth reference. Nonetheless, QuestEval substantially improves the correlation with human judgments over four evaluation dimensions (consistency, coherence, fluency, and relevance), as shown in the extensive experiments we report.