DocAsRef: An Empirical Study on Repurposing Reference-Based Summary Quality Metrics Reference-Freely

TL;DR

This study demonstrates that reference-based summary quality metrics can be adapted to function as reference-free metrics, with the repurposed zero-shot BERTScore outperforming many existing metrics across multiple datasets.

Contribution

The paper introduces a novel method to transform reference-based metrics into effective reference-free summary quality evaluators using adaptation techniques.

Findings

Repurposed zero-shot BERTScore outperforms original reference-based version.

The adapted metric surpasses most existing reference-free metrics.

It closely rivals GPT-3.5 based evaluators.

Abstract

Automated summary quality assessment falls into two categories: reference-based and reference-free. Reference-based metrics, historically deemed more accurate due to the additional information provided by human-written references, are limited by their reliance on human input. In this paper, we hypothesize that the comparison methodologies used by some reference-based metrics to evaluate a system summary against its corresponding reference can be effectively adapted to assess it against its source document, thereby transforming these metrics into reference-free ones. Experimental results support this hypothesis. After being repurposed reference-freely, the zero-shot BERTScore using the pretrained DeBERTa-large-MNLI model of <0.5B parameters consistently outperforms its original reference-based version across various aspects on the SummEval and Newsroom datasets. It also excels in comparison to most existing reference-free metrics and closely competes with zero-shot summary evaluators based on GPT-3.5.