An Automated Length-Aware Quality Metric for Summarization

TL;DR

This paper introduces NOIR, a new automated metric that evaluates summarization quality by balancing semantic retention and summary length, correlating well with human judgment and applicable across various tasks.

Contribution

The paper presents NOIR, a novel length-aware quality metric for summarization that effectively measures semantic retention and compression without human references.

Findings

NOIR correlates strongly with human judgment of summary quality.

It effectively captures the semantic and length tradeoff in summarization.

The metric is applicable to diverse summarization tasks.

Abstract

This paper proposes NOrmed Index of Retention (NOIR), a quantitative objective metric for evaluating summarization quality of arbitrary texts that relies on both the retention of semantic meaning and the summary length compression. This gives a measure of how well the recall-compression tradeoff is managed, the most important skill in summarization. Experiments demonstrate that NOIR effectively captures the token-length / semantic retention tradeoff of a summarizer and correlates to human perception of sumarization quality. Using a language model-embedding to measure semantic similarity, it provides an automated alternative for assessing summarization quality without relying on time-consuming human-generated reference summaries. The proposed metric can be applied to various summarization tasks, offering an automated tool for evaluating and improving summarization algorithms, summarization prompts, and synthetically-generated summaries.