Sensitivity of BLANC to human-scored qualities of text summaries

TL;DR

This paper investigates how well the BLANC metric aligns with human judgments of summary qualities like fluency, informativeness, and factual correctness, aiming to optimize its parameters for better sensitivity.

Contribution

The study identifies optimal BLANC parameters that make its sensitivity to summary qualities comparable to human evaluators.

Findings

BLANC sensitivity can be tuned to match human assessment levels.

Optimal parameters improve BLANC's ability to evaluate multiple summary qualities.

The research provides guidelines for setting BLANC parameters for better summary evaluation.

Abstract

We explore the sensitivity of a document summary quality estimator, BLANC, to human assessment of qualities for the same summaries. In our human evaluations, we distinguish five summary qualities, defined by how fluent, understandable, informative, compact, and factually correct the summary is. We make the case for optimal BLANC parameters, at which the BLANC sensitivity to almost all of summary qualities is about as good as the sensitivity of a human annotator.