Consistency and Coherence from Points of Contextual Similarity

TL;DR

This paper generalizes the ESTIME measure for factual consistency evaluation, making it applicable to any text-summary pairs and analyzing the layers of BERT for insights into information usefulness across different summarization qualities.

Contribution

The authors extend ESTIME to handle arbitrary text-summary pairs and analyze BERT layer contributions for various summarization qualities.

Findings

Useful information is present in most BERT layers except the lowest ones.

Layers near the top of BERT are most useful for local text details.

Different qualities like coherence and relevance involve different BERT layer patterns.

Abstract

Factual consistency is one of important summary evaluation dimensions, especially as summary generation becomes more fluent and coherent. The ESTIME measure, recently proposed specifically for factual consistency, achieves high correlations with human expert scores both for consistency and fluency, while in principle being restricted to evaluating such text-summary pairs that have high dictionary overlap. This is not a problem for current styles of summarization, but it may become an obstacle for future summarization systems, or for evaluating arbitrary claims against the text. In this work we generalize the method, and make a variant of the measure applicable to any text-summary pairs. As ESTIME uses points of contextual similarity, it provides insights into usefulness of information taken from different BERT layers. We observe that useful information exists in almost all of the layers except the several lowest ones. For consistency and fluency - qualities focused on local text details - the most useful layers are close to the top (but not at the top); for coherence and relevance we found a more complicated and interesting picture.