Evaluating Factual Consistency of Texts with Semantic Role Labeling

TL;DR

SRLScore is a reference-free, interpretable metric for evaluating the factual consistency of generated texts, especially summaries, using semantic role labeling to produce fact tuples and a flexible scoring mechanism.

Contribution

The paper introduces SRLScore, a novel factuality evaluation metric leveraging semantic role labels, which is domain-adaptive, interpretable, and competitive with state-of-the-art methods without requiring training.

Findings

SRLScore correlates well with human judgments on summarization datasets.

It generalizes across datasets without additional training.

Adding co-reference resolution offers minimal performance gains relative to computational costs.

Abstract

Automated evaluation of text generation systems has recently seen increasing attention, particularly checking whether generated text stays truthful to input sources. Existing methods frequently rely on an evaluation using task-specific language models, which in turn allows for little interpretability of generated scores. We introduce SRLScore, a reference-free evaluation metric designed with text summarization in mind. Our approach generates fact tuples constructed from Semantic Role Labels, applied to both input and summary texts. A final factuality score is computed by an adjustable scoring mechanism, which allows for easy adaption of the method across domains. Correlation with human judgments on English summarization datasets shows that SRLScore is competitive with state-of-the-art methods and exhibits stable generalization across datasets without requiring further training or hyperparameter tuning. We experiment with an optional co-reference resolution step, but find that the performance boost is mostly outweighed by the additional compute required. Our metric is available online at https://github.com/heyjing/SRLScore.