SMART: Sentences as Basic Units for Text Evaluation

TL;DR

SMART introduces a sentence-based evaluation metric for text generation that improves correlation with human judgments, especially for longer texts, and offers a resource-efficient alternative to neural models.

Contribution

The paper proposes a novel sentence-level matching metric for text evaluation that enhances correlation with human assessments and reduces resource dependency.

Findings

System-level correlation with human judgments is improved.

Sentence-based metric performs well on longer summaries.

String-based version is competitive and resource-efficient.

Abstract

Widely used evaluation metrics for text generation either do not work well with longer texts or fail to evaluate all aspects of text quality. In this paper, we introduce a new metric called SMART to mitigate such limitations. Specifically, We treat sentences as basic units of matching instead of tokens, and use a sentence matching function to soft-match candidate and reference sentences. Candidate sentences are also compared to sentences in the source documents to allow grounding (e.g., factuality) evaluation. Our results show that system-level correlations of our proposed metric with a model-based matching function outperforms all competing metrics on the SummEval summarization meta-evaluation dataset, while the same metric with a string-based matching function is competitive with current model-based metrics. The latter does not use any neural model, which is useful during model development phases where resources can be limited and fast evaluation is required. Finally, we also conducted extensive analyses showing that our proposed metrics work well with longer summaries and are less biased towards specific models.