RISE: Leveraging Retrieval Techniques for Summarization Evaluation

TL;DR

RISE is a novel evaluation method for text summaries that uses retrieval techniques to assess quality without needing reference summaries, showing high correlation with human judgments and good generalizability.

Contribution

Introduces RISE, a retrieval-based evaluation approach that works without gold references, improving correlation with human assessments and enabling cross-lingual evaluation.

Findings

RISE outperforms previous methods in correlation with human evaluations.

It demonstrates strong data-efficiency and cross-lingual generalizability.

Effective on the SummEval benchmark without requiring reference summaries.

Abstract

Evaluating automatically-generated text summaries is a challenging task. While there have been many interesting approaches, they still fall short of human evaluations. We present RISE, a new approach for evaluating summaries by leveraging techniques from information retrieval. RISE is first trained as a retrieval task using a dual-encoder retrieval setup, and can then be subsequently utilized for evaluating a generated summary given an input document, without gold reference summaries. RISE is especially well suited when working on new datasets where one may not have reference summaries available for evaluation. We conduct comprehensive experiments on the SummEval benchmark (Fabbri et al., 2021) and the results show that RISE has higher correlation with human evaluations compared to many past approaches to summarization evaluation. Furthermore, RISE also demonstrates data-efficiency and generalizability across languages.