Evaluation of Automatic Text Summarization using Synthetic Facts

TL;DR

This paper introduces a novel reference-less evaluation system for automatic text summarization that assesses factual consistency, coverage, and compression, aiming to improve reliability and practical utility.

Contribution

It presents the first evaluation method that measures summarization quality based on factuality, coverage, and compression without relying on reference summaries.

Findings

Effective measurement of factual consistency and coverage.

Improved evaluation reliability for summarization models.

Potential to enhance practical summarization applications.

Abstract

Despite some recent advances, automatic text summarization remains unreliable, elusive, and of limited practical use in applications. Two main problems with current summarization methods are well known: evaluation and factual consistency. To address these issues, we propose a new automatic reference-less text summarization evaluation system that can measure the quality of any text summarization model with a set of generated facts based on factual consistency, comprehensiveness, and compression rate. As far as we know, our evaluation system is the first system that measures the overarching quality of the text summarization models based on factuality, information coverage, and compression rate.