SEval-Ex: A Statement-Level Framework for Explainable Summarization Evaluation

TL;DR

SEval-Ex is a novel framework for summarization evaluation that offers high accuracy and interpretability by decomposing assessments into atomic statements and providing detailed evidence for its decisions.

Contribution

It introduces a statement-level evaluation framework that improves performance and explainability over existing methods in summarization quality assessment.

Findings

Achieves state-of-the-art correlation with human judgments (0.580) on SummEval.

Surpasses GPT-4 based evaluators in consistency correlation.

Demonstrates robustness against hallucination in evaluations.

Abstract

Evaluating text summarization quality remains a critical challenge in Natural Language Processing. Current approaches face a trade-off between performance and interpretability. We present SEval-Ex, a framework that bridges this gap by decomposing summarization evaluation into atomic statements, enabling both high performance and explainability. SEval-Ex employs a two-stage pipeline: first extracting atomic statements from text source and summary using LLM, then a matching between generated statements. Unlike existing approaches that provide only summary-level scores, our method generates detailed evidence for its decisions through statement-level alignments. Experiments on the SummEval benchmark demonstrate that SEval-Ex achieves state-of-the-art performance with 0.580 correlation on consistency with human consistency judgments, surpassing GPT-4 based evaluators (0.521) while maintaining interpretability. Finally, our framework shows robustness against hallucination.