LongSumEval: Question-Answering Based Evaluation and Feedback-Driven Refinement for Long Document Summarization

TL;DR

LongSumEval introduces a question-answering based framework for evaluating and improving long document summaries, providing interpretable scores and actionable feedback aligned with human judgments.

Contribution

It presents a novel QA-based evaluation method that correlates better with human judgments and enables self-refinement without retraining.

Findings

QA-based evaluation outperforms existing metrics in agreement with human judgments.

Structured feedback facilitates significant quality improvements through self-refinement.

Evaluation feedback can be used as executable instructions to guide generation.

Abstract

Evaluating long document summaries remains the primary bottleneck in summarization research. Existing metrics correlate weakly with human judgments and produce aggregate scores without explaining deficiencies or guiding improvement, preventing effective refinement in applications requiring verifiable accuracy. We introduce LongSumEval, a unified framework bridging evaluation and generation through structured question-answering feedback. The framework operationalizes summary quality as answerability and factual alignment of question-answer pairs, generating interpretable scores and actionable feedback that identifies coverage gaps and factual inconsistencies. This resolves the misalignment where evaluation operates independently of generation objectives. Meta-evaluation of our QA-based evaluation module across seven benchmarks demonstrates substantially stronger agreement with human judgments compared to established metrics. Structured feedback enables significant quality improvements through self-refinement without retraining. By demonstrating that evaluation feedback can serve as executable instructions for generation, this work establishes a generalizable paradigm for aligning assessment with improvement, with direct implications for controllable text generation requiring verifiable accuracy and transparent quality control. All code and datasets will be released in GitHub for reproducibility.