LongDocFACTScore: Evaluating the Factuality of Long Document Abstractive Summarisation

TL;DR

This paper introduces LongDocFACTScore, a new evaluation framework for assessing factual consistency in long document summarisation, addressing limitations of existing metrics and demonstrating improved correlation with human judgments.

Contribution

The paper presents LongDocFACTScore, a novel framework for evaluating factuality in long document summarisation, and introduces LongSciVerify, a human-annotated dataset for this purpose.

Findings

LongDocFACTScore outperforms existing metrics in correlating with human judgments.

The framework efficiently extends to any document length.

The dataset provides detailed factuality annotations for scientific summaries.

Abstract

Maintaining factual consistency is a critical issue in abstractive text summarisation, however, it cannot be assessed by traditional automatic metrics used for evaluating text summarisation, such as ROUGE scoring. Recent efforts have been devoted to developing improved metrics for measuring factual consistency using pre-trained language models, but these metrics have restrictive token limits, and are therefore not suitable for evaluating long document text summarisation. Moreover, there is limited research and resources available for evaluating whether existing automatic evaluation metrics are fit for purpose when applied in long document settings. In this work, we evaluate the efficacy of automatic metrics for assessing the factual consistency of long document text summarisation. We create a human-annotated data set for evaluating automatic factuality metrics, LongSciVerify, which contains fine-grained factual consistency annotations for long document summaries from the scientific domain. We also propose a new evaluation framework, LongDocFACTScore, which is suitable for evaluating long document summarisation. This framework allows metrics to be efficiently extended to any length document and outperforms existing state-of-the-art metrics in its ability to correlate with human measures of factuality when used to evaluate long document summarisation data sets. We make our code and LongSciVerify data set publicly available: https://github.com/jbshp/LongDocFACTScore.