CQSumDP: A ChatGPT-Annotated Resource for Query-Focused Abstractive Summarization Based on Debatepedia

TL;DR

This paper introduces CQSumDP, a cleaned and ChatGPT-annotated version of the Debatepedia dataset, improving its relevance and quality for query-focused abstractive summarization tasks.

Contribution

The authors propose a novel methodology leveraging ChatGPT to clean and regenerate queries in Debatepedia, enhancing its suitability for summarization models.

Findings

Annotated dataset outperforms original in relevance and quality

ChatGPT effectively regenerates relevant queries

Improved dataset benefits query-focused summarization models

Abstract

Debatepedia is a publicly available dataset consisting of arguments and counter-arguments on controversial topics that has been widely used for the single-document query-focused abstractive summarization task in recent years. However, it has been recently found that this dataset is limited by noise and even most queries in this dataset do not have any relevance to the respective document. In this paper, we present a methodology for cleaning the Debatepedia dataset by leveraging the generative power of large language models to make it suitable for query-focused abstractive summarization. More specifically, we harness the language generation capabilities of ChatGPT to regenerate its queries. We evaluate the effectiveness of the proposed ChatGPT annotated version of the Debatepedia dataset using several benchmark summarization models and demonstrate that the newly annotated version of Debatepedia outperforms the original dataset in terms of both query relevance as well as summary generation quality. We will make this annotated and cleaned version of the dataset publicly available.