Podcast Summary Assessment: A Resource for Evaluating Summary Assessment Methods

TL;DR

This paper introduces a new podcast summary assessment dataset, evaluates existing assessment methods on it, and explores its use for filtering training data for summary generation tasks.

Contribution

It presents a novel long-input podcast dataset for summary assessment, benchmarks assessment methods on this data, and demonstrates its application in data filtering for summary generation.

Findings

Existing assessment methods provide varying performance on podcast data

The dataset reveals challenges in assessing long, speech-based summaries

Assessment can effectively filter training data for better summary generation

Abstract

Automatic summary assessment is useful for both machine-generated and human-produced summaries. Automatically evaluating the summary text given the document enables, for example, summary generation system development and detection of inappropriate summaries. Summary assessment can be run in a number of modes: ranking summary generation systems; ranking summaries of a particular document; and estimating the quality of a document-summary pair on an absolute scale. Existing datasets with annotation for summary assessment are usually based on news summarization datasets such as CNN/DailyMail or XSum. In this work, we describe a new dataset, the podcast summary assessment corpus, a collection of podcast summaries that were evaluated by human experts at TREC2020. Compared to existing summary assessment data, this dataset has two unique aspects: (i) long-input, speech podcast based, documents; and (ii) an opportunity to detect inappropriate reference summaries in podcast corpus. First, we examine existing assessment methods, including model-free and model-based methods, and provide benchmark results for this long-input summary assessment dataset. Second, with the aim of filtering reference summary-document pairings for training, we apply summary assessment for data selection. The experimental results on these two aspects provide interesting insights on the summary assessment and generation tasks. The podcast summary assessment data is available.