Reinforcement Replaces Supervision: Query focused Summarization using Deep Reinforcement Learning

TL;DR

This paper introduces an RL-based approach for query-focused summarization that outperforms state-of-the-art methods on benchmark datasets by optimizing reward signals like ROUGE and semantic similarity, and also provides a new dataset for future research.

Contribution

It develops multiple policy gradient networks trained with novel reward signals, including a semantic similarity measure, and addresses RL training challenges in transformer models.

Findings

10-point improvement on ROUGE-L for ELI5 dataset

Comparable zero-shot performance on DebatePedia

Proposes a novel Passage Embedding scheme for semantic rewards

Abstract

Query-focused Summarization (QfS) deals with systems that generate summaries from document(s) based on a query. Motivated by the insight that Reinforcement Learning (RL) provides a generalization to Supervised Learning (SL) for Natural Language Generation, and thereby performs better (empirically) than SL, we use an RL-based approach for this task of QfS. Additionally, we also resolve the conflict of employing RL in Transformers with Teacher Forcing. We develop multiple Policy Gradient networks, trained on various reward signals: ROUGE, BLEU, and Semantic Similarity, which lead to a 10-point improvement over the State-of-the-Art approach on the ROUGE-L metric for a benchmark dataset (ELI5). We also show performance of our approach in zero-shot setting for another benchmark dataset (DebatePedia) -- our approach leads to results comparable to baselines, which were specifically trained on DebatePedia. To aid the RL training, we propose a better semantic similarity reward, enabled by a novel Passage Embedding scheme developed using Cluster Hypothesis. Lastly, we contribute a gold-standard test dataset to further research in QfS and Long-form Question Answering (LfQA).