Quality of syntactic implication of RL-based sentence summarization

TL;DR

This paper compares reinforcement learning and syntax-aware models for sentence summarization, showing that RL alone performs nearly as well as syntax-aware models in quality, with advantages in simplicity and training speed.

Contribution

It provides a detailed comparison of RL-based and syntax-aware summarization models, highlighting that RL alone achieves competitive quality with fewer parameters.

Findings

RL alone nearly matches syntax-aware models in quality

Combined models yield the best results overall

RL training is faster and simpler without syntactic info

Abstract

Work on summarization has explored both reinforcement learning (RL) optimization using ROUGE as a reward and syntax-aware models, such as models those input is enriched with part-of-speech (POS)-tags and dependency information. However, it is not clear what is the respective impact of these approaches beyond the standard ROUGE evaluation metric. Especially, RL-based for summarization is becoming more and more popular. In this paper, we provide a detailed comparison of these two approaches and of their combination along several dimensions that relate to the perceived quality of the generated summaries: number of repeated words, distribution of part-of-speech tags, impact of sentence length, relevance and grammaticality. Using the standard Gigaword sentence summarization task, we compare an RL self-critical sequence training (SCST) method with syntax-aware models that leverage POS tags and Dependency information. We show that on all qualitative evaluations, the combined model gives the best results, but also that only training with RL and without any syntactic information already gives nearly as good results as syntax-aware models with less parameters and faster training convergence.