Topic-Controllable Summarization: Topic-Aware Evaluation and Transformer Methods

TL;DR

This paper introduces a new topic-oriented evaluation metric and a Transformer-based method with control tokens for topic-controllable summarization, improving performance and efficiency over previous recurrent models.

Contribution

It proposes a novel topic-aware evaluation metric and a Transformer-based summarization approach using control tokens, addressing limitations of prior recurrent models.

Findings

Control tokens outperform embedding-based methods in quality and speed

The new evaluation metric correlates well with human judgments

Transformer models with control tokens achieve state-of-the-art results

Abstract

Topic-controllable summarization is an emerging research area with a wide range of potential applications. However, existing approaches suffer from significant limitations. For example, the majority of existing methods built upon recurrent architectures, which can significantly limit their performance compared to more recent Transformer-based architectures, while they also require modifications to the model's architecture for controlling the topic. At the same time, there is currently no established evaluation metric designed specifically for topic-controllable summarization. This work proposes a new topic-oriented evaluation measure to automatically evaluate the generated summaries based on the topic affinity between the generated summary and the desired topic. The reliability of the proposed measure is demonstrated through appropriately designed human evaluation. In addition, we adapt topic embeddings to work with powerful Transformer architectures and propose a novel and efficient approach for guiding the summary generation through control tokens. Experimental results reveal that control tokens can achieve better performance compared to more complicated embedding-based approaches while also being significantly faster.