Keyphrase Generation Beyond the Boundaries of Title and Abstract

TL;DR

This paper investigates how incorporating full text and summaries of scholarly articles enhances neural keyphrase generation, demonstrating significant improvements and introducing a new comprehensive dataset for the task.

Contribution

It is the first to systematically explore full text and summaries for keyphrase generation and introduces the FullTextKP dataset with full article texts.

Findings

Adding full text summaries improves keyphrase generation accuracy.

Transformer models like LED outperform traditional models on full-text data.

The new dataset enables more comprehensive research in scholarly keyphrase extraction.

Abstract

Keyphrase generation aims at generating important phrases (keyphrases) that best describe a given document. In scholarly domains, current approaches have largely used only the title and abstract of the articles to generate keyphrases. In this paper, we comprehensively explore whether the integration of additional information from the full text of a given article or from semantically similar articles can be helpful for a neural keyphrase generation model or not. We discover that adding sentences from the full text, particularly in the form of the extractive summary of the article can significantly improve the generation of both types of keyphrases that are either present or absent from the text. Experimental results with three widely used models for keyphrase generation along with one of the latest transformer models suitable for longer documents, Longformer Encoder-Decoder (LED) validate the observation. We also present a new large-scale scholarly dataset FullTextKP for keyphrase generation. Unlike prior large-scale datasets, FullTextKP includes the full text of the articles along with the title and abstract. We release the source code at https://github.com/kgarg8/FullTextKP.