An Analysis of Datasets, Metrics and Models in Keyphrase Generation

TL;DR

This paper reviews over 50 studies on keyphrase generation, revealing evaluation issues, dataset limitations, and introducing a new pre-trained model to advance future research in the field.

Contribution

It provides a comprehensive analysis of existing research, highlights critical evaluation challenges, and releases a new pre-trained model for keyphrase generation.

Findings

Evaluation practices are inconsistent and often overestimate performance.

Benchmark datasets are too similar, limiting progress.

A strong PLM-based model is released to support future work.

Abstract

Keyphrase generation refers to the task of producing a set of words or phrases that summarises the content of a document. Continuous efforts have been dedicated to this task over the past few years, spreading across multiple lines of research, such as model architectures, data resources, and use-case scenarios. Yet, the current state of keyphrase generation remains unknown as there has been no attempt to review and analyse previous work. In this paper, we bridge this gap by presenting an analysis of over 50 research papers on keyphrase generation, offering a comprehensive overview of recent progress, limitations, and open challenges. Our findings highlight several critical issues in current evaluation practices, such as the concerning similarity among commonly-used benchmark datasets and inconsistencies in metric calculations leading to overestimated performances. Additionally, we address the limited availability of pre-trained models by releasing a strong PLM-based model for keyphrase generation as an effort to facilitate future research.