Pre-trained Language Models for Keyphrase Generation: A Thorough Empirical Study

TL;DR

This paper provides a comprehensive empirical comparison of pre-trained language models versus traditional neural models for keyphrase generation, analyzing design choices and domain-specific performance.

Contribution

It systematically evaluates how different PLM configurations and design choices impact keyphrase generation performance across domains.

Findings

PLMs perform well in both high-resource and low-resource settings.

In-domain PLMs enhance keyphrase generation accuracy.

Prioritizing depth over width in models improves performance.

Abstract

Neural models that do not rely on pre-training have excelled in the keyphrase generation task with large annotated datasets. Meanwhile, new approaches have incorporated pre-trained language models (PLMs) for their data efficiency. However, there lacks a systematic study of how the two types of approaches compare and how different design choices can affect the performance of PLM-based models. To fill in this knowledge gap and facilitate a more informed use of PLMs for keyphrase extraction and keyphrase generation, we present an in-depth empirical study. Formulating keyphrase extraction as sequence labeling and keyphrase generation as sequence-to-sequence generation, we perform extensive experiments in three domains. After showing that PLMs have competitive high-resource performance and state-of-the-art low-resource performance, we investigate important design choices including in-domain PLMs, PLMs with different pre-training objectives, using PLMs with a parameter budget, and different formulations for present keyphrases. Further results show that (1) in-domain BERT-like PLMs can be used to build strong and data-efficient keyphrase generation models; (2) with a fixed parameter budget, prioritizing model depth over width and allocating more layers in the encoder leads to better encoder-decoder models; and (3) introducing four in-domain PLMs, we achieve a competitive performance in the news domain and the state-of-the-art performance in the scientific domain.