A Preliminary Empirical Study on Prompt-based Unsupervised Keyphrase Extraction

TL;DR

This study evaluates how different prompt designs influence the performance of large language models in unsupervised keyphrase extraction, revealing that prompt complexity and specific keyword choices significantly impact results.

Contribution

It systematically investigates the effect of prompt design on keyphrase extraction performance across multiple datasets and models, highlighting the importance of prompt simplicity and keyword selection.

Findings

Simple prompts can be as effective as complex ones.

Keyword modifications in prompts influence extraction quality.

Complex prompts outperform simple ones on long documents.

Abstract

Pre-trained large language models can perform natural language processing downstream tasks by conditioning on human-designed prompts. However, a prompt-based approach often requires "prompt engineering" to design different prompts, primarily hand-crafted through laborious trial and error, requiring human intervention and expertise. It is a challenging problem when constructing a prompt-based keyphrase extraction method. Therefore, we investigate and study the effectiveness of different prompts on the keyphrase extraction task to verify the impact of the cherry-picked prompts on the performance of extracting keyphrases. Extensive experimental results on six benchmark keyphrase extraction datasets and different pre-trained large language models demonstrate that (1) designing complex prompts may not necessarily be more effective than designing simple prompts; (2) individual keyword changes in the designed prompts can affect the overall performance; (3) designing complex prompts achieve better performance than designing simple prompts when facing long documents.