Evaluating the Homogeneity of Keyphrase Prediction Models

TL;DR

This paper introduces a method to evaluate the homogeneity of keyphrase prediction models, revealing that extraction methods are competitive and absent keyphrase generation may reduce homogeneity.

Contribution

The work proposes a novel evaluation method for homogeneity in keyphrase models and compares generative and extraction approaches on this metric.

Findings

Extraction methods are competitive with generative models.

Absent keyphrase generation can negatively impact homogeneity.

Homogeneity evaluation is a new benchmark for keyphrase models.

Abstract

Keyphrases which are useful in several NLP and IR applications are either extracted from text or predicted by generative models. Contrarily to keyphrase extraction approaches, keyphrase generation models can predict keyphrases that do not appear in a document's text called `absent keyphrases`. This ability means that keyphrase generation models can associate a document to a notion that is not explicitly mentioned in its text. Intuitively, this suggests that for two documents treating the same subjects, a keyphrase generation model is more likely to be homogeneous in their indexing i.e. predict the same keyphrase for both documents, regardless of those keyphrases appearing in their respective text or not; something a keyphrase extraction model would fail to do. Yet, homogeneity of keyphrase prediction models is not covered by current benchmarks. In this work, we introduce a method to evaluate the homogeneity of keyphrase prediction models and study if absent keyphrase generation capabilities actually help the model to be more homogeneous. To our surprise, we show that keyphrase extraction methods are competitive with generative models, and that the ability to generate absent keyphrases can actually have a negative impact on homogeneity. Our data, code and prompts are available on huggingface and github.