ChatGPT vs State-of-the-Art Models: A Benchmarking Study in Keyphrase Generation Task

TL;DR

This study benchmarks ChatGPT against leading models for keyphrase generation, demonstrating its superior performance across diverse datasets, domains, and document lengths, and exploring its potential for domain adaptation and long document processing.

Contribution

It provides a comprehensive comparison of ChatGPT's keyphrase generation capabilities with state-of-the-art models across multiple datasets and tasks.

Findings

ChatGPT outperforms existing models in all tested datasets.

ChatGPT adapts well to different domains and document lengths.

ChatGPT shows promise for domain adaptation and processing long documents.

Abstract

Transformer-based language models, including ChatGPT, have demonstrated exceptional performance in various natural language generation tasks. However, there has been limited research evaluating ChatGPT's keyphrase generation ability, which involves identifying informative phrases that accurately reflect a document's content. This study seeks to address this gap by comparing ChatGPT's keyphrase generation performance with state-of-the-art models, while also testing its potential as a solution for two significant challenges in the field: domain adaptation and keyphrase generation from long documents. We conducted experiments on six publicly available datasets from scientific articles and news domains, analyzing performance on both short and long documents. Our results show that ChatGPT outperforms current state-of-the-art models in all tested datasets and environments, generating high-quality keyphrases that adapt well to diverse domains and document lengths.