Attention-Seeker: Dynamic Self-Attention Scoring for Unsupervised Keyphrase Extraction

TL;DR

Attention-Seeker is an unsupervised keyphrase extraction method that uses self-attention maps from a Large Language Model to identify important phrases without manual tuning, outperforming many baselines on multiple datasets.

Contribution

It introduces a dynamic, unsupervised approach leveraging self-attention components from language models for keyphrase extraction, eliminating manual parameter tuning.

Findings

Outperforms baseline models on three datasets

Achieves state-of-the-art results in keyphrase extraction

Excels particularly with long documents

Abstract

This paper proposes Attention-Seeker, an unsupervised keyphrase extraction method that leverages self-attention maps from a Large Language Model to estimate the importance of candidate phrases. Our approach identifies specific components - such as layers, heads, and attention vectors - where the model pays significant attention to the key topics of the text. The attention weights provided by these components are then used to score the candidate phrases. Unlike previous models that require manual tuning of parameters (e.g., selection of heads, prompts, hyperparameters), Attention-Seeker dynamically adapts to the input text without any manual adjustments, enhancing its practical applicability. We evaluate Attention-Seeker on four publicly available datasets: Inspec, SemEval2010, SemEval2017, and Krapivin. Our results demonstrate that, even without parameter tuning, Attention-Seeker outperforms most baseline models, achieving state-of-the-art performance on three out of four datasets, particularly excelling in extracting keyphrases from long documents.