SEKE: Specialised Experts for Keyword Extraction

TL;DR

SEKE introduces a novel supervised keyword extraction method using a mixture of specialised experts with DeBERTa and BiLSTM, achieving state-of-the-art results and providing interpretability insights.

Contribution

It presents a new MoE-based framework for keyword extraction that enhances performance and explainability, especially on smaller datasets.

Findings

SEKE outperforms existing baselines on multiple datasets.

Experts specialise in different linguistic features like punctuation and named entities.

The approach improves keyword extraction in low-resource scenarios.

Abstract

Keyword extraction involves identifying the most descriptive words in a document, allowing automatic categorisation and summarisation of large quantities of diverse textual data. Relying on the insight that real-world keyword detection often requires handling of diverse content, we propose a novel supervised keyword extraction approach based on the mixture of experts (MoE) technique. MoE uses a learnable routing sub-network to direct information to specialised experts, allowing them to specialise in distinct regions of the input space. SEKE, a mixture of Specialised Experts for supervised Keyword Extraction, uses DeBERTa as the backbone model and builds on the MoE framework, where experts attend to each token, by integrating it with a bidirectional Long short-term memory (BiLSTM) network, to allow successful extraction even on smaller corpora, where specialisation is harder due to lack of training data. The MoE framework also provides an insight into inner workings of individual experts, enhancing the explainability of the approach. We benchmark SEKE on multiple English datasets, achieving state-of-the-art performance compared to strong supervised and unsupervised baselines. Our analysis reveals that depending on data size and type, experts specialise in distinct syntactic and semantic components, such as punctuation, stopwords, parts-of-speech, or named entities. Code is available at https://github.com/matejMartinc/SEKE_keyword_extraction