Real World Conversational Entity Linking Requires More Than Zeroshots

TL;DR

This paper examines the limitations of current zero-shot conversational entity linking models in real-world scenarios with limited resources, highlighting the need for new approaches to improve adaptability and performance.

Contribution

The study introduces a novel evaluation framework and dataset for assessing conversational entity linking models under resource constraints and domain-specific challenges.

Findings

Zero-shot EL models perform poorly on domain-specific KBs without prior training.

Existing evaluation methods do not adequately reflect real-world EL complexities.

New approaches are needed for effective conversational EL in resource-scarce environments.

Abstract

Entity linking (EL) in conversations faces notable challenges in practical applications, primarily due to the scarcity of entity-annotated conversational datasets and sparse knowledge bases (KB) containing domain-specific, long-tail entities. We designed targeted evaluation scenarios to measure the efficacy of EL models under resource constraints. Our evaluation employs two KBs: Fandom, exemplifying real-world EL complexities, and the widely used Wikipedia. First, we assess EL models' ability to generalize to a new unfamiliar KB using Fandom and a novel zero-shot conversational entity linking dataset that we curated based on Reddit discussions on Fandom entities. We then evaluate the adaptability of EL models to conversational settings without prior training. Our results indicate that current zero-shot EL models falter when introduced to new, domain-specific KBs without prior training, significantly dropping in performance. Our findings reveal that previous evaluation approaches fall short of capturing real-world complexities for zero-shot EL, highlighting the necessity for new approaches to design and assess conversational EL models to adapt to limited resources. The evaluation setup and the dataset proposed in this research are made publicly available.