CLIMATELI: Evaluating Entity Linking on Climate Change Data

TL;DR

This paper introduces CLIMATELI, a new manually annotated dataset for entity linking in climate change texts, and evaluates existing systems, revealing significant performance gaps compared to humans and the impact of filtering methods.

Contribution

The paper presents the first climate change-specific entity linking dataset and assesses current models, highlighting areas for improvement and proposing filtering techniques.

Findings

EL models perform worse than humans at token and entity levels.

Filtering non-nominal and non-CC entities affects model performance.

CLIMATELI enables better evaluation and development of climate-related NLP tools.

Abstract

Climate Change (CC) is a pressing topic of global importance, attracting increasing attention across research fields, from social sciences to Natural Language Processing (NLP). CC is also discussed in various settings and communication platforms, from academic publications to social media forums. Understanding who and what is mentioned in such data is a first critical step to gaining new insights into CC. We present CLIMATELI (CLIMATe Entity LInking), the first manually annotated CC dataset that links 3,087 entity spans to Wikipedia. Using CLIMATELI (CLIMATe Entity LInking), we evaluate existing entity linking (EL) systems on the CC topic across various genres and propose automated filtering methods for CC entities. We find that the performance of EL models notably lags behind humans at both token and entity levels. Testing within the scope of retaining or excluding non-nominal and/or non-CC entities particularly impacts the models' performances.