COMETA: A Corpus for Medical Entity Linking in the Social Media

TL;DR

COMETA is a new large-scale corpus of social media biomedical mentions linked to SNOMED CT, designed to improve entity linking in health-related language and evaluate current models' performance.

Contribution

This paper introduces COMETA, a comprehensive biomedical entity linking dataset from Reddit, and benchmarks multiple models to highlight current challenges and the need for combined approaches.

Findings

No existing model achieves perfect performance on COMETA.

Combining different data views improves entity linking accuracy.

Current models still have significant performance gaps.

Abstract

Whilst there has been growing progress in Entity Linking (EL) for general language, existing datasets fail to address the complex nature of health terminology in layman's language. Meanwhile, there is a growing need for applications that can understand the public's voice in the health domain. To address this we introduce a new corpus called COMETA, consisting of 20k English biomedical entity mentions from Reddit expert-annotated with links to SNOMED CT, a widely-used medical knowledge graph. Our corpus satisfies a combination of desirable properties, from scale and coverage to diversity and quality, that to the best of our knowledge has not been met by any of the existing resources in the field. Through benchmark experiments on 20 EL baselines from string- to neural-based models we shed light on the ability of these systems to perform complex inference on entities and concepts under 2 challenging evaluation scenarios. Our experimental results on COMETA illustrate that no golden bullet exists and even the best mainstream techniques still have a significant performance gap to fill, while the best solution relies on combining different views of data.