BELB: a Biomedical Entity Linking Benchmark

TL;DR

This paper introduces BELB, a standardized benchmark for biomedical entity linking, enabling consistent evaluation across multiple corpora and knowledge bases, and assesses current systems' performance.

Contribution

The paper presents BELB, the first unified benchmark for biomedical entity linking, facilitating reproducible evaluation across diverse datasets and knowledge bases.

Findings

Neural models show inconsistent performance across entity types.

BELB reduces preprocessing effort for testing BEL systems.

Rule-based systems perform variably depending on entity type.

Abstract

Biomedical entity linking (BEL) is the task of grounding entity mentions to a knowledge base. It plays a vital role in information extraction pipelines for the life sciences literature. We review recent work in the field and find that, as the task is absent from existing benchmarks for biomedical text mining, different studies adopt different experimental setups making comparisons based on published numbers problematic. Furthermore, neural systems are tested primarily on instances linked to the broad coverage knowledge base UMLS, leaving their performance to more specialized ones, e.g. genes or variants, understudied. We therefore developed BELB, a Biomedical Entity Linking Benchmark, providing access in a unified format to 11 corpora linked to 7 knowledge bases and spanning six entity types: gene, disease, chemical, species, cell line and variant. BELB greatly reduces preprocessing overhead in testing BEL systems on multiple corpora offering a standardized testbed for reproducible experiments. Using BELB we perform an extensive evaluation of six rule-based entity-specific systems and three recent neural approaches leveraging pre-trained language models. Our results reveal a mixed picture showing that neural approaches fail to perform consistently across entity types, highlighting the need of further studies towards entity-agnostic models.