# A practical and nuanced framework for entity linking evaluation

## TL;DR

This paper introduces a new framework for evaluating entity linking systems that considers semantic relationships and hierarchical structures, leading to more accurate assessments.

## Contribution

A novel entity linking evaluation framework that incorporates hierarchical structures and shifts focus from text-level matching to semantic comparison.

## Key findings

- The framework emphasizes hierarchical vocabulary structures and semantic comparison for better evaluation.
- Results show the importance of aligning metrics with application-specific needs.
- Structured hierarchical error analysis is provided for entity linking.

## Abstract

Entity linking maps textual mentions with entities in vocabularies. While accuracy is the primary metric for entity linking evaluation, it fails to capture the complexity of model behaviour.

We propose an entity linking evaluation framework that clarifies the target objects of metrics calculation, incorporates term hierarchy, generates performance profiles, and summarises them as model characteristics. The framework emphasises hierarchical vocabulary structures, shifting the focus from text-level label matching to semantic comparison between concepts. We illustrate the competence and utility of this framework through a case study on disease entity linking.

Our results highlight the importance of aligning evaluation metrics with application-specific requirements, and provide structured hierarchical error analysis for entity linking, paving the way for more nuanced and practical assessments of entity linking systems.