Exploring Neural Entity Representations for Semantic Information

TL;DR

This paper systematically evaluates eight neural entity embedding methods on probing and entity linking tasks, revealing their strengths and limitations in capturing semantic, relational, and frequency information across models and datasets.

Contribution

It provides a comprehensive, unified analysis of neural entity representations using diverse probing and linking tasks, addressing interpretability and generalization issues.

Findings

Certain methods effectively encode entity types and relationships.

Some embeddings capture mention frequency and factual info.

Performance varies across models and datasets.

Abstract

Neural methods for embedding entities are typically extrinsically evaluated on downstream tasks and, more recently, intrinsically using probing tasks. Downstream task-based comparisons are often difficult to interpret due to differences in task structure, while probing task evaluations often look at only a few attributes and models. We address both of these issues by evaluating a diverse set of eight neural entity embedding methods on a set of simple probing tasks, demonstrating which methods are able to remember words used to describe entities, learn type, relationship and factual information, and identify how frequently an entity is mentioned. We also compare these methods in a unified framework on two entity linking tasks and discuss how they generalize to different model architectures and datasets.