All Entities are Not Created Equal: Examining the Long Tail for Ultra-Fine Entity Typing

TL;DR

This paper investigates the limitations of pre-trained language models in ultra-fine entity typing, especially for infrequent entities, and proposes methods to better understand and address these challenges.

Contribution

It introduces a heuristic to approximate pre-training entity distribution and demonstrates the shortcomings of PLMs for long-tail entities, suggesting the need for beyond-PLM solutions.

Findings

PLMs struggle with long-tail entities in ultra-fine typing.

Knowledge-infused methods can mitigate some PLM limitations.

Understanding pre-training data distribution is crucial for improving entity typing.

Abstract

Due to their capacity to acquire world knowledge from large corpora, pre-trained language models (PLMs) are extensively used in ultra-fine entity typing tasks where the space of labels is extremely large. In this work, we explore the limitations of the knowledge acquired by PLMs by proposing a novel heuristic to approximate the pre-training distribution of entities when the pre-training data is unknown. Then, we systematically demonstrate that entity-typing approaches that rely solely on the parametric knowledge of PLMs struggle significantly with entities at the long tail of the pre-training distribution, and that knowledge-infused approaches can account for some of these shortcomings. Our findings suggest that we need to go beyond PLMs to produce solutions that perform well for infrequent entities.