NERetrieve: Dataset for Next Generation Named Entity Recognition and Retrieval

TL;DR

This paper introduces NERetrieve, a new dataset and task variants for advanced NER, including fine-grained, zero-shot, and retrieval-based recognition, to push the boundaries of current NLP capabilities.

Contribution

It presents three novel NER task variants and a large-scale dataset supporting fine-grained, zero-shot, and retrieval-oriented entity recognition, expanding beyond traditional NER approaches.

Findings

LLMs enable new NER capabilities but are not yet fully solved.

A large silver-annotated corpus of 4 million paragraphs is provided.

The proposed variants pose significant challenges for future research.

Abstract

Recognizing entities in texts is a central need in many information-seeking scenarios, and indeed, Named Entity Recognition (NER) is arguably one of the most successful examples of a widely adopted NLP task and corresponding NLP technology. Recent advances in large language models (LLMs) appear to provide effective solutions (also) for NER tasks that were traditionally handled with dedicated models, often matching or surpassing the abilities of the dedicated models. Should NER be considered a solved problem? We argue to the contrary: the capabilities provided by LLMs are not the end of NER research, but rather an exciting beginning. They allow taking NER to the next level, tackling increasingly more useful, and increasingly more challenging, variants. We present three variants of the NER task, together with a dataset to support them. The first is a move towards more fine-grained -- and intersectional -- entity types. The second is a move towards zero-shot recognition and extraction of these fine-grained types based on entity-type labels. The third, and most challenging, is the move from the recognition setup to a novel retrieval setup, where the query is a zero-shot entity type, and the expected result is all the sentences from a large, pre-indexed corpus that contain entities of these types, and their corresponding spans. We show that all of these are far from being solved. We provide a large, silver-annotated corpus of 4 million paragraphs covering 500 entity types, to facilitate research towards all of these three goals.