RuNNE-2022 Shared Task: Recognizing Nested Named Entities

TL;DR

The RuNNE-2022 shared task focuses on recognizing nested named entities in Russian texts, involving complex annotations and multiple entity types, with many submissions outperforming baseline models.

Contribution

This paper introduces a shared task dataset and evaluation framework for nested Russian NER, highlighting the challenge of recognizing overlapping and nested entities.

Findings

Half of the submissions outperform the BERT baseline.

Nested entities can be recognized with up to six levels of nesting.

The shared task provides insights into nested NER system performance.

Abstract

The RuNNE Shared Task approaches the problem of nested named entity recognition. The annotation schema is designed in such a way, that an entity may partially overlap or even be nested into another entity. This way, the named entity "The Yermolova Theatre" of type "organization" houses another entity "Yermolova" of type "person". We adopt the Russian NEREL dataset for the RuNNE Shared Task. NEREL comprises news texts written in the Russian language and collected from the Wikinews portal. The annotation schema includes 29 entity types. The nestedness of named entities in NEREL reaches up to six levels. The RuNNE Shared Task explores two setups. (i) In the general setup all entities occur more or less with the same frequency. (ii) In the few-shot setup the majority of entity types occur often in the training set. However, some of the entity types are have lower frequency, being thus challenging to recognize. In the test set the frequency of all entity types is even. This paper reports on the results of the RuNNE Shared Task. Overall the shared task has received 156 submissions from nine teams. Half of the submissions outperform a straightforward BERT-based baseline in both setups. This paper overviews the shared task setup and discusses the submitted systems, discovering meaning insights for the problem of nested NER. The links to the evaluation platform and the data from the shared task are available in our github repository: https://github.com/dialogue-evaluation/RuNNE.