RuREBus: a Case Study of Joint Named Entity Recognition and Relation Extraction from e-Government Domain

TL;DR

This paper explores the challenges of applying state-of-the-art NER and relation extraction models to a novel, non-English e-government corpus with unique annotation schemes, highlighting the need for more advanced methods.

Contribution

The study presents a comprehensive pipeline for joint NER and RE in a specialized domain, revealing limitations of current transformer-based models and emphasizing the necessity for improved techniques.

Findings

Transformer models show modest performance on the domain-specific corpus.

Fine-tuning on unlabeled data does not significantly improve results.

Current NER and RE technologies are insufficient for complex, non-English government documents.

Abstract

We show-case an application of information extraction methods, such as named entity recognition (NER) and relation extraction (RE) to a novel corpus, consisting of documents, issued by a state agency. The main challenges of this corpus are: 1) the annotation scheme differs greatly from the one used for the general domain corpora, and 2) the documents are written in a language other than English. Unlike expectations, the state-of-the-art transformer-based models show modest performance for both tasks, either when approached sequentially, or in an end-to-end fashion. Our experiments have demonstrated that fine-tuning on a large unlabeled corpora does not automatically yield significant improvement and thus we may conclude that more sophisticated strategies of leveraging unlabelled texts are demanded. In this paper, we describe the whole developed pipeline, starting from text annotation, baseline development, and designing a shared task in hopes of improving the baseline. Eventually, we realize that the current NER and RE technologies are far from being mature and do not overcome so far challenges like ours.