SciER: An Entity and Relation Extraction Dataset for Datasets, Methods, and Tasks in Scientific Documents

TL;DR

This paper introduces a comprehensive dataset for scientific entity and relation extraction, covering full-text articles with detailed annotations, to advance the development of more effective SciIE models.

Contribution

It provides a new, large-scale, full-text scientific dataset with fine-grained annotations and an out-of-distribution test set, enabling more realistic evaluation of SciIE models.

Findings

State-of-the-art models perform poorly on the dataset.

LLM-based baselines show promising results but still face challenges.

The dataset reveals complexities in extracting scientific entities and relations.

Abstract

Scientific information extraction (SciIE) is critical for converting unstructured knowledge from scholarly articles into structured data (entities and relations). Several datasets have been proposed for training and validating SciIE models. However, due to the high complexity and cost of annotating scientific texts, those datasets restrict their annotations to specific parts of paper, such as abstracts, resulting in the loss of diverse entity mentions and relations in context. In this paper, we release a new entity and relation extraction dataset for entities related to datasets, methods, and tasks in scientific articles. Our dataset contains 106 manually annotated full-text scientific publications with over 24k entities and 12k relations. To capture the intricate use and interactions among entities in full texts, our dataset contains a fine-grained tag set for relations. Additionally, we provide an out-of-distribution test set to offer a more realistic evaluation. We conduct comprehensive experiments, including state-of-the-art supervised models and our proposed LLM-based baselines, and highlight the challenges presented by our dataset, encouraging the development of innovative models to further the field of SciIE.