SoMeSci- A 5 Star Open Data Gold Standard Knowledge Graph of Software Mentions in Scientific Articles

TL;DR

This paper introduces SoMeSci, a comprehensive gold standard knowledge graph of software mentions in scientific articles, enabling improved automatic extraction, disambiguation, and analysis of software usage in research.

Contribution

It presents the first high-quality, annotated dataset of software mentions in scientific literature, including relation labels and mention types, supporting various NLP tasks.

Findings

High inter-annotator agreement (κ=0.82)

Contains 3756 software mentions in 1367 articles

Provides baseline results for NLP tasks

Abstract

Knowledge about software used in scientific investigations is important for several reasons, for instance, to enable an understanding of provenance and methods involved in data handling. However, software is usually not formally cited, but rather mentioned informally within the scholarly description of the investigation, raising the need for automatic information extraction and disambiguation. Given the lack of reliable ground truth data, we present SoMeSci (Software Mentions in Science) a gold standard knowledge graph of software mentions in scientific articles. It contains high quality annotations (IRR: $\kappa{=}.82$) of 3756 software mentions in 1367 PubMed Central articles. Besides the plain mention of the software, we also provide relation labels for additional information, such as the version, the developer, a URL or citations. Moreover, we distinguish between different types, such as application, plugin or programming environment, as well as different types of mentions, such as usage or creation. To the best of our knowledge, SoMeSci is the most comprehensive corpus about software mentions in scientific articles, providing training samples for Named Entity Recognition, Relation Extraction, Entity Disambiguation, and Entity Linking. Finally, we sketch potential use cases and provide baseline results.