Automatically Annotating Articles Towards Opening and Reusing Transparent Peer Reviews

TL;DR

This paper introduces AR-Annotator, a tool that semantically annotates articles and reviews in open peer review systems, enabling structured, reusable, and queryable representations to improve transparency and secondary analysis.

Contribution

The paper presents AR-Annotator, a novel semantic annotation system that structures open peer reviews and articles for enhanced transparency and reusability in scientific publishing.

Findings

Structured representations enable complex quality queries.

Semantic markup improves transparency and secondary analysis.

Linked Data format maximizes reusability.

Abstract

An increasing number of scientific publications are created in open and transparent peer review models: a submission is published first, and then reviewers are invited, or a submission is reviewed in a closed environment but then these reviews are published with the final article, or combinations of these. Reasons for open peer review include giving better credit to reviewers and enabling readers to better appraise the quality of a publication. In most cases, the full, unstructured text of an open review is published next to the full, unstructured text of the article reviewed. This approach prevents human readers from getting a quick impression of the quality of parts of an article, and it does not easily support secondary exploitation, e.g., for scientometrics on reviews. While document formats have been proposed for publishing structured articles including reviews, integrated tool support for entire open peer review workflows resulting in such documents is still scarce. We present AR-Annotator, the Automatic Article and Review Annotator which employs a semantic information model of an article and its reviews, using semantic markup and unique identifiers for all entities of interest. The fine-grained article structure is not only exposed to authors and reviewers but also preserved in the published version. We publish articles and their reviews in a Linked Data representation and thus maximize their reusability by third-party applications. We demonstrate this reusability by running quality-related queries against the structured representation of articles and their reviews.