FMMD: A multimodal open peer review dataset based on F1000Research

TL;DR

FMMD is a comprehensive, multimodal open peer review dataset from F1000Research that aligns reviewer comments with specific manuscript versions, supporting advanced AI research in peer review across diverse scientific fields.

Contribution

The paper introduces FMMD, a novel multimodal dataset that integrates visual, structural, and version-specific review data, addressing limitations of existing text-centric peer review datasets.

Findings

Enables fine-grained analysis of peer review lifecycle

Supports multimodal issue detection and comment generation

Bridges gap between reviewer comments and manuscript versions

Abstract

Automated scholarly paper review (ASPR) has entered the coexistence phase with traditional peer review, where artificial intelligence (AI) systems are increasingly incorporated into real-world manuscript evaluation. In parallel, research on automated and AI-assisted peer review has proliferated. Despite this momentum, empirical progress remains constrained by several critical limitations in existing datasets. While reviewers routinely evaluate figures, tables, and complex layouts to assess scientific claims, most existing datasets remain overwhelmingly text-centric. This bias is reinforced by a narrow focus on data from computer science venues. Furthermore, these datasets lack precise alignment between reviewer comments and specific manuscript versions, obscuring the iterative relationship between peer review and manuscript evolution. In response, we introduce FMMD, a multimodal and multidisciplinary open peer review dataset curated from F1000Research. The dataset bridges the current gap by integrating manuscript-level visual and structural data with version-specific reviewer reports and editorial decisions. By providing explicit alignment between reviewer comments and the exact article iteration under review, FMMD enables fine-grained analysis of the peer review lifecycle across diverse scientific domains. FMMD supports tasks such as multimodal issue detection and multimodal review comment generation. It provides a comprehensive empirical resource for the development of peer review research.