Jagged AI in Scientific Peer Review: Evidence from POMP Data Analysis

TL;DR

This study investigates the uneven performance of AI tools in scientific peer review, revealing that AI exhibits a jagged capability profile, excelling in technical error detection but struggling with interpretive and narrative assessments.

Contribution

It provides empirical evidence of AI's jagged performance in peer review across diverse tasks and shows that this pattern is inherent to the AI model, not just specific instructions.

Findings

AI reviewers detect technical errors effectively

AI struggles with interpretive and narrative errors

Jagged performance pattern is consistent across AI agents

Abstract

Despite their growing use in academic writing and statistical analysis, the performance of artificial intelligence (AI) tools in scientific peer review remains a largely unexplored area. A key challenge is jagged AI, a phenomenon where AI exhibits strong ability spikes in some domains while remaining deficient in others. To study this jaggedness in a practical data science context, we considered the task of reviewing partially observed Markov process (POMP) data analyses. POMP models, also known as state-space models or hidden Markov models, are used to fit mechanistic dynamic models to time series data in diverse applications including disease transmission, ecological dynamics, and financial risk assessment. High-quality peer review in this area entails assessment of scientific context, identification of errors in implementing complex algorithms, and decisions concerning methodological best practices. We studied 72 POMP projects from four semesters of a University of Michigan graduate time series course for which the project reports, the source code, and student peer reviews are anonymized and open-access. We compared the human reviews with four AI reviewing agents, using Claude Code with differing instructions implemented as skill files. We found that AI reviewers exhibited a jagged capability profile, proficiently catching human-overlooked technical errors and invalid inference methodology, while failing to match human standards in checking interpretive errors, narrative coherence, and domain-informed model critique. The jaggedness was found to be similar for all agents, consistent with it being primarily a property of the underlying AI model rather than the specific instructions. Skill file configuration shifted which weaknesses agents emphasized, without removing the jaggedness.