PaperMind: Benchmarking Agentic Reasoning and Critique over Scientific Papers in Multimodal LLMs

TL;DR

PaperMind is a comprehensive benchmark designed to evaluate integrated agentic reasoning and critique abilities of multimodal LLMs over scientific papers across multiple domains.

Contribution

It introduces a unified benchmark with diverse tasks to assess complex scientific reasoning and critique in multimodal language models.

Findings

Models show consistent performance gaps across tasks.

Existing models struggle with integrated scientific reasoning.

Benchmark reveals persistent challenges in multimodal scientific understanding.

Abstract

Understanding scientific papers requires more than answering isolated questions or summarizing content. It involves an integrated reasoning process that grounds textual and visual information, interprets experimental evidence, synthesizes information across sources, and critically evaluates scientific claims. However, existing benchmarks typically assess these abilities in isolation, making it difficult to evaluate scientific paper understanding as a unified set of interacting cognitive abilities. In this work, we introduce PaperMind, a benchmark designed to evaluate integrated and agent-oriented scientific reasoning over research papers. PaperMind is constructed from real scientific papers across seven domains, including agriculture, biology, chemistry, computer science, medicine, physics, and economics. It comprises four complementary task families that collectively operationalize distinct cognitive facets of scientific paper reasoning, including multimodal grounding, experimental interpretation, cross-source evidence reasoning, and critical assessment. By analyzing model behavior across multiple tasks, PaperMind enables a diagnostic evaluation of integrated scientific reasoning behaviors that are difficult to assess through isolated task evaluations. Extensive experiments on both opensource and closed-source multimodal LLMs reveal consistent performance gaps across tasks, highlighting persistent challenges in integrated scientific reasoning and critique. Our benchmark and dataset are available at https:// github.com/Yanjun-Zhao/PaperMind.