SPM-Bench: Benchmarking Large Language Models for Scanning Probe Microscopy

TL;DR

SPM-Bench is a novel, automated, multimodal benchmark designed to evaluate large language models' reasoning and understanding in the specialized domain of scanning probe microscopy, addressing previous limitations in data quality and complexity.

Contribution

The paper introduces SPM-Bench, a fully automated data synthesis pipeline, a new evaluation metric SIP-F1, and a framework for assessing LLMs' capabilities in scientific microscopy tasks.

Findings

High dataset purity achieved with AGS technology

Introduction of SIP-F1 score for rigorous evaluation

Model personality traits correlated with confidence and difficulty

Abstract

As LLMs achieved breakthroughs in general reasoning, their proficiency in specialized scientific domains reveals pronounced gaps in existing benchmarks due to data contamination, insufficient complexity, and prohibitive human labor costs. Here we present SPM-Bench, an original, PhD-level multimodal benchmark specifically designed for scanning probe microscopy (SPM). We propose a fully automated data synthesis pipeline that ensures both high authority and low-cost. By employing Anchor-Gated Sieve (AGS) technology, we efficiently extract high-value image-text pairs from arXiv and journal papers published between 2023 and 2025. Through a hybrid cloud-local architecture where VLMs return only spatial coordinates "llbox" for local high-fidelity cropping, our pipeline achieves extreme token savings while maintaining high dataset purity. To accurately and objectively evaluate the performance of the LLMs, we introduce the Strict Imperfection Penalty F1 (SIP-F1) score. This metric not only establishes a rigorous capability hierarchy but also, for the first time, quantifies model "personalities" (Conservative, Aggressive, Gambler, or Wise). By correlating these results with model-reported confidence and perceived difficulty, we expose the true reasoning boundaries of current AI in complex physical scenarios. These insights establish SPM-Bench as a generalizable paradigm for automated scientific data synthesis.