Benchmarking Cross-Scale Perception Ability of Large Multimodal Models in Material Science

TL;DR

This paper introduces CSMBench, a new benchmark dataset for evaluating large multimodal models' ability to interpret scientific figures across different physical scales in materials science, revealing significant performance variability.

Contribution

The paper presents CSMBench, a comprehensive dataset and evaluation framework specifically designed to assess hierarchical perception abilities of models in materials science.

Findings

Performance varies across physical scales due to visual differences.

Current models show limitations in understanding hierarchical scientific figures.

The benchmark highlights key directions for improving model understanding in materials research.

Abstract

Unraveling the hierarchical structure-property relationships is the central challenge of materials science, necessitating the interpretation of data across vast physical scales from micro to macro. Despite the rapid integration of Large Multimodal Models (LMMs) into scientific workflows, existing scientific benchmarks primarily focus on general chart interpretation or isolated common-sense reasoning, failing to capture reasoning ability across intricate physical dimensions. To address this, we introduce CSMBench, a dataset comprising 1,041 high-quality figures curated from premier journals up to September 2025. CSMBench categorizes data into four scientifically distinct regimes: atomic, micro, meso, and macro scales, strictly aligning with the focus and definitions in materials study. Through open-ended figure description and multiple-choice caption matching tasks, we evaluate state-of-the-art open-source and closed-source models. Our analysis identifies that performance varies significantly across physical scales due to the distinct visual characteristics, highlighting the limitations of current generalist models and identifying critical directions for achieving hierarchical and accurate understanding in materials research. The CSMBench is publicly released at: https://huggingface.co/datasets/lututu/CSMBench.