MME-SCI: A Comprehensive and Challenging Science Benchmark for Multimodal Large Language Models

TL;DR

This paper introduces MME-SCI, a new comprehensive benchmark with 1,019 high-quality scientific questions across multiple subjects, languages, and evaluation modes, to better assess the reasoning and knowledge capabilities of multimodal large language models.

Contribution

It presents MME-SCI, a challenging, multilingual, and multi-modal benchmark with fine-grained annotations, addressing gaps in existing scientific evaluation benchmarks for MLLMs.

Findings

Existing models perform poorly on MME-SCI, indicating high difficulty.

Multilingual and domain-specific weaknesses are identified in current models.

Benchmark reveals significant room for improvement in reasoning and knowledge coverage.

Abstract

Recently, multimodal large language models (MLLMs) have achieved significant advancements across various domains, and corresponding evaluation benchmarks have been continuously refined and improved. In this process, benchmarks in the scientific domain have played an important role in assessing the reasoning capabilities of MLLMs. However, existing benchmarks still face three key challenges: 1) Insufficient evaluation of models' reasoning abilities in multilingual scenarios; 2) Inadequate assessment of MLLMs' comprehensive modality coverage; 3) Lack of fine-grained annotation of scientific knowledge points. To address these gaps, we propose MME-SCI, a comprehensive and challenging benchmark. We carefully collected 1,019 high-quality question-answer pairs, which involve 3 distinct evaluation modes. These pairs cover four subjects, namely mathematics, physics, chemistry, and biology, and support five languages: Chinese, English, French, Spanish, and Japanese. We conducted extensive experiments on 16 open-source models and 4 closed-source models, and the results demonstrate that MME-SCI is widely challenging for existing MLLMs. For instance, under the Image-only evaluation mode, o4-mini achieved accuracy of only 52.11%, 24.73%, 36.57%, and 29.80% in mathematics, physics, chemistry, and biology, respectively, indicating a significantly higher difficulty level compared to existing benchmarks. More importantly, using MME-SCI's multilingual and fine-grained knowledge attributes, we analyzed existing models' performance in depth and identified their weaknesses in specific domains. The Data and Evaluation Code are available at https://github.com/JCruan519/MME-SCI.