VisScience: An Extensive Benchmark for Evaluating K12 Educational Multi-modal Scientific Reasoning

TL;DR

VisScience is a comprehensive benchmark designed to evaluate multi-modal scientific reasoning in K12 education across mathematics, physics, and chemistry, highlighting current model strengths and weaknesses.

Contribution

This work introduces VisScience, a new extensive benchmark with 3,000 questions across three disciplines, filling a gap in existing MLLM evaluation benchmarks.

Findings

Closed-source MLLMs outperform open-source models.

Best accuracy: 53.4% in math, 38.2% in physics, 47.0% in chemistry.

Performance varies significantly across disciplines and models.

Abstract

Multi-modal large language models (MLLMs) have demonstrated promising capabilities across various tasks by integrating textual and visual information to achieve visual understanding in complex scenarios. Despite the availability of several benchmarks aims to evaluating MLLMs in tasks from visual question answering to complex problem-solving, most focus predominantly on mathematics or general visual understanding tasks. This reveals a critical gap in current benchmarks, which often overlook the inclusion of other key scientific disciplines such as physics and chemistry. To address this gap, we meticulously construct a comprehensive benchmark, named VisScience, which is utilized to assess the multi-modal scientific reasoning across the three disciplines of mathematics, physics, and chemistry. This benchmark comprises 3,000 questions drawn from K12 education - spanning elementary school through high school - equally distributed across three disciplines, with 1,000 questions per discipline. The questions within VisScience span 21 distinct subjects and are categorized into five difficulty levels, offering a broad spectrum of topics within each discipline. With VisScience, we present a detailed evaluation of the performance of 25 representative MLLMs in scientific reasoning. Experimental results demonstrate that closed-source MLLMs generally outperform open-source models. The best performance observed include a 53.4\% accuracy in mathematics by Claude3.5-Sonnet, 38.2\% in physics by GPT-4o, and 47.0\% in chemistry by Gemini-1.5-Pro. These results underscore the strengths and limitations of MLLMs, suggesting areas for future improvement and highlighting the importance of developing models that can effectively handle the diverse demands of multi-modal scientific reasoning.