SpecVQA: A Benchmark for Spectral Understanding and Visual Question Answering in Scientific Images

TL;DR

SpecVQA introduces a comprehensive benchmark for evaluating multimodal models' ability to understand and reason about scientific spectral images, facilitating progress in scientific visual-language AI.

Contribution

The paper presents a new benchmark with expert-annotated QA pairs for spectral scientific images and proposes a data sampling method to improve model performance.

Findings

A spectral data sampling approach improves model accuracy.

SpecVQA covers 7 spectrum types with 3100 QA pairs.

Benchmark enables evaluation of scientific spectral understanding.

Abstract

Spectra are a prevalent yet highly information-dense form of scientific imagery, presenting substantial challenges to multimodal large language models (MLLMs) due to their unstructured and domain-specific characteristics. Here we introduce SpecVQA, a professional scientific-image benchmark for evaluating multimodal models on scientific spectral understanding, covering 7 representative spectrum types with expert-annotated question-answer pairs. The aim comprises two aspects: spectra scientific QA evaluation and corresponding underlying task evaluation. SpecVQA contains 620 figures and 3100 QA pairs curated from peer-reviewed literature, targeting both direct information extraction and domain-specific reasoning. To effectively reduce token length while preserving essential curve characteristics, we propose a spectral data sampling and interpolation reconstruction approach. Ablation studies further confirm that the approach achieves substantial performance improvements on the proposed benchmark. We test the capability of prominent MLLMs in scientific spectral understanding on our benchmark and present a leaderboard. This work represents an essential step toward enhancing spectral understanding in multimodal large models and suggests promising directions for extending visual-language models to broader scientific research and data analysis.