VeriSciQA: An Auto-Verified Dataset for Scientific Visual Question Answering

TL;DR

VeriSciQA is a large, high-quality dataset for scientific visual question answering, created through a cross-modal verification framework that ensures accurate question-answer pairs from scientific figures and their citing paragraphs.

Contribution

The paper introduces VeriSciQA, a novel dataset for SVQA, generated using a verification framework that filters out errors, improving data quality for scientific visual reasoning tasks.

Findings

Models fine-tuned on VeriSciQA outperform those trained on previous datasets.

There is a significant accuracy gap between open-source and proprietary models on SVQA.

Scaling data with VeriSciQA enhances model performance on SVQA benchmarks.

Abstract

Large Vision-Language Models (LVLMs) show promise for scientific applications, yet open-source models still struggle with Scientific Visual Question Answering (SVQA), namely answering questions about figures from scientific papers. A key bottleneck is the lack of public, large-scale, high-quality SVQA datasets. Although recent work uses LVLMs to synthesize data at scale, we identify systematic errors in their resulting QA pairs, stemming from LVLMs' inherent limitations and information asymmetry between figures and text. To address these challenges, we propose a Cross-Modal verification framework that generates questions and answers purely from figure-citing paragraphs, then verifies them against the figures themselves, leveraging the inherent text-figure alignment in scientific papers to filter out erroneous QA pairs. We instantiate this framework to curate VeriSciQA, a dataset of 20,272 QA pairs spanning 20 scientific domains and 12 figure types. Difficulty assessment reveals a notable accuracy gap between the best open-source model (65%) and the best proprietary model (80.5%), demonstrating room for improvement. Moreover, models fine-tuned on VeriSciQA achieve consistent improvements on SVQA benchmarks, with performance gains that scale with data size, surpassing models trained on existing datasets. Human evaluation further validates the improved quality of VeriSciQA. These results demonstrate that continued data expansion via our scalable framework can further advance SVQA capability in the open-source community. Our dataset is publicly available at https://huggingface.co/datasets/datajuicer/VeriSciQA.