OmniScience: A Large-scale Multi-modal Dataset for Scientific Image Understanding

TL;DR

OmniScience introduces a large, high-quality multi-modal dataset of scientific images with dense descriptions, enabling improved training and evaluation of models in scientific image understanding across multiple disciplines.

Contribution

The paper presents OmniScience, a comprehensive dataset with a novel re-captioning pipeline and quality filtering, enhancing scientific image understanding for large multimodal models.

Findings

Significant improvement in multi-modal similarity scores.

Enhanced model performance on scientific image understanding benchmarks.

High-fidelity, densely annotated dataset covering multiple scientific disciplines.

Abstract

Multimodal Large Language Models demonstrate strong performance on natural image understanding, yet exhibit limited capability in interpreting scientific images, including but not limited to schematic diagrams, experimental characterizations, and analytical charts. This limitation is particularly pronounced in open-source MLLMs. The gap largely stems from existing datasets with limited domain coverage, coarse structural annotations, and weak semantic grounding. We introduce OmniScience, a large-scale, high-fidelity multi-modal dataset comprising 1.5 million figure-caption-context triplets, spanning more than 10 major scientific disciplines. To obtain image caption data with higher information density and accuracy for multi-modal large-model training, we develop a dynamic model-routing re-captioning pipeline that leverages state-of-the-art multi-modal large language models to generate dense, self-contained descriptions by jointly synthesizing visual features, original figure captions, and corresponding in-text references authored by human scientists. The pipeline is further reinforced with rigorous quality filtering and alignment with human expert judgments, ensuring both factual accuracy and semantic completeness, and boosts the image-text multi-modal similarity score from 0.769 to 0.956. We further propose a caption QA protocol as a proxy task for evaluating visual understanding. Under this setting, Qwen2.5-VL-3B model finetuned on OmniScience show substantial gains over baselines, achieving a gain of 0.378 on MM-MT-Bench and a gain of 0.140 on MMMU.