SIQA: Toward Reliable Scientific Image Quality Assessment

TL;DR

This paper introduces SIQA, a new framework for assessing scientific image quality by evaluating both scientific correctness and perceptual clarity, addressing limitations of existing perceptual-focused IQA methods.

Contribution

The paper proposes a multidimensional SIQA framework with two evaluation protocols and a large-scale benchmark, advancing scientific image quality assessment beyond perceptual fidelity.

Findings

Models align well with expert ratings on scoring but less on understanding.

Fine-tuning improves both understanding and scoring, with scoring gains being larger.

Discrepancy between scoring and understanding highlights the need for multidimensional evaluation.

Abstract

Scientific images fundamentally differ from natural and AI-generated images in that they encode structured domain knowledge rather than merely depict visual scenes. Assessing their quality therefore requires evaluating not only perceptual fidelity but also scientific correctness and logical completeness. However, existing image quality assessment (IQA) paradigms primarily focus on perceptual distortions or image-text alignment, implicitly assuming that depicted content is factually valid. This assumption breaks down in scientific contexts, where visually plausible figures may still contain conceptual errors or incomplete reasoning. To address this gap, we introduce Scientific Image Quality Assessment (SIQA), a framework that models scientific image quality along two complementary dimensions: Knowledge (Scientific Validity and Scientific Completeness) and Perception (Cognitive Clarity and Disciplinary Conformity). To operationalize this formulation, we design two evaluation protocols: SIQA-U (Understanding), which measures semantic comprehension of scientific content through multiple-choice tasks, and SIQA-S (Scoring), which evaluates alignment with expert quality judgments. We further construct the SIQA Challenge, consisting of an expert-annotated benchmark and a large-scale training set. Experiments across representative multimodal large language models (MLLMs) reveal a consistent discrepancy between scoring alignment and scientific understanding. While models can achieve strong agreement with expert ratings under SIQA-S, their performance on SIQA-U remains substantially lower. Fine-tuning improves both metrics, yet gains in scoring consistently outpace improvements in understanding. These results suggest that rating consistency alone may not reliably reflect scientific comprehension, underscoring the necessity of multidimensional evaluation for scientific image quality assessment.