Rethinking Comprehensive Benchmark for Chart Understanding: A Perspective from Scientific Literature

TL;DR

This paper introduces SCI-CQA, a comprehensive benchmark for scientific chart understanding that includes diverse chart types, realistic questions, and contextual evaluation, aiming to better assess multimodal model capabilities in real-world scientific literature.

Contribution

The paper presents a new large-scale benchmark dataset, SCI-CQA, with a novel evaluation framework and an efficient annotation pipeline, focusing on complex scientific charts and contextual understanding.

Findings

Curated 37,607 high-quality scientific charts with context from top-tier conferences.

Developed 5,629 human-like questions, including open-ended and objective types.

Highlighted the importance of context in answering complex chart questions.

Abstract

Scientific Literature charts often contain complex visual elements, including multi-plot figures, flowcharts, structural diagrams and etc. Evaluating multimodal models using these authentic and intricate charts provides a more accurate assessment of their understanding abilities. However, existing benchmarks face limitations: a narrow range of chart types, overly simplistic template-based questions and visual elements, and inadequate evaluation methods. These shortcomings lead to inflated performance scores that fail to hold up when models encounter real-world scientific charts. To address these challenges, we introduce a new benchmark, Scientific Chart QA (SCI-CQA), which emphasizes flowcharts as a critical yet often overlooked category. To overcome the limitations of chart variety and simplistic visual elements, we curated a dataset of 202,760 image-text pairs from 15 top-tier computer science conferences papers over the past decade. After rigorous filtering, we refined this to 37,607 high-quality charts with contextual information. SCI-CQA also introduces a novel evaluation framework inspired by human exams, encompassing 5,629 carefully curated questions, both objective and open-ended. Additionally, we propose an efficient annotation pipeline that significantly reduces data annotation costs. Finally, we explore context-based chart understanding, highlighting the crucial role of contextual information in solving previously unanswerable questions.