In-Depth and In-Breadth: Pre-training Multimodal Language Models Customized for Comprehensive Chart Understanding

TL;DR

ChartScope is a novel vision-language model designed for comprehensive understanding of diverse scientific charts, utilizing synthesized data and dual-path training to improve data alignment and reasoning capabilities across many chart types.

Contribution

The paper introduces ChartScope, a new LVLM with an efficient data generation pipeline and dual-path training strategy for broad and deep chart comprehension.

Findings

Significantly improves understanding across diverse chart types.

Outperforms existing models on the ChartDQA benchmark.

Demonstrates effective reasoning over underlying chart data.

Abstract

Recent methods for customizing Large Vision Language Models (LVLMs) for domain-specific tasks have shown promising results in scientific chart comprehension. However, existing approaches face two major limitations: First, they rely on paired data from only a few chart types, limiting generalization to wide range of chart types. Secondly, they lack targeted pre-training for chart-data alignment, which hampers the model's understanding of underlying data. In this paper, we introduce ChartScope, an LVLM optimized for in-depth chart comprehension across diverse chart types. We propose an efficient data generation pipeline that synthesizes paired data for a wide range of chart types, along with a novel Dual-Path training strategy that enabling the model to succinctly capture essential data details while preserving robust reasoning capabilities by incorporating reasoning over the underlying data. Lastly, we establish ChartDQA, a new benchmark for evaluating not only question-answering at different levels but also underlying data understanding. Experimental results demonstrate that ChartScope significantly enhances comprehension on a wide range of chart types. The code and data are available at https://davidhalladay.github.io/chartscope_demo.