mChartQA: A universal benchmark for multimodal Chart Question Answer based on Vision-Language Alignment and Reasoning

TL;DR

This paper presents mChartQA, a universal benchmark and model for multimodal chart question-answering that effectively handles complex visual and linguistic chart data through alignment and reasoning, outperforming existing methods.

Contribution

Introduces a novel multimodal chart QA model with a dual-phase training approach, enhancing interpretation and analysis of complex charts involving color, structure, and textless data.

Findings

Superior performance on multiple datasets

Effective handling of color, structure, and textless charts

Outperforms traditional methods in complex scenarios

Abstract

In the fields of computer vision and natural language processing, multimodal chart question-answering, especially involving color, structure, and textless charts, poses significant challenges. Traditional methods, which typically involve either direct multimodal processing or a table-to-text conversion followed by language model analysis, have limitations in effectively handling these complex scenarios. This paper introduces a novel multimodal chart question-answering model, specifically designed to address these intricate tasks. Our model integrates visual and linguistic processing, overcoming the constraints of existing methods. We adopt a dual-phase training approach: the initial phase focuses on aligning image and text representations, while the subsequent phase concentrates on optimizing the model's interpretative and analytical abilities in chart-related queries. This approach has demonstrated superior performance on multiple public datasets, particularly in handling color, structure, and textless chart questions, indicating its effectiveness in complex multimodal tasks.