Enhancing Question Answering on Charts Through Effective Pre-training Tasks

TL;DR

This paper improves chart question answering by analyzing current models' limitations and introducing three pre-training tasks that enhance understanding of visual, structural, and numerical aspects, leading to better performance.

Contribution

It proposes three novel pre-training tasks to enhance visual and structural understanding in chart question answering models, addressing key limitations of existing approaches.

Findings

Achieved 1.7% average improvement over baseline models

Identified key weaknesses in current models' understanding of chart structure and numerical data

Validated effectiveness across multiple chart datasets

Abstract

To completely understand a document, the use of textual information is not enough. Understanding visual cues, such as layouts and charts, is also required. While the current state-of-the-art approaches for document understanding (both OCR-based and OCR-free) work well, a thorough analysis of their capabilities and limitations has not yet been performed. Therefore, in this work, we addresses the limitation of current VisualQA models when applied to charts and plots. To investigate shortcomings of the state-of-the-art models, we conduct a comprehensive behavioral analysis, using ChartQA as a case study. Our findings indicate that existing models particularly underperform in answering questions related to the chart's structural and visual context, as well as numerical information. To address these issues, we propose three simple pre-training tasks that enforce the existing model in terms of both structural-visual knowledge, as well as its understanding of numerical questions. We evaluate our pre-trained model (called MatCha-v2) on three chart datasets - both extractive and abstractive question datasets - and observe that it achieves an average improvement of 1.7% over the baseline model.