VProChart: Answering Chart Question through Visual Perception Alignment Agent and Programmatic Solution Reasoning

TL;DR

VProChart is a novel framework that improves chart question answering by aligning visual perception with programmatic reasoning, enabling better interpretation and reasoning over complex chart data.

Contribution

The paper introduces VProChart, combining a visual perception alignment agent with programmatic reasoning to enhance chart understanding and question answering performance.

Findings

Outperforms existing CQA methods on ChartQA and PlotQA datasets

Effectively models chart elements based on human visual perception principles

Enables precise numerical and logical reasoning in chart interpretation

Abstract

Charts are widely used for data visualization across various fields, including education, research, and business. Chart Question Answering (CQA) is an emerging task focused on the automatic interpretation and reasoning of data presented in charts. However, chart images are inherently difficult to interpret, and chart-related questions often involve complex logical and numerical reasoning, which hinders the performance of existing models. This paper introduces VProChart, a novel framework designed to address these challenges in CQA by integrating a lightweight Visual Perception Alignment Agent (VPAgent) and a Programmatic Solution Reasoning approach. VPAgent aligns and models chart elements based on principles of human visual perception, enhancing the understanding of chart context. The Programmatic Solution Reasoning approach leverages large language models (LLMs) to transform natural language reasoning questions into structured solution programs, facilitating precise numerical and logical reasoning. Extensive experiments on benchmark datasets such as ChartQA and PlotQA demonstrate that VProChart significantly outperforms existing methods, highlighting its capability in understanding and reasoning with charts.