Why Vision Language Models Struggle with Visual Arithmetic? Towards Enhanced Chart and Geometry Understanding

TL;DR

This paper investigates why vision language models struggle with visual arithmetic tasks and introduces CogAlign, a post-training strategy that improves their reasoning abilities, especially in chart and geometry understanding, with less data.

Contribution

The paper identifies the root causes of visual arithmetic deficiencies in VLMs and proposes CogAlign, a novel post-training method inspired by cognitive development theory to enhance reasoning.

Findings

CogAlign significantly improves VLM performance on visual arithmetic tasks.

The method outperforms supervised fine-tuning with less training data.

Results show enhanced transfer to downstream tasks like chart and geometry understanding.

Abstract

Vision Language Models (VLMs) have achieved remarkable progress in multimodal tasks, yet they often struggle with visual arithmetic, seemingly simple capabilities like object counting or length comparison, which are essential for relevant complex tasks like chart understanding and geometric reasoning. In this work, we first investigate the root causes of this deficiency through a suite of probing tasks focusing on basic visual arithmetic. Our analysis reveals that while pre-trained vision encoders typically capture sufficient information, the text decoder often fails to decode it correctly for arithmetic reasoning. To address this, we propose CogAlign, a novel post-training strategy inspired by Piaget's theory of cognitive development. CogAlign trains VLMs to recognize invariant properties under visual transformations. We demonstrate that this approach significantly improves the performance of three diverse VLMs on our proposed probing tasks. Furthermore, CogAlign enhances performance by an average of 4.6% on CHOCOLATE and 2.9% on MATH-VISION, outperforming or matching supervised fine-tuning methods while requiring only 60% less training data. These results highlight the effectiveness and generalizability of CogAlign in improving fundamental visual arithmetic capabilities and their transfer to downstream tasks.