GeomVerse: A Systematic Evaluation of Large Models for Geometric Reasoning

TL;DR

This paper systematically evaluates the reasoning abilities of vision-language models on geometry problems, revealing their limitations in complex multi-step reasoning tasks compared to prior benchmarks.

Contribution

It introduces a synthetic geometry dataset with controllable difficulty levels to systematically assess VLM reasoning capabilities across different complexity axes.

Findings

VLMs underperform on geometry reasoning tasks compared to expectations

Higher-depth problems require complex reasoning chains, exposing model limitations

The dataset enables targeted evaluation of reasoning depth in vision-language models

Abstract

Large language models have shown impressive results for multi-hop mathematical reasoning when the input question is only textual. Many mathematical reasoning problems, however, contain both text and image. With the ever-increasing adoption of vision language models (VLMs), understanding their reasoning abilities for such problems is crucial. In this paper, we evaluate the reasoning capabilities of VLMs along various axes through the lens of geometry problems. We procedurally create a synthetic dataset of geometry questions with controllable difficulty levels along multiple axes, thus enabling a systematic evaluation. The empirical results obtained using our benchmark for state-of-the-art VLMs indicate that these models are not as capable in subjects like geometry (and, by generalization, other topics requiring similar reasoning) as suggested by previous benchmarks. This is made especially clear by the construction of our benchmark at various depth levels, since solving higher-depth problems requires long chains of reasoning rather than additional memorized knowledge. We release the dataset for further research in this area.