Forgotten Polygons: Multimodal Large Language Models are Shape-Blind

TL;DR

Multimodal Large Language Models (MLLMs) exhibit significant shape recognition shortcomings, especially in geometric reasoning, but can be substantially improved with visual prompting techniques like VC-CoT, highlighting the need for better visual reasoning integration.

Contribution

This paper systematically evaluates MLLMs' geometric reasoning abilities, identifies their reliance on intuitive associations, and introduces VC-CoT prompting to significantly enhance visual-math reasoning performance.

Findings

MLLMs achieve under 50% accuracy in recognizing regular polygons.

Reliance on System 1 leads to failure in counting sides of shapes.

VC-CoT prompting boosts accuracy from 7% to 93% in side-counting tasks.

Abstract

Despite strong performance on vision-language tasks, Multimodal Large Language Models (MLLMs) struggle with mathematical problem-solving, with both open-source and state-of-the-art models falling short of human performance on visual-math benchmarks. To systematically examine visual-mathematical reasoning in MLLMs, we (1) evaluate their understanding of geometric primitives, (2) test multi-step reasoning, and (3) explore a potential solution to improve visual reasoning capabilities. Our findings reveal fundamental shortcomings in shape recognition, with top models achieving under 50% accuracy in identifying regular polygons. We analyze these failures through the lens of dual-process theory and show that MLLMs rely on System 1 (intuitive, memorized associations) rather than System 2 (deliberate reasoning). Consequently, MLLMs fail to count the sides of both familiar and novel shapes, suggesting they have neither learned the concept of sides nor effectively process visual inputs. Finally, we propose Visually Cued Chain-of-Thought (VC-CoT) prompting, which enhances multi-step mathematical reasoning by explicitly referencing visual annotations in diagrams, boosting GPT-4o's accuracy on an irregular polygon side-counting task from 7% to 93%. Our findings suggest that System 2 reasoning in MLLMs remains an open problem, and visually-guided prompting is essential for successfully engaging visual reasoning. Code available at: https://github.com/rsinghlab/Shape-Blind.