Tangram: Benchmark for Evaluating Geometric Element Recognition in Large Multimodal Models

TL;DR

Tangram introduces a benchmark for assessing large multimodal models' ability to recognize geometric elements in diagrams, revealing significant performance gaps and highlighting the need for improved perception capabilities.

Contribution

We created Tangram, a comprehensive benchmark with diverse geometric diagrams to evaluate LMMs' geometric element recognition, an area underexplored in current research.

Findings

Top model accuracy is only 53% on geometric recognition tasks.

Current LMMs struggle with basic geometric perception.

The benchmark exposes significant gaps in multimodal models' understanding.

Abstract

Significant advancements in Large Multimodal Models (LMMs) have enabled them to tackle complex problems involving visual-mathematical reasoning. However, their ability to identify geometric elements remains underexplored. To address this gap, we introduce Tangram, a novel benchmark designed to evaluate the performance of LMMs on geometric element recognition. Tangram comprises 1,080 diverse geometric diagrams sourced from primary and secondary school exams, competitions, and textbooks, ranging from simple geometric shapes to complex combinations. Each diagram is paired with four questions, resulting in 4,320 visual-question-answer pairs. Unlike existing benchmarks that emphasize higher-level cognition and reasoning, Tangram focuses on understanding geometric elements, requiring models to perform a ``simple yet challenging" counting task. Systematic evaluation of 13 prominent LMMs, such as GPT-4o and Claude 3.5 Sonnet, reveals that these models face significant challenges even in seemingly straightforward tasks. The top-performing model achieves an accuracy of only 53.0%, highlighting a substantial gap compared to human performance. These findings underscore the limitations of current multimodal AI systems in handling basic perception tasks and serve to inspire the development of the next generation of expert-level multimodal foundational models. The data and code will be released soon.