TurtleBench: A Visual Programming Benchmark in Turtle Geometry

TL;DR

TurtleBench is a new benchmark designed to evaluate large multimodal models' ability to interpret geometric patterns and generate code, revealing significant gaps in current AI capabilities compared to human understanding.

Contribution

The paper introduces TurtleBench, a novel benchmark based on turtle geometry to assess LMMs' visual reasoning and code generation in geometric tasks.

Findings

Leading LMMs perform poorly on TurtleBench tasks.

GPT-4o achieves only 19% accuracy on simple tasks.

Few-shot prompting yields less than 2% improvement.

Abstract

Humans have the ability to reason about geometric patterns in images and scenes from a young age. However, developing large multimodal models (LMMs) capable of similar reasoning remains a challenge, highlighting the need for robust evaluation methods to assess these capabilities. We introduce \Turtle, a benchmark designed to evaluate LMMs' capacity to interpret geometric patterns -- given visual examples, textual instructions, or both -- and generate precise code outputs. Inspired by turtle geometry, a notion used to teach children foundational coding and geometric concepts, TurtleBench features tasks with patterned shapes that have underlying algorithmic logic. Our evaluation reveals that leading LMMs struggle significantly with these tasks, with GPT-4o achieving only 19\% accuracy on the simplest tasks and few-shot prompting only marginally improves their performance ($<2\%$). \Turtle highlights the gap between human and AI performance in intuitive and visual geometrical understanding, setting the stage for future research in this area. \Turtle stands as one of the few benchmarks to evaluate the integration of visual understanding and code generation capabilities in LMMs, setting the stage for future research. Code and Dataset for this paper is provided here: \href{https://github.com/sinaris76/TurtleBench}{https://github.com/sinaris76/TurtleBench}