NoReGeo: Non-Reasoning Geometry Benchmark

TL;DR

NoReGeo is a benchmark that evaluates large language models' innate geometric understanding without reasoning, revealing current models' limited ability to grasp geometric concepts directly.

Contribution

The paper introduces NoReGeo, a new benchmark for assessing intrinsic geometric understanding in LLMs, highlighting the gap in native geometric cognition.

Findings

State-of-the-art models achieve up to 65% accuracy on geometric tasks.

Geometric understanding does not improve significantly through fine-tuning.

Current models lack innate geometric comprehension, as shown by benchmark results.

Abstract

We present NoReGeo, a novel benchmark designed to evaluate the intrinsic geometric understanding of large language models (LLMs) without relying on reasoning or algebraic computation. Unlike existing benchmarks that primarily assess models' proficiency in reasoning-based geometry-where solutions are derived using algebraic methods-NoReGeo focuses on evaluating whether LLMs can inherently encode spatial relationships and recognize geometric properties directly. Our benchmark comprises 2,500 trivial geometric problems spanning 25 categories, each carefully crafted to be solvable purely through native geometric understanding, assuming known object locations. We assess a range of state-of-the-art models on NoReGeo, including frontier models like GPT-4, observing that even the most advanced systems achieve an overall maximum of 65% accuracy in binary classification tasks. Further, our ablation experiments demonstrate that such geometric understanding does not emerge through fine-tuning alone, indicating that effective training for geometric comprehension requires a specialized approach from the outset. Our findings highlight a significant gap in current LLMs' ability to natively grasp geometric concepts, providing a foundation for future research toward models with true geometric cognition.