GeoDANO: Geometric VLM with Domain Agnostic Vision Encoder

TL;DR

GeoDANO is a novel geometric vision-language model with a domain-agnostic encoder that effectively recognizes geometric features and generalizes across diagram styles, outperforming existing methods in plane geometry problem solving.

Contribution

We introduce GeoDANO, a geometric VLM with a domain-agnostic encoder, and a benchmark for geometric feature recognition, addressing limitations of general-purpose vision encoders.

Findings

GeoCLIP outperforms existing vision encoders in recognizing geometric features.

GeoDANO surpasses specialized methods and GPT-4o on MathVerse.

Our benchmark reveals general-purpose VLMs struggle with geometric feature detection.

Abstract

We introduce GeoDANO, a geometric vision-language model (VLM) with a domain-agnostic vision encoder, for solving plane geometry problems. Although VLMs have been employed for solving geometry problems, their ability to recognize geometric features remains insufficiently analyzed. To address this gap, we propose a benchmark that evaluates the recognition of visual geometric features, including primitives such as dots and lines, and relations such as orthogonality. Our preliminary study shows that vision encoders often used in general-purpose VLMs, e.g., OpenCLIP, fail to detect these features and struggle to generalize across domains. To overcome the limitation, we develop GeoCLIP, a CLIP-based model trained on synthetic geometric diagram--caption pairs. Benchmark results show that GeoCLIP outperforms existing vision encoders in recognizing geometric features. We then propose our VLM, GeoDANO, which augments GeoCLIP with a domain adaptation strategy for unseen diagram styles. GeoDANO outperforms specialized methods for plane geometry problems and GPT-4o on MathVerse. The implementation is available at https://github.com/ml-postech/GeoDANO.