Understanding Depth and Height Perception in Large Visual-Language Models

TL;DR

This paper evaluates the geometric understanding of large Vision Language Models, focusing on their ability to perceive depth and height, revealing significant shortcomings and proposing a benchmark for future improvements.

Contribution

Introduces GeoMeter, a benchmark suite for assessing depth and height perception in VLMs, and benchmarks 18 models to identify their limitations in geometric reasoning.

Findings

VLMs excel at shape and size perception

Models struggle with depth and height perception

Depth and height reasoning are limited in current VLMs

Abstract

Geometric understanding - including depth and height perception - is fundamental to intelligence and crucial for navigating our environment. Despite the impressive capabilities of large Vision Language Models (VLMs), it remains unclear how well they possess the geometric understanding required for practical applications in visual perception. In this work, we focus on evaluating the geometric understanding of these models, specifically targeting their ability to perceive the depth and height of objects in an image. To address this, we introduce GeoMeter, a suite of benchmark datasets - encompassing 2D and 3D scenarios - to rigorously evaluate these aspects. By benchmarking 18 state-of-the-art VLMs, we found that although they excel in perceiving basic geometric properties like shape and size, they consistently struggle with depth and height perception. Our analysis reveal that these challenges stem from shortcomings in their depth and height reasoning capabilities and inherent biases. This study aims to pave the way for developing VLMs with enhanced geometric understanding by emphasizing depth and height perception as critical components necessary for real-world applications.