Spatial Reasoning is Not a Free Lunch: A Controlled Study on LLaVA

TL;DR

This study investigates how current vision-language model design choices, like image encoding and positional encoding, impact spatial reasoning abilities, revealing persistent gaps despite various modifications.

Contribution

It provides a controlled diagnostic analysis showing that encoder objectives and positional encoding influence spatial understanding but do not fully solve existing limitations.

Findings

Consistent spatial performance gaps across models

Encoder objectives and positional encoding affect spatial behavior

Modifications do not fully resolve spatial reasoning issues

Abstract

Vision-language models (VLMs) have advanced rapidly, yet they still struggle with basic spatial reasoning. Despite strong performance on general benchmarks, modern VLMs remain brittle at understanding 2D spatial relationships such as relative position, layout, and counting. We argue that this failure is not merely a data problem, but is closely tied to dominant design choices in current VLM pipelines: reliance on CLIP-style image encoders and the flattening of images into 1D token sequences with 1D positional encoding. We present a controlled diagnostic study within the LLaVA framework to isolate how these choices affect spatial grounding. We evaluate frontier models and LLaVA variants on a suite of spatial benchmarks, comparing CLIP-based encoders against alternatives trained with denser or generative objectives, as well as variants augmented with 2D positional encoding. Our results show consistent spatial performance gaps across models, and indicate that encoder objectives and positional structure shape spatial behavior, but do not fully resolve it.