On the Intrinsic Limits of Transformer Image Embeddings in Non-Solvable Spatial Reasoning

TL;DR

This paper investigates the fundamental limitations of Vision Transformers in spatial reasoning, showing that their architecture inherently struggles with non-solvable group structures like 3D rotations due to complexity bounds.

Contribution

It formalizes spatial understanding as learning a group homomorphism and proves that ViTs cannot efficiently embed non-solvable groups due to their limited logical depth.

Findings

ViTs fail on non-solvable spatial tasks as depth increases

Complexity bounds restrict ViT's ability to learn certain algebraic structures

Empirical evidence shows structural collapse in ViT representations for complex spatial tasks

Abstract

Vision Transformers (ViTs) excel in semantic recognition but exhibit systematic failures in spatial reasoning tasks such as mental rotation. While often attributed to data scale, we propose that this limitation arises from the intrinsic circuit complexity of the architecture. We formalize spatial understanding as learning a Group Homomorphism: mapping image sequences to a latent space that preserves the algebraic structure of the underlying transformation group. We demonstrate that for non-solvable groups (e.g., the 3D rotation group $\mathrm{SO}(3)$), maintaining such a structure-preserving embedding is computationally lower-bounded by the Word Problem, which is $\mathsf{NC^1}$-complete. In contrast, we prove that constant-depth ViTs with polynomial precision are strictly bounded by $\mathsf{TC^0}$. Under the conjecture $\mathsf{TC^0} \subsetneq \mathsf{NC^1}$, we establish a complexity boundary: constant-depth ViTs fundamentally lack the logical depth to efficiently capture non-solvable spatial structures. We validate this complexity gap via latent-space probing, demonstrating that ViT representations suffer a structural collapse on non-solvable tasks as compositional depth increases.