Global Geometry Is Not Enough for Vision Representations

TL;DR

This paper demonstrates that global geometric metrics are insufficient for assessing vision representations' ability to encode compositional structure, highlighting the importance of functional sensitivity measured by input-output Jacobian.

Contribution

It reveals the limitations of global geometry in predicting compositional binding and advocates for functional sensitivity as a key measure for representational competence.

Findings

Global geometry metrics do not correlate with compositional binding.

Input-output Jacobian reliably predicts compositional capabilities.

Existing training objectives constrain geometry but not local input-output mappings.

Abstract

A common assumption in representation learning is that globally well-distributed embeddings support robust and generalizable representations. This focus has shaped both training objectives and evaluation protocols, implicitly treating global geometry as a proxy for representational competence. While global geometry effectively encodes which elements are present, it is often insensitive to how they are composed. We investigate this limitation by testing the ability of geometric metrics to predict compositional binding across 21 vision encoders. We find that standard geometry-based statistics exhibit near-zero correlation with compositional binding. In contrast, functional sensitivity, as measured by the input-output Jacobian, reliably tracks this capability. We further provide an analytic account showing that this disparity arises from objective design, as existing losses explicitly constrain embedding geometry but leave the local input-output mapping unconstrained. These results suggest that global embedding geometry captures only a partial view of representational competence and establish functional sensitivity as a critical complementary axis for modeling composite structure.