Visual Language Hypothesis

TL;DR

This paper proposes a topological framework for visual representation learning, suggesting that understanding requires a semantic language and specific structural model architecture to capture the organization of visual observations.

Contribution

It introduces a novel topological perspective on visual understanding, linking semantic invariance to the structure of the observation space and model architecture requirements.

Findings

Visual observation space has a fiber bundle structure with nuisance and semantic components.

Semantic invariance necessitates non-smooth, discriminative targets like labels or multimodal alignment.

Model architecture must support topology change through expand and snap processes.

Abstract

We study visual representation learning from a structural and topological perspective. We begin from a single hypothesis: that visual understanding presupposes a semantic language for vision, in which many perceptual observations correspond to a small number of discrete semantic states. Together with widely assumed premises on transferability and abstraction in representation learning, this hypothesis implies that the visual observation space must be organized in a fiber bundle like structure, where nuisance variation populates fibers and semantics correspond to a quotient base space. From this structure we derive two theoretical consequences. First, the semantic quotient X/G is not a submanifold of X and cannot be obtained through smooth deformation alone, semantic invariance requires a non homeomorphic, discriminative target for example, supervision via labels, cross-instance identification, or multimodal alignment that supplies explicit semantic equivalence. Second, we show that approximating the quotient also places structural demands on the model architecture. Semantic abstraction requires not only an external semantic target, but a representation mechanism capable of supporting topology change: an expand and snap process in which the manifold is first geometrically expanded to separate structure and then collapsed to form discrete semantic regions. We emphasize that these results are interpretive rather than prescriptive: the framework provides a topological lens that aligns with empirical regularities observed in large-scale discriminative and multimodal models, and with classical principles in statistical learning theory.