The Wittgensteinian Representation Hypothesis: Is Language the Attractor of Multimodal Convergence?

TL;DR

This paper introduces a novel asymmetric analysis method revealing that non-language modalities tend to converge towards language-like representations, suggesting language acts as an attractor in multimodal learning.

Contribution

It presents directional convergence analysis using cycle-kNN, uncovering asymmetric patterns in multimodal models and proposing the Wittgensteinian Representation Hypothesis.

Findings

Non-language modalities move toward language representations more than vice versa.

Symmetric similarity measures fail to detect this directional asymmetry.

Language representations occupy the most compact regions of the representational space.

Abstract

Understanding why independently trained neural networks from different modalities converge toward shared representations, and where this convergence leads, remains an open question in representation learning. All existing evidence relies on symmetric similarity measures, which can detect convergence but are structurally blind to its direction. We introduce directional convergence analysis using cycle-kNN, an asymmetric alignment measure, applied across dozens of independently trained unimodal models spanning point clouds, vision, and language. We uncover a consistent directional asymmetry: non-language modalities move toward the neighborhood structure of language significantly more than the reverse, and this pattern holds across all model families and scales--yet is entirely invisible to symmetric measures. Mechanistic analysis traces the directionality to feature density asymmetry, whereby language representations occupy the most compact regions of representational space. The Information Bottleneck framework provides a principled interpretation: optimization under compression drives representations toward discrete, compositional structures characteristic of language. We formalize this as the Wittgensteinian Representation Hypothesis: the semantic structure of language is the asymptotic attractor of multimodal representation convergence.