Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale

TL;DR

This paper critically examines the claim that neural networks trained on different modalities converge to similar representations, finding that such alignment is fragile, dataset-dependent, and weaker than previously thought.

Contribution

The study challenges the robustness of cross-modal representational convergence claims, highlighting the importance of evaluation regimes and dataset scale.

Findings

Alignment degrades with larger datasets.

Remaining alignment reflects coarse semantic overlap.

Newer models do not show increased alignment.

Abstract

The Platonic Representation Hypothesis suggests that neural networks trained on different modalities (e.g., text and images) align and eventually converge toward the same representation of reality. If true, this has significant implications for whether modality choice matters at all. We show that the experimental evidence for this hypothesis is fragile and depends critically on the evaluation regime. Alignment is measured using mutual nearest neighbors on small datasets ($\approx$1K samples) and degrades substantially as the dataset is scaled to millions of samples. The alignment that remains between model representations reflects coarse semantic overlap rather than consistent fine-grained structure. Moreover, the evaluations in Huh et al. are done in a one-to-one image-caption setting, a constraint that breaks down in realistic many-to-many settings and further reduces alignment. We also find that the reported trend of stronger language models increasingly aligning with vision does not appear to hold for newer models. Overall, our findings suggest that the current evidence for cross-modal representational convergence is considerably weaker than subsequent works have taken it to be. Models trained on different modalities may learn equally rich representations of the world, just not the same one.