When Does Closeness in Distribution Imply Representational Similarity? An Identifiability Perspective

TL;DR

This paper investigates the relationship between distributional closeness and representational similarity in neural networks, revealing that small distributional divergence does not guarantee similar representations, and proposing a new distributional distance measure.

Contribution

It introduces an identifiability-based framework to analyze when distributional similarity implies representational similarity and proposes a new distance measure that aligns these concepts.

Findings

Small KL divergence does not ensure similar representations.

Models with high likelihood can learn dissimilar representations.

Wider networks tend to have closer distributions and more similar representations.

Abstract

When and why representations learned by different deep neural networks are similar is an active research topic. We choose to address these questions from the perspective of identifiability theory, which suggests that a measure of representational similarity should be invariant to transformations that leave the model distribution unchanged. Focusing on a model family which includes several popular pre-training approaches, e.g., autoregressive language models, we explore when models which generate distributions that are close have similar representations. We prove that a small Kullback--Leibler divergence between the model distributions does not guarantee that the corresponding representations are similar. This has the important corollary that models with near-maximum data likelihood can still learn dissimilar representations -- a phenomenon mirrored in our experiments with models trained on CIFAR-10. We then define a distributional distance for which closeness implies representational similarity, and in synthetic experiments, we find that wider networks learn distributions which are closer with respect to our distance and have more similar representations. Our results thus clarify the link between closeness in distribution and representational similarity.