The Triangle of Similarity: A Multi-Faceted Framework for Comparing Neural Network Representations

TL;DR

This paper introduces the Triangle of Similarity, a comprehensive framework combining three perspectives to compare neural network representations, enhancing understanding of model similarities and differences across architectures and tasks.

Contribution

It presents a novel multi-faceted framework that integrates static, functional, and sparsity-based similarity measures for neural network comparison.

Findings

Architectural family influences representational similarity, forming distinct clusters.

CKA self-similarity correlates with task accuracy during pruning.

Pruning can regularize representations, revealing shared computational cores.

Abstract

Comparing neural network representations is essential for understanding and validating models in scientific applications. Existing methods, however, often provide a limited view. We propose the Triangle of Similarity, a framework that combines three complementary perspectives: static representational similarity (CKA/Procrustes), functional similarity (Linear Mode Connectivity or Predictive Similarity), and sparsity similarity (robustness under pruning). Analyzing a range of CNNs, Vision Transformers, and Vision-Language Models using both in-distribution (ImageNetV2) and out-of-distribution (CIFAR-10) testbeds, our initial findings suggest that: (1) architectural family is a primary determinant of representational similarity, forming distinct clusters; (2) CKA self-similarity and task accuracy are strongly correlated during pruning, though accuracy often degrades more sharply; and (3) for some model pairs, pruning appears to regularize representations, exposing a shared computational core. This framework offers a more holistic approach for assessing whether models have converged on similar internal mechanisms, providing a useful tool for model selection and analysis in scientific research.