On Privileged and Convergent Bases in Neural Network Representations

TL;DR

This paper investigates the nature of neural network representations, showing they lack a unique convergent basis and that basis correlation is influenced by network width and layer freezing, challenging assumptions about invariance and convergence.

Contribution

It demonstrates that neural networks do not converge to a unique basis and that basis correlation is affected by network width and early layer freezing.

Findings

Neural representations are not rotationally invariant.

Networks trained with different initializations do not converge to a single basis.

Basis correlation increases when early layers are frozen.

Abstract

In this study, we investigate whether the representations learned by neural networks possess a privileged and convergent basis. Specifically, we examine the significance of feature directions represented by individual neurons. First, we establish that arbitrary rotations of neural representations cannot be inverted (unlike linear networks), indicating that they do not exhibit complete rotational invariance. Subsequently, we explore the possibility of multiple bases achieving identical performance. To do this, we compare the bases of networks trained with the same parameters but with varying random initializations. Our study reveals two findings: (1) Even in wide networks such as WideResNets, neural networks do not converge to a unique basis; (2) Basis correlation increases significantly when a few early layers of the network are frozen identically. Furthermore, we analyze Linear Mode Connectivity, which has been studied as a measure of basis correlation. Our findings give evidence that while Linear Mode Connectivity improves with increased network width, this improvement is not due to an increase in basis correlation.