Stimulus symmetries can confound representational similarity analyses

TL;DR

This paper reveals that symmetries in neural network inputs can distort representational similarity analyses, complicating the interpretation of neural codes and their geometries.

Contribution

It demonstrates how stimulus symmetries and training dynamics can lead to confounding RSMs, highlighting challenges in comparing nonlinear neural representations.

Findings

Symmetries can cause different RSMs for functionally equivalent representations.

Training methods like gradient descent induce drifting codes and RSMs.

Latent symmetries in image-encoding networks affect RSM analysis.

Abstract

What can representational similarity matrices (RSMs) tell us about a neural code? As the popularity of these summary statistics grows, so too does the need for a more complete characterization of their properties. Here, we show that symmetries in network inputs can confound RSM-based analyses. Stimulus symmetries render many representations functionally equivalent, but these different configurations can lead to different RSMs. These different RSMs reflect qualitatively different representational geometries. We show that stochastic gradient descent or energetic regularization can generate sparse, drifting codes, leading in turn to drifting RSMs. Moreover, we demonstrate that these phenomena are present in networks trained to encode image data, where the symmetry is latent. Our results illustrate the challenges inherent in comparing nonlinear neural codes, when functionally-equivalent representations are not related by a simple rotation.