Representation biases: will we achieve complete understanding by analyzing representations?

TL;DR

This paper examines how biases in neural representations, especially in learned features, can hinder our understanding of neural systems, highlighting challenges in analysis methods and the potential for dissociation between representations and computations.

Contribution

It demonstrates how feature representation biases affect common analysis techniques and explores homomorphic encryption as an example of dissociation between representations and computations.

Findings

Representation biases can lead to misleading inferences in PCA, regression, and RSA.

Simple features tend to be over-represented compared to complex features.

Homomorphic encryption illustrates potential dissociation between representations and underlying computations.

Abstract

A common approach in neuroscience is to study neural representations as a means to understand a system -- increasingly, by relating the neural representations to the internal representations learned by computational models. However, a recent work in machine learning (Lampinen, 2024) shows that learned feature representations may be biased to over-represent certain features, and represent others more weakly and less-consistently. For example, simple (linear) features may be more strongly and more consistently represented than complex (highly nonlinear) features. These biases could pose challenges for achieving full understanding of a system through representational analysis. In this perspective, we illustrate these challenges -- showing how feature representation biases can lead to strongly biased inferences from common analyses like PCA, regression, and RSA. We also present homomorphic encryption as a simple case study of the potential for strong dissociation between patterns of representation and computation. We discuss the implications of these results for representational comparisons between systems, and for neuroscience more generally.