A Spectral Theory of Neural Prediction and Alignment

TL;DR

This paper introduces a spectral framework to analyze and interpret how deep neural networks predict neural responses, revealing multiple geometries that lead to low prediction error and enhancing understanding of model-neural alignment.

Contribution

It applies a spectral decomposition approach to neural prediction errors, offering new geometrical measures for interpreting model-neural alignment and differentiating among models with similar predictive performance.

Findings

Multiple geometries can produce low neural prediction error.

Spectral decomposition reveals how model eigenspectra relate to neural response prediction.

The approach enhances interpretability of neural network models in neuroscience.

Abstract

The representations of neural networks are often compared to those of biological systems by performing regression between the neural network responses and those measured from biological systems. Many different state-of-the-art deep neural networks yield similar neural predictions, but it remains unclear how to differentiate among models that perform equally well at predicting neural responses. To gain insight into this, we use a recent theoretical framework that relates the generalization error from regression to the spectral properties of the model and the target. We apply this theory to the case of regression between model activations and neural responses and decompose the neural prediction error in terms of the model eigenspectra, alignment of model eigenvectors and neural responses, and the training set size. Using this decomposition, we introduce geometrical measures to interpret the neural prediction error. We test a large number of deep neural networks that predict visual cortical activity and show that there are multiple types of geometries that result in low neural prediction error as measured via regression. The work demonstrates that carefully decomposing representational metrics can provide interpretability of how models are capturing neural activity and points the way towards improved models of neural activity.