System identification of neural systems: If we got it right, would we know?

TL;DR

This paper critically examines the effectiveness of current system identification methods in neural modeling, revealing their limitations and variability in accurately capturing brain computation and architecture.

Contribution

It evaluates common comparison techniques using ground truth models, highlighting their limitations and the influence of external factors on identification accuracy.

Findings

System identification performance varies significantly.

External factors like stimuli images impact identification accuracy.

Functional similarity scores have limitations in identifying architectural motifs.

Abstract

Artificial neural networks are being proposed as models of parts of the brain. The networks are compared to recordings of biological neurons, and good performance in reproducing neural responses is considered to support the model's validity. A key question is how much this system identification approach tells us about brain computation. Does it validate one model architecture over another? We evaluate the most commonly used comparison techniques, such as a linear encoding model and centered kernel alignment, to correctly identify a model by replacing brain recordings with known ground truth models. System identification performance is quite variable; it also depends significantly on factors independent of the ground truth architecture, such as stimuli images. In addition, we show the limitations of using functional similarity scores in identifying higher-level architectural motifs.