How 'Neural' is a Neural Foundation Model?

TL;DR

This paper investigates the internal neural representations of a state-of-the-art foundation model of neural activity, analyzing its processing stages and proposing design improvements for biological plausibility.

Contribution

It introduces a novel analysis method to interpret the internal neurons of a neural foundation model in biological terms, revealing differences across processing stages.

Findings

Recurrent modules enhance stimulus pattern separation.

Different model stages show distinct representational structures.

Proposed design changes could improve biological alignment.

Abstract

Foundation models have shown remarkable success in fitting biological visual systems; however, their black-box nature inherently limits their utility for understanding brain function. Here, we peek inside a SOTA foundation model of neural activity (Wang et al., 2025) as a physiologist might, characterizing each 'neuron' based on its temporal response properties to parametric stimuli. We analyze how different stimuli are represented in neural activity space by building decoding manifolds, and we analyze how different neurons are represented in stimulus-response space by building neural encoding manifolds. We find that the different processing stages of the model (i.e., the feedforward encoder, recurrent, and readout modules) each exhibit qualitatively different representational structures in these manifolds. The recurrent module shows a jump in capabilities over the encoder module by 'pushing apart' the representations of different temporal stimulus patterns. Our 'tubularity' metric quantifies this stimulus-dependent development of neural activity as biologically plausible. The readout module achieves high fidelity by using numerous specialized feature maps rather than biologically plausible mechanisms. Overall, this study provides a window into the inner workings of a prominent neural foundation model, gaining insights into the biological relevance of its internals through the novel analysis of its neurons' joint temporal response patterns. Our findings suggest design changes that could bring neural foundation models into closer alignment with biological systems: introducing recurrence in early encoder stages, and constraining features in the readout module.