AVM: Towards Structure-Preserving Neural Response Modeling in the Visual Cortex Across Stimuli and Individuals

TL;DR

The paper introduces AVM, a neural response model that preserves structure and enables condition-aware adaptation, improving neural response prediction across stimuli and individuals using a modular, transformer-based framework.

Contribution

AVM is a novel structure-preserving neural model that separates stable visual encoding from condition-specific adaptation, enhancing generalization and interpretability.

Findings

AVM outperforms state-of-the-art models by ~2% in predictive correlation.

AVM achieves a 9.1% improvement in explained variance in cross-dataset adaptation.

AVM demonstrates robust generalization across stimuli, subjects, and datasets.

Abstract

While deep learning models have shown strong performance in simulating neural responses, they often fail to clearly separate stable visual encoding from condition-specific adaptation, which limits their ability to generalize across stimuli and individuals. We introduce the Adaptive Visual Model (AVM), a structure-preserving framework that enables condition-aware adaptation through modular subnetworks, without modifying the core representation. AVM keeps a Vision Transformer-based encoder frozen to capture consistent visual features, while independently trained modulation paths account for neural response variations driven by stimulus content and subject identity. We evaluate AVM in three experimental settings, including stimulus-level variation, cross-subject generalization, and cross-dataset adaptation, all of which involve structured changes in inputs and individuals. Across two large-scale mouse V1 datasets, AVM outperforms the state-of-the-art V1T model by approximately 2% in predictive correlation, demonstrating robust generalization, interpretable condition-wise modulation, and high architectural efficiency. Specifically, AVM achieves a 9.1% improvement in explained variance (FEVE) under the cross-dataset adaptation setting. These results suggest that AVM provides a unified framework for adaptive neural modeling across biological and experimental conditions, offering a scalable solution under structural constraints. Its design may inform future approaches to cortical modeling in both neuroscience and biologically inspired AI systems.