Cross-Modal Alignment between Visual Stimuli and Neural Responses in the Visual Cortex

TL;DR

This paper introduces a cross-modal alignment method called Visual-Neural Alignment (VNA) that improves the mapping between visual stimuli and neural responses in the visual cortex, outperforming traditional encoding and decoding approaches.

Contribution

The paper proposes a novel cross-modal alignment approach for better visual-neural mapping, addressing overfitting issues in neural response modeling and demonstrating superior performance across multiple datasets.

Findings

VNA outperforms direct encoding and decoding methods.

VNA achieves higher accuracy in discriminative tasks.

The approach generalizes well across species and datasets.

Abstract

Investigating the mapping between visual stimuli and neural responses in the visual cortex contributes to a deeper understanding of biological visual processing mechanisms. Most existing studies characterize this mapping by training models to directly encode visual stimuli into neural responses or decode neural responses into visual stimuli. However, due to neural response variability and limited neural recording techniques, these studies suffer from overfitting and lack generalizability. Motivated by this challenge, in this paper we shift the tasks from conventional direct encoding and decoding to discriminative encoding and decoding, which are more reasonable. And on top of this we propose a cross-modal alignment approach, named Visual-Neural Alignment (VNA). To thoroughly test the performance of the three methods (direct encoding, direct decoding, and our proposed VNA) on discriminative encoding and decoding tasks, we conduct extensive experiments on three invasive visual cortex datasets, involving two types of subject mammals (mice and macaques). The results demonstrate that our VNA generally outperforms direct encoding and direct decoding, indicating our VNA can most precisely characterize the above visual-neural mapping among the three methods.