Visualizing and Controlling Cortical Responses Using Voxel-Weighted Activation Maximization

TL;DR

This paper introduces a method that uses activation maximization on DNN-based encoding models to generate images that reveal and influence responses in the human visual cortex, providing new insights into neural representations.

Contribution

It adapts activation maximization to DNN-based brain encoding models, enabling visualization and control of cortical responses without needing generative models.

Findings

Generated images correspond with known cortical selectivity.

Images reliably drive activity in targeted visual regions.

Method works across multiple visual areas and subjects.

Abstract

Deep neural networks (DNNs) trained on visual tasks develop feature representations that resemble those in the human visual system. Although DNN-based encoding models can accurately predict brain responses to visual stimuli, they offer limited insight into the specific features driving these responses. Here, we demonstrate that activation maximization -- a technique designed to interpret vision DNNs -- can be applied to DNN-based encoding models of the human brain. We extract and adaptively downsample activations from multiple layers of a pretrained Inception V3 network, then use linear regression to predict fMRI responses. This yields a full image-computable model of brain responses. Next, we apply activation maximization to generate images optimized for predicted responses in individual cortical voxels. We find that these images contain visual characteristics that qualitatively correspond with known selectivity and enable exploration of selectivity across the visual cortex. We further extend our method to whole regions of interest (ROIs) of the brain and validate its efficacy by presenting these images to human participants in an fMRI study. We find that the generated images reliably drive activity in targeted regions across both low- and high-level visual areas and across subjects. These results demonstrate that activation maximization can be successfully applied to DNN-based encoding models. By addressing key limitations of alternative approaches that require natively generative models, our approach enables flexible characterization and modulation of responses across the human visual system.