Parallel Backpropagation for Shared-Feature Visualization

TL;DR

This paper introduces a deep-learning-based method for visualizing shared features in neural responses, enhancing understanding of neuron selectivity in high-level visual brain regions by identifying common features across preferred and out-of-category stimuli.

Contribution

It proposes a novel parallel backpropagation approach that highlights shared visual features driving neuron responses, improving interpretability of neural selectivity in brain regions.

Findings

Revealed object parts resembling macaque bodies in neural responses.

Identified shared features explaining neuron activation by different stimuli.

Enhanced visualization of neural feature selectivity.

Abstract

High-level visual brain regions contain subareas in which neurons appear to respond more strongly to examples of a particular semantic category, like faces or bodies, rather than objects. However, recent work has shown that while this finding holds on average, some out-of-category stimuli also activate neurons in these regions. This may be due to visual features common among the preferred class also being present in other images. Here, we propose a deep-learning-based approach for visualizing these features. For each neuron, we identify relevant visual features driving its selectivity by modelling responses to images based on latent activations of a deep neural network. Given an out-of-category image which strongly activates the neuron, our method first identifies a reference image from the preferred category yielding a similar feature activation pattern. We then backpropagate latent activations of both images to the pixel level, while enhancing the identified shared dimensions and attenuating non-shared features. The procedure highlights image regions containing shared features driving responses of the model neuron. We apply the algorithm to novel recordings from body-selective regions in macaque IT cortex in order to understand why some images of objects excite these neurons. Visualizations reveal object parts which resemble parts of a macaque body, shedding light on neural preference of these objects.