Reconstructed spatial receptive field structures by reverse correlation technique explains the visual feature selectivity of units in deep convolutional neural networks

TL;DR

This study applies neuroscience-inspired reverse correlation techniques to analyze and visualize the spatial receptive fields of units in VGG16, revealing their feature selectivity and offering insights into the network's internal representations.

Contribution

It introduces a method to reconstruct and interpret the receptive fields of CNN units using activation-weighted analyses, bridging neuroscience and deep learning.

Findings

AWC analysis successfully reconstructed receptive fields in middle layers

Reconstructed fields predicted units' visual feature selectivity

Method offers a new way to interpret CNN internal representations

Abstract

An important issue in dealing with Deep Convolutional Neural Networks (DCNN) is the 'black box problem', which represents the unknowns about internal information representation and processing, especially in the middle and higher layers. In this study, we adopted a systems neuroscience methodology to measure the visual feature selectivity and visualize the spatial receptive field of the units in VGG16. Orientation and spatial frequency tunings of each unit were measured using sinusoidal grating stimuli. The image category selectivity of each unit was also measured using natural image stimuli. The spatial structures of the receptive fields of all convolutional units were estimated by activation-weighted average (AWA) and activation-weighted covariance (AWC) analyses. In the middle layers (convolutional layers in block3 and block4), AWC analysis successfully reconstructed the receptive field that predicted the visual feature selectivity of the unit. Those results suggested the possibility that analyzing the reconstructed receptive field structure can be used to interpret the functional significance of the units and layers of a DCNN.