Similarities and differences between stimulus tuning in the inferotemporal visual cortex and convolutional networks

TL;DR

This paper compares the activity patterns of deep CNNs and primate inferotemporal cortex, highlighting both similarities in population responses and differences in object selectivity, informing future neural network improvements.

Contribution

It provides a detailed comparison of CNN layer activity with IT electrophysiology, identifying key similarities and differences in stimulus tuning.

Findings

Similar population response sparseness in CNNs and IT

Differences in object selectivity statistics

Partial parallels in scale invariance and clutter responses

Abstract

Deep convolutional neural networks (CNNs) trained for object classification have a number of striking similarities with the primate ventral visual stream. In particular, activity in early, intermediate, and late layers is closely related to activity in V1, V4, and the inferotemporal cortex (IT). This study further compares activity in late layers of object-classification CNNs to activity patterns reported in the IT electrophysiology literature. There are a number of close similarities, including the distributions of population response sparseness across stimuli, and the distribution of size tuning bandwidth. Statisics of scale invariance, responses to clutter and occlusion, and orientation tuning are less similar. Statistics of object selectivity are quite different. These results agree with recent studies that highlight strong parallels between object-categorization CNNs and the ventral stream, and also highlight differences that could perhaps be reduced in future CNNs.