Comparing object recognition in humans and deep convolutional neural networks -- An eye tracking study

TL;DR

This study compares human and deep neural network object recognition using eye tracking, revealing differences in visual processing and showing that biologically inspired networks align more closely with human viewing behavior.

Contribution

It introduces a novel comparison method using eye tracking and saliency maps, highlighting differences in spatial processing and demonstrating the effectiveness of biologically plausible networks.

Findings

Biologically inspired vNet shows higher agreement with human viewing behavior.

Image factors like category and arousal influence recognition priorities.

Differences in visualization methods impact comparison insights.

Abstract

Deep convolutional neural networks (DCNNs) and the ventral visual pathway share vast architectural and functional similarities in visual challenges such as object recognition. Recent insights have demonstrated that both hierarchical cascades can be compared in terms of both exerted behavior and underlying activation. However, these approaches ignore key differences in spatial priorities of information processing. In this proof-of-concept study, we demonstrate a comparison of human observers (N = 45) and three feedforward DCNNs through eye tracking and saliency maps. The results reveal fundamentally different resolutions in both visualization methods that need to be considered for an insightful comparison. Moreover, we provide evidence that a DCNN with biologically plausible receptive field sizes called vNet reveals higher agreement with human viewing behavior as contrasted with a standard ResNet architecture. We find that image-specific factors such as category, animacy, arousal, and valence have a direct link to the agreement of spatial object recognition priorities in humans and DCNNs, while other measures such as difficulty and general image properties do not. With this approach, we try to open up new perspectives at the intersection of biological and computer vision research.