CIFAR10 to Compare Visual Recognition Performance between Deep Neural Networks and Humans

TL;DR

This study compares human and deep neural network performance on CIFAR10, revealing that while CNNs are efficient, they still lag behind humans in generalization, especially on images humans find easy.

Contribution

The paper provides a thorough, fair comparison of human and neural network recognition on CIFAR10, highlighting differences in difficulty levels and identifying images that can improve future models.

Findings

CNNs outperform humans on some images but lack generalization.

Humans excel at recognizing easy images that CNNs find challenging.

A subset of CIFAR10 images can be used to benchmark and enhance neural networks.

Abstract

Visual object recognition plays an essential role in human daily life. This ability is so efficient that we can recognize a face or an object seemingly without effort, though they may vary in position, scale, pose, and illumination. In the field of computer vision, a large number of studies have been carried out to build a human-like object recognition system. Recently, deep neural networks have shown impressive progress in object classification performance, and have been reported to surpass humans. Yet there is still lack of thorough and fair comparison between humans and artificial recognition systems. While some studies consider artificially degraded images, human recognition performance on dataset widely used for deep neural networks has not been fully evaluated. The present paper carries out an extensive experiment to evaluate human classification accuracy on CIFAR10, a well-known dataset of natural images. This then allows for a fair comparison with the state-of-the-art deep neural networks. Our CIFAR10-based evaluations show very efficient object recognition of recent CNNs but, at the same time, prove that they are still far from human-level capability of generalization. Moreover, a detailed investigation using multiple levels of difficulty reveals that easy images for humans may not be easy for deep neural networks. Such images form a subset of CIFAR10 that can be employed to evaluate and improve future neural networks.