Evaluation of Deep Learning on an Abstract Image Classification Dataset

TL;DR

This paper introduces a new abstract image classification dataset designed to be easy for humans but challenging for CNNs, and evaluates popular deep learning models on it to identify research opportunities.

Contribution

It presents a novel dataset with abstract classes and assesses CNN performance, highlighting challenges and future research directions.

Findings

CNNs perform poorly on the abstract dataset compared to humans.

Variations of the dataset reveal specific weaknesses of current CNN architectures.

The dataset offers new challenges for developing more robust image classification models.

Abstract

Convolutional Neural Networks have become state of the art methods for image classification over the last couple of years. By now they perform better than human subjects on many of the image classification datasets. Most of these datasets are based on the notion of concrete classes (i.e. images are classified by the type of object in the image). In this paper we present a novel image classification dataset, using abstract classes, which should be easy to solve for humans, but variations of it are challenging for CNNs. The classification performance of popular CNN architectures is evaluated on this dataset and variations of the dataset that might be interesting for further research are identified.