Efficient Image Dataset Classification Difficulty Estimation for Predicting Deep-Learning Accuracy

TL;DR

This paper introduces a fast method to estimate the difficulty of image classification datasets, helping to efficiently select suitable neural network configurations without extensive training.

Contribution

It proposes a novel dataset difficulty estimation technique that is 27 times faster than training models, aiding in quicker neural network selection and hyper-parameter tuning.

Findings

Estimates dataset difficulty 27x faster than training.

Helps guide neural network architecture and hyper-parameter search.

Reduces computational cost in model selection process.

Abstract

In the deep-learning community new algorithms are published at an incredible pace. Therefore, solving an image classification problem for new datasets becomes a challenging task, as it requires to re-evaluate published algorithms and their different configurations in order to find a close to optimal classifier. To facilitate this process, before biasing our decision towards a class of neural networks or running an expensive search over the network space, we propose to estimate the classification difficulty of the dataset. Our method computes a single number that characterizes the dataset difficulty 27x faster than training state-of-the-art networks. The proposed method can be used in combination with network topology and hyper-parameter search optimizers to efficiently drive the search towards promising neural-network configurations.