On the Effectiveness of Neural Ensembles for Image Classification with Small Datasets

TL;DR

This paper demonstrates that ensembling small neural networks improves image classification performance on small datasets, outperforming deeper or wider single models within the same computational budget.

Contribution

It is the first comprehensive study showing neural ensembling's effectiveness for small data image classification, highlighting its simplicity and sample efficiency.

Findings

Ensembling shallow networks outperforms state-of-the-art methods on small datasets.

Neural ensembles are more sample efficient by learning simpler functions.

Ensembling is effective within fixed computational budgets.

Abstract

Deep neural networks represent the gold standard for image classification. However, they usually need large amounts of data to reach superior performance. In this work, we focus on image classification problems with a few labeled examples per class and improve data efficiency by using an ensemble of relatively small networks. For the first time, our work broadly studies the existing concept of neural ensembling in domains with small data, through extensive validation using popular datasets and architectures. We compare ensembles of networks to their deeper or wider single competitors given a total fixed computational budget. We show that ensembling relatively shallow networks is a simple yet effective technique that is generally better than current state-of-the-art approaches for learning from small datasets. Finally, we present our interpretation according to which neural ensembles are more sample efficient because they learn simpler functions.