Unravelling Small Sample Size Problems in the Deep Learning World

TL;DR

This paper reviews deep learning challenges with small sample sizes and introduces a Dynamic Attention Pooling method that enhances model performance on limited data by focusing on discriminative features.

Contribution

It provides a comprehensive review of algorithms for small sample size problems and proposes a novel Dynamic Attention Pooling technique to improve deep learning performance in such scenarios.

Findings

Dynamic Attention Pooling improves accuracy on small datasets

The method outperforms traditional pooling in limited data settings

Effective on datasets like SVHN, C10, C100, TinyImageNet

Abstract

The growth and success of deep learning approaches can be attributed to two major factors: availability of hardware resources and availability of large number of training samples. For problems with large training databases, deep learning models have achieved superlative performances. However, there are a lot of \textit{small sample size or $S^3$} problems for which it is not feasible to collect large training databases. It has been observed that deep learning models do not generalize well on $S^3$ problems and specialized solutions are required. In this paper, we first present a review of deep learning algorithms for small sample size problems in which the algorithms are segregated according to the space in which they operate, i.e. input space, model space, and feature space. Secondly, we present Dynamic Attention Pooling approach which focuses on extracting global information from the most discriminative sub-part of the feature map. The performance of the proposed dynamic attention pooling is analyzed with state-of-the-art ResNet model on relatively small publicly available datasets such as SVHN, C10, C100, and TinyImageNet.