How Important is Importance Sampling for Deep Budgeted Training?

TL;DR

This paper investigates the effectiveness of importance sampling in deep neural network training under budget constraints, finding that data augmentation often outperforms importance sampling in limited training scenarios.

Contribution

The study demonstrates that importance sampling does not consistently improve training efficiency under budget restrictions and highlights the effectiveness of data augmentation techniques.

Findings

Importance sampling offers limited benefits under training budgets.

Data augmentation maintains accuracy when training budgets are reduced.

Variety in training data is more beneficial than importance sampling in budgeted training.

Abstract

Long iterative training processes for Deep Neural Networks (DNNs) are commonly required to achieve state-of-the-art performance in many computer vision tasks. Importance sampling approaches might play a key role in budgeted training regimes, i.e. when limiting the number of training iterations. These approaches aim at dynamically estimating the importance of each sample to focus on the most relevant and speed up convergence. This work explores this paradigm and how a budget constraint interacts with importance sampling approaches and data augmentation techniques. We show that under budget restrictions, importance sampling approaches do not provide a consistent improvement over uniform sampling. We suggest that, given a specific budget, the best course of action is to disregard the importance and introduce adequate data augmentation; e.g. when reducing the budget to a 30% in CIFAR-10/100, RICAP data augmentation maintains accuracy, while importance sampling does not. We conclude from our work that DNNs under budget restrictions benefit greatly from variety in the training set and that finding the right samples to train on is not the most effective strategy when balancing high performance with low computational requirements. Source code available at https://git.io/JKHa3 .