Swift Sampler: Efficient Learning of Sampler by 10 Parameters

TL;DR

The paper introduces Swift Sampler (SS), an efficient automatic method for learning data samplers that improves training performance and can be applied to large datasets with minimal computational cost.

Contribution

The paper proposes a novel low-dimensional hyper-parameter formulation and an approximated local minimum approach for fast, automatic sampler learning, outperforming previous heuristic and trial-based methods.

Findings

Achieves 1.5% accuracy improvement on ImageNet

Demonstrates transferability across different neural networks

Operates efficiently on large-scale datasets

Abstract

Data selection is essential for training deep learning models. An effective data sampler assigns proper sampling probability for training data and helps the model converge to a good local minimum with high performance. Previous studies in data sampling are mainly based on heuristic rules or learning through a huge amount of time-consuming trials. In this paper, we propose an automatic \textbf{swift sampler} search algorithm, \textbf{SS}, to explore automatically learning effective samplers efficiently. In particular, \textbf{SS} utilizes a novel formulation to map a sampler to a low dimension of hyper-parameters and uses an approximated local minimum to quickly examine the quality of a sampler. Benefiting from its low computational expense, \textbf{SS} can be applied on large-scale data sets with high efficiency. Comprehensive experiments on various tasks demonstrate that \textbf{SS} powered sampling can achieve obvious improvements (e.g., 1.5\% on ImageNet) and transfer among different neural networks. Project page: https://github.com/Alexander-Yao/Swift-Sampler.