# ADASS: Adaptive Sample Selection for Training Acceleration

## TL;DR

ADASS is a novel adaptive sample selection method that accelerates machine learning training by selectively sampling data based on Lipschitz constants, reducing training time while maintaining accuracy.

## Contribution

It introduces a new adaptive sampling technique integrated with existing optimizers, enabling faster training with fewer samples per epoch.

## Key findings

- Accelerates training without accuracy loss
- Compatible with SGD and momentum SGD
- Effective on shallow and deep models

## Abstract

Stochastic gradient decent~(SGD) and its variants, including some accelerated variants, have become popular for training in machine learning. However, in all existing SGD and its variants, the sample size in each iteration~(epoch) of training is the same as the size of the full training set. In this paper, we propose a new method, called \underline{ada}ptive \underline{s}ample \underline{s}election~(ADASS), for training acceleration. During different epoches of training, ADASS only need to visit different training subsets which are adaptively selected from the full training set according to the Lipschitz constants of the loss functions on samples. It means that in ADASS the sample size in each epoch of training can be smaller than the size of the full training set, by discarding some samples. ADASS can be seamlessly integrated with existing optimization methods, such as SGD and momentum SGD, for training acceleration. Theoretical results show that the learning accuracy of ADASS is comparable to that of counterparts with full training set. Furthermore, empirical results on both shallow models and deep models also show that ADASS can accelerate the training process of existing methods without sacrificing accuracy.