Faster Learning by Reduction of Data Access Time

TL;DR

This paper addresses the big data challenge in machine learning by proposing systematic and cyclic sampling methods to reduce data access time, resulting in significantly faster training without sacrificing convergence.

Contribution

It introduces systematic and cyclic sampling techniques for mini-batch selection, demonstrating their effectiveness in speeding up training in empirical risk minimization.

Findings

Up to six times faster training times observed.

Theoretical convergence proven for proposed sampling methods.

Effective on benchmark datasets with strong convexity and smoothness assumptions.

Abstract

Nowadays, the major challenge in machine learning is the Big Data challenge. The big data problems due to large number of data points or large number of features in each data point, or both, the training of models have become very slow. The training time has two major components: Time to access the data and time to process (learn from) the data. So far, the research has focused only on the second part, i.e., learning from the data. In this paper, we have proposed one possible solution to handle the big data problems in machine learning. The idea is to reduce the training time through reducing data access time by proposing systematic sampling and cyclic/sequential sampling to select mini-batches from the dataset. To prove the effectiveness of proposed sampling techniques, we have used Empirical Risk Minimization, which is commonly used machine learning problem, for strongly convex and smooth case. The problem has been solved using SAG, SAGA, SVRG, SAAG-II and MBSGD (Mini-batched SGD), each using two step determination techniques, namely, constant step size and backtracking line search method. Theoretical results prove the same convergence for systematic sampling, cyclic sampling and the widely used random sampling technique, in expectation. Experimental results with bench marked datasets prove the efficacy of the proposed sampling techniques and show up to six times faster training.