Accelerating Deep Learning with Shrinkage and Recall

TL;DR

This paper introduces sDLr, a method inspired by SVM and LASSO techniques, to accelerate deep learning training by shrinking parameters and recalling them, achieving over 2x speedup without sacrificing accuracy.

Contribution

The paper proposes a novel shrinking and recall technique for deep learning that significantly speeds up training while maintaining competitive performance.

Findings

Speedup exceeds 2.0 times on multiple datasets.

Method is applicable to DNN, DBN, and CNN architectures.

Maintains competitive classification accuracy.

Abstract

Deep Learning is a very powerful machine learning model. Deep Learning trains a large number of parameters for multiple layers and is very slow when data is in large scale and the architecture size is large. Inspired from the shrinking technique used in accelerating computation of Support Vector Machines (SVM) algorithm and screening technique used in LASSO, we propose a shrinking Deep Learning with recall (sDLr) approach to speed up deep learning computation. We experiment shrinking Deep Learning with recall (sDLr) using Deep Neural Network (DNN), Deep Belief Network (DBN) and Convolution Neural Network (CNN) on 4 data sets. Results show that the speedup using shrinking Deep Learning with recall (sDLr) can reach more than 2.0 while still giving competitive classification performance.