Accelerating Training using Tensor Decomposition

TL;DR

This paper introduces a novel method to accelerate training from scratch by applying tensor decomposition after initial epochs, achieving up to 2x faster training with minimal accuracy loss.

Contribution

The paper proposes a new approach to reduce training time from scratch using tensor decomposition, with optional architecture recovery, demonstrating significant speedups on standard datasets.

Findings

Up to 2x training speedup on CIFAR10 and ImageNet.

Accuracy drop of up to 1.5%, otherwise no loss.

Method is hardware-independent, effective on CPU and GPU.

Abstract

Tensor decomposition is one of the well-known approaches to reduce the latency time and number of parameters of a pre-trained model. However, in this paper, we propose an approach to use tensor decomposition to reduce training time of training a model from scratch. In our approach, we train the model from scratch (i.e., randomly initialized weights) with its original architecture for a small number of epochs, then the model is decomposed, and then continue training the decomposed model till the end. There is an optional step in our approach to convert the decomposed architecture back to the original architecture. We present results of using this approach on both CIFAR10 and Imagenet datasets, and show that there can be upto 2x speed up in training time with accuracy drop of upto 1.5% only, and in other cases no accuracy drop. This training acceleration approach is independent of hardware and is expected to have similar speed ups on both CPU and GPU platforms.