Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization Framework

TL;DR

This paper introduces a systematic ADMM-based framework for tensor decomposition model compression that maintains high accuracy in CNNs and RNNs, achieving significant compression ratios with minimal or no accuracy loss.

Contribution

It proposes a novel optimization-based approach for tensor decomposition in DNNs, enabling high-accuracy compression for both CNNs and RNNs using ADMM.

Findings

Achieved 2.3X and 2.4X compression with higher accuracy on CIFAR-100.

Reduced FLOPs by 2.47X on ResNet-18 without accuracy loss on ImageNet.

Framework is general and adaptable to various tensor decomposition methods.

Abstract

Advanced tensor decomposition, such as Tensor train (TT) and Tensor ring (TR), has been widely studied for deep neural network (DNN) model compression, especially for recurrent neural networks (RNNs). However, compressing convolutional neural networks (CNNs) using TT/TR always suffers significant accuracy loss. In this paper, we propose a systematic framework for tensor decomposition-based model compression using Alternating Direction Method of Multipliers (ADMM). By formulating TT decomposition-based model compression to an optimization problem with constraints on tensor ranks, we leverage ADMM technique to systemically solve this optimization problem in an iterative way. During this procedure, the entire DNN model is trained in the original structure instead of TT format, but gradually enjoys the desired low tensor rank characteristics. We then decompose this uncompressed model to TT format and fine-tune it to finally obtain a high-accuracy TT-format DNN model. Our framework is very general, and it works for both CNNs and RNNs, and can be easily modified to fit other tensor decomposition approaches. We evaluate our proposed framework on different DNN models for image classification and video recognition tasks. Experimental results show that our ADMM-based TT-format models demonstrate very high compression performance with high accuracy. Notably, on CIFAR-100, with 2.3X and 2.4X compression ratios, our models have 1.96% and 2.21% higher top-1 accuracy than the original ResNet-20 and ResNet-32, respectively. For compressing ResNet-18 on ImageNet, our model achieves 2.47X FLOPs reduction without accuracy loss.