Wide Compression: Tensor Ring Nets

TL;DR

Tensor Ring Networks (TR-Nets) leverage tensor ring factorization to significantly compress deep neural networks, reducing memory and computation needs while maintaining accuracy, thus enabling deployment on resource-constrained devices.

Contribution

The paper introduces TR-Nets, a novel tensor ring factorization-based method for compressing both fully connected and convolutional layers in deep neural networks.

Findings

Compressed LeNet-5 by 11x without accuracy loss

Achieved 243x compression of Wide ResNet with 2.3% accuracy degradation

Demonstrated potential for resource-constrained device deployment

Abstract

Deep neural networks have demonstrated state-of-the-art performance in a variety of real-world applications. In order to obtain performance gains, these networks have grown larger and deeper, containing millions or even billions of parameters and over a thousand layers. The trade-off is that these large architectures require an enormous amount of memory, storage, and computation, thus limiting their usability. Inspired by the recent tensor ring factorization, we introduce Tensor Ring Networks (TR-Nets), which significantly compress both the fully connected layers and the convolutional layers of deep neural networks. Our results show that our TR-Nets approach {is able to compress LeNet-5 by $11\times$ without losing accuracy}, and can compress the state-of-the-art Wide ResNet by $243\times$ with only 2.3\% degradation in {Cifar10 image classification}. Overall, this compression scheme shows promise in scientific computing and deep learning, especially for emerging resource-constrained devices such as smartphones, wearables, and IoT devices.