Deep Neural Network Approximation using Tensor Sketching

TL;DR

This paper introduces a tensor sketching method to approximate deep neural network layers, enabling significant parameter reduction while maintaining comparable classification accuracy, thus addressing resource constraints.

Contribution

It proposes a novel randomized tensor sketching technique for approximating convolutional and fully connected layers in deep networks, facilitating model compression.

Findings

Reduces effective parameters in CNNs significantly.

Smaller networks trained with the method achieve similar accuracy.

Framework applicable to both convolutional and fully connected layers.

Abstract

Deep neural networks are powerful learning models that achieve state-of-the-art performance on many computer vision, speech, and language processing tasks. In this paper, we study a fundamental question that arises when designing deep network architectures: Given a target network architecture can we design a smaller network architecture that approximates the operation of the target network? The question is, in part, motivated by the challenge of parameter reduction (compression) in modern deep neural networks, as the ever increasing storage and memory requirements of these networks pose a problem in resource constrained environments. In this work, we focus on deep convolutional neural network architectures, and propose a novel randomized tensor sketching technique that we utilize to develop a unified framework for approximating the operation of both the convolutional and fully connected layers. By applying the sketching technique along different tensor dimensions, we design changes to the convolutional and fully connected layers that substantially reduce the number of effective parameters in a network. We show that the resulting smaller network can be trained directly, and has a classification accuracy that is comparable to the original network.