Weight Matrix Dimensionality Reduction in Deep Learning via Kronecker Multi-layer Architectures

TL;DR

This paper introduces a novel deep learning architecture that employs Kronecker product-based matrix decomposition to reduce model complexity and computational costs while maintaining accuracy.

Contribution

The paper presents a new neural network architecture utilizing Kronecker product decomposition for efficient dimensionality reduction in deep learning models.

Findings

Significant reduction in training time and computational resources.

Achieves comparable error levels to traditional networks.

Effective sparsification of fully connected layers.

Abstract

Deep learning using neural networks is an effective technique for generating models of complex data. However, training such models can be expensive when networks have large model capacity resulting from a large number of layers and nodes. For training in such a computationally prohibitive regime, dimensionality reduction techniques ease the computational burden, and allow implementations of more robust networks. We propose a novel type of such dimensionality reduction via a new deep learning architecture based on fast matrix multiplication of a Kronecker product decomposition; in particular our network construction can be viewed as a Kronecker product-induced sparsification of an "extended" fully connected network. Analysis and practical examples show that this architecture allows a neural network to be trained and implemented with a significant reduction in computational time and resources, while achieving a similar error level compared to a traditional feedforward neural network.