Deep Unitary Convolutional Neural Networks

TL;DR

This paper introduces deep unitary convolutional neural networks that address vanishing and exploding gradients, offering faster inference and reduced storage without sacrificing accuracy by extending the Lie algebra-based unitary framework.

Contribution

It extends the unitary framework to neural networks of any dimensionality, overcoming previous limitations and improving inference speed and storage efficiency.

Findings

Up to 32% faster inference speeds.

Up to 50% reduction in storage space.

Maintains competitive prediction accuracy.

Abstract

Deep neural networks can suffer from the exploding and vanishing activation problem, in which the networks fail to train properly because the neural signals either amplify or attenuate across the layers and become saturated. While other normalization methods aim to fix the stated problem, most of them have inference speed penalties in those applications that require running averages of the neural activations. Here we extend the unitary framework based on Lie algebra to neural networks of any dimensionalities, overcoming the major constraints of the prior arts that limit synaptic weights to be square matrices. Our proposed unitary convolutional neural networks deliver up to 32% faster inference speeds and up to 50% reduction in permanent hard disk space while maintaining competitive prediction accuracy.