A Projection Algorithm for the Unitary Weights

TL;DR

This paper introduces a novel backpropagation-based projection algorithm using Lie algebra to efficiently initialize unitary weights in neural networks, reducing training time while preserving inference speed benefits.

Contribution

The paper presents a new algorithm that approximates unitary weights from pre-trained non-unitary weights, improving training efficiency of unitary neural networks.

Findings

Faster training of unitary networks with the proposed method.

Maintains inference speed advantages of unitary networks.

Applicable to architectures with pre-trained weights.

Abstract

Unitary neural networks are promising alternatives for solving the exploding and vanishing activation/gradient problem without the need for explicit normalization that reduces the inference speed. However, they often require longer training time due to the additional unitary constraints on their weight matrices. Here we show a novel algorithm using a backpropagation technique with Lie algebra for computing approximated unitary weights from their pre-trained, non-unitary counterparts. The unitary networks initialized with these approximations can reach the desired accuracies much faster, mitigating their training time penalties while maintaining inference speedups. Our approach will be instrumental in the adaptation of unitary networks, especially for those neural architectures where pre-trained weights are freely available.