Efficient Finite Initialization with Partial Norms for Tensorized Neural Networks and Tensor Networks Algorithms

TL;DR

This paper introduces two algorithms for initializing tensorized neural networks and tensor network algorithms using partial Frobenius norm computations, improving scalability and reusing intermediate calculations.

Contribution

The authors propose a novel initialization method based on partial norms and subnetworks, applicable to various tensor network types, with demonstrated scalability and efficiency.

Findings

Effective normalization of tensor network layers using partial norms.

Scalable initialization method tested on MPS/TT and MPO/TT-M layers.

Code implementation is publicly available.

Abstract

We present two algorithms to initialize layers of tensorized neural networks and general tensor network algorithms using partial computations of their Frobenius norms and positive lineal entrywise sums, depending on the type of tensor network involved. The core of this method is the use of the norm of subnetworks of the tensor network in an iterative way, so that we normalize by the finite values of the norms that led to the divergence or zero norm. In addition, the method benefits from the reuse of intermediate calculations. We have also applied it to the Matrix Product State/Tensor Train (MPS/TT) and Matrix Product Operator/Tensor Train Matrix (MPO/TT-M) layers and have seen its scaling versus the number of nodes, bond dimension, and physical dimension. All code is publicly available.