Adaptive Learning of Tensor Network Structures

TL;DR

This paper introduces an adaptive, greedy algorithm for jointly learning tensor network structures and parameters, significantly improving efficiency and effectiveness in tensor decomposition, completion, and neural network compression tasks.

Contribution

It presents a novel, generic method to adaptively identify tensor network structures from data, outperforming existing topology search and tensor compression approaches.

Findings

Outperforms state-of-the-art evolutionary topology search in tensor decomposition

Efficiently compresses neural networks with superior results

Faster and more effective than previous methods

Abstract

Tensor Networks (TN) offer a powerful framework to efficiently represent very high-dimensional objects. TN have recently shown their potential for machine learning applications and offer a unifying view of common tensor decomposition models such as Tucker, tensor train (TT) and tensor ring (TR). However, identifying the best tensor network structure from data for a given task is challenging. In this work, we leverage the TN formalism to develop a generic and efficient adaptive algorithm to jointly learn the structure and the parameters of a TN from data. Our method is based on a simple greedy approach starting from a rank one tensor and successively identifying the most promising tensor network edges for small rank increments. Our algorithm can adaptively identify TN structures with small number of parameters that effectively optimize any differentiable objective function. Experiments on tensor decomposition, tensor completion and model compression tasks demonstrate the effectiveness of the proposed algorithm. In particular, our method outperforms the state-of-the-art evolutionary topology search [Li and Sun, 2020] for tensor decomposition of images (while being orders of magnitude faster) and finds efficient tensor network structures to compress neural networks outperforming popular TT based approaches [Novikov et al., 2015].