Low-Rank Tensor Decompositions for the Theory of Neural Networks

TL;DR

This paper reviews how low-rank tensor decompositions are fundamental in understanding deep neural networks, covering their theoretical implications for expressivity, learnability, generalization, and identifiability.

Contribution

It provides a unified overview of tensor methods' role in deep neural network theory, connecting diverse approaches across disciplines.

Findings

Tensor decompositions have strong uniqueness guarantees.

Polynomial time algorithms exist for tensor decomposition.

Tensor methods explain neural networks' expressivity and learnability.

Abstract

The groundbreaking performance of deep neural networks (NNs) promoted a surge of interest in providing a mathematical basis to deep learning theory. Low-rank tensor decompositions are specially befitting for this task due to their close connection to NNs and their rich theoretical results. Different tensor decompositions have strong uniqueness guarantees, which allow for a direct interpretation of their factors, and polynomial time algorithms have been proposed to compute them. Through the connections between tensors and NNs, such results supported many important advances in the theory of NNs. In this review, we show how low-rank tensor methods--which have been a core tool in the signal processing and machine learning communities--play a fundamental role in theoretically explaining different aspects of the performance of deep NNs, including their expressivity, algorithmic learnability and computational hardness, generalization, and identifiability. Our goal is to give an accessible overview of existing approaches (developed by different communities, ranging from computer science to mathematics) in a coherent and unified way, and to open a broader perspective on the use of low-rank tensor decompositions for the theory of deep NNs.