Tensor Decompositions in Deep Learning

TL;DR

This paper surveys tensor decompositions in deep learning, highlighting their role in model compression, data representation, and discussing open challenges in the field.

Contribution

It provides a comprehensive overview of tensor methods in deep learning, emphasizing recent advances and future research directions.

Findings

Tensor decompositions effectively compress deep learning models.

Tensor methods enable richer adaptive data representations.

Open research challenges remain in applying tensor techniques.

Abstract

The paper surveys the topic of tensor decompositions in modern machine learning applications. It focuses on three active research topics of significant relevance for the community. After a brief review of consolidated works on multi-way data analysis, we consider the use of tensor decompositions in compressing the parameter space of deep learning models. Lastly, we discuss how tensor methods can be leveraged to yield richer adaptive representations of complex data, including structured information. The paper concludes with a discussion on interesting open research challenges.