Enhancing Deep Learning Models through Tensorization: A Comprehensive Survey and Framework

TL;DR

This paper reviews tensorization techniques that enhance deep learning by leveraging multidimensional data, demonstrating improved expressiveness and efficiency through multiway analysis in various applications.

Contribution

It provides a comprehensive survey of tensorization methods, illustrating their benefits and integration with deep neural networks, supported by practical case studies.

Findings

Multiway analysis is more expressive than 2D algorithms.

Tensorization reduces model parameters and accelerates processing.

Multidimensional data captures complex interrelationships effectively.

Abstract

The burgeoning growth of public domain data and the increasing complexity of deep learning model architectures have underscored the need for more efficient data representation and analysis techniques. This paper is motivated by the work of (Helal, 2023) and aims to present a comprehensive overview of tensorization. This transformative approach bridges the gap between the inherently multidimensional nature of data and the simplified 2-dimensional matrices commonly used in linear algebra-based machine learning algorithms. This paper explores the steps involved in tensorization, multidimensional data sources, various multiway analysis methods employed, and the benefits of these approaches. A small example of Blind Source Separation (BSS) is presented comparing 2-dimensional algorithms and a multiway algorithm in Python. Results indicate that multiway analysis is more expressive. Contrary to the intuition of the dimensionality curse, utilising multidimensional datasets in their native form and applying multiway analysis methods grounded in multilinear algebra reveal a profound capacity to capture intricate interrelationships among various dimensions while, surprisingly, reducing the number of model parameters and accelerating processing. A survey of the multi-away analysis methods and integration with various Deep Neural Networks models is presented using case studies in different application domains.