Tensor Methods for Generating Compact Uncertainty Quantification and Deep Learning Models

TL;DR

This paper reviews how tensor methods can create compact models for uncertainty quantification and deep learning, reducing computational costs and enabling deployment on resource-limited hardware.

Contribution

It summarizes recent advances in applying tensor techniques to develop efficient, low-rank models for uncertainty analysis and neural network compression.

Findings

Tensor methods significantly reduce simulation and measurement costs.

Tensorized neural networks can automatically determine optimal tensor ranks.

Tensor techniques enable deployment of deep learning models on resource-constrained devices.

Abstract

Tensor methods have become a promising tool to solve high-dimensional problems in the big data era. By exploiting possible low-rank tensor factorization, many high-dimensional model-based or data-driven problems can be solved to facilitate decision making or machine learning. In this paper, we summarize the recent applications of tensor computation in obtaining compact models for uncertainty quantification and deep learning. In uncertainty analysis where obtaining data samples is expensive, we show how tensor methods can significantly reduce the simulation or measurement cost. To enable the deployment of deep learning on resource-constrained hardware platforms, tensor methods can be used to significantly compress an over-parameterized neural network model or directly train a small-size model from scratch via optimization or statistical techniques. Recent Bayesian tensorized neural networks can automatically determine their tensor ranks in the training process.