Rademacher Random Projections with Tensor Networks

TL;DR

This paper introduces Rademacher-based tensorized random projections using Tensor Train and MPO formats, demonstrating their effectiveness and limitations in high-dimensional tensor dimension reduction.

Contribution

It proposes a novel Rademacher tensorized random projection method in TT format and analyzes its theoretical properties and practical performance compared to Gaussian projections.

Findings

Rademacher TT projections can replace Gaussian ones with similar embedding size

Tensorized Rademacher RP outperforms Gaussian RP in experiments

MPO format tensorized RP does not satisfy Johnson-Lindenstrauss properties

Abstract

Random projection (RP) have recently emerged as popular techniques in the machine learning community for their ability in reducing the dimension of very high-dimensional tensors. Following the work in [30], we consider a tensorized random projection relying on Tensor Train (TT) decomposition where each element of the core tensors is drawn from a Rademacher distribution. Our theoretical results reveal that the Gaussian low-rank tensor represented in compressed form in TT format in [30] can be replaced by a TT tensor with core elements drawn from a Rademacher distribution with the same embedding size. Experiments on synthetic data demonstrate that tensorized Rademacher RP can outperform the tensorized Gaussian RP studied in [30]. In addition, we show both theoretically and experimentally, that the tensorized RP in the Matrix Product Operator (MPO) format is not a Johnson-Lindenstrauss transform (JLT) and therefore not a well-suited random projection map