Tensor Network Based Feature Learning Model

TL;DR

This paper introduces a tensor network-based feature learning model that efficiently learns feature hyperparameters alongside model parameters, significantly speeding up training while maintaining prediction accuracy.

Contribution

The paper proposes the FL model using CPD and ALS to automatically learn feature hyperparameters, addressing a key challenge in tensor-based kernel methods.

Findings

FL model trains 3-5 times faster than standard cross-validation

FL maintains comparable prediction accuracy to traditional methods

Effective on real datasets of various sizes and dimensions

Abstract

Many approximations were suggested to circumvent the cubic complexity of kernel-based algorithms, allowing their application to large-scale datasets. One strategy is to consider the primal formulation of the learning problem by mapping the data to a higher-dimensional space using tensor-product structured polynomial and Fourier features. The curse of dimensionality due to these tensor-product features was effectively solved by a tensor network reparameterization of the model parameters. However, another important aspect of model training - identifying optimal feature hyperparameters - has not been addressed and is typically handled using the standard cross-validation approach. In this paper, we introduce the Feature Learning (FL) model, which addresses this issue by representing tensor-product features as a learnable Canonical Polyadic Decomposition (CPD). By leveraging this CPD structure, we efficiently learn the hyperparameters associated with different features alongside the model parameters using an Alternating Least Squares (ALS) optimization method. We prove the effectiveness of the FL model through experiments on real data of various dimensionality and scale. The results show that the FL model can be consistently trained 3-5 times faster than and have the prediction quality on par with a standard cross-validated model.