Structured Low-Rank Tensors for Generalized Linear Models

TL;DR

This paper introduces a Low Separation Rank (LSR) tensor model for generalized linear models, providing a new approach that reduces sample complexity and improves estimation accuracy over existing tensor and vector-based methods.

Contribution

It proposes the LSR tensor model for GLMs, develops a block coordinate descent algorithm, and derives a minimax lower bound on estimation error, demonstrating theoretical and practical advantages.

Findings

The LSR model outperforms Tucker and CP models in synthetic experiments.

The minimax lower bound scales with the intrinsic degrees of freedom.

Numerical experiments show improved performance on medical imaging datasets.

Abstract

Recent works have shown that imposing tensor structures on the coefficient tensor in regression problems can lead to more reliable parameter estimation and lower sample complexity compared to vector-based methods. This work investigates a new low-rank tensor model, called Low Separation Rank (LSR), in Generalized Linear Model (GLM) problems. The LSR model -- which generalizes the well-known Tucker and CANDECOMP/PARAFAC (CP) models, and is a special case of the Block Tensor Decomposition (BTD) model -- is imposed onto the coefficient tensor in the GLM model. This work proposes a block coordinate descent algorithm for parameter estimation in LSR-structured tensor GLMs. Most importantly, it derives a minimax lower bound on the error threshold on estimating the coefficient tensor in LSR tensor GLM problems. The minimax bound is proportional to the intrinsic degrees of freedom in the LSR tensor GLM problem, suggesting that its sample complexity may be significantly lower than that of vectorized GLMs. This result can also be specialised to lower bound the estimation error in CP and Tucker-structured GLMs. The derived bounds are comparable to tight bounds in the literature for Tucker linear regression, and the tightness of the minimax lower bound is further assessed numerically. Finally, numerical experiments on synthetic datasets demonstrate the efficacy of the proposed LSR tensor model for three regression types (linear, logistic and Poisson). Experiments on a collection of medical imaging datasets demonstrate the usefulness of the LSR model over other tensor models (Tucker and CP) on real, imbalanced data with limited available samples.