Asymptotic Optimism for Tensor Regression Models with Applications to Neural Network Compression

TL;DR

This paper develops a rank-selection method for low-rank tensor regression models, demonstrating its effectiveness in model selection and neural network compression through theoretical analysis and real-world applications.

Contribution

It introduces a prediction-oriented rank-selection rule based on asymptotic optimism, applicable to tensor regression and neural network compression, with theoretical justification and practical validation.

Findings

Optimism is minimized at the true tensor rank for CP and Tucker models.

The proposed method aligns with cross-validation for rank selection.

Application to neural network compression shows practical utility.

Abstract

We study rank selection for low-rank tensor regression under random covariates design. Under a Gaussian random-design model and some mild conditions, we derive population expressions for the expected training-testing discrepancy (optimism) for both CP and Tucker decomposition. We further demonstrate that the optimism is minimized at the true tensor rank for both CP and Tucker regression. This yields a prediction-oriented rank-selection rule that aligns with cross-validation and extends naturally to tensor-model averaging. We also discuss conditions under which under- or over-ranked models may appear preferable, thereby clarifying the scope of the method. Finally, we showcase its practical utility on a real-world image regression task and extend its application to tensor-based compression of neural network, highlighting its potential for model selection in deep learning.