Subquadratic Kronecker Regression with Applications to Tensor Decomposition

TL;DR

This paper introduces a novel subquadratic algorithm for Kronecker regression that significantly speeds up tensor decomposition tasks like Tucker decomposition, with proven efficiency and accuracy on real-world data.

Contribution

It presents the first subquadratic-time algorithm for Kronecker regression with approximation guarantees, extending to related tensor decomposition problems.

Findings

Achieves subquadratic running time for Kronecker regression

Effective on synthetic and real-world tensor data

Improves efficiency of Tucker decomposition steps

Abstract

Kronecker regression is a highly-structured least squares problem $\min_{\mathbf{x}} \lVert \mathbf{K}\mathbf{x} - \mathbf{b} \rVert_{2}^2$, where the design matrix $\mathbf{K} = \mathbf{A}^{(1)} \otimes \cdots \otimes \mathbf{A}^{(N)}$ is a Kronecker product of factor matrices. This regression problem arises in each step of the widely-used alternating least squares (ALS) algorithm for computing the Tucker decomposition of a tensor. We present the first subquadratic-time algorithm for solving Kronecker regression to a $(1+\varepsilon)$-approximation that avoids the exponential term $O(\varepsilon^{-N})$ in the running time. Our techniques combine leverage score sampling and iterative methods. By extending our approach to block-design matrices where one block is a Kronecker product, we also achieve subquadratic-time algorithms for (1) Kronecker ridge regression and (2) updating the factor matrices of a Tucker decomposition in ALS, which is not a pure Kronecker regression problem, thereby improving the running time of all steps of Tucker ALS. We demonstrate the speed and accuracy of this Kronecker regression algorithm on synthetic data and real-world image tensors.