Multi-step Inertial Accelerated Doubly Stochastic Gradient Methods for Block Term Tensor Decomposition

TL;DR

This paper introduces Midas-LL1, a novel accelerated stochastic gradient method for tensor decomposition that improves convergence speed and solution quality over existing algorithms, especially for structured multilinear rank models.

Contribution

It develops a unified multi-step inertial accelerated doubly stochastic gradient algorithm with convergence analysis for structured tensor decomposition.

Findings

Midas-LL1 converges to an $ ext{ε}$-stationary point within $ ext{O(ε}^{-2})$ iterations.

The algorithm outperforms state-of-the-art methods in speed and accuracy.

Experimental results validate the theoretical convergence and efficiency improvements.

Abstract

In this paper, we explore a specific optimization problem that combines a differentiable nonconvex function with a nondifferentiable function for multi-block variables, which is particularly relevant to tackle the multilinear rank-($L_r$,$L_r$,1) block-term tensor decomposition model with a regularization term. While existing algorithms often suffer from high per-iteration complexity and slow convergence, this paper employs a unified multi-step inertial accelerated doubly stochastic gradient descent method tailored for structured rank-$\left(L_r, L_r, 1\right)$ tensor decomposition, referred to as Midas-LL1. We also introduce an extended multi-step variance-reduced stochastic estimator framework. Our analysis under this new framework demonstrates the subsequential and sequential convergence of the proposed algorithm under certain conditions and illustrates the sublinear convergence rate of the subsequence, showing that the Midas-LL1 algorithm requires at most $\mathcal{O}(\varepsilon^{-2})$ iterations in expectation to reach an $\varepsilon$-stationary point. The proposed algorithm is evaluated on several datasets, and the results indicate that Midas-LL1 outperforms existing state-of-the-art algorithms in terms of both computational speed and solution quality.