Online Tensor Learning: Computational and Statistical Trade-offs, Adaptivity and Optimal Regret

TL;DR

This paper introduces an online tensor learning algorithm, oRGrad, that is computationally efficient, memory-saving, and capable of achieving optimal regret and statistical error rates, with an adaptive version that does not require knowing the time horizon.

Contribution

The paper proposes a unified online Riemannian gradient descent algorithm for tensor learning, achieving optimal regret and statistical error rates, and introduces an adaptive variant that does not need the time horizon.

Findings

oRGrad achieves $O(T^{1/2})$ regret with fixed step size.

Adaptive-oRGrad attains $O( ext{log } T)$ regret without knowing the horizon.

Numerical simulations show superior performance in space weather prediction.

Abstract

Large tensor learning algorithms are typically computationally expensive and require storing a vast amount of data. In this paper, we propose a unified online Riemannian gradient descent (oRGrad) algorithm for tensor learning, which is computationally efficient, consumes much less memory, and can handle sequentially arriving data while making timely predictions. The algorithm is applicable to both linear and generalized linear models. If the time horizon T is known, oRGrad achieves statistical optimality by choosing an appropriate fixed step size. We find that noisy tensor completion particularly benefits from online algorithms by avoiding the trimming procedure and ensuring sharp entry-wise statistical error, which is often technically challenging for offline methods. The regret of oRGrad is analyzed, revealing a fascinating trilemma concerning the computational convergence rate, statistical error, and regret bound. By selecting an appropriate constant step size, oRGrad achieves an $O(T^{1/2})$ regret. We then introduce the adaptive-oRGrad algorithm, which can achieve the optimal $O(\log T)$ regret by adaptively selecting step sizes, regardless of whether the time horizon is known. The adaptive-oRGrad algorithm can attain a statistically optimal error rate without knowing the horizon. Comprehensive numerical simulations corroborate our theoretical findings. We show that oRGrad significantly outperforms its offline counterpart in predicting the solar F10.7 index with tensor predictors that monitor space weather impacts.