Implicit Regularization for Tubal Tensor Factorizations via Gradient Descent

TL;DR

This paper rigorously analyzes how gradient descent implicitly regularizes overparametrized tubal tensor factorizations, favoring low tubal rank solutions, especially relevant for image data, beyond the lazy training regime.

Contribution

It provides the first theoretical proof of implicit regularization for tensor factorizations under gradient descent, extending beyond gradient flow and lazy training regimes.

Findings

Gradient descent biases solutions towards low tubal rank.

Small random initialization is crucial for the implicit regularization.

Numerical simulations confirm the theoretical predictions.

Abstract

We provide a rigorous analysis of implicit regularization in an overparametrized tensor factorization problem beyond the lazy training regime. For matrix factorization problems, this phenomenon has been studied in a number of works. A particular challenge has been to design universal initialization strategies which provably lead to implicit regularization in gradient-descent methods. At the same time, it has been argued by Cohen et. al. 2016 that more general classes of neural networks can be captured by considering tensor factorizations. However, in the tensor case, implicit regularization has only been rigorously established for gradient flow or in the lazy training regime. In this paper, we prove the first tensor result of its kind for gradient descent rather than gradient flow. We focus on the tubal tensor product and the associated notion of low tubal rank, encouraged by the relevance of this model for image data. We establish that gradient descent in an overparametrized tensor factorization model with a small random initialization exhibits an implicit bias towards solutions of low tubal rank. Our theoretical findings are illustrated in an extensive set of numerical simulations show-casing the dynamics predicted by our theory as well as the crucial role of using a small random initialization.