Implicit Regularization with Polynomial Growth in Deep Tensor Factorization

TL;DR

This paper investigates how implicit regularization in deep tensor factorization leads to polynomial growth in regularization effects with network depth, improving estimation accuracy and convergence.

Contribution

It reveals that implicit regularization in deep tensor factorization grows polynomially with depth, extending previous quadratic growth understanding.

Findings

Implicit regularization grows polynomially with depth.

Deeper networks yield more accurate tensor estimations.

Enhanced convergence properties observed in experiments.

Abstract

We study the implicit regularization effects of deep learning in tensor factorization. While implicit regularization in deep matrix and 'shallow' tensor factorization via linear and certain type of non-linear neural networks promotes low-rank solutions with at most quadratic growth, we show that its effect in deep tensor factorization grows polynomially with the depth of the network. This provides a remarkably faithful description of the observed experimental behaviour. Using numerical experiments, we demonstrate the benefits of this implicit regularization in yielding a more accurate estimation and better convergence properties.