Understanding Deflation Process in Over-parametrized Tensor Decomposition

TL;DR

This paper analyzes the training dynamics of gradient flow in over-parametrized tensor decomposition, showing it naturally follows a deflation process and can recover tensor components, advancing understanding of implicit regularization.

Contribution

It proves that gradient flow on orthogonally decomposable tensors mimics a deflation process, revealing how over-parametrized models implicitly regularize low-rank tensor learning.

Findings

Gradient flow first fits larger tensor components then smaller ones.

Gradient flow can recover all tensor components for orthogonal tensors.

The process is analogous to greedy low-rank learning in matrices.

Abstract

In this paper we study the training dynamics for gradient flow on over-parametrized tensor decomposition problems. Empirically, such training process often first fits larger components and then discovers smaller components, which is similar to a tensor deflation process that is commonly used in tensor decomposition algorithms. We prove that for orthogonally decomposable tensor, a slightly modified version of gradient flow would follow a tensor deflation process and recover all the tensor components. Our proof suggests that for orthogonal tensors, gradient flow dynamics works similarly as greedy low-rank learning in the matrix setting, which is a first step towards understanding the implicit regularization effect of over-parametrized models for low-rank tensors.