A Complete Loss Landscape Analysis of Regularized Deep Matrix Factorization

TL;DR

This paper provides a comprehensive analysis of the loss landscape in regularized deep matrix factorization, revealing critical point structures and conditions for convergence, supported by numerical visualizations.

Contribution

It offers a complete characterization of all critical points and conditions for their nature in regularized DMF, explaining convergence behavior of gradient methods.

Findings

All critical points are characterized explicitly.

Conditions distinguishing local, global minima, and saddle points are established.

Numerical experiments support the theoretical landscape analysis.

Abstract

Despite its wide range of applications across various domains, the optimization foundations of deep matrix factorization (DMF) remain largely open. In this work, we aim to fill this gap by conducting a comprehensive study of the loss landscape of the regularized DMF problem. Toward this goal, we first provide a closed-form characterization of all critical points of the problem. Building on this, we establish precise conditions under which a critical point is a local minimizer, a global minimizer, a strict saddle point, or a non-strict saddle point. Leveraging these results, we derive a necessary and sufficient condition under which every critical point is either a local minimizer or a strict saddle point. This provides insights into why gradient-based methods almost always converge to a local minimizer of the regularized DMF problem. Finally, we conduct numerical experiments to visualize its loss landscape to support our theory.