Bayesian Extensive-Rank Matrix Factorization with Rotational Invariant Priors

TL;DR

This paper develops a Bayesian framework for matrix factorization with rotational invariant priors, deriving formulas for optimal estimators and confirming their conjectured optimality through numerical experiments.

Contribution

It introduces a novel Bayesian approach with rotational invariant priors for large-rank matrix factorization and provides analytical formulas for optimal estimators in this setting.

Findings

Derived analytical formulas for Rotation Invariant Estimators.

Conjectured and numerically supported their optimality in large dimensions.

Validated estimators against Oracle Estimators to confirm minimal mean-square-error.

Abstract

We consider a statistical model for matrix factorization in a regime where the rank of the two hidden matrix factors grows linearly with their dimension and their product is corrupted by additive noise. Despite various approaches, statistical and algorithmic limits of such problems have remained elusive. We study a Bayesian setting with the assumptions that (a) one of the matrix factors is symmetric, (b) both factors as well as the additive noise have rotational invariant priors, (c) the priors are known to the statistician. We derive analytical formulas for Rotation Invariant Estimators to reconstruct the two matrix factors, and conjecture that these are optimal in the large-dimension limit, in the sense that they minimize the average mean-square-error. We provide numerical checks which confirm the optimality conjecture when confronted to Oracle Estimators which are optimal by definition, but involve the ground-truth. Our derivation relies on a combination of tools, namely random matrix theory transforms, spherical integral formulas, and the replica method from statistical mechanics.