Variable projection framework for the reduced-rank matrix approximation problem by weighted least-squares

TL;DR

This paper reviews and develops advanced variable projection algorithms for the weighted low-rank approximation problem, addressing robustness, efficiency, and scalability issues, and providing new insights into the problem's geometry and solution landscape.

Contribution

It introduces new formulas for Jacobian and Hessian matrices, analyzes their properties, and enhances second-order algorithms for large-scale WLRA problems, connecting various existing methods.

Findings

New Jacobian and Hessian formulas with specific properties

Analysis of the geometric and landscape features of WLRA

Enhanced second-order algorithms for large datasets

Abstract

In this monograph, we review and develop variable projection Gauss-Newton, Levenberg-Marquardt and Newton methods for the Weighted Low-Rank Approximation (WLRA) problem, which has now an increasing number of applications in many scientific fields. Particular attention is drawn at the robustness, efficiency and scalability of these variable projection second-order algorithms such that they can be used also on larger datasets now commonly found in many practical problems for which only first-order algorithms based on sequential repetitions of local optimization (e.g., majorization, Expectation-Maximization or alternating least-squares methods) or variations of gradient descent (e.g., conjugate, proximal or stochastic gradient descent methods), or hybrid algorithms from these two classes of methods, were only feasible due to their lower cost and memory requirement per iteration. In parallel with this review of variable projection algorithms, we develop new formulae for the Jacobian and Hessian matrices involved in these variable projection methods and demonstrate their very specific properties such as the uniform rank deficiency of the Jacobian matrix or the rank deficiency of the Hessian matrix at the (local) minimizers of the cost function associated with the WLRA problem. These systematic deficiencies must be taken into account in any practical implementations of the algorithms. These different properties and the very particular geometry of the WLRA problem have not been well appreciated in the past and have been the main obstacles in the development of robust variable projection second-order algorithms for solving the WLRA problem. In addition, we demonstrate that the variable projection framework gives original insights on the solvability, the landscape and the non-smoothness of the WLRA problem. It also helps to describe the tight links between previously unrelated methods, which have been proposed to solve it. Specifically, we illustrate the closed links between the variable projection framework and Riemannian optimization on the Grassmann manifold for the WLRA problem. We expect that software's developers and practitioners in different fields such as computer vision, signal processing, recommender systems, machine learning, multivariate statistics and geophysical sciences will benefit from the results in this monograph in order to devise more robust and accurate algorithms to solve the WLRA problem.