Efficient Alternating Minimization with Applications to Weighted Low Rank Approximation

TL;DR

This paper presents an efficient, robust alternating minimization framework for weighted low rank approximation, significantly improving runtime and providing provable guarantees for this NP-hard problem with broad applications in machine learning.

Contribution

It introduces a framework that allows approximate updates in alternating minimization, reducing runtime from W_0 k^2 to W_0 k for weighted low rank approximation.

Findings

Runtime improved from W_0 k^2 to W_0 k.

Provides provable guarantees for approximate alternating minimization.

Develops a high-accuracy regression solver and robust analysis.

Abstract

Weighted low rank approximation is a fundamental problem in numerical linear algebra, and it has many applications in machine learning. Given a matrix $M \in \mathbb{R}^{n \times n}$, a non-negative weight matrix $W \in \mathbb{R}_{\geq 0}^{n \times n}$, a parameter $k$, the goal is to output two matrices $X,Y\in \mathbb{R}^{n \times k}$ such that $\| W \circ (M - X Y^\top) \|_F$ is minimized, where $\circ$ denotes the Hadamard product. It naturally generalizes the well-studied low rank matrix completion problem. Such a problem is known to be NP-hard and even hard to approximate assuming the Exponential Time Hypothesis [GG11, RSW16]. Meanwhile, alternating minimization is a good heuristic solution for weighted low rank approximation. In particular, [LLR16] shows that, under mild assumptions, alternating minimization does provide provable guarantees. In this work, we develop an efficient and robust framework for alternating minimization that allows the alternating updates to be computed approximately. For weighted low rank approximation, this improves the runtime of [LLR16] from $\|W\|_0k^2$ to $\|W\|_0 k$ where $\|W\|_0$ denotes the number of nonzero entries of the weight matrix. At the heart of our framework is a high-accuracy multiple response regression solver together with a robust analysis of alternating minimization.