When Can We Solve the Weighted Low Rank Approximation Problem in Truly Subquadratic Time?

TL;DR

This paper investigates conditions under which the weighted low-rank approximation problem can be solved in nearly linear time, challenging the traditional quadratic time complexity for dense matrices.

Contribution

It identifies a specific regime where dense matrices allow for almost linear time solutions, advancing understanding of computational complexity in low-rank approximation.

Findings

Potential for subquadratic algorithms in certain dense matrix regimes

Breakthrough in understanding when weighted low-rank approximation is computationally feasible

Challenges the assumption that dense matrices always require quadratic time algorithms

Abstract

The weighted low-rank approximation problem is a fundamental numerical linear algebra problem and has many applications in machine learning. Given a $n \times n$ weight matrix $W$ and a $n \times n$ matrix $A$, the goal is to find two low-rank matrices $U, V \in \mathbb{R}^{n \times k}$ such that the cost of $\| W \circ (U V^\top - A) \|_F^2$ is minimized. Previous work has to pay $\Omega(n^2)$ time when matrices $A$ and $W$ are dense, e.g., having $\Omega(n^2)$ non-zero entries. In this work, we show that there is a certain regime, even if $A$ and $W$ are dense, we can still hope to solve the weighted low-rank approximation problem in almost linear $n^{1+o(1)}$ time.