Relative Errors for Deterministic Low-Rank Matrix Approximations

TL;DR

This paper analyzes the Frequent Directions streaming algorithm for low-rank matrix approximation, providing bounds on approximation errors and discussing limitations in sparsity adaptation.

Contribution

It establishes error bounds for the deterministic Frequent Directions algorithm and discusses its limitations in creating sparse approximations.

Findings

Achieves (1+eps) approximation of the best rank-k approximation.

Provides bounds on the Frobenius norm errors of the approximation.

Shows limitations in adapting the algorithm for sparse matrices.

Abstract

We consider processing an n x d matrix A in a stream with row-wise updates according to a recent algorithm called Frequent Directions (Liberty, KDD 2013). This algorithm maintains an l x d matrix Q deterministically, processing each row in O(d l^2) time; the processing time can be decreased to O(d l) with a slight modification in the algorithm and a constant increase in space. We show that if one sets l = k+ k/eps and returns Q_k, a k x d matrix that is the best rank k approximation to Q, then we achieve the following properties: ||A - A_k||_F^2 <= ||A||_F^2 - ||Q_k||_F^2 <= (1+eps) ||A - A_k||_F^2 and where pi_{Q_k}(A) is the projection of A onto the rowspace of Q_k then ||A - pi_{Q_k}(A)||_F^2 <= (1+eps) ||A - A_k||_F^2. We also show that Frequent Directions cannot be adapted to a sparse version in an obvious way that retains the l original rows of the matrix, as opposed to a linear combination or sketch of the rows.