Fast approximation of matrix coherence and statistical leverage

TL;DR

This paper introduces a fast randomized algorithm to approximate statistical leverage scores and coherence of large matrices, significantly reducing computational complexity for applications in matrix approximation and data analysis.

Contribution

The paper presents a novel $O(nd \, \log n)$ time randomized algorithm for relative-error approximation of leverage scores, improving over the traditional $O(nd^2)$ approach.

Findings

Algorithm achieves relative-error approximations efficiently.

Extends to cross-leverage scores and streaming data.

Applicable to matrices with $n \approx d$.

Abstract

The statistical leverage scores of a matrix $A$ are the squared row-norms of the matrix containing its (top) left singular vectors and the coherence is the largest leverage score. These quantities are of interest in recently-popular problems such as matrix completion and Nystr\"{o}m-based low-rank matrix approximation as well as in large-scale statistical data analysis applications more generally; moreover, they are of interest since they define the key structural nonuniformity that must be dealt with in developing fast randomized matrix algorithms. Our main result is a randomized algorithm that takes as input an arbitrary $n \times d$ matrix $A$, with $n \gg d$, and that returns as output relative-error approximations to all $n$ of the statistical leverage scores. The proposed algorithm runs (under assumptions on the precise values of $n$ and $d$) in $O(n d \log n)$ time, as opposed to the $O(nd^2)$ time required by the na\"{i}ve algorithm that involves computing an orthogonal basis for the range of $A$. Our analysis may be viewed in terms of computing a relative-error approximation to an underconstrained least-squares approximation problem, or, relatedly, it may be viewed as an application of Johnson-Lindenstrauss type ideas. Several practically-important extensions of our basic result are also described, including the approximation of so-called cross-leverage scores, the extension of these ideas to matrices with $n \approx d$, and the extension to streaming environments.