Iterative Row Sampling

TL;DR

This paper introduces an iterative row sampling algorithm for regression problems with tall, thin matrices, providing improved theoretical guarantees and efficiency by leveraging iterative importance sampling based on leverage scores.

Contribution

It develops a novel iterative leverage score computation method that enhances existing algorithms, achieving better theoretical guarantees and efficiency in solving regression problems.

Findings

Algorithm runs in $O(nnz(A) + d^{ heta + ext{constant}} \, \epsilon^{-2})$ time.

Iterative approach produces geometrically smaller matrix instances.

Provides guarantees comparable or better than current state-of-the-art methods.

Abstract

There has been significant interest and progress recently in algorithms that solve regression problems involving tall and thin matrices in input sparsity time. These algorithms find shorter equivalent of a n*d matrix where n >> d, which allows one to solve a poly(d) sized problem instead. In practice, the best performances are often obtained by invoking these routines in an iterative fashion. We show these iterative methods can be adapted to give theoretical guarantees comparable and better than the current state of the art. Our approaches are based on computing the importances of the rows, known as leverage scores, in an iterative manner. We show that alternating between computing a short matrix estimate and finding more accurate approximate leverage scores leads to a series of geometrically smaller instances. This gives an algorithm that runs in $O(nnz(A) + d^{\omega + \theta} \epsilon^{-2})$ time for any $\theta > 0$, where the $d^{\omega + \theta}$ term is comparable to the cost of solving a regression problem on the small approximation. Our results are built upon the close connection between randomized matrix algorithms, iterative methods, and graph sparsification.