Turnstile $\ell_p$ leverage score sampling with applications

TL;DR

This paper introduces a novel turnstile data stream algorithm for $\, ext{l}_p$ leverage score sampling, enabling efficient, approximate subsampling for regression problems, including the first turnstile algorithm for logistic regression with near-optimal guarantees.

Contribution

It develops the first turnstile $ ext{l}_p$ leverage score sampling algorithm with provable guarantees, extending to logistic regression and improving over prior methods.

Findings

Algorithm achieves $(1+ ext{epsilon})$ approximation for logistic regression in turnstile streams.

Extends to $ ext{l}_p$ leverage score sampling with low overhead.

Experimental results show improvements over oblivious sketching and previous sampling methods.

Abstract

The turnstile data stream model offers the most flexible framework where data can be manipulated dynamically, i.e., rows, columns, and even single entries of an input matrix can be added, deleted, or updated multiple times in a data stream. We develop a novel algorithm for sampling rows $a_i$ of a matrix $A\in\mathbb{R}^{n\times d}$, proportional to their $\ell_p$ norm, when $A$ is presented in a turnstile data stream. Our algorithm not only returns the set of sampled row indexes, it also returns slightly perturbed rows $\tilde{a}_i \approx a_i$, and approximates their sampling probabilities up to $\varepsilon$ relative error. When combined with preconditioning techniques, our algorithm extends to $\ell_p$ leverage score sampling over turnstile data streams. With these properties in place, it allows us to simulate subsampling constructions of coresets for important regression problems to operate over turnstile data streams with very little overhead compared to their respective off-line subsampling algorithms. For logistic regression, our framework yields the first algorithm that achieves a $(1+\varepsilon)$ approximation and works in a turnstile data stream using polynomial sketch/subsample size, improving over $O(1)$ approximations, or $\exp(1/\varepsilon)$ sketch size of previous work. We compare experimentally to plain oblivious sketching and plain leverage score sampling algorithms for $\ell_p$ and logistic regression.