Iterative Hessian Sketch in Input Sparsity Time

TL;DR

This paper introduces an efficient iterative Hessian sketching method using CountSketch and Johnson-Lindenstrauss transforms, enabling significantly faster solutions for large-scale constrained regression problems with high accuracy.

Contribution

The paper demonstrates that fast sketching techniques like CountSketch improve the speed and accuracy of iterative Hessian sketching for convex regression, outperforming previous methods.

Findings

Achieves roughly 100x speedup on sparse data

Achieves roughly 10x speedup on dense data

Solutions are within machine precision of the optimal

Abstract

Scalable algorithms to solve optimization and regression tasks even approximately, are needed to work with large datasets. In this paper we study efficient techniques from matrix sketching to solve a variety of convex constrained regression problems. We adopt "Iterative Hessian Sketching" (IHS) and show that the fast CountSketch and sparse Johnson-Lindenstrauss Transforms yield state-of-the-art accuracy guarantees under IHS, while drastically improving the time cost. As a result, we obtain significantly faster algorithms for constrained regression, for both sparse and dense inputs. Our empirical results show that we can summarize data roughly 100x faster for sparse data, and, surprisingly, 10x faster on dense data! Consequently, solutions accurate to within machine precision of the optimal solution can be found much faster than the previous state of the art.