Adaptive Sketches for Robust Regression with Importance Sampling

TL;DR

This paper presents a novel data structure and algorithm for efficient importance sampling in robust regression, enabling faster stochastic gradient descent with reduced variance using only a single data pass.

Contribution

It introduces an approximate importance sampling method for robust regression that operates with sublinear space and a single data pass, improving upon traditional methods.

Findings

Efficient importance sampling reduces variance in SGD.

Algorithm operates with sublinear space and one data pass.

Extends to second-order optimization importance sampling.

Abstract

We introduce data structures for solving robust regression through stochastic gradient descent (SGD) by sampling gradients with probability proportional to their norm, i.e., importance sampling. Although SGD is widely used for large scale machine learning, it is well-known for possibly experiencing slow convergence rates due to the high variance from uniform sampling. On the other hand, importance sampling can significantly decrease the variance but is usually difficult to implement because computing the sampling probabilities requires additional passes over the data, in which case standard gradient descent (GD) could be used instead. In this paper, we introduce an algorithm that approximately samples $T$ gradients of dimension $d$ from nearly the optimal importance sampling distribution for a robust regression problem over $n$ rows. Thus our algorithm effectively runs $T$ steps of SGD with importance sampling while using sublinear space and just making a single pass over the data. Our techniques also extend to performing importance sampling for second-order optimization.