Ridge Regression with Frequent Directions: Statistical and Optimization Perspectives

TL;DR

This paper establishes theoretical guarantees for Frequent Directions in large-scale ridge regression, demonstrating its statistical accuracy and optimization efficiency, and introduces enhancements with Robust Frequent Directions.

Contribution

It provides the first constant factor relative error bounds for sketched ridge regression using Frequent Directions and proposes an iterative scheme for high-accuracy solutions.

Findings

Frequent Directions achieves constant factor relative error bounds in ridge regression.

An iterative scheme using FD yields high-accuracy solutions in optimization.

Robust FD further improves statistical and optimization performance.

Abstract

Despite its impressive theory \& practical performance, Frequent Directions (\acrshort{fd}) has not been widely adopted for large-scale regression tasks. Prior work has shown randomized sketches (i) perform worse in estimating the covariance matrix of the data than \acrshort{fd}; (ii) incur high error when estimating the bias and/or variance on sketched ridge regression. We give the first constant factor relative error bounds on the bias \& variance for sketched ridge regression using \acrshort{fd}. We complement these statistical results by showing that \acrshort{fd} can be used in the optimization setting through an iterative scheme which yields high-accuracy solutions. This improves on randomized approaches which need to compromise the need for a new sketch every iteration with speed of convergence. In both settings, we also show using \emph{Robust Frequent Directions} further enhances performance.