Adaptive and Oblivious Randomized Subspace Methods for High-Dimensional Optimization: Sharp Analysis and Lower Bounds

TL;DR

This paper introduces adaptive and oblivious randomized subspace methods for high-dimensional convex optimization, providing sharp analysis, lower bounds, and demonstrating significant practical speed-ups in machine learning tasks.

Contribution

It develops novel adaptive subspace sampling strategies, analyzes their approximation properties, and establishes lower bounds, advancing the understanding of randomized optimization in high dimensions.

Findings

Adaptive subspace sampling outperforms oblivious methods.

Theoretical bounds relate error to data spectrum and Gaussian width.

Experimental results show substantial speed-ups in ML applications.

Abstract

We propose novel randomized optimization methods for high-dimensional convex problems based on restrictions of variables to random subspaces. We consider oblivious and data-adaptive subspaces and study their approximation properties via convex duality and Fenchel conjugates. A suitable adaptive subspace can be generated by sampling a correlated random matrix whose second order statistics mirror the input data. We illustrate that the adaptive strategy can significantly outperform the standard oblivious sampling method, which is widely used in the recent literature. We show that the relative error of the randomized approximations can be tightly characterized in terms of the spectrum of the data matrix and Gaussian width of the dual tangent cone at optimum. We develop lower bounds for both optimization and statistical error measures based on concentration of measure and Fano's inequality. We then present the consequences of our theory with data matrices of varying spectral decay profiles. Experimental results show that the proposed approach enables significant speed ups in a wide variety of machine learning and optimization problems including logistic regression, kernel classification with random convolution layers and shallow neural networks with rectified linear units.