Randomized Sketches of Convex Programs with Sharp Guarantees

TL;DR

This paper analyzes how random projections can efficiently approximate convex programs, providing sharp guarantees on the approximation quality while reducing computational and memory costs, with applications in privacy and compressed sensing.

Contribution

It offers a theoretical analysis of random projection methods for convex optimization, establishing bounds based on the geometry of the constraint set and demonstrating broad applicability.

Findings

Approximation ratio bounded by the geometry of the constraint set.

Data can be projected to the statistical dimension of the tangent cone.

Applicable to various problems including SVMs, low-rank estimation, and compressed sensing.

Abstract

Random projection (RP) is a classical technique for reducing storage and computational costs. We analyze RP-based approximations of convex programs, in which the original optimization problem is approximated by the solution of a lower-dimensional problem. Such dimensionality reduction is essential in computation-limited settings, since the complexity of general convex programming can be quite high (e.g., cubic for quadratic programs, and substantially higher for semidefinite programs). In addition to computational savings, random projection is also useful for reducing memory usage, and has useful properties for privacy-sensitive optimization. We prove that the approximation ratio of this procedure can be bounded in terms of the geometry of constraint set. For a broad class of random projections, including those based on various sub-Gaussian distributions as well as randomized Hadamard and Fourier transforms, the data matrix defining the cost function can be projected down to the statistical dimension of the tangent cone of the constraints at the original solution, which is often substantially smaller than the original dimension. We illustrate consequences of our theory for various cases, including unconstrained and $\ell_1$-constrained least squares, support vector machines, low-rank matrix estimation, and discuss implications on privacy-sensitive optimization and some connections with de-noising and compressed sensing.