Optimal Sketching Bounds for Exp-concave Stochastic Minimization

TL;DR

This paper establishes optimal bounds for exp-concave stochastic minimization, linking statistical efficiency and computational complexity to effective dimension and sketching techniques, with implications for stability and ridge leverage scores.

Contribution

It introduces the first optimal bounds for exp-concave minimization based on effective dimension and connects stability analysis to sketching methods.

Findings

Effective dimension can be significantly smaller than ambient dimension for common eigendecay patterns.

A novel relationship between stability of empirical risk minimization and ridge leverage scores is identified.

A fast sketch-to-precondition algorithm for exp-concave empirical risk minimization is developed.

Abstract

We derive optimal statistical and computational complexity bounds for exp-concave stochastic minimization in terms of the effective dimension. For common eigendecay patterns of the population covariance matrix, this quantity is significantly smaller than the ambient dimension. Our results reveal interesting connections to sketching results in numerical linear algebra. In particular, our statistical analysis highlights a novel and natural relationship between algorithmic stability of empirical risk minimization and ridge leverage scores, which play significant role in sketching-based methods. Our main computational result is a fast implementation of a sketch-to-precondition approach in the context of exp-concave empirical risk minimization.