Uniform Convergence of Gradients for Non-Convex Learning and Optimization

TL;DR

This paper studies how gradients in non-convex learning tasks converge to their true values, introduces vector-valued Rademacher complexities for analysis, and explores implications for optimization and convergence rates.

Contribution

It introduces a new analysis framework using vector-valued Rademacher complexities for uniform convergence of gradients in non-convex learning, and applies it to optimize sample complexity in high-dimensional settings.

Findings

Gradient convergence rates depend on distributional assumptions.

Dimension-independent rates are achievable under certain conditions.

In non-smooth models, worst-case dimension-independent convergence is impossible, but positive results exist under specific assumptions.

Abstract

We investigate 1) the rate at which refined properties of the empirical risk---in particular, gradients---converge to their population counterparts in standard non-convex learning tasks, and 2) the consequences of this convergence for optimization. Our analysis follows the tradition of norm-based capacity control. We propose vector-valued Rademacher complexities as a simple, composable, and user-friendly tool to derive dimension-free uniform convergence bounds for gradients in non-convex learning problems. As an application of our techniques, we give a new analysis of batch gradient descent methods for non-convex generalized linear models and non-convex robust regression, showing how to use any algorithm that finds approximate stationary points to obtain optimal sample complexity, even when dimension is high or possibly infinite and multiple passes over the dataset are allowed. Moving to non-smooth models we show----in contrast to the smooth case---that even for a single ReLU it is not possible to obtain dimension-independent convergence rates for gradients in the worst case. On the positive side, it is still possible to obtain dimension-independent rates under a new type of distributional assumption.