Basic Inequalities for First-Order Optimization with Applications to Statistical Risk Analysis

TL;DR

This paper develops basic inequalities for first-order optimization algorithms, providing a unified framework that links optimization steps to statistical risk, and applies it to analyze various algorithms and models.

Contribution

It introduces a versatile inequality framework that connects optimization dynamics with statistical regularization, offering new insights and refinements for multiple algorithms.

Findings

Refined analysis of gradient descent dynamics.

New results for mirror descent and exponentiated gradient.

Experimental validation on generalized linear models.

Abstract

We introduce \textit{basic inequalities} for first-order iterative optimization algorithms, forming a simple and versatile framework that connects implicit and explicit regularization. While related inequalities appear in the literature, we isolate and highlight a specific form and develop it as a well-rounded tool for statistical analysis. Let $f$ denote the objective function to be optimized. Given a first-order iterative algorithm initialized at $\theta_0$ with current iterate $\theta_T$, the basic inequality upper bounds $f(\theta_T)-f(z)$ for any reference point $z$ in terms of the accumulated step sizes and the distances between $\theta_0$, $\theta_T$, and $z$. The bound translates the number of iterations into an effective regularization coefficient in the loss function. We demonstrate this framework through analyses of training dynamics and prediction risk bounds. In addition to revisiting and refining known results on gradient descent, we provide new results for mirror descent with Bregman divergence projection, for generalized linear models trained by gradient descent and exponentiated gradient descent, and for randomized predictors. We illustrate and supplement these theoretical findings with experiments on generalized linear models.