Nesterov's accelerated gradient for unbounded convex functions finds the minimum-norm point in the dual space

TL;DR

This paper analyzes the divergence of gradient methods on unbounded convex functions and reveals that Nesterov's accelerated gradient method effectively finds the minimum-norm point in the dual space, even when the primal problem is unbounded.

Contribution

It establishes a connection between gradient methods on unbounded convex functions and dual norm-minimization, demonstrating that Nesterov's method solves both problems at accelerated rates.

Findings

Gradient descent diverges on unbounded convex functions.

Nesterov's accelerated method finds the dual minimum-norm point at a faster rate.

The dual problem provides insights into the divergence behavior of primal methods.

Abstract

We study the behavior of first-order methods applied to a lower-unbounded convex function $f$, i.e., $\inf f = -\infty$. Such a setting has received little attention since the trajectories of gradient descent and Nesterov's accelerated gradient method diverge. In this paper, we establish quantitative convergence results describing their speeds and directions of divergence, with implications for unboundedness judgment. A key idea is a relation to a norm-minimization problem in the dual space: minimize $\|p\|^2/2$ over $p \in \mathrm{dom}f^\ast$, which can be naturally solved via mirror descent by taking the Legendre--Fenchel conjugate $f^\ast$ as the distance-generating function. It then turns out that gradient descent for $f$ coincides with mirror descent for this norm-minimization problem, and thus it simultaneously solves both problems at $\mathcal{O}(k^{-1})$. This result admits acceleration; Nesterov's accelerated gradient method, without any modifications, simultaneously solves the original minimization and the dual norm-minimization problems at $\mathcal{O}(k^{-2})$, providing a quantitative characterization of divergence in unbounded convex optimization.