A Novel Unified Parametric Assumption for Nonconvex Optimization

TL;DR

This paper introduces a new unified parametric assumption for nonconvex functions, enabling a general convergence analysis for gradient methods and bridging the gap between theory and practice in nonconvex optimization.

Contribution

The paper proposes a versatile parametric assumption that generalizes many nonconvex function classes, facilitating a unified convergence analysis for gradient-based algorithms.

Findings

The assumption recovers several existing function classes as special cases.

A unified convergence theorem is derived for both deterministic and stochastic methods.

Experiments show the assumption holds in practical optimization trajectories.

Abstract

Nonconvex optimization is central to modern machine learning, but the general framework of nonconvex optimization yields weak convergence guarantees that are too pessimistic compared to practice. On the other hand, while convexity enables efficient optimization, it is of limited applicability to many practical problems. To bridge this gap and better understand the practical success of optimization algorithms in nonconvex settings, we introduce a novel unified parametric assumption. Our assumption is general enough to encompass a broad class of nonconvex functions while also being specific enough to enable the derivation of a unified convergence theorem for gradient-based methods. Notably, by tuning the parameters of our assumption, we demonstrate its versatility in recovering several existing function classes as special cases and in identifying functions amenable to efficient optimization. We derive our convergence theorem for both deterministic and stochastic optimization, and conduct experiments to verify that our assumption can hold practically over optimization trajectories.