Consistent Approximations in Composite Optimization

TL;DR

This paper develops a framework for consistent approximations in complex composite optimization problems, ensuring solution stability and convergence across various non-convex and non-smooth scenarios.

Contribution

It introduces conditions for well-behaved approximations in broad classes of composite problems, including non-convex and non-smooth cases, with practical algorithmic and convergence insights.

Findings

Framework applies to stochastic optimization, machine learning, and robust optimization.

Provides convergence rates for approximation methods.

Demonstrates enhanced proximal method for composite problems.

Abstract

Approximations of optimization problems arise in computational procedures and sensitivity analysis. The resulting effect on solutions can be significant, with even small approximations of components of a problem translating into large errors in the solutions. We specify conditions under which approximations are well behaved in the sense of minimizers, stationary points, and level-sets and this leads to a framework of consistent approximations. The framework is developed for a broad class of composite problems, which are neither convex nor smooth. We demonstrate the framework using examples from stochastic optimization, neural-network based machine learning, distributionally robust optimization, penalty and augmented Lagrangian methods, interior-point methods, homotopy methods, smoothing methods, extended nonlinear programming, difference-of-convex programming, and multi-objective optimization. An enhanced proximal method illustrates the algorithmic possibilities. A quantitative analysis supplements the development by furnishing rates of convergence.