First-order variational analysis of non-amenable composite functions

TL;DR

This paper develops a first-order variational analysis framework for non-convex, non-differentiable functions using directional derivatives, enabling new algorithms for constrained optimization with convergence guarantees.

Contribution

It introduces exact chain and sum rules for non-amenable functions via subderivatives, facilitating auto-differentiation and first-order algorithms for complex optimization problems.

Findings

Established chain and sum rules for non-regular functions.

Proposed a first-order algorithm with $O( ext{epsilon}^{-2})$ convergence rate.

Identified semi-differentiability of distance functions for constraint sets.

Abstract

This paper is devoted to studying the first-order variational analysis of non-convex and non-differentiable functions that may not be subdifferentially regular. To achieve this goal, we entirely rely on two concepts of directional derivatives known as subderivative and semi-derivative. We establish the exact chain and sum rules for this class of functions via these directional derivatives. These calculus rules provide an implementable auto-differentiation process such as back-propagation in composite functions. The latter calculus rules can be used to identify the directional stationary points defined by the subderivative. We show that the distance function of a geometrically derivable constraint set is semi-differentiable, which opens the door for designing first-order algorithms for non-Clarke regular constrained optimization problems. We propose a first-order algorithm to find a directional stationary point of non-Clarke regular and perhaps non-Lipschitz functions. We introduce a descent property under which we establish the non-asymptotic convergence of our method with rate $O(\varepsilon^{-2})$, akin to gradient descent for smooth minimization. We show that the latter descent property holds for free in some interesting non-amenable composite functions, in particular, it holds for the Moreau envelope of any bounded-below function.