Using second-order information in gradient sampling methods for nonsmooth optimization

TL;DR

This paper introduces a new second-order information concept for nonsmooth functions, leading to a descent method that converges in convex cases and shows promising superlinear behavior in experiments.

Contribution

It proposes a novel second-order information framework based on eps-subdifferentials and develops a descent method with convergence guarantees for convex or max-type objectives.

Findings

Convergence proven for convex and max-type functions.

Numerical experiments indicate superlinear convergence.

A practical sampling scheme for the second-order model.

Abstract

In this article, we introduce a novel concept for second-order information of a nonsmooth function inspired by the Goldstein eps-subdifferential. It comprises the coefficients of all existing second-order Taylor expansions in an eps-ball around a given point. Based on this concept, we define a model of the objective as the maximum of these Taylor expansions, and derive a sampling scheme for its approximation in practice. Minimization of this model induces a simple descent method, for which we show convergence for the case where the objective is convex or of max-type. While we do not prove any rate of convergence of this method, numerical experiments suggest superlinear behavior with respect to the number of oracle calls of the objective.