Generalized Damped Newton Algorithms in Nonsmooth Optimization via Second-Order Subdifferentials

TL;DR

This paper introduces new globally convergent damped Newton algorithms for nonsmooth optimization, utilizing second-order subdifferentials and variational analysis, with applications to machine learning problems like Lasso.

Contribution

The paper develops novel globally convergent damped Newton algorithms for nonsmooth optimization using second-order subdifferentials, extending classical methods to nonsmooth and structured problems.

Findings

Algorithms demonstrate superlinear convergence in theory.

Numerical experiments show competitive performance on Lasso problems.

Methods outperform some existing first- and second-order algorithms.

Abstract

The paper proposes and develops new globally convergent algorithms of the generalized damped Newton type for solving important classes of nonsmooth optimization problems. These algorithms are based on the theory and calculations of second-order subdifferentials of nonsmooth functions with employing the machinery of second-order variational analysis and generalized differentiation. First we develop a globally superlinearly convergent damped Newton-type algorithm for the class of continuously differentiable functions with Lipschitzian gradients, which are nonsmooth of second order. Then we design such a globally convergent algorithm to solve a structured class of nonsmooth quadratic composite problems with extended-real-valued cost functions, which typically arise in machine learning and statistics. Finally, we present the results of numerical experiments and compare the performance of our main algorithm applied to an important class of Lasso problems with those achieved by other first-order and second-order optimization algorithms.