Non-Convex Self-Concordant Functions: Practical Algorithms and Complexity Analysis

TL;DR

This paper extends self-concordance to non-convex functions, proposing second-order algorithms with convergence guarantees and demonstrating their efficiency in large-scale and neural network optimization.

Contribution

It introduces weakly and F-based self-concordant function classes and develops algorithms with global convergence guarantees for non-convex optimization.

Findings

Algorithms achieve $O(\epsilon^{-2})$ convergence to first-order stationary points.

Adaptive method attains second-order stationary points with negative curvature detection.

Experimental results show improved robustness and efficiency over existing methods.

Abstract

We extend the standard notion of self-concordance to non-convex optimization and develop a family of second-order algorithms with global convergence guarantees. In particular, two function classes -- \textit{weakly self-concordant} functions and \textit{$F$-based self-concordant} functions -- generalize the self-concordant framework beyond convexity, without assuming the Lipschitz continuity of the gradient or Hessian. For these function classes, we propose a regularized Newton method and an adaptive regularization method that achieve an $\epsilon$-approximate first-order stationary point in $O(\epsilon^{-2})$ iterations. Equipped with an oracle capable of detecting negative curvature, the adaptive algorithm can further attain convergence to an approximate second-order stationary point. Our experimental results demonstrate that the proposed methods offer superior robustness and computational efficiency compared to cubic regularization and trust-region approaches, underscoring the broad potential of self-concordant regularization for large-scale and neural network optimization problems.