Convergence of high-index saddle dynamics for degenerate saddle points on critical manifolds

TL;DR

This paper provides a rigorous analysis of the high-index saddle dynamics (HiSD) method for degenerate saddle points on critical manifolds, proving convergence and explaining gradient alignment, with validation on neural network loss landscapes.

Contribution

It extends HiSD analysis to degenerate saddle points using Morse-Bott functions, establishing convergence and gradient alignment properties.

Findings

Proves local convergence of continuous HiSD for degenerate saddle points.

Establishes linear convergence rate of discrete HiSD algorithm.

Demonstrates rapid convergence of momentum-accelerated HiSD variants on neural networks.

Abstract

The high-index saddle dynamics (HiSD) method provides a powerful framework for finding saddle points and constructing solution landscapes. While originally derived for nondegenerate critical points, HiSD has demonstrated empirical success in degenerate cases, where the Hessian matrix exhibits zero eigenvalues. However, the mathematical and numerical analysis of HiSD for degenerate saddle points remains unexplored. In this paper, utilizing Morse-Bott functions, we present a rigorous analysis of HiSD for computing degenerate saddle points on a critical manifold. We prove the local convergence of the continuous HiSD and establish the linear convergence rate of the discrete HiSD algorithm. Furthermore, we provide a theoretical explanation for the gradient alignment tendency, revealing that the gradient direction asymptotically aligns with a specific Hessian eigenvector. Our analysis also elucidates the flexibility in selecting the index for HiSD in the context of degenerate saddle points. We validate our analytical results through numerical experiments on neural-network loss landscapes and demonstrate that momentum-accelerated variants of HiSD achieve rapid convergence to degenerate saddle points.