Gradient extremals, talwegs, valleys, and directional alignment for generic gradient descent

TL;DR

This paper explores the geometric structure of gradient descent paths, focusing on gradient extremals, valleys, and talwegs, and analyzes how trajectories align with these structures under various conditions.

Contribution

It introduces a geometric framework connecting gradient extremals, valleys, and talwegs, and studies their influence on the alignment and concentration of gradient flow trajectories.

Findings

Gradient flow trajectories align with tangent spaces of gradient extremals and talwegs.

Alignment rates depend on spectral gaps or Hessian eigenvalues under non-resonance.

Large-time behavior shows concentration of initial condition images inside valleys and around talwegs.

Abstract

Gradient extremals are loci along which the gradient is an eigenvector of the Hessian. These objects provide a natural geometric framework connecting several notions, notably valleys and talwegs, which we analyze from a variational viewpoint in the generic case. We then show that trajectories of the gradient flow and of its discrete counterpart exhibit directional alignment with the tangent spaces to gradient extremals, and generically to the talweg. Under non-resonance assumptions, and in contrast with the quadratic case, alignment rates are governed either by the first spectral gap or by the smallest eigenvalue of the Hessian at the limit point. Nonlinearities and the step length may both distort these rates in a complex manner. We further prove a volume concentration phenomenon emphasizing the structuring role of gradient extremals: for large times, the images of sets of initial conditions concentrate inside valleys and asymptotically around talwegs.