SAD Neural Networks: Divergent Gradient Flows and Asymptotic Optimality via o-minimal Structures

TL;DR

This paper analyzes the behavior of gradient flows in neural networks with common activation functions, showing conditions for convergence or divergence, and establishing asymptotic optimality using o-minimal structures, supported by numerical experiments.

Contribution

It introduces a rigorous geometric framework using o-minimal structures to analyze gradient flow dynamics and proves divergence or convergence properties in neural network training.

Findings

Gradient flows either converge to critical points or diverge to infinity.

A threshold exists below which the loss converges to the optimal value.

For large architectures and data, the optimal loss is asymptotically zero.

Abstract

We study gradient flows for loss landscapes of fully connected feedforward neural networks with commonly used continuously differentiable activation functions such as the logistic, hyperbolic tangent, softplus or GELU function. We prove that the gradient flow either converges to a critical point or diverges to infinity while the loss converges to an asymptotic critical value. Moreover, we prove the existence of a threshold $\varepsilon>0$ such that the loss value of any gradient flow initialized at most $\varepsilon$ above the optimal level converges to it. For polynomial target functions and sufficiently big architecture and data set, we prove that the optimal loss value is zero and can only be realized asymptotically. From this setting, we deduce our main result that any gradient flow with sufficiently good initialization diverges to infinity. Our proof heavily relies on the geometry of o-minimal structures. We confirm these theoretical findings with numerical experiments and extend our investigation to more realistic scenarios, where we observe an analogous behavior.