Absence of Closed-Form Descriptions for Gradient Flow in Two-Layer Narrow Networks

TL;DR

This paper proves that the training dynamics of two-layer narrow neural networks cannot be expressed in closed-form solutions due to their non-integrable nature, highlighting the need for numerical approaches.

Contribution

It demonstrates the non-integrability of gradient flow in two-layer narrow networks using differential Galois theory, establishing the absence of closed-form solutions.

Findings

Gradient flow dynamics are non-integrable.

No closed-form solutions exist for these dynamics.

Numerical methods are necessary for training analysis.

Abstract

In the field of machine learning, comprehending the intricate training dynamics of neural networks poses a significant challenge. This paper explores the training dynamics of neural networks, particularly whether these dynamics can be expressed in a general closed-form solution. We demonstrate that the dynamics of the gradient flow in two-layer narrow networks is not an integrable system. Integrable systems are characterized by trajectories confined to submanifolds defined by level sets of first integrals (invariants), facilitating predictable and reducible dynamics. In contrast, non-integrable systems exhibit complex behaviors that are difficult to predict. To establish the non-integrability, we employ differential Galois theory, which focuses on the solvability of linear differential equations. We demonstrate that under mild conditions, the identity component of the differential Galois group of the variational equations of the gradient flow is non-solvable. This result confirms the system's non-integrability and implies that the training dynamics cannot be represented by Liouvillian functions, precluding a closed-form solution for describing these dynamics. Our findings highlight the necessity of employing numerical methods to tackle optimization problems within neural networks. The results contribute to a deeper understanding of neural network training dynamics and their implications for machine learning optimization strategies.