Extracting Global Dynamics of Loss Landscape in Deep Learning Models

TL;DR

This paper introduces DOODL3, a toolkit that models neural network training as a dynamical system to analyze and visualize the global loss landscape dynamics, aiding in understanding and improving training stability.

Contribution

It presents a novel empirical framework that captures the global dynamics of deep learning loss landscapes using topological and geometric analysis.

Findings

Provides a global view of training trajectories in the loss landscape

Identifies unstable regions and elongated training states

Guides neural network training based on landscape dynamics

Abstract

Deep learning models evolve through training to learn the manifold in which the data exists to satisfy an objective. It is well known that evolution leads to different final states which produce inconsistent predictions of the same test data points. This calls for techniques to be able to empirically quantify the difference in the trajectories and highlight problematic regions. While much focus is placed on discovering what models learn, the question of how a model learns is less studied beyond theoretical landscape characterizations and local geometric approximations near optimal conditions. Here, we present a toolkit for the Dynamical Organization Of Deep Learning Loss Landscapes, or DOODL3. DOODL3 formulates the training of neural networks as a dynamical system, analyzes the learning process, and presents an interpretable global view of trajectories in the loss landscape. Our approach uses the coarseness of topology to capture the granularity of geometry to mitigate against states of instability or elongated training. Overall, our analysis presents an empirical framework to extract the global dynamics of a model and to use that information to guide the training of neural networks.