# Learning time-scales in two-layers neural networks

## TL;DR

This paper investigates the multi-scale and intermittent learning dynamics of two-layer neural networks in high-dimensional settings, revealing how different phases of training occur on distinct time scales.

## Contribution

It provides a new theoretical framework for understanding the separation of time scales and intermittency in neural network training dynamics.

## Key findings

- Identification of multiple learning time scales.
- Demonstration of intermittency in gradient flow.
- Validation through numerical simulations.

## Abstract

Gradient-based learning in multi-layer neural networks displays a number of striking features. In particular, the decrease rate of empirical risk is non-monotone even after averaging over large batches. Long plateaus in which one observes barely any progress alternate with intervals of rapid decrease. These successive phases of learning often take place on very different time scales. Finally, models learnt in an early phase are typically `simpler' or `easier to learn' although in a way that is difficult to formalize. Although theoretical explanations of these phenomena have been put forward, each of them captures at best certain specific regimes. In this paper, we study the gradient flow dynamics of a wide two-layer neural network in high-dimension, when data are distributed according to a single-index model (i.e., the target function depends on a one-dimensional projection of the covariates). Based on a mixture of new rigorous results, non-rigorous mathematical derivations, and numerical simulations, we propose a scenario for the learning dynamics in this setting. In particular, the proposed evolution exhibits separation of timescales and intermittency. These behaviors arise naturally because the population gradient flow can be recast as a singularly perturbed dynamical system.