Phases of learning dynamics in artificial neural networks: with or without mislabeled data

TL;DR

This paper uses a statistical physics framework to analyze the learning dynamics of neural networks trained with SGD, revealing distinct phases in learning with or without mislabeled data and their impact on generalization.

Contribution

It introduces a novel physics-inspired framework to characterize SGD dynamics and identifies phase transitions in learning, especially in the presence of mislabeled data.

Findings

SGD dynamics transition from fast learning to slow exploration without mislabeled data.

With mislabeled data, four distinct learning phases are identified.

Sample loss separation during phase II aids in eliminating mislabeled samples.

Abstract

Despite tremendous success of deep neural network in machine learning, the underlying reason for its superior learning capability remains unclear. Here, we present a framework based on statistical physics to study dynamics of stochastic gradient descent (SGD) that drives learning in neural networks. By using the minibatch gradient ensemble, we construct order parameters to characterize dynamics of weight updates in SGD. Without mislabeled data, we find that the SGD learning dynamics transitions from a fast learning phase to a slow exploration phase, which is associated with large changes in order parameters that characterize the alignment of SGD gradients and their mean amplitude. In the case with randomly mislabeled samples, SGD learning dynamics falls into four distinct phases. The system first finds solutions for the correctly labeled samples in phase I, it then wanders around these solutions in phase II until it finds a direction to learn the mislabeled samples during phase III, after which it finds solutions that satisfy all training samples during phase IV. Correspondingly, the test error decreases during phase I and remains low during phase II; however, it increases during phase III and reaches a high plateau during phase IV. The transitions between different phases can be understood by changes of order parameters that characterize the alignment of mean gradients for the correctly and incorrectly labeled samples and their (relative) strength during learning. We find that individual sample losses for the two datasets are most separated during phase II, which leads to a cleaning process to eliminate mislabeled samples for improving generalization.