Dynamics of Learning with Restricted Training Sets I: General Theory

TL;DR

This paper develops a theoretical framework using dynamical replica theory to analyze the learning dynamics of single-layer neural networks trained on restricted datasets, revealing spin-glass behavior and predicting performance metrics.

Contribution

It introduces a novel application of dynamical replica theory to model the learning dynamics with limited training data in neural networks.

Findings

Predicts evolution of training and generalization errors

Shows spin-glass nature of learning dynamics with restricted datasets

Extends formalism to finite training set sizes

Abstract

We study the dynamics of supervised learning in layered neural networks, in the regime where the size $p$ of the training set is proportional to the number $N$ of inputs. Here the local fields are no longer described by Gaussian probability distributions and the learning dynamics is of a spin-glass nature, with the composition of the training set playing the role of quenched disorder. We show how dynamical replica theory can be used to predict the evolution of macroscopic observables, including the two relevant performance measures (training error and generalization error), incorporating the old formalism developed for complete training sets in the limit $\alpha=p/N\to\infty$ as a special case. For simplicity we restrict ourselves in this paper to single-layer networks and realizable tasks.