Dynamical and Stationary Properties of On-line Learning from Finite Training Sets

TL;DR

This paper analyzes the dynamics and stationary behavior of online learning from finite datasets using the cavity method, providing analytical solutions and insights into errors, learning rates, and comparisons with batch learning.

Contribution

It introduces a cavity method-based analytical framework for understanding online learning dynamics from finite datasets, including solutions for errors and critical learning rates.

Findings

Analytical equations for macroscopic parameters in online learning.

Predictions of training and generalization errors match simulations.

Identification of the critical learning rate and comparison with batch learning.

Abstract

The dynamical and stationary properties of on-line learning from finite training sets are analysed using the cavity method. For large input dimensions, we derive equations for the macroscopic parameters, namely, the student-teacher correlation, the student-student autocorrelation and the learning force uctuation. This enables us to provide analytical solutions to Adaline learning as a benchmark. Theoretical predictions of training errors in transient and stationary states are obtained by a Monte Carlo sampling procedure. Generalization and training errors are found to agree with simulations. The physical origin of the critical learning rate is presented. Comparison with batch learning is discussed throughout the paper.