On-Line Learning with Restricted Training Sets: An Exactly Solvable Case

TL;DR

This paper provides an exact solution for the dynamics of on-line Hebbian learning in large perceptrons with training sets proportional to input size, serving as a benchmark for more complex models.

Contribution

It offers an exact analytical solution for Hebbian learning dynamics with restricted training sets, a case previously unsolved, aiding the development of advanced learning theories.

Findings

Exact solution for noiseless and noisy teachers

Benchmark for testing more general learning theories

Insights into learning dynamics with limited training data

Abstract

We solve the dynamics of on-line Hebbian learning in large perceptrons exactly, for the regime where the size of the training set scales linearly with the number of inputs. We consider both noiseless and noisy teachers. Our calculation cannot be extended to non-Hebbian rules, but the solution provides a convenient and welcome benchmark with which to test more general and advanced theories for solving the dynamics of learning with restricted training sets.