Unified Description of Learning Dynamics in the Soft Committee Machine from Finite to Ultra-Wide Regimes

TL;DR

This paper provides a unified theoretical framework describing the learning dynamics of the soft committee machine with ReLU activation across different network widths, revealing phase transitions and asymptotic error scaling.

Contribution

It introduces a reduced set of macroscopic parameters that unify the description from narrow to ultra-wide networks, extending understanding of SCM learning behavior.

Findings

Identifies a phase transition at a critical dataset size for small hidden-to-input unit ratios.

Shows the transition disappears for finite ratios, with smooth error decrease.

Demonstrates asymptotic error scales as 1/alpha, independent of network width.

Abstract

We study the learning dynamics of the soft committee machine (SCM) with Rectified Linear Unit (ReLU) activation using a statistical-mechanics approach within the annealed approximation. The SCM consists of a student network with $N$ input units and $K$ hidden units trained to reproduce the output of a teacher network with $M$ hidden units. We introduce a reduced set of macroscopic order parameters that yields a unified description valid from the conventional regime $K \ll N$ to the ultra-wide limit $K \ge N$. The control parameter $\alpha$, proportional to the ratio of training samples to adjustable weights, serves as an effective measure of dataset size. For small $\gamma = M/N$, we recover a continuous phase transition at $\alpha_{c} \approx 2\pi$ from an unspecialized, permutation-symmetric state to a specialized state in which student units align with the teacher. For finite $\gamma$, the transition disappears and the generalization error decreases smoothly with dataset size, reaching a low plateau when $\gamma=1$. In the asymptotic limit $\alpha \to \infty$, the error scales as $\varepsilon_{g} \propto 1/\alpha$, independent of $\gamma$ and $K$. The results highlight the central role of network dimensions in SCM learning and provide a framework extendable to other activations and quenched analyses.