Disentangle Sample Size and Initialization Effect on Perfect Generalization for Single-Neuron Target

TL;DR

This paper investigates how initialization scale and sample size affect perfect generalization in overparameterized neural networks, using a single-neuron target as a simplified model to reveal critical thresholds and dynamics.

Contribution

It introduces the concept of the initial imbalance ratio and identifies sample size thresholds that govern target recovery, supported by theoretical proofs and empirical validation.

Findings

Smaller initialization improves generalization.

Identifies optimistic and separation sample size thresholds.

Recovery of the target function depends on sample size and initialization.

Abstract

Overparameterized models like deep neural networks have the intriguing ability to recover target functions with fewer sampled data points than parameters (see arXiv:2307.08921). To gain insights into this phenomenon, we concentrate on a single-neuron target recovery scenario, offering a systematic examination of how initialization and sample size influence the performance of two-layer neural networks. Our experiments reveal that a smaller initialization scale is associated with improved generalization, and we identify a critical quantity called the "initial imbalance ratio" that governs training dynamics and generalization under small initialization, supported by theoretical proofs. Additionally, we empirically delineate two critical thresholds in sample size--termed the "optimistic sample size" and the "separation sample size"--that align with the theoretical frameworks established by (see arXiv:2307.08921 and arXiv:2309.00508). Our results indicate a transition in the model's ability to recover the target function: below the optimistic sample size, recovery is unattainable; at the optimistic sample size, recovery becomes attainable albeit with a set of initialization of zero measure. Upon reaching the separation sample size, the set of initialization that can successfully recover the target function shifts from zero to positive measure. These insights, derived from a simplified context, provide a perspective on the intricate yet decipherable complexities of perfect generalization in overparameterized neural networks.