Geometric Regularization from Overparameterization
Nicholas J. Teague

TL;DR
This paper proposes a geometric regularization conjecture explaining overparameterization effects like double descent through volume contraction in high-dimensional weight distributions, linking geometric properties to model capacity phenomena.
Contribution
It introduces a new geometric regularization conjecture and connects volume contraction in high-dimensional spaces to the double descent phenomenon in neural networks.
Findings
Volume contraction explains implicit regularization.
Data complexity influences double descent thresholds.
Geometric forms suggest universality in model capacity behavior.
Abstract
The volume of the distribution of weight sets associated with a loss value may be the source of implicit regularization from overparameterization due to the phenomenon of contracting volume with increasing dimensions for geometric figures demonstrated by hyperspheres. We introduce the geometric regularization conjecture and extract to an explanation for the double descent phenomenon by considering a similar property resulting from shrinking intrinsic dimensionality of the distribution of potential weight set updates available along training path, where if that distribution retracts across a volume verses dimensionality curve peak when approaching the global minima we could expect geometric regularization to re-emerge. We illustrate how data fidelity representational complexity may influence model capacity double descent interpolation thresholds. The existence of epoch and model capacity…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsMorphological variations and asymmetry · Topological and Geometric Data Analysis · Advanced Vision and Imaging
