Critical Points of Degenerate Metrics on Algebraic Varieties: A Tale of Overparametrization

TL;DR

This paper explores the critical points of degenerate quadratic optimization problems on algebraic varieties, linking algebraic geometry with machine learning, especially in overparametrized models, and extends existing mathematical tools to this setting.

Contribution

It introduces a method to relate degenerate optimization problems to nondegenerate ones via projection, and analyzes the role of ramification loci in highly-degenerate regimes.

Findings

Established a connection between degenerate and nondegenerate problems through projection.

Provided tools for counting critical points on projective varieties.

Discussed applications to deep learning models.

Abstract

We study the critical points over an algebraic variety of an optimization problem defined by a quadratic objective that is degenerate. This scenario arises in machine learning when the dataset size is small with respect to the model, and is typically referred to as overparametrization. Our main result relates the degenerate optimization problem to a nondegenerate one via a projection. In the highly-degenerate regime, we find that a central role is played by the ramification locus of the projection. Additionally, we provide tools for counting the number of critical points over projective varieties, and discuss specific cases arising from deep learning. Our work bridges tools from algebraic geometry with ideas from machine learning, and it extends the line of literature around the Euclidean distance degree to the degenerate setting.