Uncovering Critical Sets of Deep Neural Networks via Sample-Independent Critical Lifting

TL;DR

This paper introduces a novel sample-independent critical lifting operator for neural networks, revealing new critical points and saddle points that are not captured by previous methods, thus deepening understanding of neural network criticality.

Contribution

The paper proposes a new sample-independent critical lifting operator and demonstrates its ability to identify previously unrecognized critical points in neural networks.

Findings

Sample-independent lifted critical points exist beyond previous embeddings.

Saddles appear among sample-dependent lifted critical points.

Sample size influences the emergence of critical points and saddles.

Abstract

This paper investigates the sample dependence of critical points for neural networks. We introduce a sample-independent critical lifting operator that associates a parameter of one network with a set of parameters of another, thus defining sample-dependent and sample-independent lifted critical points. We then show by example that previously studied critical embeddings do not capture all sample-independent lifted critical points. Finally, we demonstrate the existence of sample-dependent lifted critical points for sufficiently large sample sizes and prove that saddles appear among them.