Deep Learning is Singular, and That's Good

TL;DR

This paper discusses the significance of singular models in deep learning, highlighting how their unique properties challenge traditional inference methods and proposing singular learning theory as a promising framework for understanding neural networks.

Contribution

It introduces the relevance of singular learning theory to deep learning, combining theoretical insights and experiments to motivate its application in practice.

Findings

Neural networks are singular models with complex parameter spaces.

Classical inference methods are unsuitable for singular models.

Singular learning theory offers a promising framework for deep learning understanding.

Abstract

In singular models, the optimal set of parameters forms an analytic set with singularities and classical statistical inference cannot be applied to such models. This is significant for deep learning as neural networks are singular and thus "dividing" by the determinant of the Hessian or employing the Laplace approximation are not appropriate. Despite its potential for addressing fundamental issues in deep learning, singular learning theory appears to have made little inroads into the developing canon of deep learning theory. Via a mix of theory and experiment, we present an invitation to singular learning theory as a vehicle for understanding deep learning and suggest important future work to make singular learning theory directly applicable to how deep learning is performed in practice.