The energy landscape of a simple neural network

TL;DR

This paper investigates the energy landscape of a simple neural network, revealing that the effective complexity is much lower than the number of parameters, which aids in understanding generalization.

Contribution

It provides new insights into the energy landscape of neural networks and demonstrates the beneficial role of implicit regularization in model generalization.

Findings

Empirical complexity is much less than naive parameter count

Implicit regularization improves generalization

Energy landscape analysis reveals simpler effective models

Abstract

We explore the energy landscape of a simple neural network. In particular, we expand upon previous work demonstrating that the empirical complexity of fitted neural networks is vastly less than a naive parameter count would suggest and that this implicit regularization is actually beneficial for generalization from fitted models.