Correlation between entropy and generalizability in a neural network

TL;DR

This paper investigates the relationship between entropy in neural network parameter space and their ability to generalize, using a novel entropy calculation method to shed light on why neural networks perform well despite their complexity.

Contribution

It introduces a method to calculate entropy related to neural network generalizability and demonstrates entropical forces' role in aiding generalization, even in simple models.

Findings

Entropy correlates with test accuracy and training loss.

Entropical forces influence neural network generalizability.

Method applicable to more complex networks in future studies.

Abstract

Although neural networks can solve very complex machine-learning problems, the theoretical reason for their generalizability is still not fully understood. Here we use Wang-Landau Mote Carlo algorithm to calculate the entropy (logarithm of the volume of a part of the parameter space) at a given test accuracy, and a given training loss function value or training accuracy. Our results show that entropical forces help generalizability. Although our study is on a very simple application of neural networks (a spiral dataset and a small, fully-connected neural network), our approach should be useful in explaining the generalizability of more complicated neural networks in future works.