Density of states in neural networks: an in-depth exploration of learning in parameter space

TL;DR

This paper introduces a novel computational approach using Wang-Landau sampling to analyze the density of states in neural network weight space, revealing how data structure influences the loss landscape across various network architectures.

Contribution

It presents a new, efficient method to explore the entire loss landscape of neural networks, bridging machine learning and soft matter physics insights.

Findings

Density of states varies with data structure and network architecture.

The method enables comprehensive analysis of loss landscape geometry.

Results show how data features influence network configuration space.

Abstract

Learning in neural networks critically hinges on the intricate geometry of the loss landscape associated with a given task. Traditionally, most research has focused on finding specific weight configurations that minimize the loss. In this work, born from the cross-fertilization of machine learning and theoretical soft matter physics, we introduce a novel, computationally efficient approach to examine the weight space across all loss values. Employing the Wang-Landau enhanced sampling algorithm, we explore the neural network density of states - the number of network parameter configurations that produce a given loss value - and analyze how it depends on specific features of the training set. Using both real-world and synthetic data, we quantitatively elucidate the relation between data structure and network density of states across different sizes and depths of binary-state networks.