Black holes and the loss landscape in machine learning

TL;DR

This paper draws an analogy between black hole entropy landscapes in string theory and loss landscapes in machine learning, suggesting insights into local minima and mode connectivity, with potential implications for optimization.

Contribution

It introduces a novel perspective linking black hole microstate counting to the structure of loss landscapes in neural networks, providing a theoretical framework for understanding minima.

Findings

Black hole entropy landscapes resemble neural network loss landscapes.

Exact counts of minima are known from string theory dualities.

Stochastic Gradient Descent can find many minima in these landscapes.

Abstract

Understanding the loss landscape is an important problem in machine learning. One key feature of the loss function, common to many neural network architectures, is the presence of exponentially many low lying local minima. Physical systems with similar energy landscapes may provide useful insights. In this work, we point out that black holes naturally give rise to such landscapes, owing to the existence of black hole entropy. For definiteness, we consider 1/8 BPS black holes in $\mathcal{N} = 8$ string theory. These provide an infinite family of potential landscapes arising in the microscopic descriptions of corresponding black holes. The counting of minima amounts to black hole microstate counting. Moreover, the exact numbers of the minima for these landscapes are a priori known from dualities in string theory. Some of the minima are connected by paths of low loss values, resembling mode connectivity. We estimate the number of runs needed to find all the solutions. Initial explorations suggest that Stochastic Gradient Descent can find a significant fraction of the minima.