Barcodes as Summary of Loss Function Topology

TL;DR

This paper applies topological data analysis, specifically barcodes of Morse complexes, to study the topology of neural network loss surfaces, revealing insights about minima distribution and effects of network size.

Contribution

It introduces a novel application of Morse complex barcodes to analyze neural network loss landscapes and provides experimental validation on benchmark functions and small networks.

Findings

Barcodes of local minima are concentrated in the lower loss range.

Increasing network depth and width lowers the loss of local minima.

Loss surface topology insights have implications for learning and generalization.

Abstract

We propose to study neural networks' loss surfaces by methods of topological data analysis. We suggest to apply barcodes of Morse complexes to explore topology of loss surfaces. An algorithm for calculations of the loss function's barcodes of local minima is described. We have conducted experiments for calculating barcodes of local minima for benchmark functions and for loss surfaces of small neural networks. Our experiments confirm our two principal observations for neural networks' loss surfaces. First, the barcodes of local minima are located in a small lower part of the range of values of neural networks' loss function. Secondly, increase of the neural network's depth and width lowers the barcodes of local minima. This has some natural implications for the neural network's learning and for its generalization properties.