Understanding Deep Learning using Topological Dynamical Systems, Index Theory, and Homology

TL;DR

This paper applies advanced topological and dynamical systems theories to analyze deep learning models, revealing their decision surfaces and the dynamical behavior of learned probability densities using algebraic topology and differential equations.

Contribution

It introduces a novel framework combining topological dynamical systems, index theory, and homology to understand neural network decision surfaces and their underlying geometry.

Findings

Neurons correspond to simplexes in a simplicial complex approximation.

Homology groups can be explicitly computed for decision manifolds.

The probability density gradient creates a dynamical system with identifiable sinks and saddles.

Abstract

In this paper we investigate Deep Learning Models using topological dynamical systems, index theory, and computational homology. These mathematical machinery was invented initially by Henri Poincare around 1900 and developed over time to understand shapes and dynamical systems whose structure and behavior is too complicated to solve for analytically but can be understood via global relationships. In particular, we show how individual neurons in a neural network can correspond to simplexes in a simplicial complex manifold approximation to the decision surface learned by the NN, and how these simplexes can be used to compute topological invariants from algebraic topology for the decision manifold with an explicit computation of homology groups by hand in a simple case. We also show how the gradient of the probability density function learned by the NN creates a dynamical system, which can be analyzed by a myriad of topological tools such as Conley Index Theory, Morse Theory, and Stable Manifolds. We solve analytically for associated the differential equation for a trained NN with a single hidden layer of 256 Neurons applied to the MINST digit dataset, and approximately numerically that it a sink and basin of attraction for each of the 10 classes, but the sinks and strong attracting manifolds lie in regions not corresponding to images of actual digits. Index theory implies the existence of saddles. Level sets of the probability functions are 783-dimensional manifolds which can only change topology at critical points of the dynamical system, and these changes in topology can be investigated with Morse Theory.