Combinatorial Regularity for Relatively Perfect Discrete Morse Gradient Vector Fields of ReLU Neural Networks

TL;DR

This paper introduces algorithms to analyze the topological structure of ReLU neural networks by translating their piecewise linear functions into discrete Morse functions, enabling better understanding of critical points and sublevel set topology.

Contribution

It develops a constructive scheme and algorithms to convert ReLU neural network functions into compatible discrete Morse functions on their polyhedral complexes.

Findings

Algorithm for identifying Morse critical points in ReLU networks

Method for constructing discrete Morse pairings on complex cells

New realizability results for sublevel set topology in shallow networks

Abstract

One common function class in machine learning is the class of ReLU neural networks. ReLU neural networks induce a piecewise linear decomposition of their input space called the canonical polyhedral complex. It has previously been established that it is decidable whether a ReLU neural network is piecewise linear Morse. In order to expand computational tools for analyzing the topological properties of ReLU neural networks, and to harness the strengths of discrete Morse theory, we introduce a schematic for translating between a given piecewise linear Morse function (e.g. parameters of a ReLU neural network) on a canonical polyhedral complex and a compatible (``relatively perfect") discrete Morse function on the same complex. Our approach is constructive, producing an algorithm that can be used to determine if a given vertex in a canonical polyhedral complex corresponds to a piecewise linear Morse critical point. Furthermore we provide an algorithm for constructing a consistent discrete Morse pairing on cells in the canonical polyhedral complex which contain this vertex. We additionally provide some new realizability results with respect to sublevel set topology in the case of shallow ReLU neural networks.