Topology and Geometry of the Learning Space of ReLU Networks: Connectivity and Singularities

TL;DR

This paper investigates the structure of the parameter space in ReLU networks, revealing how topology and singularities influence training dynamics, with implications for network design and pruning strategies.

Contribution

It provides a comprehensive characterization of connectedness and singularities in the parameter space of DAG-structured ReLU networks, extending previous theoretical results.

Findings

Parameter space forms an algebraic variety after initialization.

Connectivity depends on bottleneck nodes and balance conditions.

Singularities relate to the network's DAG topology and sub-networks.

Abstract

Understanding the properties of the parameter space in feed-forward ReLU networks is critical for effectively analyzing and guiding training dynamics. After initialization, training under gradient flow decisively restricts the parameter space to an algebraic variety that emerges from the homogeneous nature of the ReLU activation function. In this study, we examine two key challenges associated with feed-forward ReLU networks built on general directed acyclic graph (DAG) architectures: the (dis)connectedness of the parameter space and the existence of singularities within it. We extend previous results by providing a thorough characterization of connectedness, highlighting the roles of bottleneck nodes and balance conditions associated with specific subsets of the network. Our findings clearly demonstrate that singularities are intricately connected to the topology of the underlying DAG and its induced sub-networks. We discuss the reachability of these singularities and establish a principled connection with differentiable pruning. We validate our theory with simple numerical experiments.