Pruning at Initialisation through the lens of Graphon Limit: Convergence, Expressivity, and Generalisation

TL;DR

This paper establishes a theoretical framework connecting pruning at initialization to graphon limits, revealing how different pruning strategies influence network expressivity and generalisation through continuous graphon models.

Contribution

It introduces a graphon-based limit framework for pruning methods, providing new insights into their convergence, expressivity, and generalisation properties.

Findings

Unstructured pruning converges to homogeneous graphons.

Data-driven pruning converges to heterogeneous graphons.

Universal approximation and generalisation bounds derived.

Abstract

Pruning at Initialisation methods discover sparse, trainable subnetworks before training, but their theoretical mechanisms remain elusive. Existing analyses are often limited to finite-width statistics, lacking a rigorous characterisation of the global sparsity patterns that emerge as networks grow large. In this work, we connect discrete pruning heuristics to graph limit theory via graphons, establishing the graphon limit of PaI masks. We introduce a Factorised Saliency Model that encompasses popular pruning criteria and prove that, under regularity conditions, the discrete masks generated by these algorithms converge to deterministic bipartite graphons. This limit framework establishes a novel topological taxonomy for sparse networks: while unstructured methods (e.g., Random, Magnitude) converge to homogeneous graphons representing uniform connectivity, data-driven methods (e.g., SNIP, GraSP) converge to heterogeneous graphons that encode implicit feature selection. Leveraging this continuous characterisation, we derive two fundamental theoretical results: (i) a Universal Approximation Theorem for sparse networks that depends only on the intrinsic dimension of active coordinate subspaces; and (ii) a Graphon-NTK generalisation bound demonstrating how the limit graphon modulates the kernel geometry to align with informative features. Our results transform the study of sparse neural networks from combinatorial graph problems into a rigorous framework of continuous operators, offering a new mechanism for analysing expressivity and generalisation in sparse neural networks.