Sparse Super-Regular Networks

TL;DR

Sparse Super-Regular Networks (SRNs) leverage super-regularity properties to outperform fully-connected networks in certain tasks, offering structural guarantees, reduced regularization needs, and comparable performance to existing models like X-Nets.

Contribution

This paper introduces SRNs, a novel neural network architecture based on super-regular pairs, providing theoretical guarantees and demonstrating comparable performance to X-Nets without regularization.

Findings

SRNs guarantee edge uniformity and degree properties

SRNs perform similarly to X-Nets in experiments

SRNs eliminate the need for Dropout regularization

Abstract

It has been argued by Thom and Palm that sparsely-connected neural networks (SCNs) show improved performance over fully-connected networks (FCNs). Super-regular networks (SRNs) are neural networks composed of a set of stacked sparse layers of (epsilon, delta)-super-regular pairs, and randomly permuted node order. Using the Blow-up Lemma, we prove that as a result of the individual super-regularity of each pair of layers, SRNs guarantee a number of properties that make them suitable replacements for FCNs for many tasks. These guarantees include edge uniformity across all large-enough subsets, minimum node in- and out-degree, input-output sensitivity, and the ability to embed pre-trained constructs. Indeed, SRNs have the capacity to act like FCNs, and eliminate the need for costly regularization schemes like Dropout. We show that SRNs perform similarly to X-Nets via readily reproducible experiments, and offer far greater guarantees and control over network structure.