Compositional Sparsity as an Inductive Bias for Neural Architecture Design

TL;DR

This paper introduces Homological Neural Networks (HNNs), which leverage compositional sparsity as an inductive bias to create interpretable, sparse neural architectures that outperform dense models on various tasks.

Contribution

The paper formalizes a novel pipeline combining Information Filtering Networks with HNNs, resulting in sparser, more interpretable neural networks that maintain or improve performance.

Findings

HNNs are significantly sparser than standard DNNs.

HNNs recover underlying compositional structures in synthetic tasks.

HNNs outperform or match dense models on real-world datasets with fewer parameters.

Abstract

Identifying the structural priors that enable Deep Neural Networks (DNNs) to overcome the curse of dimensionality is a fundamental challenge in machine learning theory. Existing literature suggests that effective high-dimensional learning is driven by compositional sparsity, where target functions decompose into constituents supported on low-dimensional variable subsets. To investigate this hypothesis, we combine Information Filtering Networks (IFNs), which extract sparse dependency structures via constrained information maximisation, with Homological Neural Networks (HNNs), which map the inferred topology into fixed-wiring sparse neural graphs. We formalise the design principles underlying this construction and present an interpretable pipeline in which abstraction emerges through hierarchical composition. HNNs are orders of magnitude sparser than standard DNNs and require only minimal hyperparameter tuning. On synthetic tasks with known sparse hierarchies, HNNs recover the underlying compositional structure and remain stable in regimes where dense alternatives degrade as dimensionality increases. Across a broad suite of real-world datasets, HNNs consistently match or outperform dense baselines while using far fewer parameters, exhibiting lower variance and showing reduced sensitivity to hyperparameters.