Spectral Higher-Order Neural Networks

TL;DR

Spectral Higher-Order Neural Networks (SHONNs) introduce a spectral reformulation to incorporate higher-order interactions into general feedforward neural networks, addressing stability and scaling issues.

Contribution

The paper proposes SHONNs, a novel spectral approach enabling higher-order interactions in standard neural networks without relying on explicit hypergraph structures.

Findings

SHONNs effectively model higher-order interactions in neural networks.

Spectral reformulation improves stability and parameter scaling.

The approach generalizes beyond graph neural network limitations.

Abstract

Neural networks are fundamental tools of modern machine learning. The standard paradigm assumes binary interactions (across feedforward linear passes) between inter-tangled units, organized in sequential layers. Generalized architectures have been also designed that move beyond pairwise interactions, so as to account for higher-order couplings among computing neurons. Higher-order networks are however usually deployed as augmented graph neural networks (GNNs), and, as such, prove solely advantageous in contexts where the input exhibits an explicit hypergraph structure. Here, we present Spectral Higher-Order Neural Networks (SHONNs), a new algorithmic strategy to incorporate higher-order interactions in general-purpose, feedforward, network structures. SHONNs leverages a reformulation of the model in terms of spectral attributes. This allows to mitigate the common stability and parameter scaling problems that come along weighted, higher-order, forward propagations.