Algebraic Neural Networks: Stability to Deformations

TL;DR

This paper introduces algebraic neural networks (AlgNNs) that unify various architectures and analyzes their stability to deformations, showing that stable filters have frequency responses with inversely proportional derivatives, leading to enhanced stability.

Contribution

The paper develops a unified algebraic framework for neural networks, analyzes their stability to deformations, and explains why AlgNNs outperform algebraic filters in stability and empirical performance.

Findings

Stable algebraic filters have frequency responses with derivatives inversely proportional to frequency.

AlgNNs are more stable than algebraic filters for a given discriminability level.

The stability property is a deep algebraic characteristic shared by multiple neural network architectures.

Abstract

We study algebraic neural networks (AlgNNs) with commutative algebras which unify diverse architectures such as Euclidean convolutional neural networks, graph neural networks, and group neural networks under the umbrella of algebraic signal processing. An AlgNN is a stacked layered information processing structure where each layer is conformed by an algebra, a vector space and a homomorphism between the algebra and the space of endomorphisms of the vector space. Signals are modeled as elements of the vector space and are processed by convolutional filters that are defined as the images of the elements of the algebra under the action of the homomorphism. We analyze stability of algebraic filters and AlgNNs to deformations of the homomorphism and derive conditions on filters that lead to Lipschitz stable operators. We conclude that stable algebraic filters have frequency responses -- defined as eigenvalue domain representations -- whose derivative is inversely proportional to the frequency -- defined as eigenvalue magnitudes. It follows that for a given level of discriminability, AlgNNs are more stable than algebraic filters, thereby explaining their better empirical performance. This same phenomenon has been proven for Euclidean convolutional neural networks and graph neural networks. Our analysis shows that this is a deep algebraic property shared by a number of architectures.