Product Interaction: An Algebraic Formalism for Deep Learning Architectures

TL;DR

This paper introduces product interactions, an algebraic framework that unifies various neural network architectures through compositions of multiplication operators, revealing their algebraic structure and symmetry properties.

Contribution

It presents a novel algebraic formalism for neural networks, unifying linear, quadratic, and higher-order interactions within a single framework.

Findings

Convolutional and equivariant networks are linear product interactions.

Attention and Mamba are higher-order product interactions.

Provides a unified algebraic perspective on neural network architectures.

Abstract

In this paper, we introduce product interactions, an algebraic formalism in which neural network layers are constructed from compositions of a multiplication operator defined over suitable algebras. Product interactions provide a principled way to generate and organize algebraic expressions by increasing interaction order. Our central observation is that algebraic expressions in modern neural networks admit a unified construction in terms of linear, quadratic, and higher-order product interactions. Convolutional and equivariant networks arise as symmetry-constrained linear product interactions, while attention and Mamba correspond to higher-order product interactions.