The Representation Theory of Neural Networks

TL;DR

This paper introduces a novel algebraic framework using quiver representation theory to model neural networks, providing exact mathematical descriptions and insights into their structure and data processing capabilities.

Contribution

It establishes that neural networks can be precisely represented as quiver representations, connecting neural network concepts with algebraic structures for the first time.

Findings

Neural networks are equivalent to quiver representations with activation functions.

Network quivers can model common neural network components like convolution and residual connections.

Neural networks map data to moduli spaces via quiver representations, revealing their algebraic structure.

Abstract

In this work, we show that neural networks can be represented via the mathematical theory of quiver representations. More specifically, we prove that a neural network is a quiver representation with activation functions, a mathematical object that we represent using a network quiver. Also, we show that network quivers gently adapt to common neural network concepts such as fully-connected layers, convolution operations, residual connections, batch normalization, pooling operations and even randomly wired neural networks. We show that this mathematical representation is by no means an approximation of what neural networks are as it exactly matches reality. This interpretation is algebraic and can be studied with algebraic methods. We also provide a quiver representation model to understand how a neural network creates representations from the data. We show that a neural network saves the data as quiver representations, and maps it to a geometrical space called the moduli space, which is given in terms of the underlying oriented graph of the network, i.e., its quiver. This results as a consequence of our defined objects and of understanding how the neural network computes a prediction in a combinatorial and algebraic way. Overall, representing neural networks through the quiver representation theory leads to 9 consequences and 4 inquiries for future research that we believe are of great interest to better understand what neural networks are and how they work.