Equivariant neural networks and piecewise linear representation theory

TL;DR

This paper explores the theoretical structure of equivariant neural networks using group representation theory, revealing a filtration akin to Fourier series that aids in understanding their behavior.

Contribution

It introduces a novel theoretical framework that decomposes equivariant neural network layers into simple representations and generalizes Fourier series concepts.

Findings

Decomposition of network layers into simple representations.

ReLU induces piecewise linear equivariant maps.

Filtration of networks generalizing Fourier series.

Abstract

Equivariant neural networks are neural networks with symmetry. Motivated by the theory of group representations, we decompose the layers of an equivariant neural network into simple representations. The nonlinear activation functions lead to interesting nonlinear equivariant maps between simple representations. For example, the rectified linear unit (ReLU) gives rise to piecewise linear maps. We show that these considerations lead to a filtration of equivariant neural networks, generalizing Fourier series. This observation might provide a useful tool for interpreting equivariant neural networks.