Equivariance Through Parameter-Sharing

TL;DR

This paper explores how to enforce equivariance in neural networks by leveraging parameter symmetries, proposing schemes that tie parameters to induce specific group symmetries and ensure sensitivity to other permutations.

Contribution

It establishes a theoretical link between network equivariance and parameter symmetries, and introduces parameter-sharing schemes to achieve desired equivariance properties.

Findings

Parameter-sharing schemes induce $ ext{G}$-equivariance in neural networks.

Proposed methods guarantee sensitivity to all permutation groups outside $ ext{G}$ under certain conditions.

The approach provides a systematic way to incorporate symmetry into neural network design.

Abstract

We propose to study equivariance in deep neural networks through parameter symmetries. In particular, given a group $\mathcal{G}$ that acts discretely on the input and output of a standard neural network layer $\phi_{W}: \Re^{M} \to \Re^{N}$, we show that $\phi_{W}$ is equivariant with respect to $\mathcal{G}$-action iff $\mathcal{G}$ explains the symmetries of the network parameters $W$. Inspired by this observation, we then propose two parameter-sharing schemes to induce the desirable symmetry on $W$. Our procedures for tying the parameters achieve $\mathcal{G}$-equivariance and, under some conditions on the action of $\mathcal{G}$, they guarantee sensitivity to all other permutation groups outside $\mathcal{G}$.