Activation Functions for "A Feedforward Unitary Equivariant Neural Network"

TL;DR

This paper generalizes activation functions for a feedforward unitary equivariant neural network, expanding flexibility while preserving equivariance, thus enabling more versatile neural network architectures.

Contribution

It introduces a unified functional form for activation functions that maintains equivariance and broadens design options for such neural networks.

Findings

Unified activation function form preserves equivariance

Enhanced flexibility in neural network architecture

Applicable to a broad class of functions

Abstract

In our previous work [Ma and Chan (2023)], we presented a feedforward unitary equivariant neural network. We proposed three distinct activation functions tailored for this network: a softsign function with a small residue, an identity function, and a Leaky ReLU function. While these functions demonstrated the desired equivariance properties, they limited the neural network's architecture. This short paper generalises these activation functions to a single functional form. This functional form represents a broad class of functions, maintains unitary equivariance, and offers greater flexibility for the design of equivariant neural networks.