Drawback of Enforcing Equivariance and its Compensation via the Lens of Expressive Power

TL;DR

This paper analyzes how enforcing equivariance in neural networks can limit their expressive power and shows that increasing model size can compensate for this drawback, potentially improving generalization.

Contribution

It provides a theoretical investigation into the expressive limitations of equivariant neural networks and quantifies how enlarging the model can offset these limitations.

Findings

Enforcing equivariance constraints can reduce the expressive power of neural networks.

Compensating for this reduction requires enlarging the model size by a quantifiable amount.

Enlarged models may have lower hypothesis space dimensionality, enhancing generalizability.

Abstract

Equivariant neural networks encode the intrinsic symmetry of data as an inductive bias, which has achieved impressive performance in wide domains. However, the understanding to their expressive power remains premature. Focusing on 2-layer ReLU networks, this paper investigates the impact of enforcing equivariance constraints on the expressive power. By examining the boundary hyperplanes and the channel vectors, we constructively demonstrate that enforcing equivariance constraints could undermine the expressive power. Naturally, this drawback can be compensated for by enlarging the model size -- we further prove upper bounds on the required enlargement for compensation. Surprisingly, we show that the enlarged neural architectures have reduced hypothesis space dimensionality, implying even better generalizability.