Exponential Separations in Symmetric Neural Networks

TL;DR

This paper proves that certain symmetric neural network architectures can efficiently approximate specific functions, while others require exponentially larger widths, highlighting fundamental representational differences.

Contribution

It introduces a novel exponential separation result between Relational Networks and DeepSets for symmetric function approximation.

Findings

Relational Networks can efficiently approximate certain symmetric functions.

DeepSets require exponential width to approximate the same functions.

The separation holds under analytic activation functions.

Abstract

In this work we demonstrate a novel separation between symmetric neural network architectures. Specifically, we consider the Relational Network~\parencite{santoro2017simple} architecture as a natural generalization of the DeepSets~\parencite{zaheer2017deep} architecture, and study their representational gap. Under the restriction to analytic activation functions, we construct a symmetric function acting on sets of size $N$ with elements in dimension $D$, which can be efficiently approximated by the former architecture, but provably requires width exponential in $N$ and $D$ for the latter.