Provable Benefits of Sinusoidal Activation for Modular Addition

TL;DR

This paper demonstrates that sinusoidal activation functions enable neural networks to efficiently learn modular addition with minimal width, outperforming ReLU networks in expressivity and generalization, supported by theoretical bounds and empirical validation.

Contribution

It introduces the first sharp expressivity gap showing sine networks' advantages in modular addition, along with novel generalization bounds and empirical evidence of superior performance.

Findings

Sine MLPs can exactly realize modular addition with width 2.

ReLU networks require width scaling linearly with input length.

Sine networks exhibit better generalization and length extrapolation.

Abstract

This paper studies the role of activation functions in learning modular addition with two-layer neural networks. We first establish a sharp expressivity gap: sine MLPs admit width-$2$ exact realizations for any fixed length $m$ and, with bias, width-$2$ exact realizations uniformly over all lengths. In contrast, the width of ReLU networks must scale linearly with $m$ to interpolate, and they cannot simultaneously fit two lengths with different residues modulo $p$. We then provide a novel Natarajan-dimension generalization bound for sine networks, yielding nearly optimal sample complexity $\widetilde{\mathcal{O}}(p)$ for ERM over constant-width sine networks. We also derive width-independent, margin-based generalization for sine networks in the overparametrized regime and validate it. Empirically, sine networks generalize consistently better than ReLU networks across regimes and exhibit strong length extrapolation.