Exploring the Approximation Capabilities of Multiplicative Neural Networks for Smooth Functions

TL;DR

This paper analyzes how neural networks with multiplicative layers can more efficiently approximate smooth functions like bandlimited signals and Sobolev functions, outperforming standard ReLU networks in layer and neuron efficiency.

Contribution

It provides theoretical results showing multiplicative neural networks can approximate certain classes of smooth functions with fewer layers and neurons than traditional ReLU networks.

Findings

Multiplicative neural networks require fewer layers for approximation.

They achieve better approximation with fewer neurons.

Outperform standard ReLU networks in efficiency.

Abstract

Multiplication layers are a key component in various influential neural network modules, including self-attention and hypernetwork layers. In this paper, we investigate the approximation capabilities of deep neural networks with intermediate neurons connected by simple multiplication operations. We consider two classes of target functions: generalized bandlimited functions, which are frequently used to model real-world signals with finite bandwidth, and Sobolev-Type balls, which are embedded in the Sobolev Space $\mathcal{W}^{r,2}$. Our results demonstrate that multiplicative neural networks can approximate these functions with significantly fewer layers and neurons compared to standard ReLU neural networks, with respect to both input dimension and approximation error. These findings suggest that multiplicative gates can outperform standard feed-forward layers and have potential for improving neural network design.