Approximation with SiLU Networks: Constant Depth and Exponential Rates for Basic Operations

TL;DR

This paper demonstrates how properly tuned SiLU neural networks can efficiently approximate functions like $x^2$ with constant depth and exponential rates, emphasizing the importance of hyperparameter optimization.

Contribution

It introduces SiLU network constructions that achieve approximation with constant depth and exponential rates by optimal hyperparameter tuning, extending to Sobolev spaces.

Findings

Two-layer networks approximate $x^2$ with error $oldsymbol{ ext{ε}}$ using weights scaling as $eta^{ ext{±}k}$.

Networks with depth $oldsymbol{ ext{O}(1)}$ and $oldsymbol{ ext{O}( ext{ε}^{-d/n})}$ parameters approximate Sobolev functions.

Hyperparameter tuning critically influences the approximation efficiency of SiLU networks.

Abstract

We present SiLU network constructions whose approximation efficiency depends critically on proper hyperparameter tuning. For the square function $x^2$, with optimally chosen shift $a$ and scale $\beta$, we achieve approximation error $\varepsilon$ using a two-layer network of constant width, where weights scale as $\beta^{\pm k}$ with $k = \mathcal{O}(\ln(1/\varepsilon))$. We then extend this approach through functional composition to Sobolev spaces, we obtain networks with depth $\mathcal{O}(1)$ and $\mathcal{O}(\varepsilon^{-d/n})$ parameters under optimal hyperparameters settings. Our work highlights the trade-off between architectural depth and activation parameter optimization in neural network approximation theory.