Parallel Layer Normalization for Universal Approximation

TL;DR

This paper demonstrates that neural networks with parallel layer normalization (PLN) layers can universally approximate functions, surpassing standard LN networks, with theoretical analysis and empirical validation across various architectures.

Contribution

It introduces PLN-Nets, a novel architecture that achieves universal approximation, and extends the analysis to RMSNorm and complex models like Transformers.

Findings

PLN-Nets achieve universal approximation.

Analysis of approximation rates in different norms.

Empirical evidence supports PLN-Nets' potential.

Abstract

This paper studies the approximation capabilities of neural networks that combine layer normalization (LN) with linear layers. We prove that networks consisting of two linear layers with parallel layer normalizations (PLNs) inserted between them (referred to as PLN-Nets) achieve universal approximation, whereas architectures that use only standard LN exhibit strictly limited expressive power.We further analyze approximation rates of shallow and deep PLN-Nets under the $L^\infty$ norm as well as in Sobolev norms. Our analysis extends beyond LN to RMSNorm, and from standard MLPs to position-wise feed-forward networks, the core building blocks used in RNNs and Transformers.Finally, we provide empirical experiments to explore other possible potentials of PLN-Nets.