Rational Neural Networks have Expressivity Advantages

TL;DR

This paper demonstrates that neural networks with trainable rational activation functions are more expressive and parameter-efficient than traditional activations, with theoretical and practical advantages shown across various architectures.

Contribution

It introduces trainable rational activation functions, proving their superior expressivity and efficiency both theoretically and empirically compared to standard activations.

Findings

Rational-activation networks approximate fixed-activation networks with exponentially fewer parameters.

Rational activations outperform fixed activations in practical training scenarios.

Theoretical separation results show exponential advantages in approximation complexity.

Abstract

We study neural networks with trainable low-degree rational activation functions and show that they are more expressive and parameter-efficient than modern piecewise-linear and smooth activations such as ELU, LeakyReLU, LogSigmoid, PReLU, ReLU, SELU, CELU, Sigmoid, SiLU, Mish, Softplus, Tanh, Softmin, Softmax, and LogSoftmax. For an error target of $\varepsilon>0$, we establish approximation-theoretic separations: Any network built from standard fixed activations can be uniformly approximated on compact domains by a rational-activation network with only $\mathrm{poly}(\log\log(1/\varepsilon))$ overhead in size, while the converse provably requires $\Omega(\log(1/\varepsilon))$ parameters in the worst case. This exponential gap persists at the level of full networks and extends to gated activations and transformer-style nonlinearities. In practice, rational activations integrate seamlessly into standard architectures and training pipelines, allowing rationals to match or outperform fixed activations under identical architectures and optimizers.