SignReLU neural network and its approximation ability

TL;DR

This paper introduces SignReLU, a new activation function for deep neural networks, demonstrating its superior approximation capabilities both theoretically and empirically compared to existing functions like ReLU.

Contribution

The paper presents the SignReLU activation function and proves its enhanced approximation ability over rational and ReLU networks, supported by numerical experiments.

Findings

SignReLU networks outperform ReLU, Leaky ReLU, and ELU in approximation tasks.

Theoretical analysis confirms SignReLU's superior approximation power.

Numerical experiments show competitive practical performance of SignReLU.

Abstract

Deep neural networks (DNNs) have garnered significant attention in various fields of science and technology in recent years. Activation functions define how neurons in DNNs process incoming signals for them. They are essential for learning non-linear transformations and for performing diverse computations among successive neuron layers. In the last few years, researchers have investigated the approximation ability of DNNs to explain their power and success. In this paper, we explore the approximation ability of DNNs using a different activation function, called SignReLU. Our theoretical results demonstrate that SignReLU networks outperform rational and ReLU networks in terms of approximation performance. Numerical experiments are conducted comparing SignReLU with the existing activations such as ReLU, Leaky ReLU, and ELU, which illustrate the competitive practical performance of SignReLU.