Leveraging Continuously Differentiable Activation Functions for Learning in Quantized Noisy Environments

TL;DR

This paper shows that using continuously differentiable activation functions like GELU and SiLU improves the robustness and accuracy of deep learning models in noisy, quantized analog environments, compared to traditional ReLU.

Contribution

It introduces the analysis and validation that differentiable activations enhance noise resilience in quantized analog systems, guiding activation choice for hardware implementations.

Findings

GELU and SiLU outperform ReLU in noisy environments.

Gradient errors are 100x lower with GELU near zero.

Differentiable activations improve model robustness in analog hardware.

Abstract

Real-world analog systems intrinsically suffer from noise that can impede model convergence and accuracy on a variety of deep learning models. We demonstrate that differentiable activations like GELU and SiLU enable robust propagation of gradients which help to mitigate analog quantization error that is ubiquitous to all analog systems. We perform analysis and training of convolutional, linear, and transformer networks in the presence of quantized noise. Here, we are able to demonstrate that continuously differentiable activation functions are significantly more noise resilient over conventional rectified activations. As in the case of ReLU, the error in gradients are 100x higher than those in GELU near zero. Our findings provide guidance for selecting appropriate activations to realize performant and reliable hardware implementations across several machine learning domains such as computer vision, signal processing, and beyond. Code available at: \href{https://github.com/Vivswan/GeLUReLUInterpolation}{https://github.com/Vivswan/GeLUReLUInterpolation}.}