Stochastic Adaptive Activation Function

TL;DR

This paper introduces Adaptive Swish (ASH), a novel trainable activation function that models neuron variability, improves deep learning performance across tasks, and offers a generalized mathematical framework over existing functions.

Contribution

The study proposes ASH, a flexible, context-aware activation function that generalizes Swish and enhances deep learning models' accuracy and convergence.

Findings

ASH improves prediction accuracy across various tasks.

ASH enables earlier convergence in training.

ASH demonstrates robustness in multiple deep learning models.

Abstract

The simulation of human neurons and neurotransmission mechanisms has been realized in deep neural networks based on the theoretical implementations of activation functions. However, recent studies have reported that the threshold potential of neurons exhibits different values according to the locations and types of individual neurons, and that the activation functions have limitations in terms of representing this variability. Therefore, this study proposes a simple yet effective activation function that facilitates different thresholds and adaptive activations according to the positions of units and the contexts of inputs. Furthermore, the proposed activation function mathematically exhibits a more generalized form of Swish activation function, and thus we denoted it as Adaptive SwisH (ASH). ASH highlights informative features that exhibit large values in the top percentiles in an input, whereas it rectifies low values. Most importantly, ASH exhibits trainable, adaptive, and context-aware properties compared to other activation functions. Furthermore, ASH represents general formula of the previously studied activation function and provides a reasonable mathematical background for the superior performance. To validate the effectiveness and robustness of ASH, we implemented ASH into many deep learning models for various tasks, including classification, detection, segmentation, and image generation. Experimental analysis demonstrates that our activation function can provide the benefits of more accurate prediction and earlier convergence in many deep learning applications.