ULU: A Unified Activation Function

TL;DR

This paper introduces ULU, a new non-monotonic activation function that outperforms ReLU and Mish in image tasks, with an adaptive variant AULU that learns separate responses for positive and negative inputs.

Contribution

The paper proposes ULU, a novel activation function with a learnable adaptive variant AULU, and introduces the LIB metric to quantify model inductive bias.

Findings

ULU outperforms ReLU and Mish in image classification and detection.

AULU adapts responses for positive and negative inputs via learnable parameters.

The LIB metric effectively measures the model's inductive bias.

Abstract

We propose \textbf{ULU}, a novel non-monotonic, piecewise activation function defined as $\{f(x;\alpha_1),x<0; f(x;\alpha_2),x>=0 \}$, where $f(x;\alpha)=0.5x(tanh(\alpha x)+1),\alpha >0$. ULU treats positive and negative inputs differently. Extensive experiments demonstrate ULU significantly outperforms ReLU and Mish across image classification and object detection tasks. Its variant Adaptive ULU (\textbf{AULU}) is expressed as $\{f(x;\beta_1^2),x<0; f(x;\beta_2^2),x>=0 \}$, where $\beta_1$ and $\beta_2$ are learnable parameters, enabling it to adapt its response separately for positive and negative inputs. Additionally, we introduce the LIB (Like Inductive Bias) metric from AULU to quantitatively measure the inductive bias of the model.