SAU: Smooth activation function using convolution with approximate identities

TL;DR

This paper introduces the Smooth Activation Unit (SAU), a new smooth approximation of Leaky ReLU created by convolution with approximate identities, leading to improved performance across datasets.

Contribution

The paper presents a novel method for smoothing activation functions via convolution with approximate identities, specifically improving Leaky ReLU with demonstrated empirical benefits.

Findings

SAU outperforms traditional activation functions on multiple datasets.

Replacing ReLU with SAU improves ShuffleNet V2 accuracy by 5.12% on CIFAR100.

SAU provides a differentiable alternative to non-smooth activation functions.

Abstract

Well-known activation functions like ReLU or Leaky ReLU are non-differentiable at the origin. Over the years, many smooth approximations of ReLU have been proposed using various smoothing techniques. We propose new smooth approximations of a non-differentiable activation function by convolving it with approximate identities. In particular, we present smooth approximations of Leaky ReLU and show that they outperform several well-known activation functions in various datasets and models. We call this function Smooth Activation Unit (SAU). Replacing ReLU by SAU, we get 5.12% improvement with ShuffleNet V2 (2.0x) model on CIFAR100 dataset.