Learning specialized activation functions with the Piecewise Linear Unit

TL;DR

This paper introduces PWLU, a novel learnable activation function that adapts to specific datasets and architectures, outperforming existing functions like Swish on large-scale benchmarks with efficient implementation.

Contribution

The paper proposes PWLU, a new piecewise linear activation function with a designed learning method, enabling specialized activations and achieving state-of-the-art results.

Findings

PWLU outperforms Swish on ImageNet and COCO datasets.

PWLU improves top-1 accuracy by up to 1.7% over Swish.

PWLU is easy to implement and efficient at inference.

Abstract

The choice of activation functions is crucial for modern deep neural networks. Popular hand-designed activation functions like Rectified Linear Unit(ReLU) and its variants show promising performance in various tasks and models. Swish, the automatically discovered activation function, has been proposed and outperforms ReLU on many challenging datasets. However, it has two main drawbacks. First, the tree-based search space is highly discrete and restricted, which is difficult for searching. Second, the sample-based searching method is inefficient, making it infeasible to find specialized activation functions for each dataset or neural architecture. To tackle these drawbacks, we propose a new activation function called Piecewise Linear Unit(PWLU), which incorporates a carefully designed formulation and learning method. It can learn specialized activation functions and achieves SOTA performance on large-scale datasets like ImageNet and COCO. For example, on ImageNet classification dataset, PWLU improves 0.9%/0.53%/1.0%/1.7%/1.0% top-1 accuracy over Swish for ResNet-18/ResNet-50/MobileNet-V2/MobileNet-V3/EfficientNet-B0. PWLU is also easy to implement and efficient at inference, which can be widely applied in real-world applications.