Competition-based Adaptive ReLU for Deep Neural Networks

TL;DR

This paper introduces CAReLU, a novel activation function that models competition between positive and negative inputs, improving performance across various deep learning tasks by adaptively scaling inputs.

Contribution

The paper proposes CAReLU, a new adaptive activation function that incorporates competition between positive and negative values, trained jointly with network parameters.

Findings

CAReLU outperforms traditional activation functions in multiple tasks.

Replacing ReLU with CAReLU in ResNet-18 improves CIFAR-100 accuracy.

CAReLU offers a new perspective on activation function design through competition modeling.

Abstract

Activation functions introduce nonlinearity into deep neural networks. Most popular activation functions allow positive values to pass through while blocking or suppressing negative values. From the idea that positive values and negative values are equally important, and they must compete for activation, we proposed a new Competition-based Adaptive ReLU (CAReLU). CAReLU scales the input values based on the competition results between positive values and negative values. It defines two parameters to adjust the scaling strategy and can be trained uniformly with other network parameters. We verify the effectiveness of CAReLU on image classification, super-resolution, and natural language processing tasks. In the experiment, our method performs better than other widely used activation functions. In the case of replacing ReLU in ResNet-18 with our proposed activation function, it improves the classification accuracy on the CIFAR-100 dataset. The effectiveness and the new perspective on the utilization of competition results between positive values and negative values make CAReLU a promising activation function.