ANAct: Adaptive Normalization for Activation Functions

TL;DR

This paper introduces ANAct, a dynamic normalization method for activation functions that maintains consistent gradient variance during training, leading to improved neural network performance.

Contribution

We propose ANAct, a novel normalization technique that adaptively adjusts activation functions based on mini-batch statistics to enhance training stability and accuracy.

Findings

Normalized Swish outperforms vanilla Swish by 1.4% on Tiny ImageNet with ResNet50.

ANAct improves CNN and residual network performance across datasets.

Convergence rate correlates with the normalization property.

Abstract

In this paper, we investigate the negative effect of activation functions on forward and backward propagation and how to counteract this effect. First, We examine how activation functions affect the forward and backward propagation of neural networks and derive a general form for gradient variance that extends the previous work in this area. We try to use mini-batch statistics to dynamically update the normalization factor to ensure the normalization property throughout the training process, rather than only accounting for the state of the neural network after weight initialization. Second, we propose ANAct, a method that normalizes activation functions to maintain consistent gradient variance across layers and demonstrate its effectiveness through experiments. We observe that the convergence rate is roughly related to the normalization property. We compare ANAct with several common activation functions on CNNs and residual networks and show that ANAct consistently improves their performance. For instance, normalized Swish achieves 1.4\% higher top-1 accuracy than vanilla Swish on ResNet50 with the Tiny ImageNet dataset and more than 1.2\% higher with CIFAR-100.