Improving Deep Learning by Inverse Square Root Linear Units (ISRLUs)

TL;DR

This paper introduces the ISRLU activation function, which accelerates learning and improves generalization in deep neural networks, outperforming ELU and ReLU, with efficient variants suitable for RNNs.

Contribution

The paper proposes the ISRLU activation function and its efficient variant ISRU, demonstrating improved training speed and performance in CNNs and RNNs over existing functions.

Findings

ISRLU outperforms ELU and ReLU in CNN training.

ISRLU enables faster learning and better generalization.

ISRU offers a computationally efficient alternative for RNNs.

Abstract

We introduce the "inverse square root linear unit" (ISRLU) to speed up learning in deep neural networks. ISRLU has better performance than ELU but has many of the same benefits. ISRLU and ELU have similar curves and characteristics. Both have negative values, allowing them to push mean unit activation closer to zero, and bring the normal gradient closer to the unit natural gradient, ensuring a noise-robust deactivation state, lessening the over fitting risk. The significant performance advantage of ISRLU on traditional CPUs also carry over to more efficient HW implementations on HW/SW codesign for CNNs/RNNs. In experiments with TensorFlow, ISRLU leads to faster learning and better generalization than ReLU on CNNs. This work also suggests a computationally efficient variant called the "inverse square root unit" (ISRU) which can be used for RNNs. Many RNNs use either long short-term memory (LSTM) and gated recurrent units (GRU) which are implemented with tanh and sigmoid activation functions. ISRU has less com- putational complexity but still has a similar curve to tanh and sigmoid.