Nonlinearity Enhanced Adaptive Activation Functions

TL;DR

This paper introduces a general method for adding learned nonlinearities to activation functions, improving neural network accuracy on datasets like MNIST and CNN benchmarks with minimal extra computation.

Contribution

It proposes a novel, general approach for parametric, learned nonlinear activation functions that enhance neural network performance.

Findings

Improved accuracy on MNIST dataset

Enhanced CNN benchmark performance

Minimal additional computational cost

Abstract

A general procedure for introducing parametric, learned, nonlinearity into activation functions is found to enhance the accuracy of representative neural networks without requiring significant additional computational resources. Examples are given based on the standard rectified linear unit (ReLU) as well as several other frequently employed activation functions. The associated accuracy improvement is quantified both in the context of the MNIST digit data set and a convolutional neural network (CNN) benchmark example.