Efficient Non-linear Calculators

TL;DR

This paper introduces a new multiplier-less algorithm for smooth, quadratic nonlinearities suitable for digital hardware, optimized for use in neural network activation functions and edge computing devices.

Contribution

The paper presents a novel, multiplier-less implementation of quadratic nonlinearities that are efficient for FPGA/ASIC hardware and suitable for neural network activation functions.

Findings

Hardware implementations are resource-efficient and suitable for edge devices.

Benchmarking shows competitive performance with existing nonlinearities.

Integrated scaling eliminates the need for additional multipliers.

Abstract

A novel algorithm for producing smooth nonlinearities on digital hardware is presented. The non-linearities are inherently quadratic and have both symmetrical and asymmetrical variants. The integer (and fixed point) implementation is highly amenable for use with digital gates on an ASIC or FPGA. The implementations are multiplier-less. Scaling of the non-linear output, as required in an LSTM cell, is integrated into the implementation. This too does not require a multiplier. The non-linearities are useful as activation functions in a variety of ANN architectures. The floating point mappings have been compared with other non-linearities and have been benchmarked. Results show that these functions should be considered in the ANN design phase. The hardware resource usage of the implementations have been thoroughly investigated. Our results make a strong case for implementions in edge applications. This document summarizes the findings and serves to give a quick overview of the outcomes of our research\footnote{The authors peer-reviewed manuscripts (available at https://doi.org/10.1016/j.neucom.2021.02.030) offer more detail and may be better suited for a thorough consideration}.