Reduced Softmax Unit for Deep Neural Network Accelerators

TL;DR

This paper introduces a simplified Softmax unit for DNN accelerators that replaces exponential calculations with a comparator, maintaining classification accuracy while reducing computational complexity.

Contribution

It proposes a reduced Softmax unit that simplifies the activation layer by using only a comparator, eliminating exponential calculations in DNN accelerators.

Findings

Classification results are identical to traditional Softmax.

Reduces computational complexity in DNN accelerators.

Simplifies hardware implementation of Softmax layer.

Abstract

The Softmax activation layer is a very popular Deep Neural Network (DNN) component when dealing with multi-class prediction problems. However, in DNN accelerator implementations it creates additional complexities due to the need for computation of the exponential for each of its inputs. In this brief we propose a simplified version of the activation unit for accelerators, where only a comparator unit produces the classification result, by choosing the maximum among its inputs. Due to the nature of the activation function, we show that this result is always identical to the classification produced by the Softmax layer.