NN-LUT: Neural Approximation of Non-Linear Operations for Efficient Transformer Inference

TL;DR

This paper introduces NN-LUT, a neural network-based approximation framework that replaces costly non-linear operations in Transformer models with a hardware-efficient solution, significantly reducing area, power, and latency.

Contribution

It proposes a novel neural approximation framework, NN-LUT, that accurately replaces non-linear operations in Transformers with a LUT-based structure for efficient inference.

Findings

Achieves accurate approximation of non-linear operations in BERT models.

Reduces hardware area, power consumption, and latency significantly.

Provides a universal approach adaptable to various Transformer components.

Abstract

Non-linear operations such as GELU, Layer normalization, and Softmax are essential yet costly building blocks of Transformer models. Several prior works simplified these operations with look-up tables or integer computations, but such approximations suffer inferior accuracy or considerable hardware cost with long latency. This paper proposes an accurate and hardware-friendly approximation framework for efficient Transformer inference. Our framework employs a simple neural network as a universal approximator with its structure equivalently transformed into a LUT. The proposed framework called NN-LUT can accurately replace all the non-linear operations in popular BERT models with significant reductions in area, power consumption, and latency.