Pushing the Limits of BFP on Narrow Precision LLM Inference

TL;DR

This paper introduces a hardware-software co-design framework, DB-Attn, that enhances nonlinear operation efficiency in large language models using an advanced BFP format, achieving significant speedups with minimal accuracy loss.

Contribution

The paper presents DBFP, an improved BFP format, and DH-LUT, a lookup table algorithm, along with an RTL engine, to optimize nonlinear operations in LLM inference.

Findings

74% GPU speedup on Softmax of LLaMA

10x performance improvement over state-of-the-art designs

Negligible accuracy loss with the proposed methods

Abstract

The substantial computational and memory demands of Large Language Models (LLMs) hinder their deployment. Block Floating Point (BFP) has proven effective in accelerating linear operations, a cornerstone of LLM workloads. However, as sequence lengths grow, nonlinear operations, such as Attention, increasingly become performance bottlenecks due to their quadratic computational complexity. These nonlinear operations are predominantly executed using inefficient floating-point formats, which renders the system challenging to optimize software efficiency and hardware overhead. In this paper, we delve into the limitations and potential of applying BFP to nonlinear operations. Given our findings, we introduce a hardware-software co-design framework (DB-Attn), including: (i) DBFP, an advanced BFP version, overcomes nonlinear operation challenges with a pivot-focus strategy for diverse data and an adaptive grouping strategy for flexible exponent sharing. (ii) DH-LUT, a novel lookup table algorithm dedicated to accelerating nonlinear operations with DBFP format. (iii) An RTL-level DBFP-based engine is implemented to support DB-Attn, applicable to FPGA and ASIC. Results show that DB-Attn provides significant performance improvements with negligible accuracy loss, achieving 74% GPU speedup on Softmax of LLaMA and 10x low overhead performance improvement over SOTA designs.