FP8-BERT: Post-Training Quantization for Transformer

TL;DR

This paper demonstrates that FP8-based post-training quantization can significantly reduce model size and inference costs for BERT without accuracy loss, outperforming traditional INT8 quantization methods.

Contribution

The paper empirically validates FP8 as an effective format for post-training quantization of BERT, achieving accuracy comparable to full-precision models.

Findings

FP8 PTQ maintains full-precision accuracy on BERT.

FP8 outperforms INT8 in accuracy retention.

Simple calibration enables effective FP8 quantization.

Abstract

Transformer-based models, such as BERT, have been widely applied in a wide range of natural language processing tasks. However, one inevitable side effect is that they require massive memory storage and inference cost when deployed in production. Quantization is one of the popularized ways to alleviate the cost. However, the previous 8-bit quantization strategy based on INT8 data format either suffers from the degradation of accuracy in a Post-Training Quantization (PTQ) fashion or requires an expensive Quantization-Aware Training (QAT) process. Recently, a new numeric format FP8 (i.e. floating-point of 8-bits) has been proposed and supported in commercial AI computing platforms such as H100. In this paper, we empirically validate the effectiveness of FP8 as a way to do Post-Training Quantization without significant loss of accuracy, with a simple calibration and format conversion process. We adopt the FP8 standard proposed by NVIDIA Corp. (2022) in our extensive experiments of BERT variants on GLUE and SQuAD v1.1 datasets, and show that PTQ with FP8 can significantly improve the accuracy upon that with INT8, to the extent of the full-precision model.