Integer-only Quantized Transformers for Embedded FPGA-based Time-series Forecasting in AIoT

TL;DR

This paper develops an integer-only quantized Transformer hardware accelerator for embedded FPGA-based time-series forecasting in AIoT, achieving high efficiency with minimal accuracy loss.

Contribution

It introduces a complete FPGA implementation of 4-bit and 6-bit quantized Transformers optimized for AIoT, demonstrating significant improvements in speed and energy efficiency.

Findings

4-bit Transformer increases test loss by only 0.63%

Operates up to 132.33x faster than 8-bit models

Consumes 48.19x less energy than 8-bit models

Abstract

This paper presents the design of a hardware accelerator for Transformers, optimized for on-device time-series forecasting in AIoT systems. It integrates integer-only quantization and Quantization-Aware Training with optimized hardware designs to realize 6-bit and 4-bit quantized Transformer models, which achieved precision comparable to 8-bit quantized models from related research. Utilizing a complete implementation on an embedded FPGA (Xilinx Spartan-7 XC7S15), we examine the feasibility of deploying Transformer models on embedded IoT devices. This includes a thorough analysis of achievable precision, resource utilization, timing, power, and energy consumption for on-device inference. Our results indicate that while sufficient performance can be attained, the optimization process is not trivial. For instance, reducing the quantization bitwidth does not consistently result in decreased latency or energy consumption, underscoring the necessity of systematically exploring various optimization combinations. Compared to an 8-bit quantized Transformer model in related studies, our 4-bit quantized Transformer model increases test loss by only 0.63%, operates up to 132.33x faster, and consumes 48.19x less energy. Relevant source code is provided in the accompanying GitHub repository: https://github.com/tianheng-ling/TinyTransformer4TS.