The Feasibility of Implementing Large-Scale Transformers on Multi-FPGA Platforms

TL;DR

This paper investigates the potential of using multiple FPGAs to implement large transformer models, developing a scalable platform and tools, and demonstrating feasibility with a multi-FPGA I-BERT prototype.

Contribution

It introduces a scalable multi-FPGA platform and tools for large ML applications, and validates the approach with a multi-FPGA I-BERT implementation.

Findings

Multi-FPGA implementation of I-BERT is feasible.

FPGAs can be competitive with GPUs for large ML models.

The platform shows promising performance potential.

Abstract

FPGAs are rarely mentioned when discussing the implementation of large machine learning applications, such as Large Language Models (LLMs), in the data center. There has been much evidence showing that single FPGAs can be competitive with GPUs in performance for some computations, especially for low latency, and often much more efficient when power is considered. This suggests that there is merit to exploring the use of multiple FPGAs for large machine learning applications. The challenge with using multiple FPGAs is that there is no commonly-accepted flow for developing and deploying multi-FPGA applications, i.e., there are no tools to describe a large application, map it to multiple FPGAs and then deploy the application on a multi-FPGA platform. In this paper, we explore the feasibility of implementing large transformers using multiple FPGAs by developing a scalable multi-FPGA platform and some tools to map large applications to the platform. We validate our approach by designing an efficient multi-FPGA version of the I-BERT transformer and implement one encoder using six FPGAs as a working proof-of-concept to show that our platform and tools work. Based on our proof-of-concept prototype and the estimations of performance using the latest FPGAs compared to GPUs, we conclude that there can be a place for FPGAs in the world of large machine learning applications. We demonstrate a promising first step that shows that with the right infrastructure and tools it is reasonable to continue to explore the possible benefits of using FPGAs for applications such as LLMs.