Exploring FPGA designs for MX and beyond

TL;DR

This paper presents the first open-source FPGA implementation of the MX standard for low-precision neural network computation, including hardware components and a quantization library, demonstrating advantages over GPUs for certain formats.

Contribution

It introduces an open-source FPGA library supporting MX formats, provides a quantization tool integrated with PyTorch, and offers concrete FPGA design choices for the MX standard.

Findings

MX formats like INT5 and FP6 outperform GPU support in neural networks.

The FPGA implementation supports all standard formats and custom formats.

Open-source tools enable community development of MX-based neural networks.

Abstract

A number of companies recently worked together to release the new Open Compute Project MX standard for low-precision computation, aimed at efficient neural network implementation. In this paper, we describe and evaluate the first open-source FPGA implementation of the arithmetic defined in the standard. Our designs fully support all the standard's concrete formats for conversion into and out of MX formats and for the standard-defined arithmetic operations, as well as arbitrary fixed-point and floating-point formats. Certain elements of the standard are left as implementation-defined, and we present the first concrete FPGA-inspired choices for these elements, which we outline in the paper. Our library of optimized hardware components is available open source, and can be used to build larger systems. For this purpose, we also describe and release an open-source Pytorch library for quantization into the new standard, integrated with the Brevitas library so that the community can develop novel neural network designs quantized with MX formats in mind. We demonstrate the usability and efficacy of our libraries via the implementation of example neural networks such as ResNet-18 on the ImageNet ILSVRC12 dataset. Our testing shows that MX is very effective for formats such as INT5 or FP6 which are not natively supported on GPUs. This gives FPGAs an advantage as they have the flexibility to implement a custom datapath and take advantage of the smaller area footprints offered by these formats.