Multi-FPGA Designs and Scaling of HPC Challenge Benchmarks via MPI and Circuit-Switched Inter-FPGA Networks

TL;DR

This paper extends the HPC Challenge benchmark suite to support multiple FPGAs, explores direct FPGA-to-FPGA communication, and evaluates performance on large-scale FPGA clusters, advancing multi-FPGA HPC applications.

Contribution

It introduces multi-FPGA support and new benchmarks for FPGA HPC, and investigates circuit-switched FPGA interconnects for improved communication performance.

Findings

Multi-FPGA benchmarks established on current FPGA boards.

Potential of direct FPGA-to-FPGA communication demonstrated.

Performance evaluation conducted on up to 26 FPGA boards.

Abstract

While FPGA accelerator boards and their respective high-level design tools are maturing, there is still a lack of multi-FPGA applications, libraries, and not least, benchmarks and reference implementations towards sustained HPC usage of these devices. As in the early days of GPUs in HPC, for workloads that can reasonably be decoupled into loosely coupled working sets, multi-accelerator support can be achieved by using standard communication interfaces like MPI on the host side. However, for performance and productivity, some applications can profit from a tighter coupling of the accelerators. FPGAs offer unique opportunities here when extending the dataflow characteristics to their communication ininterfaces. In this work, we extend the HPCC FPGA benchmark suite by multi-FPGA support and three missing benchmarks that particularly characterize or stress inter-device communication: b_eff, PTRANS, and LINPACK. With all benchmarks implemented for current boards with Intel and Xilinx FPGAs, we established a baseline for multi-FPGA performance. Additionally, for the communication-centric benchmarks, we explored the potential of direct FPGA-to-FPGA communication with a circuit-switched inter-FPGA network that is currently only available for one of the boards. The evaluation with parallel execution on up to 26 FPGA boards makes use of one of the largest academic FPGA installations.