FSHMEM: Supporting Partitioned Global Address Space on FPGAs for Large-Scale Hardware Acceleration Infrastructure

TL;DR

This paper introduces FSHMEM, a framework enabling PGAS programming on FPGAs, achieving high bandwidth and low latency, and demonstrating significant speedups in HPC applications with FPGA-based accelerators.

Contribution

It presents the first FPGA implementation of PGAS core functions, integrating GASNet specification, with high performance and compatibility for HPC systems.

Findings

Peak bandwidth of 3813 MB/s, over 95% of theoretical maximum.

Remote write/read latency of 0.35us/0.59us.

Speedup of nearly 2x in matrix multiplication and convolution.

Abstract

By providing highly efficient one-sided communication with globally shared memory space, Partitioned Global Address Space (PGAS) has become one of the most promising parallel computing models in high-performance computing (HPC). Meanwhile, FPGA is getting attention as an alternative compute platform for HPC systems with the benefit of custom computing and design flexibility. However, the exploration of PGAS has not been conducted on FPGAs, unlike the traditional message passing interface. This paper proposes FSHMEM, a software/hardware framework that enables the PGAS programming model on FPGAs. We implement the core functions of GASNet specification on FPGA for native PGAS integration in hardware, while its programming interface is designed to be highly compatible with legacy software. Our experiments show that FSHMEM achieves the peak bandwidth of 3813 MB/s, which is more than 95% of the theoretical maximum, outperforming the prior works by 9.5$\times$. It records 0.35$us$ and 0.59$us$ latency for remote write and read operations, respectively. Finally, we conduct a case study on the two Intel D5005 FPGA nodes integrating Intel's deep learning accelerator. The two-node system programmed by FSHMEM achieves 1.94$\times$ and 1.98$\times$ speedup for matrix multiplication and convolution operation, respectively, showing its scalability potential for HPC infrastructure.