An FPGA-based Torus Communication Network

TL;DR

This paper presents the design and FPGA implementation of a 3D torus network for scalable, high-performance scientific computing, including hardware, software, and integration details with real systems.

Contribution

It introduces a novel FPGA-based 3D torus network with custom communication APIs, integrated into existing parallel systems for improved scalability.

Findings

Successful integration into QPACE and AuroraScience systems

Performance results demonstrating efficiency and scalability

Open-source hardware and software implementation

Abstract

We describe the design and FPGA implementation of a 3D torus network (TNW) to provide nearest-neighbor communications between commodity multi-core processors. The aim of this project is to build up tightly interconnected and scalable parallel systems for scientific computing. The design includes the VHDL code to implement on latest FPGA devices a network processor, which can be accessed by the CPU through a PCIe interface and which controls the external PHYs of the physical links. Moreover, a Linux driver and a library implementing custom communication APIs are provided. The TNW has been successfully integrated in two recent parallel machine projects, QPACE and AuroraScience. We describe some details of the porting of the TNW for the AuroraScience system and report performance results.