Efficient and Scalable Barrier over Quadrics and Myrinet with a New NIC-Based Collective Message Passing Protocol

TL;DR

This paper presents a new NIC-based collective protocol that significantly improves barrier operation latency over high-performance interconnects like Quadrics and Myrinet, demonstrating scalability and efficiency for large clusters.

Contribution

The paper introduces a novel NIC-based collective protocol that simplifies communication processing, enabling scalable and efficient barrier operations on high-performance interconnects.

Findings

Barrier latency of 5.60μs over 8 nodes on Quadrics

Barrier latency of 14.20μs over 8 nodes on Myrinet

Estimated 22.13μs latency for 1024-node cluster on Quadrics

Abstract

Modern interconnects often have programmable processors in the network interface that can be utilized to offload communication processing from host CPU. In this paper, we explore different schemes to support collective operations at the network interface and propose a new collective protocol. With barrier as an initial case study, we have demontrated that much of the communication processing can be greatly simplified with this collective protocol. Accordingly, %with our proposed collective processing scheme, we have designed and implemented efficient and scalable NIC-based barrier operations over two high performance interconnects, Quadrics and Myrinet. Our evaluation shows that, over a Quadrics cluster of 8 nodes with ELan3 Network, the NIC-based barrier operation achieves a barrier latency of only 5.60$\mu$s. This result is a 2.48 factor of improvement over the Elanlib tree-based barrier operation. Over a Myrinet cluster of 8 nodes with LANai-XP NIC cards, a barrier latency of 14.20$\mu$s over 8 nodes is achieved. This is a 2.64 factor of improvement over the host-based barrier algorithm. Furthermore, an analytical model developed for the proposed scheme indicates that a NIC-based barrier operation on a 1024-node cluster can be performed with only 22.13$\mu$s latency over Quadrics and with 38.94$\mu$s latency over Myrinet. These results indicate the potential for developing high performance communication subsystems for next generation clusters.