Optimized Graph Based Routing Algorithm for the Angara Interconnect

TL;DR

This paper introduces an optimized deadlock-free routing algorithm for the Angara 4D torus interconnect, improving communication performance in real-world cluster systems by leveraging a shortest path approach.

Contribution

It proposes a novel routing algorithm based on a fast shortest path method that enhances performance and guarantees deadlock freedom in the Angara interconnect.

Findings

11.1% performance improvement for Alltoall communication

Over 5% performance gain for FT and IS kernels

Validated on a 32-node Desmos cluster system

Abstract

JSC NICEVT has developed the Angara high-speed interconnect with 4D torus topology. The Angara interconnect router implements deterministic routing based on the bubble flow control, a direction order routing (DOR) and direction bits rules. The router chip also supports non standard First Step / Last Step for bypassing failed nodes and links, these steps can violate the DOR rule. In the previous work we have proposed an algorithm for generation and analysis of routing tables that guarantees no deadlocks in the Angara interconnect. It is based on a breadth-first search algorithm in a graph and it practically does not take into consideration communication channel load. Also we have never evaluated the influence of routing table generation algorithm on the performance of a real-world Angara based cluster. In this paper we present a routing graph notation that provides a possibility to build routes in the torus topology of the Angara interconnect. We propose a deadlock-free routing algorithm based on a fast single-source shortest path algorithm for the deterministic Angara routing with a single virtual channel. We evaluated the considered routing algorithms on a 32-node Desmos cluster system and benchmarked the proposed algorithm performance improvement of 11.1% for the Alltoall communication pattern and of more than 5% for the FT and IS application kernels.