FLIT-level InfiniBand network simulations of the DAQ system of the LHCb experiment for Run-3

TL;DR

This paper presents detailed FLIT-level InfiniBand network simulations for the LHCb experiment's data acquisition system, aiding in optimizing network configuration for high data rates and congestion management.

Contribution

It provides an accurate low-level simulation model of InfiniBand networks with event building traffic, enabling performance analysis and optimization before physical deployment.

Findings

Simulation results closely match real system performance.

Network configuration impacts link utilization and congestion.

Parameter variations significantly affect system throughput.

Abstract

The LHCb (Large Hadron Collider beauty) experiment is designed to study differences between particles and anti-particles as well as very rare decays in the charm and beauty sector at the LHC (Large Hadron Collider). The detector will be upgraded in 2019 and a new trigger-less readout system will be implemented in order to significantly increase its efficiency and take advantage of the increased machine luminosity. In the upgraded system, both event building and event filtering will be performed in software for all the data produced in every bunch-crossing of the LHC. In order to transport the full data rate of 32 Tb/s we will use custom FPGA readout boards (PCIe40) and state of the art off-the-shelf network technologies. The full event building system will require around 500 nodes interconnected together. From a networking point of view, event building traffic has an all-to-all pattern, therefore it tends to create high network congestion. In order to maximize the link utilization different techniques can be adopted in various areas like traffic shaping, network topology and routing optimization. The size of the system makes it very difficult to test at production scale, before the actual procurement. We resort therefore to network simulations as a powerful tool for finding the optimal configuration. We will present an accurate low level description of an InfiniBand based network with event building like traffic. We will show comparison between simulated and real systems and how changes in the input parameters affect performances.