Towards In-transit Analysis on Supercomputing Environments

TL;DR

This paper explores in-transit analysis in supercomputing environments, focusing on data transfer strategies using RDMA to optimize performance during simulation result analysis.

Contribution

It investigates the impact of varying block sizes and data transfer methods like RDMA on in-transit analysis performance in supercomputing.

Findings

RDMA reduces performance interference during data transfer

Varying block sizes affect transfer efficiency

In-memory transfer strategies improve analysis throughput

Abstract

The drive towards exascale computing is opening an enormous opportunity for more realistic and precise simulations of natural phenomena. The process of simulation, however, involves not only the numerical computation of predictions but also the analysis of results both to evaluate the simulation quality and interpret the simulated phenomenon. In this context, one may consider the duality between transaction and analytical processing to be repositioned in this new context. The co-habitation of simulation computation and analysis has been named after in situ analysis, whereas the separation in different systems considered as in-transit analysis. In this paper we focus in the latter model and study the impact of transferring varying block size data from the simulation system to the analytical one. We use the Remote Direct Memory Access protocol (RDMA) that reduces the interference on performance caused by data copies and context switching. It adopts an in-memory data transfer strategy combined with TCP, using the BSD sockets API and the Linux splice(2) syscall. We present a performance evaluation with our work and traditional utilities.