In Situ In Transit Hybrid Analysis with Catalyst-ADIOS2

TL;DR

This paper introduces a hybrid in situ and in transit data analysis method using Catalyst-ADIOS2 that reduces bandwidth and costs by performing data reduction before visualization, adaptable to various workflows.

Contribution

It presents a novel hybrid analysis approach leveraging Catalyst-ADIOS2, enabling seamless switching between in situ, in transit, and hybrid analysis without code modifications.

Findings

Large cost reductions in visualization pipelines.

Effective data reduction before transmission.

Flexible workflows depending on resources and data size.

Abstract

In this short paper, we present an innovative approach to limit the required bandwidth when transferring data during in transit analysis. This approach is called hybrid because it combines existing in situ and in transit solutions. It leverages the stable ABI of Catalyst version 2 and the Catalyst-ADIOS2 implementation to seamlessly switch from in situ, in transit and hybrid analysis without modifying the numerical simulation code. The typical use case is to perform data reduction in situ then generate a visualization in transit on the reduced data. This approach makes the numerical simulation workflows very flexible depending on the size of the data, the available computing resources or the analysis type. Our experiment with this hybrid approach, reducing data before sending it, demonstrated large cost reductions for some visualization pipelines compared to in situ and in transit solutions. The implementation is available under an open source permissive license to be usable broadly in any scientific community.