Large scale in transit computation of quantiles for ensemble runs

TL;DR

This paper presents an on-the-fly, scalable method for computing quantiles in large ensemble simulations at exascale, using an adaptive Robbins-Monro algorithm integrated with the Melissa framework.

Contribution

It introduces a novel iterative quantile computation approach suitable for exascale data, overcoming limitations of traditional methods requiring full sample availability.

Findings

Successfully computed spatio-temporal percentile maps from 11 TB of fluid dynamics data.

Demonstrated scalability and fault-tolerance of the method on large-scale simulations.

Achieved real-time quantile estimation during ensemble runs.

Abstract

The classical approach for quantiles computation requires availability of the full sample before ranking it. In uncertainty quantification of numerical simulation models, this approach is not suitable at exascale as large ensembles of simulation runs would need to gather a prohibitively large amount of data. This problem is solved thanks to an on-the-fly and iterative approach based on the Robbins-Monro algorithm. This approach relies on Melissa, a file avoiding, adaptive, fault-tolerant and elastic framework. On a validation case producing 11 TB of data, which consists in 3000 fluid dynamics parallel simulations on a 6M cell mesh, it allows on-line computation of spatio-temporal maps of percentiles.