Partitioning a Large Simulation as It Runs

TL;DR

This paper presents an online in situ method that adaptively selects key simulation time steps to reduce data storage and transfer needs while maintaining high fidelity for post-processing, demonstrated on a NASA simulation.

Contribution

It introduces a novel online in situ approach combining adaptive time step selection with data reduction for large-scale simulations.

Findings

Reduces data transfer and storage requirements significantly.

Improves fidelity of saved data for post-processing.

Validated on NASA lunar crater simulation.

Abstract

As computer simulations continue to grow in size and complexity, they present a particularly challenging class of big data problems. Many application areas are moving toward exascale computing systems, systems that perform $10^{18}$ FLOPS (FLoating-point Operations Per Second) --- a billion billion calculations per second. Simulations at this scale can generate output that exceeds both the storage capacity and the bandwidth available for transfer to storage, making post-processing and analysis challenging. One approach is to embed some analyses in the simulation while the simulation is running --- a strategy often called in situ analysis --- to reduce the need for transfer to storage. Another strategy is to save only a reduced set of time steps rather than the full simulation. Typically the selected time steps are evenly spaced, where the spacing can be defined by the budget for storage and transfer. This paper combines both of these ideas to introduce an online in situ method for identifying a reduced set of time steps of the simulation to save. Our approach significantly reduces the data transfer and storage requirements, and it provides improved fidelity to the simulation to facilitate post-processing and reconstruction. We illustrate the method using a computer simulation that supported NASA's 2009 Lunar Crater Observation and Sensing Satellite mission.