High Throughput Training of Deep Surrogates from Large Ensemble Runs

TL;DR

This paper introduces an open-source framework for online training of deep surrogate models using large ensemble simulation data, significantly improving training efficiency and accuracy.

Contribution

It presents a novel streaming training framework that leverages parallelism and a reservoir buffer to efficiently train deep surrogates from large datasets.

Findings

Enabled training on 8TB of data in 2 hours

Achieved 47% accuracy improvement

Increased batch throughput by 13 times

Abstract

Recent years have seen a surge in deep learning approaches to accelerate numerical solvers, which provide faithful but computationally intensive simulations of the physical world. These deep surrogates are generally trained in a supervised manner from limited amounts of data slowly generated by the same solver they intend to accelerate. We propose an open-source framework that enables the online training of these models from a large ensemble run of simulations. It leverages multiple levels of parallelism to generate rich datasets. The framework avoids I/O bottlenecks and storage issues by directly streaming the generated data. A training reservoir mitigates the inherent bias of streaming while maximizing GPU throughput. Experiment on training a fully connected network as a surrogate for the heat equation shows the proposed approach enables training on 8TB of data in 2 hours with an accuracy improved by 47% and a batch throughput multiplied by 13 compared to a traditional offline procedure.