Parallelizing Training of Deep Generative Models on Massive Scientific Datasets

TL;DR

This paper introduces a scalable training framework for deep generative models on massive scientific datasets, leveraging HPC systems and a novel tournament training method to significantly improve training speed and efficiency.

Contribution

It presents a new tournament-style training algorithm integrated with LBANN, enabling efficient parallel training of deep generative models on supercomputers.

Findings

Achieved 70.2x speedup with 64 trainers (1024 GPUs) over a single trainer.

Demonstrated effective 109% parallel efficiency.

Successfully trained complex models on multi-variate physics data from supercomputers.

Abstract

Training deep neural networks on large scientific data is a challenging task that requires enormous compute power, especially if no pre-trained models exist to initialize the process. We present a novel tournament method to train traditional as well as generative adversarial networks built on LBANN, a scalable deep learning framework optimized for HPC systems. LBANN combines multiple levels of parallelism and exploits some of the worlds largest supercomputers. We demonstrate our framework by creating a complex predictive model based on multi-variate data from high-energy-density physics containing hundreds of millions of images and hundreds of millions of scalar values derived from tens of millions of simulations of inertial confinement fusion. Our approach combines an HPC workflow and extends LBANN with optimized data ingestion and the new tournament-style training algorithm to produce a scalable neural network architecture using a CORAL-class supercomputer. Experimental results show that 64 trainers (1024 GPUs) achieve a speedup of 70.2 over a single trainer (16 GPUs) baseline, and an effective 109% parallel efficiency.