Analyzing I/O Performance of a Hierarchical HPC Storage System for Distributed Deep Learning

TL;DR

This paper quantitatively analyzes how hierarchical HPC storage systems impact I/O performance for large-scale distributed deep learning workloads, highlighting the potential improvements needed to meet performance goals.

Contribution

It provides a detailed performance analysis of hierarchical storage systems for DDNN workloads on supercomputers, offering insights for future HPC storage design.

Findings

Hierarchical storage can significantly improve I/O performance for DDNN workloads.

Performance gains depend on storage volume and system configuration.

Quantitative benchmarks inform future HPC storage system design.

Abstract

Today, deep learning is an essential technology for our life. To solve more complex problems with deep learning, both sizes of training datasets and neural networks are increasing. To train a model with large datasets and networks, distributed deep neural network (DDNN) training technique is necessary. For large-scale DDNN training, HPC clusters are a promising computation environment. In large-scale DDNN on HPC clusters, I/O performance is critical because it is becoming a bottleneck. Most flagship-class HPC clusters have hierarchical storage systems. For designing future HPC storage systems, it is necessary to quantify the performance improvement effect of the hierarchical storage system on the workloads. This paper demonstrates the quantitative performance analysis of the hierarchical storage system for DDNN workload in a flagship-class supercomputer. Our analysis shows how much performance improvement and volume increment of the storage will be required to meet the performance goal.