FastPersist: Accelerating Model Checkpointing in Deep Learning

TL;DR

FastPersist significantly accelerates model checkpointing in deep learning by optimizing storage I/O, parallelizing writes, and overlapping checkpointing with training, achieving up to 116x faster checkpoint creation.

Contribution

FastPersist introduces three novel techniques to drastically reduce checkpointing time in deep learning training workflows.

Findings

Up to 116x faster checkpoint creation compared to baseline.

Enables per-iteration checkpointing with negligible overhead.

Effective for both dense and sparse models.

Abstract

Model checkpoints are critical Deep Learning (DL) artifacts that enable fault tolerance for training and downstream applications, such as inference. However, writing checkpoints to persistent storage, and other I/O aspects of DL training, are mostly ignored by compute-focused optimization efforts for faster training of rapidly growing models and datasets. Towards addressing this imbalance, we propose FastPersist to accelerate checkpoint creation in DL training. FastPersist combines three novel techniques: (i) NVMe optimizations for faster checkpoint writes to SSDs, (ii) efficient write parallelism using the available SSDs in training environments, and (iii) overlapping checkpointing with independent training computations. Our evaluation using real world dense and sparse DL models shows that FastPersist creates checkpoints in persistent storage up to 116x faster than baseline, and enables per-iteration checkpointing with negligible overhead.