L3: Accelerator-Friendly Lossless Image Format for High-Resolution, High-Throughput DNN Training

TL;DR

This paper introduces L3, a lossless image format optimized for high-resolution DNN training, significantly increasing data preparation throughput and overall training efficiency by enabling parallel decoding on accelerators.

Contribution

L3 is a novel lightweight, lossless image format designed for high-throughput DNN training, with parallel decoding capabilities that reduce CPU bottlenecks during data preparation.

Findings

L3 achieves 9.29x higher decoding throughput than PNG on Cityscapes.

L3 improves end-to-end training throughput by up to 1.71x compared to PNG.

L3 outperforms JPEG and WebP in training throughput at similar accuracy levels.

Abstract

The training process of deep neural networks (DNNs) is usually pipelined with stages for data preparation on CPUs followed by gradient computation on accelerators like GPUs. In an ideal pipeline, the end-to-end training throughput is eventually limited by the throughput of the accelerator, not by that of data preparation. In the past, the DNN training pipeline achieved a near-optimal throughput by utilizing datasets encoded with a lightweight, lossy image format like JPEG. However, as high-resolution, losslessly-encoded datasets become more popular for applications requiring high accuracy, a performance problem arises in the data preparation stage due to low-throughput image decoding on the CPU. Thus, we propose L3, a custom lightweight, lossless image format for high-resolution, high-throughput DNN training. The decoding process of L3 is effectively parallelized on the accelerator, thus minimizing CPU intervention for data preparation during DNN training. L3 achieves a 9.29x higher data preparation throughput than PNG, the most popular lossless image format, for the Cityscapes dataset on NVIDIA A100 GPU, which leads to 1.71x higher end-to-end training throughput. Compared to JPEG and WebP, two popular lossy image formats, L3 provides up to 1.77x and 2.87x higher end-to-end training throughput for ImageNet, respectively, at equivalent metric performance.