Progressive Compressed Records: Taking a Byte out of Deep Learning Data

TL;DR

Progressive Compressed Records (PCRs) introduce a novel data format that uses progressive compression to reduce data fetching overhead, enabling faster deep learning training without increasing dataset size.

Contribution

PCRs combine progressive compression with an efficient storage layout, allowing datasets to be viewed at multiple fidelities without dataset size increase, improving training efficiency.

Findings

Datasets can tolerate over 50% compression for many tasks.

Automatic selection of compression levels is feasible and efficient.

PCRs can halve training bandwidth, potentially doubling training speed.

Abstract

Deep learning accelerators efficiently train over vast and growing amounts of data, placing a newfound burden on commodity networks and storage devices. A common approach to conserve bandwidth involves resizing or compressing data prior to training. We introduce Progressive Compressed Records (PCRs), a data format that uses compression to reduce the overhead of fetching and transporting data, effectively reducing the training time required to achieve a target accuracy. PCRs deviate from previous storage formats by combining progressive compression with an efficient storage layout to view a single dataset at multiple fidelities---all without adding to the total dataset size. We implement PCRs and evaluate them on a range of datasets, training tasks, and hardware architectures. Our work shows that: (i) the amount of compression a dataset can tolerate exceeds 50% of the original encoding for many DL training tasks; (ii) it is possible to automatically and efficiently select appropriate compression levels for a given task; and (iii) PCRs enable tasks to readily access compressed data at runtime---utilizing as little as half the training bandwidth and thus potentially doubling training speed.