Scaling Training Data with Lossy Image Compression

TL;DR

This paper introduces a storage scaling law for lossy image compression in training data, enabling optimization of compression levels to improve model performance under storage constraints.

Contribution

It proposes and empirically validates a new law describing the relationship between test error, sample size, and bits per image, optimizing data compression for machine learning.

Findings

The law accurately predicts test error across compression levels.

Optimally compressed images lead to lower test error at fixed storage.

Randomizing compression levels offers potential benefits.

Abstract

Empirically-determined scaling laws have been broadly successful in predicting the evolution of large machine learning models with training data and number of parameters. As a consequence, they have been useful for optimizing the allocation of limited resources, most notably compute time. In certain applications, storage space is an important constraint, and data format needs to be chosen carefully as a consequence. Computer vision is a prominent example: images are inherently analog, but are always stored in a digital format using a finite number of bits. Given a dataset of digital images, the number of bits $L$ to store each of them can be further reduced using lossy data compression. This, however, can degrade the quality of the model trained on such images, since each example has lower resolution. In order to capture this trade-off and optimize storage of training data, we propose a `storage scaling law' that describes the joint evolution of test error with sample size and number of bits per image. We prove that this law holds within a stylized model for image compression, and verify it empirically on two computer vision tasks, extracting the relevant parameters. We then show that this law can be used to optimize the lossy compression level. At given storage, models trained on optimally compressed images present a significantly smaller test error with respect to models trained on the original data. Finally, we investigate the potential benefits of randomizing the compression level.