An Efficient Compression of Deep Neural Network Checkpoints Based on Prediction and Context Modeling

TL;DR

This paper introduces a novel checkpoint compression method for deep neural networks that combines prediction, pruning, and quantization to significantly reduce storage needs while maintaining model performance.

Contribution

It presents a new compression technique that integrates prediction-based arithmetic coding with pruning and quantization for neural network checkpoints.

Findings

Achieves substantial bit size reduction.

Enables near-lossless recovery of training states.

Preserves model performance after compression.

Abstract

This paper is dedicated to an efficient compression of weights and optimizer states (called checkpoints) obtained at different stages during a neural network training process. First, we propose a prediction-based compression approach, where values from the previously saved checkpoint are used for context modeling in arithmetic coding. Second, in order to enhance the compression performance, we also propose to apply pruning and quantization of the checkpoint values. Experimental results show that our approach achieves substantial bit size reduction, while enabling near-lossless training recovery from restored checkpoints, preserving the model's performance and making it suitable for storage-limited environments.