Efficient training for compact compression models via sequential distillation

TL;DR

This paper introduces a sequential distillation method to train lightweight autoencoder models for image compression, improving stability and efficiency while maintaining quality in resource-constrained settings.

Contribution

It presents a novel sequential distillation approach that enhances training stability and performance of compact compression models compared to traditional methods.

Findings

Lightweight autoencoders retain quality with the proposed method.

Sequential distillation improves training stability in resource-limited environments.

The approach preserves reconstruction quality better than standard training.

Abstract

Deep learning models for image compression often face practical limitations in hardware-constrained applications. Although these models achieve high-quality reconstructions, they are typically complex, heavyweight, and require substantial training data and computational resources. We propose a methodology to significantly reduce autoencoder-based compression networks in a more stable Knowledge Distillation process. The intuition is that highly reduced architectures benefit from simplified optimization objectives in early training, with complexity gradually introduced later. Therefore, our approach begins with a sequential encoder--decoder distillation stage that provides a robust initialization for the lightweight model. This is followed by standard training that can be regularized with latent distillation. We evaluate the resulting lightweight autoencoders across two different architectures on the image compression task. Experiments show that our method preserves reconstruction quality and statistical fidelity in early epochs better than training lightweight autoencoders with the original loss, making it practical for resource-limited environments.