An Empirical Analysis of Recurrent Learning Algorithms In Neural Lossy Image Compression Systems

TL;DR

This paper systematically compares various training algorithms for neural image compression models, revealing that sparse attentive backtracking (SAB) can outperform traditional backpropagation through time in terms of convergence speed and image quality.

Contribution

It provides the first large-scale evaluation of training strategies for neural lossy image compression, highlighting SAB's advantages over BPTT and other methods.

Findings

SAB outperforms BPTT in convergence speed and peak signal-to-noise ratio.

Hybrid neural decoders trained with SAB achieve better results than previous models.

Evaluation across six datasets demonstrates the robustness of SAB in neural compression training.

Abstract

Recent advances in deep learning have resulted in image compression algorithms that outperform JPEG and JPEG 2000 on the standard Kodak benchmark. However, they are slow to train (due to backprop-through-time) and, to the best of our knowledge, have not been systematically evaluated on a large variety of datasets. In this paper, we perform the first large-scale comparison of recent state-of-the-art hybrid neural compression algorithms, while exploring the effects of alternative training strategies (when applicable). The hybrid recurrent neural decoder is a former state-of-the-art model (recently overtaken by a Google model) that can be trained using backprop-through-time (BPTT) or with alternative algorithms like sparse attentive backtracking (SAB), unbiased online recurrent optimization (UORO), and real-time recurrent learning (RTRL). We compare these training alternatives along with the Google models (GOOG and E2E) on 6 benchmark datasets. Surprisingly, we found that the model trained with SAB performs better (outperforming even BPTT), resulting in faster convergence and a better peak signal-to-noise ratio.