Hierarchical Residual Learning Based Vector Quantized Variational Autoencoder for Image Reconstruction and Generation

TL;DR

The paper introduces HR-VQVAE, a hierarchical vector quantized autoencoder that learns layered discrete representations for improved image reconstruction and generation, outperforming existing models in quality and diversity.

Contribution

It presents a novel hierarchical residual learning framework with a new objective function, enabling better discrete representations and addressing codebook collapse issues.

Findings

Reconstructs high-quality images with less distortion.

Generates diverse, high-quality images surpassing state-of-the-art models.

Reduces decoding time and scales codebook size effectively.

Abstract

We propose a multi-layer variational autoencoder method, we call HR-VQVAE, that learns hierarchical discrete representations of the data. By utilizing a novel objective function, each layer in HR-VQVAE learns a discrete representation of the residual from previous layers through a vector quantized encoder. Furthermore, the representations at each layer are hierarchically linked to those at previous layers. We evaluate our method on the tasks of image reconstruction and generation. Experimental results demonstrate that the discrete representations learned by HR-VQVAE enable the decoder to reconstruct high-quality images with less distortion than the baseline methods, namely VQVAE and VQVAE-2. HR-VQVAE can also generate high-quality and diverse images that outperform state-of-the-art generative models, providing further verification of the efficiency of the learned representations. The hierarchical nature of HR-VQVAE i) reduces the decoding search time, making the method particularly suitable for high-load tasks and ii) allows to increase the codebook size without incurring the codebook collapse problem.