Vector quantization loss analysis in VQGANs: a single-GPU ablation study for image-to-image synthesis

TL;DR

This paper conducts an ablation study on VQGANs for image-to-image synthesis using a single GPU, analyzing how vector quantization loss and various parameters influence reconstruction quality and artifacts.

Contribution

It provides detailed insights into the effects of codebook size, latent dimensions, and positional encodings on VQGAN performance under resource constraints.

Findings

VQGAN behavior varies significantly with codebook size and latent dimensions.

Introducing 2D positional encodings reduces artifacts and improves image quality.

Optimal parameter settings depend on dataset size and desired balance between clarity and overfitting.

Abstract

This study performs an ablation analysis of Vector Quantized Generative Adversarial Networks (VQGANs), concentrating on image-to-image synthesis utilizing a single NVIDIA A100 GPU. The current work explores the nuanced effects of varying critical parameters including the number of epochs, image count, and attributes of codebook vectors and latent dimensions, specifically within the constraint of limited resources. Notably, our focus is pinpointed on the vector quantization loss, keeping other hyperparameters and loss components (GAN loss) fixed. This was done to delve into a deeper understanding of the discrete latent space, and to explore how varying its size affects the reconstruction. Though, our results do not surpass the existing benchmarks, however, our findings shed significant light on VQGAN's behaviour for a smaller dataset, particularly concerning artifacts, codebook size optimization, and comparative analysis with Principal Component Analysis (PCA). The study also uncovers the promising direction by introducing 2D positional encodings, revealing a marked reduction in artifacts and insights into balancing clarity and overfitting.