The Utility of Decorrelating Colour Spaces in Vector Quantised Variational Autoencoders

TL;DR

This paper investigates how converting images into decorrelated colour spaces affects the performance and interpretability of vector quantised variational autoencoders, showing improvements in colour accuracy and classification tasks.

Contribution

It introduces a colour space conversion task for VQ-VAE, demonstrating that decorrelated colour spaces improve representation quality and interpretability of embeddings.

Findings

Decorrelated colour spaces reduce colour difference by 1-2 Delta-E.

Classification accuracy improves by 5-10% with colour space conversion.

Embedding space becomes more interpretable in colour opponent models.

Abstract

Vector quantised variational autoencoders (VQ-VAE) are characterised by three main components: 1) encoding visual data, 2) assigning $k$ different vectors in the so-called embedding space, and 3) decoding the learnt features. While images are often represented in RGB colour space, the specific organisation of colours in other spaces also offer interesting features, e.g. CIE L*a*b* decorrelates chromaticity into opponent axes. In this article, we propose colour space conversion, a simple quasi-unsupervised task, to enforce a network learning structured representations. To this end, we trained several instances of VQ-VAE whose input is an image in one colour space, and its output in another, e.g. from RGB to CIE L*a*b* (in total five colour spaces were considered). We examined the finite embedding space of trained networks in order to disentangle the colour representation in VQ-VAE models. Our analysis suggests that certain vectors encode hue and others luminance information. We further evaluated the quality of reconstructed images at low-level using pixel-wise colour metrics, and at high-level by inputting them to image classification and scene segmentation networks. We conducted experiments in three benchmark datasets: ImageNet, COCO and CelebA. Our results show, with respect to the baseline network (whose input and output are RGB), colour conversion to decorrelated spaces obtains 1-2 Delta-E lower colour difference and 5-10% higher classification accuracy. We also observed that the learnt embedding space is easier to interpret in colour opponent models.