Autoencoding beyond pixels using a learned similarity metric

TL;DR

This paper introduces a novel autoencoder that combines variational autoencoders with GANs to utilize learned feature representations for improved data similarity measurement, enhancing image quality and enabling high-level feature manipulation.

Contribution

It proposes a new autoencoding approach that replaces pixel-wise errors with learned feature-wise errors using GAN discriminators, improving data representation and interpretability.

Findings

Outperforms traditional VAEs in visual fidelity on face images

Learns embeddings allowing high-level feature modifications

Demonstrates invariance to translation and other transformations

Abstract

We present an autoencoder that leverages learned representations to better measure similarities in data space. By combining a variational autoencoder with a generative adversarial network we can use learned feature representations in the GAN discriminator as basis for the VAE reconstruction objective. Thereby, we replace element-wise errors with feature-wise errors to better capture the data distribution while offering invariance towards e.g. translation. We apply our method to images of faces and show that it outperforms VAEs with element-wise similarity measures in terms of visual fidelity. Moreover, we show that the method learns an embedding in which high-level abstract visual features (e.g. wearing glasses) can be modified using simple arithmetic.