Sample what you cant compress

TL;DR

This paper introduces SWYCC, a novel autoencoder framework combining diffusion models with representation learning, achieving higher compression, improved reconstruction quality, and more flexible generation than GAN-based methods.

Contribution

It is the first to jointly learn a continuous encoder and decoder using diffusion-based loss, demonstrating superior performance and easier tuning compared to existing GAN-based autoencoders.

Findings

Better reconstruction quality than GAN-based autoencoders

Easier to tune than adversarial methods

Representation more amenable to latent diffusion modeling

Abstract

For learned image representations, basic autoencoders often produce blurry results. Reconstruction quality can be improved by incorporating additional penalties such as adversarial (GAN) and perceptual losses. Arguably, these approaches lack a principled interpretation. Concurrently, in generative settings diffusion has demonstrated a remarkable ability to create crisp, high quality results and has solid theoretical underpinnings (from variational inference to direct study as the Fisher Divergence). Our work combines autoencoder representation learning with diffusion and is, to our knowledge, the first to demonstrate jointly learning a continuous encoder and decoder under a diffusion-based loss and showing that it can lead to higher compression and better generation. We demonstrate that this approach yields better reconstruction quality as compared to GAN-based autoencoders while being easier to tune. We also show that the resulting representation is easier to model with a latent diffusion model as compared to the representation obtained from a state-of-the-art GAN-based loss. Since our decoder is stochastic, it can generate details not encoded in the otherwise deterministic latent representation; we therefore name our approach "Sample what you can't compress", or SWYCC for short.