Learning to Generate Images with Perceptual Similarity Metrics

TL;DR

This paper introduces the use of the multiscale structural-similarity score (MS-SSIM) as a perceptually aligned loss function for training image synthesis networks, leading to images preferred by humans over traditional pixel-wise losses.

Contribution

It demonstrates that MS-SSIM, being differentiable, improves image quality in synthesis tasks and aligns better with human perception compared to pixel-wise loss functions.

Findings

Humans prefer images generated with MS-SSIM loss over pixel-wise loss.

MS-SSIM-optimized models outperform pixel-wise models in image reconstruction quality.

Perceptually-optimized representations enhance performance in image classification and super-resolution.

Abstract

Deep networks are increasingly being applied to problems involving image synthesis, e.g., generating images from textual descriptions and reconstructing an input image from a compact representation. Supervised training of image-synthesis networks typically uses a pixel-wise loss (PL) to indicate the mismatch between a generated image and its corresponding target image. We propose instead to use a loss function that is better calibrated to human perceptual judgments of image quality: the multiscale structural-similarity score (MS-SSIM). Because MS-SSIM is differentiable, it is easily incorporated into gradient-descent learning. We compare the consequences of using MS-SSIM versus PL loss on training deterministic and stochastic autoencoders. For three different architectures, we collected human judgments of the quality of image reconstructions. Observers reliably prefer images synthesized by MS-SSIM-optimized models over those synthesized by PL-optimized models, for two distinct PL measures ($\ell_1$ and $\ell_2$ distances). We also explore the effect of training objective on image encoding and analyze conditions under which perceptually-optimized representations yield better performance on image classification. Finally, we demonstrate the superiority of perceptually-optimized networks for super-resolution imaging. Just as computer vision has advanced through the use of convolutional architectures that mimic the structure of the mammalian visual system, we argue that significant additional advances can be made in modeling images through the use of training objectives that are well aligned to characteristics of human perception.