The Contextual Loss for Image Transformation with Non-Aligned Data

TL;DR

This paper introduces a novel loss function for image transformation tasks that does not require spatial alignment of training data, enabling effective style and content transfer even with unaligned image pairs.

Contribution

It proposes a contextual loss based on semantic and contextual similarity, allowing training on unaligned image pairs for various image transformation tasks.

Findings

Effective for unaligned image data

Preserves semantic regions during style transfer

Applicable to diverse image transformation problems

Abstract

Feed-forward CNNs trained for image transformation problems rely on loss functions that measure the similarity between the generated image and a target image. Most of the common loss functions assume that these images are spatially aligned and compare pixels at corresponding locations. However, for many tasks, aligned training pairs of images will not be available. We present an alternative loss function that does not require alignment, thus providing an effective and simple solution for a new space of problems. Our loss is based on both context and semantics -- it compares regions with similar semantic meaning, while considering the context of the entire image. Hence, for example, when transferring the style of one face to another, it will translate eyes-to-eyes and mouth-to-mouth. Our code can be found at https://www.github.com/roimehrez/contextualLoss