Harnessing the Conditioning Sensorium for Improved Image Translation

TL;DR

This paper introduces Sensorium, a method for multi-modal image translation that leverages pre-trained models to define content, enabling more flexible and higher quality translations across diverse and complex scenes.

Contribution

It proposes a novel approach that uses off-the-shelf pre-trained models to define content, simplifying training and enhancing control over image translation.

Findings

Outperforms existing methods on traditional datasets like CelebA-HQ.

Effective on complex scenes in new datasets ClassicTV and FFHQ-Wild.

Provides intuitive control over content preservation during translation.

Abstract

Multi-modal domain translation typically refers to synthesizing a novel image that inherits certain localized attributes from a 'content' image (e.g. layout, semantics, or geometry), and inherits everything else (e.g. texture, lighting, sometimes even semantics) from a 'style' image. The dominant approach to this task is attempting to learn disentangled 'content' and 'style' representations from scratch. However, this is not only challenging, but ill-posed, as what users wish to preserve during translation varies depending on their goals. Motivated by this inherent ambiguity, we define 'content' based on conditioning information extracted by off-the-shelf pre-trained models. We then train our style extractor and image decoder with an easy to optimize set of reconstruction objectives. The wide variety of high-quality pre-trained models available and simple training procedure makes our approach straightforward to apply across numerous domains and definitions of 'content'. Additionally it offers intuitive control over which aspects of 'content' are preserved across domains. We evaluate our method on traditional, well-aligned, datasets such as CelebA-HQ, and propose two novel datasets for evaluation on more complex scenes: ClassicTV and FFHQ-Wild. Our approach, Sensorium, enables higher quality domain translation for more complex scenes.