Progressive Energy-Based Cooperative Learning for Multi-Domain Image-to-Image Translation

TL;DR

This paper introduces a novel energy-based cooperative learning framework for multi-domain image-to-image translation, enabling diversified and stylized image generation through a joint training process involving multiple components.

Contribution

The paper proposes a new energy-based cooperative learning framework with four components for multi-domain image translation, including a multi-head energy-based descriptor and a diversified image generator.

Findings

Effective multi-domain image translation with diversified outputs.

Joint training via multi-domain MCMC teaching improves model performance.

Framework achieves realistic and diverse stylized images.

Abstract

This paper studies a novel energy-based cooperative learning framework for multi-domain image-to-image translation. The framework consists of four components: descriptor, translator, style encoder, and style generator. The descriptor is a multi-head energy-based model that represents a multi-domain image distribution. The components of translator, style encoder, and style generator constitute a diversified image generator. Specifically, given an input image from a source domain, the translator turns it into a stylised output image of the target domain according to a style code, which can be inferred by the style encoder from a reference image or produced by the style generator from a random noise. Since the style generator is represented as an domain-specific distribution of style codes, the translator can provide a one-to-many transformation (i.e., diversified generation) between source domain and target domain. To train our framework, we propose a likelihood-based multi-domain cooperative learning algorithm to jointly train the multi-domain descriptor and the diversified image generator (including translator, style encoder, and style generator modules) via multi-domain MCMC teaching, in which the descriptor guides the diversified image generator to shift its probability density toward the data distribution, while the diversified image generator uses its randomly translated images to initialize the descriptor's Langevin dynamics process for efficient sampling.