Learning Common and Salient Generative Factors Between Two Image Datasets

TL;DR

This paper introduces a novel contrastive analysis framework to separate common and dataset-specific generative factors in images, applicable to GANs and diffusion models, improving interpretability and synthesis quality.

Contribution

The paper proposes a new weakly supervised method for separating shared and salient factors across datasets, adaptable to multiple generative models, with enhanced separation and image quality.

Findings

Superior separation of common and salient factors compared to prior methods.

High-quality image synthesis across diverse datasets.

Effective adaptation to both GAN and diffusion models.

Abstract

Recent advancements in image synthesis have enabled high-quality image generation and manipulation. Most works focus on: 1) conditional manipulation, where an image is modified conditioned on a given attribute, or 2) disentangled representation learning, where each latent direction should represent a distinct semantic attribute. In this paper, we focus on a different and less studied research problem, called Contrastive Analysis (CA). Given two image datasets, we want to separate the common generative factors, shared across the two datasets, from the salient ones, specific to only one dataset. Compared to existing methods, which use attributes as supervised signals for editing (e.g., glasses, gender), the proposed method is weaker, since it only uses the dataset signal. We propose a novel framework for CA, that can be adapted to both GAN and Diffusion models, to learn both common and salient factors. By defining new and well-adapted learning strategies and losses, we ensure a relevant separation between common and salient factors, preserving a high-quality generation. We evaluate our approach on diverse datasets, covering human faces, animal images and medical scans. Our framework demonstrates superior separation ability and image quality synthesis compared to prior methods.