Conditional Diffusion on Web-Scale Image Pairs leads to Diverse Image Variations

TL;DR

This paper introduces Semantica, a diffusion model trained on web-scale image pairs for diverse and contextually consistent image variations, highlighting a new pretraining strategy and evaluation metrics.

Contribution

The paper presents a novel pretraining approach for diffusion models using web-scale image pairs to generate diverse image variations with semantic consistency.

Findings

Semantica can generate diverse variations from dataset images.

The model's performance depends on the choice of image encoder.

Proposed metrics better evaluate image variations than standard metrics.

Abstract

Generating image variations, where a model produces variations of an input image while preserving the semantic context has gained increasing attention. Current image variation techniques involve adapting a text-to-image model to reconstruct an input image conditioned on the same image. We first demonstrate that a diffusion model trained to reconstruct an input image from frozen embeddings, can reconstruct the image with minor variations. Second, inspired by how text-to-image models learn from web-scale text-image pairs, we explore a new pretraining strategy to generate image variations using a large collection of image pairs. Our diffusion model \textit{Semantica} receives a random (encoded) image from a webpage as conditional input and denoises another noisy random image from the same webpage. We carefully examine various design choices for the image encoder, given its crucial role in extracting relevant context from the input image. Once trained, \textit{Semantica} can adaptively generate new images from a dataset by simply using images from that dataset as input. Finally, we identify limitations in standard image consistency metrics for evaluating image variations and propose alternative metrics based on few-shot generation.