Controllable Coupled Image Generation via Diffusion Models

TL;DR

This paper introduces a diffusion model-based method for generating multiple coupled images with shared backgrounds and flexible foreground objects, controlled through attention disentanglement and time-varying parameters.

Contribution

It proposes a novel attention-level control technique that disentangles background and object features for coupled image generation with adjustable background consistency.

Findings

Outperforms existing methods in background coupling and image quality

Effectively balances background similarity and foreground diversity

Demonstrates superior alignment with text prompts

Abstract

We provide an attention-level control method for the task of coupled image generation, where "coupled" means that multiple simultaneously generated images are expected to have the same or very similar backgrounds. While backgrounds coupled, the centered objects in the generated images are still expected to enjoy the flexibility raised from different text prompts. The proposed method disentangles the background and entity components in the model's cross-attention modules, attached with a sequence of time-varying weight control parameters depending on the time step of sampling. We optimize this sequence of weight control parameters with a combined objective that assesses how coupled the backgrounds are as well as text-to-image alignment and overall visual quality. Empirical results demonstrate that our method outperforms existing approaches across these criteria.