Obtaining Favorable Layouts for Multiple Object Generation

TL;DR

This paper introduces a novel diffusion-based method for multi-object image generation that improves layout control and subject separation, addressing limitations of existing models in multi-subject scene synthesis.

Contribution

It proposes a layout-guided diffusion approach with new loss functions to better capture multiple subjects and spatial arrangements in generated images.

Findings

Enhanced fidelity in multi-object image generation

Better adherence to specified layouts and masks

Reduced overlap and clearer spatial separation

Abstract

Large-scale text-to-image models that can generate high-quality and diverse images based on textual prompts have shown remarkable success. These models aim ultimately to create complex scenes, and addressing the challenge of multi-subject generation is a critical step towards this goal. However, the existing state-of-the-art diffusion models face difficulty when generating images that involve multiple subjects. When presented with a prompt containing more than one subject, these models may omit some subjects or merge them together. To address this challenge, we propose a novel approach based on a guiding principle. We allow the diffusion model to initially propose a layout, and then we rearrange the layout grid. This is achieved by enforcing cross-attention maps (XAMs) to adhere to proposed masks and by migrating pixels from latent maps to new locations determined by us. We introduce new loss terms aimed at reducing XAM entropy for clearer spatial definition of subjects, reduce the overlap between XAMs, and ensure that XAMs align with their respective masks. We contrast our approach with several alternative methods and show that it more faithfully captures the desired concepts across a variety of text prompts.