3DIS: Depth-Driven Decoupled Instance Synthesis for Text-to-Image Generation

TL;DR

3DIS introduces a two-stage, depth-driven framework for text-to-image generation that improves instance layout accuracy and attribute rendering without additional training, enhancing multi-instance generation capabilities.

Contribution

It proposes a novel decoupled approach that separates scene layout and attribute rendering, enabling robust, training-free multi-instance image synthesis with compatibility across models.

Findings

Outperforms existing methods in layout precision and attribute rendering.

Demonstrates robustness and adaptability across diverse foundational models.

Achieves significant improvements on COCO benchmarks.

Abstract

The increasing demand for controllable outputs in text-to-image generation has spurred advancements in multi-instance generation (MIG), allowing users to define both instance layouts and attributes. However, unlike image-conditional generation methods such as ControlNet, MIG techniques have not been widely adopted in state-of-the-art models like SD2 and SDXL, primarily due to the challenge of building robust renderers that simultaneously handle instance positioning and attribute rendering. In this paper, we introduce Depth-Driven Decoupled Instance Synthesis (3DIS), a novel framework that decouples the MIG process into two stages: (i) generating a coarse scene depth map for accurate instance positioning and scene composition, and (ii) rendering fine-grained attributes using pre-trained ControlNet on any foundational model, without additional training. Our 3DIS framework integrates a custom adapter into LDM3D for precise depth-based layouts and employs a finetuning-free method for enhanced instance-level attribute rendering. Extensive experiments on COCO-Position and COCO-MIG benchmarks demonstrate that 3DIS significantly outperforms existing methods in both layout precision and attribute rendering. Notably, 3DIS offers seamless compatibility with diverse foundational models, providing a robust, adaptable solution for advanced multi-instance generation. The code is available at: https://github.com/limuloo/3DIS.