MuLan: Multimodal-LLM Agent for Progressive and Interactive Multi-Object Diffusion

TL;DR

MuLan is a training-free multimodal LLM agent that progressively generates multi-object images with intricate spatial and attribute control, enabling better multi-object image synthesis and human-AI collaboration.

Contribution

MuLan introduces a novel, training-free, multi-step approach combining LLM and VLM to generate multi-object images with precise spatial and attribute control, enhancing flexibility and collaboration.

Findings

Outperforms baselines in multi-object image generation

Enables interactive human-in-the-loop editing

Demonstrates superior creativity and control

Abstract

Existing text-to-image models still struggle to generate images of multiple objects, especially in handling their spatial positions, relative sizes, overlapping, and attribute bindings. To efficiently address these challenges, we develop a training-free Multimodal-LLM agent (MuLan), as a human painter, that can progressively generate multi-object with intricate planning and feedback control. MuLan harnesses a large language model (LLM) to decompose a prompt to a sequence of sub-tasks, each generating only one object by stable diffusion, conditioned on previously generated objects. Unlike existing LLM-grounded methods, MuLan only produces a high-level plan at the beginning while the exact size and location of each object are determined upon each sub-task by an LLM and attention guidance. Moreover, MuLan adopts a vision-language model (VLM) to provide feedback to the image generated in each sub-task and control the diffusion model to re-generate the image if it violates the original prompt. Hence, each model in every step of MuLan only needs to address an easy sub-task it is specialized for. The multi-step process also allows human users to monitor the generation process and make preferred changes at any intermediate step via text prompts, thereby improving the human-AI collaboration experience. We collect 200 prompts containing multi-objects with spatial relationships and attribute bindings from different benchmarks to evaluate MuLan. The results demonstrate the superiority of MuLan in generating multiple objects over baselines and its creativity when collaborating with human users. The code is available at https://github.com/measure-infinity/mulan-code.