MIGC++: Advanced Multi-Instance Generation Controller for Image Synthesis

TL;DR

This paper introduces MIGC++ and associated algorithms for precise multi-instance image synthesis, enabling detailed control over position, attributes, and quantity of multiple objects within a single image, addressing key challenges in attribute leakage and consistency.

Contribution

The paper presents MIGC++, a novel multi-instance generation controller that supports diverse attribute and position controls, along with algorithms ensuring consistency during iterative modifications.

Findings

Outperforms existing methods on COCO-MIG and Multimodal-MIG benchmarks.

Achieves precise control over position, attributes, and quantity.

Demonstrates robustness in iterative instance modifications.

Abstract

We introduce the Multi-Instance Generation (MIG) task, which focuses on generating multiple instances within a single image, each accurately placed at predefined positions with attributes such as category, color, and shape, strictly following user specifications. MIG faces three main challenges: avoiding attribute leakage between instances, supporting diverse instance descriptions, and maintaining consistency in iterative generation. To address attribute leakage, we propose the Multi-Instance Generation Controller (MIGC). MIGC generates multiple instances through a divide-and-conquer strategy, breaking down multi-instance shading into single-instance tasks with singular attributes, later integrated. To provide more types of instance descriptions, we developed MIGC++. MIGC++ allows attribute control through text \& images and position control through boxes \& masks. Lastly, we introduced the Consistent-MIG algorithm to enhance the iterative MIG ability of MIGC and MIGC++. This algorithm ensures consistency in unmodified regions during the addition, deletion, or modification of instances, and preserves the identity of instances when their attributes are changed. We introduce the COCO-MIG and Multimodal-MIG benchmarks to evaluate these methods. Extensive experiments on these benchmarks, along with the COCO-Position benchmark and DrawBench, demonstrate that our methods substantially outperform existing techniques, maintaining precise control over aspects including position, attribute, and quantity. Project page: https://github.com/limuloo/MIGC.