OmniBooth: Learning Latent Control for Image Synthesis with Multi-modal Instruction

TL;DR

OmniBooth introduces a versatile image synthesis framework that allows precise spatial control and multi-modal customization using text and image guidance, significantly enhancing controllability and fidelity in image generation.

Contribution

The paper proposes latent control signals that unify spatial, textual, and image conditions, extending ControlNet for instance-level open-vocabulary generation and personalized identity control.

Findings

Improved image synthesis fidelity and alignment across tasks.

Enhanced controllability with multi-modal conditions.

Effective integration of spatial, textual, and image guidance.

Abstract

We present OmniBooth, an image generation framework that enables spatial control with instance-level multi-modal customization. For all instances, the multimodal instruction can be described through text prompts or image references. Given a set of user-defined masks and associated text or image guidance, our objective is to generate an image, where multiple objects are positioned at specified coordinates and their attributes are precisely aligned with the corresponding guidance. This approach significantly expands the scope of text-to-image generation, and elevates it to a more versatile and practical dimension in controllability. In this paper, our core contribution lies in the proposed latent control signals, a high-dimensional spatial feature that provides a unified representation to integrate the spatial, textual, and image conditions seamlessly. The text condition extends ControlNet to provide instance-level open-vocabulary generation. The image condition further enables fine-grained control with personalized identity. In practice, our method empowers users with more flexibility in controllable generation, as users can choose multi-modal conditions from text or images as needed. Furthermore, thorough experiments demonstrate our enhanced performance in image synthesis fidelity and alignment across different tasks and datasets. Project page: https://len-li.github.io/omnibooth-web/