DreamOmni: Unified Image Generation and Editing

TL;DR

DreamOmni is a unified model that combines image generation and editing capabilities, leveraging synthetic data and joint training to improve performance across tasks in computer vision.

Contribution

The paper introduces DreamOmni, a novel unified framework that integrates text-to-image generation and editing tasks, along with a synthetic data pipeline for efficient dataset creation.

Findings

Enhanced image editing performance through joint training

Effective synthetic data pipeline for high-quality datasets

Unified framework improves generation and editing quality

Abstract

Currently, the success of large language models (LLMs) illustrates that a unified multitasking approach can significantly enhance model usability, streamline deployment, and foster synergistic benefits across different tasks. However, in computer vision, while text-to-image (T2I) models have significantly improved generation quality through scaling up, their framework design did not initially consider how to unify with downstream tasks, such as various types of editing. To address this, we introduce DreamOmni, a unified model for image generation and editing. We begin by analyzing existing frameworks and the requirements of downstream tasks, proposing a unified framework that integrates both T2I models and various editing tasks. Furthermore, another key challenge is the efficient creation of high-quality editing data, particularly for instruction-based and drag-based editing. To this end, we develop a synthetic data pipeline using sticker-like elements to synthesize accurate, high-quality datasets efficiently, which enables editing data scaling up for unified model training. For training, DreamOmni jointly trains T2I generation and downstream tasks. T2I training enhances the model's understanding of specific concepts and improves generation quality, while editing training helps the model grasp the nuances of the editing task. This collaboration significantly boosts editing performance. Extensive experiments confirm the effectiveness of DreamOmni. The code and model will be released.