UniWorld-V1: High-Resolution Semantic Encoders for Unified Visual Understanding and Generation

TL;DR

UniWorld-V1 introduces high-resolution semantic encoders for unified visual understanding and generation, demonstrating strong performance across diverse tasks with limited training data, and emphasizing the importance of semantic features over VAEs.

Contribution

The paper presents UniWorld-V1, a novel unified generative framework leveraging semantic encoders from multimodal models, outperforming traditional methods in image perception and manipulation tasks.

Findings

UniWorld-V1 achieves high performance with only 2.7M training data.

Semantic encoders are more effective than VAEs for image manipulation.

Open-source release promotes reproducibility and further research.

Abstract

Although existing unified models achieve strong performance in vision-language understanding and text-to-image generation, they remain limited in addressing image perception and manipulation -- capabilities increasingly demanded in practical applications. Recently, OpenAI introduced the powerful GPT-4o-Image model, which showcases advanced capabilities in comprehensive image perception and manipulation, sparking widespread interest. Through carefully designed experiments, we observe that GPT-4o-Image likely relies on semantic encoders rather than VAEs for feature extraction, despite VAEs being commonly regarded as crucial for image manipulation tasks. Inspired by this insight, we propose UniWorld-V1, a unified generative framework built upon semantic features extracted from powerful multimodal large language models and contrastive semantic encoders. Using only 2.7M training data, UniWorld-V1 achieves impressive performance across diverse tasks, including image understanding, generation, manipulation, and perception. We fully open-source the UniWorld-V1 framework, including model weights, training and evaluation scripts, and datasets to promote reproducibility and further research.