UniLiP: Adapting CLIP for Unified Multimodal Understanding, Generation and Editing

TL;DR

UniLIP is a unified framework that enhances CLIP with reconstruction, reasoning, and editing capabilities, enabling it to excel in multimodal understanding, generation, and editing with high fidelity and efficiency.

Contribution

The paper introduces a novel two-stage training scheme and dual-condition architecture that adapt CLIP for high-quality reconstruction, reasoning, and editing, surpassing larger models.

Findings

Achieves state-of-the-art scores on GenEval, WISE, and ImgEdit benchmarks.

Outperforms larger models like BAGEL and Uniworld-V1 with fewer parameters.

Demonstrates strong instruction following and editing fidelity.

Abstract

In this paper, we propose UniLIP, a unified framework that adapts CLIP for multimodal understanding, generation and editing. Although CLIP excels at understanding, it lacks reconstruction abilities required to be a unified visual encoder. However, previous CLIP-based unified methods fail to balance understanding and reconstruction, leading to semantic degradation or inconsistent reconstructions. In contrast, we introduce a novel two-stage training scheme with a self-distillation strategy that progressively endows CLIP with high-fidelity reconstruction abilities while preserving its original comprehension performance. For enhanced reasoning and consistency in generation and editing, we further develop a dual-condition architecture built upon the MetaQuery framework. Our architecture jointly utilizes multimodal hidden states for rich contextual details and learnable query embeddings to harness the powerful reasoning abilities of Multimodal Large Language Models (MLLMs). Leveraging advanced image representation and architectural design, UniLIP demonstrates superior instruction following and edit fidelity. With only 1B and 3B parameters, UniLIP can outperform larger unified models such as BAGEL (7B) and Uniworld-V1 (12B), achieving state-of-the-art performance of 0.90 on GenEval, 0.63 on WISE, and 3.94 on ImgEdit. These results demonstrate that UniLIP successfully expands the application of CLIP, establishing its continuous features to not only serve as the optimal choice for understanding tasks but also achieve highly competitive performance in generation and editing tasks. Code and models are available at https://github.com/nnnth/UniLIP.