BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset

TL;DR

This paper introduces BLIP3-o, a unified multimodal model that combines image understanding and generation using a diffusion transformer and a sequential training strategy, achieving state-of-the-art results and open-sourcing all resources.

Contribution

The paper presents a novel diffusion transformer architecture, a sequential pretraining approach, and a curated dataset for unified multimodal modeling, advancing both image understanding and generation capabilities.

Findings

Higher training efficiency and improved generative quality with diffusion transformer.

Sequential pretraining preserves understanding while enhancing generation.

State-of-the-art performance on multiple benchmarks.

Abstract

Unifying image understanding and generation has gained growing attention in recent research on multimodal models. Although design choices for image understanding have been extensively studied, the optimal model architecture and training recipe for a unified framework with image generation remain underexplored. Motivated by the strong potential of autoregressive and diffusion models for high-quality generation and scalability, we conduct a comprehensive study of their use in unified multimodal settings, with emphasis on image representations, modeling objectives, and training strategies. Grounded in these investigations, we introduce a novel approach that employs a diffusion transformer to generate semantically rich CLIP image features, in contrast to conventional VAE-based representations. This design yields both higher training efficiency and improved generative quality. Furthermore, we demonstrate that a sequential pretraining strategy for unified models-first training on image understanding and subsequently on image generation-offers practical advantages by preserving image understanding capability while developing strong image generation ability. Finally, we carefully curate a high-quality instruction-tuning dataset BLIP3o-60k for image generation by prompting GPT-4o with a diverse set of captions covering various scenes, objects, human gestures, and more. Building on our innovative model design, training recipe, and datasets, we develop BLIP3-o, a suite of state-of-the-art unified multimodal models. BLIP3-o achieves superior performance across most of the popular benchmarks spanning both image understanding and generation tasks. To facilitate future research, we fully open-source our models, including code, model weights, training scripts, and pretraining and instruction tuning datasets.