Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generation

TL;DR

Mogao is a novel omni-modal foundation model that enables interleaved multi-modal generation of text and images through a causal, unified framework with advanced architectural features and large-scale training.

Contribution

The paper introduces Mogao, a unified model for interleaved multi-modal generation, combining autoregressive and diffusion techniques with new architectural innovations and a large-scale training dataset.

Findings

Achieves state-of-the-art multi-modal understanding and generation.

Excels in zero-shot image editing and compositional tasks.

Produces high-quality, coherent interleaved outputs.

Abstract

Recent progress in unified models for image understanding and generation has been impressive, yet most approaches remain limited to single-modal generation conditioned on multiple modalities. In this paper, we present Mogao, a unified framework that advances this paradigm by enabling interleaved multi-modal generation through a causal approach. Mogao integrates a set of key technical improvements in architecture design, including a deep-fusion design, dual vision encoders, interleaved rotary position embeddings, and multi-modal classifier-free guidance, which allow it to harness the strengths of both autoregressive models for text generation and diffusion models for high-quality image synthesis. These practical improvements also make Mogao particularly effective to process interleaved sequences of text and images arbitrarily. To further unlock the potential of unified models, we introduce an efficient training strategy on a large-scale, in-house dataset specifically curated for joint text and image generation. Extensive experiments show that Mogao not only achieves state-of-the-art performance in multi-modal understanding and text-to-image generation, but also excels in producing high-quality, coherent interleaved outputs. Its emergent capabilities in zero-shot image editing and compositional generation highlight Mogao as a practical omni-modal foundation model, paving the way for future development and scaling the unified multi-modal systems.