DuoGen: Towards General Purpose Interleaved Multimodal Generation

TL;DR

DuoGen is a versatile interleaved multimodal generation framework that combines large-scale instruction tuning, a novel architecture leveraging pretrained models, and comprehensive evaluation to outperform existing models in quality and fidelity.

Contribution

The paper introduces DuoGen, a new framework that integrates multimodal LLMs with diffusion transformers, along with a large high-quality dataset, to advance interleaved multimodal generation capabilities.

Findings

Outperforms prior open-source models in text quality and image fidelity.

Achieves state-of-the-art results in text-to-image and image editing tasks.

Demonstrates effective general-purpose interleaved multimodal generation.

Abstract

Interleaved multimodal generation enables capabilities beyond unimodal generation models, such as step-by-step instructional guides, visual planning, and generating visual drafts for reasoning. However, the quality of existing interleaved generation models under general instructions remains limited by insufficient training data and base model capacity. We present DuoGen, a general-purpose interleaved generation framework that systematically addresses data curation, architecture design, and evaluation. On the data side, we build a large-scale, high-quality instruction-tuning dataset by combining multimodal conversations rewritten from curated raw websites, and diverse synthetic examples covering everyday scenarios. Architecturally, DuoGen leverages the strong visual understanding of a pretrained multimodal LLM and the visual generation capabilities of a diffusion transformer (DiT) pretrained on video generation, avoiding costly unimodal pretraining and enabling flexible base model selection. A two-stage decoupled strategy first instruction-tunes the MLLM, then aligns DiT with it using curated interleaved image-text sequences. Across public and newly proposed benchmarks, DuoGen outperforms prior open-source models in text quality, image fidelity, and image-context alignment, and also achieves state-of-the-art performance on text-to-image and image editing among unified generation models. Data and code will be released at https://research.nvidia.com/labs/dir/duogen/.