UniCMs: A Unified Consistency Model For Efficient Multimodal Generation and Understanding

TL;DR

UniCMs introduces a unified consistency model for multimodal generation and understanding, achieving superior performance and faster sampling in text-to-image and image-to-text tasks by combining discrete diffusion and autoregressive decoding.

Contribution

The paper proposes a novel unified consistency model that integrates discrete diffusion and autoregressive decoding for multimodal tasks, improving efficiency and performance.

Findings

Outperforms SD3 on GenEval, Image Reward, and CLIP Score

Requires only 1/8 of the sampling time compared to SD3

Surpasses Show-o on the MMMU benchmark with faster long-sequence generation

Abstract

Consistency models (CMs) have shown promise in the efficient generation of both image and text. This raises the natural question of whether we can learn a unified CM for efficient multimodal generation (e.g., text-to-image) and understanding (e.g., image-to-text). Intuitively, such a model could be acquired by applying the consistency distillation (CD) to existing unified multimodal models. However, the key challenge is establishing a unified denoising perspective for both image and text generation, which is essential for establishing the consistency mapping. To tackle this, at the representation level, we advocate for discrete tokens for both modalities to best preserve language modeling capabilities. Critically, instead of defining the text denoising trajectory via recent discrete diffusion language modeling principles, we specify it using the parallel decoding trace of an autoregressive language model, benefiting from the latter's superior performance in general text generation tasks. The denoising trajectory of image tokens adheres to standard discrete diffusion. We train our unified consistency models (UniCMs) on these combined multimodal trajectories simultaneously with a unified objective. We introduce a trajectory segmentation strategy to further improve the training convergence. Empirically, in text-to-image generation, UniCMs outperform SD3 on GenEval, Image Reward, and CLIP Score metrics, while requiring only approximately ${1}/{8}$ of the sampling time. Meanwhile, in image-to-text generation, UniCMs surpass Show-o on the MMMU benchmark while being $1.5 \times$ faster at long-sequence generating speed. The code is available at https://github.com/zhijie-group/UniCMs.