Orthus: Autoregressive Interleaved Image-Text Generation with Modality-Specific Heads

TL;DR

Orthus is a novel autoregressive transformer that effectively generates images from text, answers questions based on visual inputs, and creates complex interleaved image-text content by handling discrete and continuous modalities with modality-specific heads.

Contribution

The paper introduces Orthus, a new AR multimodal model with modality-specific heads that improves image-text generation and understanding, using a soft alternative to VQ and a diffusion head for continuous image features.

Findings

Outperforms baselines like Show-o and Chameleon on standard benchmarks.

Achieves a GenEval score of 0.58 and MME-P score of 1265.8 with 7B parameters.

Efficient training within 72 A100 GPU hours.

Abstract

We introduce Orthus, an autoregressive (AR) transformer that excels in generating images given textual prompts, answering questions based on visual inputs, and even crafting lengthy image-text interleaved contents. Unlike prior arts on unified multimodal modeling, Orthus simultaneously copes with discrete text tokens and continuous image features under the AR modeling principle. The continuous treatment of visual signals minimizes the information loss for both image understanding and generation while the fully AR formulation renders the characterization of the correlation between modalities straightforward. The key mechanism enabling Orthus to leverage these advantages lies in its modality-specific heads -- one regular language modeling (LM) head predicts discrete text tokens and one diffusion head generates continuous image features conditioning on the output of the backbone. We devise an efficient strategy for building Orthus -- by substituting the Vector Quantization (VQ) operation in the existing unified AR model with a soft alternative, introducing a diffusion head, and tuning the added modules to reconstruct images, we can create an Orthus-base model effortlessly (e.g., within mere 72 A100 GPU hours). Orthus-base can further embrace post-training to better model interleaved images and texts. Empirically, Orthus surpasses competing baselines including Show-o and Chameleon across standard benchmarks, achieving a GenEval score of 0.58 and an MME-P score of 1265.8 using 7B parameters. Orthus also shows exceptional mixed-modality generation capabilities, reflecting the potential for handling intricate practical generation tasks.