Cheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation

TL;DR

Cheers introduces a unified multimodal model that decouples patch details from semantics, enabling efficient and high-quality visual understanding and generation with improved fidelity and reduced training costs.

Contribution

The paper presents Cheers, a novel model that unifies multimodal tasks by decoupling visual patch details from semantic representations, enhancing efficiency and performance.

Findings

Matches or surpasses state-of-the-art in visual understanding and generation.

Achieves 4x token compression for high-resolution image processing.

Outperforms Tar-1.5B on GenEval and MMBench benchmarks with less training cost.

Abstract

A recent cutting-edge topic in multimodal modeling is to unify visual comprehension and generation within a single model. However, the two tasks demand mismatched decoding regimes and visual representations, making it non-trivial to jointly optimize within a shared feature space. In this work, we present Cheers, a unified multimodal model that decouples patch-level details from semantic representations, thereby stabilizing semantics for multimodal understanding and improving fidelity for image generation via gated detail residuals. Cheers includes three key components: (i) a unified vision tokenizer that encodes and compresses image latent states into semantic tokens for efficient LLM conditioning, (ii) an LLM-based Transformer that unifies autoregressive decoding for text generation and diffusion decoding for image generation, and (iii) a cascaded flow matching head that decodes visual semantics first and then injects semantically gated detail residuals from the vision tokenizer to refine high-frequency content. Experiments on popular benchmarks demonstrate that Cheers matches or surpasses advanced UMMs in both visual understanding and generation. Cheers also achieves 4x token compression, enabling more efficient high-resolution image encoding and generation. Notably, Cheers outperforms the Tar-1.5B on the popular benchmarks GenEval and MMBench, while requiring only 20% of the training cost, indicating effective and efficient (i.e., 4x token compression) unified multimodal modeling. We will release all code and data for future research.