HBridge: H-Shape Bridging of Heterogeneous Experts for Unified Multimodal Understanding and Generation

TL;DR

HBridge introduces an asymmetric H-shaped architecture for multimodal models, selectively bridging layers to improve efficiency and generation quality by leveraging pretrained priors and semantic reconstruction tokens.

Contribution

The paper proposes HBridge, a novel asymmetric bridging architecture that enhances multimodal understanding and generation by selectively connecting layers and incorporating semantic tokens.

Findings

HBridge reduces attention sharing by over 40%, improving efficiency.

HBridge outperforms prior symmetric fusion models on multiple benchmarks.

Selective layer bridging enhances semantic alignment and generation quality.

Abstract

Recent unified models integrate understanding experts (e.g., LLMs) with generative experts (e.g., diffusion models), achieving strong multimodal performance. However, recent advanced methods such as BAGEL and LMFusion follow the Mixture-of-Transformers (MoT) paradigm, adopting a symmetric design that mirrors one expert to another for convenient initialization and fusion, which remains suboptimal due to inherent modality discrepancies. In this work, we propose HBridge, an asymmetric H-shaped architecture that enables heterogeneous experts to optimally leverage pretrained priors from their respective modality domains. Unlike prior dense fusion strategies that straightforwardly connect all layers between experts via shared attention, HBridge selectively bridges intermediate layers, reducing over 40% attention sharing, which improves efficiency and enhances generation quality. Shallow and deep layers, which capture modality-specific representations, are decoupled, while mid-layer bridging promotes semantic alignment. To further strengthen cross-modal coherence, we introduce semantic reconstruction tokens that explicitly guide the generative expert to reconstruct visual semantic tokens of the target image. Extensive experiments across multiple benchmarks demonstrate the effectiveness and superior performance of HBridge, establishing a new paradigm for unified multimodal generation.