HaploOmni: Unified Single Transformer for Multimodal Video Understanding and Generation

TL;DR

HaploOmni is a unified single transformer model that efficiently handles multimodal video understanding and generation by integrating novel training strategies and techniques, achieving competitive results with limited training costs.

Contribution

The paper introduces HaploOmni, a new single multimodal transformer with a multimodal warmup strategy and feature pre-scaling techniques, enabling efficient training and broad capabilities.

Findings

Achieves competitive performance on multiple benchmarks.

Utilizes limited training resources effectively.

Introduces novel multimodal integration techniques.

Abstract

With the advancement of language models, unified multimodal understanding and generation have made significant strides, with model architectures evolving from separated components to unified single-model frameworks. This paper explores an efficient training paradigm to build a single transformer for unified multimodal understanding and generation. Specifically, we propose a multimodal warmup strategy utilizing prior knowledge to extend capabilities. To address cross-modal compatibility challenges, we introduce feature pre-scaling and multimodal AdaLN techniques. Integrating the proposed technologies, we present the HaploOmni, a new single multimodal transformer. With limited training costs, HaploOmni achieves competitive performance across multiple image and video understanding and generation benchmarks over advanced unified models. All codes will be made public at https://github.com/Tencent/HaploVLM.