SynerGen-VL: Towards Synergistic Image Understanding and Generation with Vision Experts and Token Folding

TL;DR

SynerGen-VL is a simple, encoder-free multimodal model that effectively combines image understanding and generation, using novel token folding and progressive alignment strategies to improve performance and training efficiency.

Contribution

It introduces token folding and vision-expert-based pretraining to enhance encoder-free MLLMs, achieving competitive results with simpler architecture.

Findings

Outperforms existing encoder-free MLLMs in various tasks.

Reduces training complexity while maintaining high-resolution understanding.

Narrower gap with task-specific state-of-the-art models.

Abstract

The remarkable success of Large Language Models (LLMs) has extended to the multimodal domain, achieving outstanding performance in image understanding and generation. Recent efforts to develop unified Multimodal Large Language Models (MLLMs) that integrate these capabilities have shown promising results. However, existing approaches often involve complex designs in model architecture or training pipeline, increasing the difficulty of model training and scaling. In this paper, we propose SynerGen-VL, a simple yet powerful encoder-free MLLM capable of both image understanding and generation. To address challenges identified in existing encoder-free unified MLLMs, we introduce the token folding mechanism and the vision-expert-based progressive alignment pretraining strategy, which effectively support high-resolution image understanding while reducing training complexity. After being trained on large-scale mixed image-text data with a unified next-token prediction objective, SynerGen-VL achieves or surpasses the performance of existing encoder-free unified MLLMs with comparable or smaller parameter sizes, and narrows the gap with task-specific state-of-the-art models, highlighting a promising path toward future unified MLLMs. Our code and models shall be released.