Unified Multimodal Understanding via Byte-Pair Visual Encoding

TL;DR

This paper introduces a unified multimodal understanding framework that applies byte-pair encoding to visual tokens, enhancing cross-modal reasoning and performance in vision-language tasks.

Contribution

It proposes a novel byte-pair visual encoding method with priority-guided encoding and curriculum-driven training, unifying visual and textual representations in multimodal models.

Findings

Improved performance on diverse vision-language tasks

Better cross-modal relationship modeling

Enhanced visual token structural encoding

Abstract

Multimodal large language models (MLLMs) have made significant progress in vision-language understanding, yet effectively aligning different modalities remains a fundamental challenge. We present a framework that unifies multimodal understanding by applying byte-pair encoding to visual tokens. Unlike conventional approaches that rely on modality-specific encoders, our method directly incorporates structural information into visual tokens, mirroring successful tokenization strategies in text-only language models. We introduce a priority-guided encoding scheme that considers both frequency and spatial consistency, coupled with a multi-stage training procedure based on curriculum-driven data composition. These enhancements enable the transformer model to better capture cross-modal relationships and reason with visual information. Comprehensive experiments demonstrate improved performance across diverse vision-language tasks. By bridging the gap between visual and textual representations, our approach contributes to the advancement of more capable and efficient multimodal foundation models.