From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities

TL;DR

This paper presents a novel image tokenizer using Byte-Pair Encoding for visual data, improving multimodal understanding in large language models by integrating structural information directly into image tokens.

Contribution

The paper introduces a BPE-based image tokenizer that incorporates structural priors, enabling more effective multimodal learning without relying on separate visual encoders.

Findings

Enhanced multimodal understanding demonstrated by experiments

Superior performance of Being-VL-0 across benchmarks

Effective learning with limited training data

Abstract

Multimodal Large Language Models have made significant strides in integrating visual and textual information, yet they often struggle with effectively aligning these modalities. We introduce a novel image tokenizer that bridges this gap by applying the principle of Byte-Pair Encoding (BPE) to visual data. Unlike conventional approaches that rely on separate visual encoders, our method directly incorporates structural prior information into image tokens, mirroring the successful tokenization strategies used in text-only Large Language Models. This innovative approach enables Transformer models to more effectively learn and reason across modalities. Through theoretical analysis and extensive experiments, we demonstrate that our BPE Image Tokenizer significantly enhances MLLMs' multimodal understanding capabilities, even with limited training data. Leveraging this method, we develop Being-VL-0, a model that demonstrates superior performance across various benchmarks and shows promising scalability, potentially paving the way for more efficient and capable multimodal foundation models.