From Text to Pixel: Advancing Long-Context Understanding in MLLMs

TL;DR

SEEKER is a novel multimodal large language model that efficiently encodes long textual information into visual pixel space, significantly improving long-context understanding and outperforming existing models in multimodal tasks.

Contribution

The paper introduces SEEKER, a new approach that compresses long text into images for better long-context processing in MLLMs, addressing a key limitation of current models.

Findings

SEEKER outperforms existing MLLMs in long-context tasks.

SEEKER uses fewer image tokens to encode the same textual information.

SEEKER demonstrates superior efficiency and accuracy in long-form multimodal understanding.

Abstract

The rapid progress in Multimodal Large Language Models (MLLMs) has significantly advanced their ability to process and understand complex visual and textual information. However, the integration of multiple images and extensive textual contexts remains a challenge due to the inherent limitation of the models' capacity to handle long input sequences efficiently. In this paper, we introduce SEEKER, a multimodal large language model designed to tackle this issue. SEEKER aims to optimize the compact encoding of long text by compressing the text sequence into the visual pixel space via images, enabling the model to handle long text within a fixed token-length budget efficiently. Our empirical experiments on six long-context multimodal tasks demonstrate that SEEKER can leverage fewer image tokens to convey the same amount of textual information compared with the OCR-based approach, and is more efficient in understanding long-form multimodal input and generating long-form textual output, outperforming all existing proprietary and open-source MLLMs by large margins.