PPE: Positional Preservation Embedding for Token Compression in Multimodal Large Language Models

TL;DR

This paper introduces PPE, a novel positional preservation embedding that maintains spatiotemporal structure during token compression in multimodal large language models, leading to improved performance across various vision-language tasks.

Contribution

PPE is a parameter-free, generic encoding operator that explicitly preserves positional information during token merging, enhancing the effectiveness of token compression in MLLMs.

Findings

Achieves 2-5% performance improvements on multiple benchmarks.

Supports cascade clustering for progressive token compression.

Effectively preserves spatial and temporal cues during token reduction.

Abstract

Multimodal large language models (MLLMs) have achieved strong performance on vision-language tasks, yet often suffer from inefficiencies due to redundant visual tokens. Existing token merging methods reduce sequence length but frequently disrupt spatial layouts and temporal continuity by disregarding positional relationships. In this work, we propose a novel encoding operator dubbed as \textbf{P}ositional \textbf{P}reservation \textbf{E}mbedding (\textbf{PPE}), which has the main hallmark of preservation of spatiotemporal structure during visual token compression. PPE explicitly introduces the disentangled encoding of 3D positions in the token dimension, enabling each compressed token to encapsulate different positions from multiple original tokens. Furthermore, we show that PPE can effectively support cascade clustering -- a progressive token compression strategy that leads to better performance retention. PPE is a parameter-free and generic operator that can be seamlessly integrated into existing token merging methods without any adjustments. Applied to state-of-the-art token merging framework, PPE achieves consistent improvements of $2\%\sim5\%$ across multiple vision-language benchmarks, including MMBench (general vision understanding), TextVQA (layout understanding) and VideoMME (temporal understanding). These results demonstrate that preserving positional cues is critical for efficient and effective MLLM reasoning. Our code is available at https://github.com/MouxiaoHuang/PPE.