DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference

TL;DR

DUET-VLM introduces a dual-stage token reduction framework for vision-language models, significantly reducing tokens while maintaining high accuracy, leading to more efficient multimodal understanding and reasoning.

Contribution

It proposes a versatile, plug-and-play dual compression method that preserves semantics during aggressive token reduction in VLM training and inference.

Findings

Maintains over 99% accuracy with 67% token reduction on LLaVA-1.5-7B.

Surpasses prior methods with 97.6% accuracy at 89% token reduction.

Achieves >100% accuracy with 53.1% token reduction on Video-LLaVA-7B.

Abstract

Vision-language models (VLMs) have achieved remarkable multimodal understanding and reasoning capabilities, yet remain computationally expensive due to dense visual tokenization. Existing efficiency approaches either merge redundant visual tokens or drop them progressively in language backbone, often trading accuracy for speed. In this work, we propose DUET-VLM, a versatile plug-and-play dual compression framework that consists of (a) vision-only redundancy aware compression of vision encoder's output into information-preserving tokens, followed by (b) layer-wise, salient text-guided dropping of visual tokens within the language backbone to progressively prune less informative tokens. This coordinated token management enables aggressive compression while retaining critical semantics. On LLaVA-1.5-7B, our approach maintains over 99% of baseline accuracy with 67% fewer tokens, and still retains >97% even at 89% reduction. With this dual-stage compression during training, it achieves 99.7% accuracy at 67% and 97.6% at 89%, surpassing prior SoTA visual token reduction methods across multiple benchmarks. When integrated into Video-LLaVA-7B, it even surpasses the baseline -- achieving >100% accuracy with a substantial 53.1% token reduction and retaining 97.6% accuracy under an extreme 93.4% setting. These results highlight end-to-end training with DUET-VLM, enabling robust adaptation to reduced visual (image/video) input without sacrificing accuracy, producing compact yet semantically rich representations within the same computational budget. Our code is available at https://github.com/AMD-AGI/DUET-VLM.