How Many Visual Tokens Do Multimodal Language Models Need? Scaling Visual Token Pruning with F^3A

TL;DR

This paper introduces F^3A, a training-free method for task-conditioned visual token pruning in multimodal models, optimizing token allocation under fixed budgets without additional training or inference overhead.

Contribution

F^3A provides a novel, training-free approach to visual token pruning that improves efficiency by task-conditioned evidence search and token allocation in multimodal models.

Findings

F^3A effectively reduces visual tokens without retraining.

It maintains model performance while lowering inference costs.

F^3A operates without extra model training or inference passes.

Abstract

Vision-language models improve perception by feeding increasingly long visual token sequences into language backbones, but the resulting inference cost raises a basic scaling question: as multimodal models grow, how many visual tokens are actually needed, and how should they be allocated under a fixed visual token budget? Existing training-free pruning methods typically answer this with one-shot proxies such as decoder attention, visual similarity, or conditional diversity. We argue that visual token pruning is better viewed as task-conditioned evidence search, especially under aggressive compression and across model scales. We propose F^3A, a training-free router for visual token pruning that operates before the language model consumes image tokens. F^3A builds lightweight question-conditioned cues, matches them to visual-grid tokens through frozen sparse sensing heads, and allocates a fixed vision token budget via coarse evidence localization, local refinement, coverage-preserving competition, and recovery of under-covered regions. It requires no model training, no extra LLM forward pass and preserves the original multimodal prompting and decoding pipeline.