When LLaVA Meets Objects: Token Composition for Vision-Language-Models

TL;DR

Mask-LLaVA introduces a flexible vision-language model that combines multi-level visual features to reduce token usage, enabling efficient inference without retraining and maintaining competitive performance.

Contribution

It proposes a novel framework that combines mask-based object representations with global and local tokens, allowing dynamic token reduction during inference.

Findings

Achieves competitive benchmark results with fewer tokens.

Enables dynamic token selection at test time.

Maintains performance without retraining when reducing tokens.

Abstract

Current autoregressive Vision Language Models (VLMs) usually rely on a large number of visual tokens to represent images, resulting in a need for more compute especially at inference time. To address this problem, we propose Mask-LLaVA, a framework that leverages different levels of visual features to create a compact yet information-rich visual representation for autoregressive VLMs. Namely, we combine mask-based object representations together with global tokens and local patch tokens. While all tokens are used during training, it shows that the resulting model can flexibly drop especially the number of mask-based object-tokens at test time, allowing to adapt the number of tokens during inference without the need to retrain the model and without a significant drop in performance. We evaluate the proposed approach on a suite of standard benchmarks showing results competitive to current token efficient methods and comparable to the original LLaVA baseline using only a fraction of visual tokens. Our analysis demonstrates that combining multi-level features enables efficient learning with fewer tokens while allowing dynamic token selection at test time for good performance.