Viper-F1: Fast and Fine-Grained Multimodal Understanding with Cross-Modal State-Space Modulation

TL;DR

Viper-F1 introduces an efficient multimodal model that replaces traditional attention with state-space dynamics and a correlation module, enabling fine-grained vision-language understanding at lower computational costs.

Contribution

The paper proposes Viper-F1, a hybrid state-space model with a novel correlation module, achieving efficient and precise multimodal understanding unlike previous attention-based methods.

Findings

Outperforms existing models on multiple benchmarks.

Achieves linear-time inference with high accuracy.

Effectively captures fine-grained visual details.

Abstract

Recent advances in multimodal large language models (MLLMs) have enabled impressive progress in vision-language understanding, yet their high computational cost limits deployment in resource-constrained scenarios such as robotic manipulation, personal assistants, and smart cameras. Most existing methods rely on Transformer-based cross-attention, whose quadratic complexity hinders efficiency. Moreover, small vision-language models often struggle to precisely capture fine-grained, task-relevant visual regions, leading to degraded performance on fine-grained reasoning tasks that limit their effectiveness in the real world. To address these issues, we introduce Viper-F1, a Hybrid State-Space Vision-Language Model that replaces attention with efficient Liquid State-Space Dynamics. To further enhance visual grounding, we propose a Token-Grid Correlation Module, which computes lightweight correlations between text tokens and image patches and modulates the state-space dynamics via FiLM conditioning. This enables the model to selectively emphasize visual regions relevant to the textual prompt while maintaining linear-time inference. Experimental results across multiple benchmarks demonstrate that Viper-F1 achieves accurate, fine-grained understanding with significantly improved efficiency.