ViCA: Efficient Multimodal LLMs with Vision-Only Cross-Attention

TL;DR

ViCA introduces a minimal multimodal language model architecture that significantly reduces visual processing overhead by using sparse cross-attention, maintaining high accuracy while greatly improving inference speed.

Contribution

The paper proposes ViCA, a novel architecture that bypasses dense visual self-attention, leading to more efficient multimodal LLMs with minimal accuracy loss.

Findings

Preserves 98% of baseline accuracy

Reduces visual-side computation to 4%

Achieves over 3.5x speedup in single-batch inference

Abstract

Modern multimodal large language models (MLLMs) adopt a unified self-attention design that processes visual and textual tokens at every Transformer layer, incurring substantial computational overhead. In this work, we revisit the necessity of such dense visual processing and show that projected visual embeddings are already well-aligned with the language space, while effective vision-language interaction occurs in only a small subset of layers. Based on these insights, we propose ViCA (Vision-only Cross-Attention), a minimal MLLM architecture in which visual tokens bypass all self-attention and feed-forward layers, interacting with text solely through sparse cross-attention at selected layers. Extensive evaluations across three MLLM backbones, nine multimodal benchmarks, and 26 pruning-based baselines show that ViCA preserves 98% of baseline accuracy while reducing visual-side computation to 4%, consistently achieving superior performance-efficiency trade-offs. Moreover, ViCA provides a regular, hardware-friendly inference pipeline that yields over 3.5x speedup in single-batch inference and over 10x speedup in multi-batch inference, reducing visual grounding to near-zero overhead compared with text-only LLMs. It is also orthogonal to token pruning methods and can be seamlessly combined for further efficiency gains. Our code is available at https://github.com/EIT-NLP/ViCA.