CASA: Cross-Attention over Self-Attention for Efficient Vision-Language Fusion

TL;DR

This paper investigates cross-attention mechanisms in vision-language models, demonstrating their efficiency and competitiveness over token insertion methods, especially for real-time video captioning applications.

Contribution

It provides a detailed analysis of cross-attention versus self-attention, trains effective cross-attention VLMs, and showcases their advantages in real-time video captioning.

Findings

Cross-attention models are more efficient than token insertion in VLMs.

Simple cross-attention approaches can match or outperform token insertion methods.

Cross-attention enables low-latency, memory-efficient real-time video captioning.

Abstract

Vision-language models (VLMs) are commonly trained by directly inserting image tokens from a pretrained vision encoder into the text stream of a language model. This allows text and image information to fully attend to one another within the model, but becomes rapidly costly for long multi-image conversations or streaming video applications, both in terms of memory and compute. VLMs leveraging cross-attention (CA) are an efficient alternative to token insertion as image tokens are not added to the KV cache. Despite being introduced early on, multimodal CA models are scarce in the current VLM literature and often underperform their token insertion counterparts. In this work, we reinvestigate the effectiveness of cross-attention for vision-language modeling: (i) We analyze the core differences between the cross-attention and self-attention mechanisms, (ii) we train cross-attention VLMs both from a text-only LLM and by adapting a pretrained insertion-based VLM, showing that simple cross-attention is far more competitive with token insertion than previously reported, and (iii) we demonstrate the practical advantages of cross-attention on real-time video captioning, where it naturally maintains low latency and near-constant memory cost. For samples and code, please see our project page at https://kyutai.org/casa .