VLCache: Computing 2% Vision Tokens and Reusing 98% for Vision-Language Inference

TL;DR

VLCache is a framework that reduces computation in vision-language models by reusing 98% of tokens through cache mechanisms, achieving near full accuracy with only 2-5% recomputation and substantial speedups.

Contribution

It introduces a formal approach to cache reuse error minimization and a dynamic layer-aware strategy for efficient, accurate vision-language inference.

Findings

Achieves 1.2x-16x speedup over full recomputation.

Maintains accuracy comparable to full recomputation.

Reuses 98% of tokens, drastically reducing computation.

Abstract

This paper presents VLCache, a cache reuse framework that exploits both Key-Value (KV) cache and encoder cache from prior multimodal inputs to eliminate costly recomputation when the same multimodal inputs recur. Unlike previous heuristic approaches, we formally identify the cumulative reuse error effect and demonstrate how to minimize the non-prefix cache reuse error effectively. We further analyze the varying importance of model layers and propose a dynamic, layer-aware recomputation strategy to balance accuracy and efficiency. Experimental results show that VLCache achieves an accuracy on par with full recomputation, while requiring only 2-5% of the tokens to compute, yielding 1.2x-16x TTFT speedups. We develop an experimental implementation of the proposed VLCache pipeline based on SGLang, enabling significantly faster inference in practical deployments.