Reducing Peak Memory Usage for Modern Multimodal Large Language Model Pipelines

TL;DR

This paper introduces a method to reduce peak memory usage in multimodal large language models by applying structure-aware cache compression during inference, enabling more efficient multimodal processing.

Contribution

It proposes a sequential input-compression mechanism that controls memory growth throughout inference by exploiting inherent redundancies in vision tokens.

Findings

Significantly reduces peak memory during inference.

Maintains generative performance with minimal degradation.

Enables more practical multimodal inference in large models.

Abstract

Multimodal large language models (MLLMs) have recently demonstrated strong capabilities in understanding and generating responses from diverse visual inputs, including high-resolution images and long video sequences. As these models scale to richer visual representations, inference increasingly relies on storing large numbers of vision tokens in the key-value (KV) cache, making memory consumption a central bottleneck. Existing methods address this issue by identifying redundancy in vision tokens and compressing the cache, but such compression is typically applied only after all inputs are processed, resulting in high peak memory usage during the prefill stage. In this work, we show that MLLMs exhibit inherent structural regularities and representational redundancy that can be exploited to control memory growth throughout inference. Based on this insight, we propose a sequential input-compression mechanism that enforces a fixed memory budget by performing structure-aware key-value cache compression during the prefill process. This approach substantially reduces peak memory usage while maintaining generative performance with only minimal degradation, enabling more practical and memory-efficient multimodal inference.