Enhancing Large Multimodal Models with Adaptive Sparsity and KV Cache Compression

TL;DR

This paper introduces an adaptive search algorithm that optimizes sparsity and KV cache compression in large multimodal models, significantly improving their efficiency for edge deployment without losing accuracy.

Contribution

It presents a novel method combining pruning and KV cache quantization with dynamic adjustment, eliminating the need for fine-tuning and outperforming existing compression techniques.

Findings

Achieves superior compression efficiency over SparseGPT and Wanda.

Maintains high accuracy with significant memory savings.

Automatically allocates KV cache resources for optimal performance.

Abstract

Large multimodal models (LMMs) have advanced significantly by integrating visual encoders with extensive language models, enabling robust reasoning capabilities. However, compressing LMMs for deployment on edge devices remains a critical challenge. In this work, we propose an adaptive search algorithm that optimizes sparsity and KV cache compression to enhance LMM efficiency. Utilizing the Tree-structured Parzen Estimator, our method dynamically adjusts pruning ratios and KV cache quantization bandwidth across different LMM layers, using model performance as the optimization objective. This approach uniquely combines pruning with key-value cache quantization and incorporates a fast pruning technique that eliminates the need for additional fine-tuning or weight adjustments, achieving efficient compression without compromising accuracy. Comprehensive evaluations on benchmark datasets, including LLaVA-1.5 7B and 13B, demonstrate our method superiority over state-of-the-art techniques such as SparseGPT and Wanda across various compression levels. Notably, our framework automatic allocation of KV cache compression resources sets a new standard in LMM optimization, delivering memory efficiency without sacrificing much performance.