Efficient Inference of Vision Instruction-Following Models with Elastic Cache

TL;DR

This paper introduces Elastic Cache, a novel cache management strategy for vision-language models that improves efficiency by importance-driven cache merging, preserving contextual information and outperforming existing pruning methods.

Contribution

The paper proposes Elastic Cache, a new cache management approach that applies importance-driven merging to enhance efficiency without losing critical contextual information in LVLMs.

Findings

Boosts efficiency of LVLMs during inference.

Outperforms existing pruning methods in language generation tasks.

Maintains contextual information while accelerating inference.

Abstract

In the field of instruction-following large vision-language models (LVLMs), the efficient deployment of these models faces challenges, notably due to the high memory demands of their key-value (KV) caches. Conventional cache management strategies for LLMs focus on cache eviction, which often fails to address the specific needs of multimodal instruction-following models. Recognizing this gap, in this paper, we introduce Elastic Cache, a novel approach that benefits from applying distinct acceleration methods for instruction encoding and output generation stages. We investigate the metrics of importance in different stages and propose an importance-driven cache merging strategy to prune redundancy caches. Instead of discarding less important caches, our strategy identifies important key/value vectors as anchor points. Surrounding less important caches are then merged with these anchors, enhancing the preservation of contextual information in the KV caches while yielding an arbitrary acceleration ratio. For instruction encoding, we utilize the frequency to evaluate the importance of caches. Regarding output generation, we prioritize tokens based on their distance with an offset, by which both the initial and most recent tokens are retained. Results on a range of LVLMs demonstrate that Elastic Cache not only boosts efficiency but also notably outperforms existing pruning methods in language generation across various tasks. Code is available at https://github.com/liuzuyan/ElasticCache