Efficient LLM Inference using Dynamic Input Pruning and Cache-Aware Masking

TL;DR

This paper introduces Dynamic Input Pruning (DIP), a novel method for reducing memory and increasing throughput in large language model inference on mobile devices by dynamically sparsifying inputs without predictor training.

Contribution

The work presents DIP, a predictor-free dynamic sparsification technique that maintains accuracy with minimal fine-tuning and introduces cache-aware masking to enhance cache utilization during LLM inference.

Findings

DIP achieves 46% memory reduction and 40% throughput increase on Phi-3-Medium.

DIP maintains less than 0.1 perplexity loss compared to dense streaming.

DIP outperforms previous sparsification methods in accuracy, memory, and throughput.

Abstract

While mobile devices provide ever more compute power, improvements in DRAM bandwidth are much slower. This is unfortunate for large language model (LLM) token generation, which is heavily memory-bound. Previous work has proposed to leverage natural dynamic activation sparsity in ReLU-activated LLMs to reduce effective DRAM bandwidth per token. However, more recent LLMs use SwiGLU instead of ReLU, which results in little inherent sparsity. While SwiGLU activations can be pruned based on magnitude, the resulting sparsity patterns are difficult to predict, rendering previous approaches ineffective. To circumvent this issue, our work introduces Dynamic Input Pruning (DIP): a predictor-free dynamic sparsification approach, which preserves accuracy with minimal fine-tuning. DIP can further use lightweight LoRA adapters to regain some performance lost during sparsification. Lastly, we describe a novel cache-aware masking strategy, which considers the cache state and activation magnitude to further increase cache hit rate, improving LLM token rate on mobile devices. DIP outperforms other methods in terms of accuracy, memory and throughput trade-offs across simulated hardware settings. On Phi-3-Medium, DIP achieves a 46\% reduction in memory and 40\% increase in throughput with $<$ 0.1 loss in perplexity when compared to streaming the dense model from Flash. The open source code for HW simulator, methods, and experiments in this paper is available at https://github.com/Qualcomm-AI-research/dynamic-sparsity .