DP-LLM: Runtime Model Adaptation with Dynamic Layer-wise Precision Assignment

TL;DR

DP-LLM introduces a dynamic layer-wise precision assignment mechanism for on-device large language models, optimizing the trade-off between performance and latency by adapting to input sensitivity during decoding.

Contribution

It proposes a novel dynamic precision assignment method that adjusts layer quantization in real-time based on input sensitivity, enhancing model efficiency.

Findings

Outperforms prior methods in performance-latency trade-offs

Demonstrates effectiveness across multiple models and benchmarks

Achieves better resource utilization through dynamic adaptation

Abstract

How can we effectively handle queries for on-device large language models (LLMs) with varying runtime constraints, such as latency and accuracy? Multi-scale quantization addresses this challenge by enabling memory-efficient runtime model adaptation of LLMs through the overlaying of multiple model variants quantized to different bitwidths. Meanwhile, an important question still remains open-ended: how can models be properly configured to match a target precision or latency? While mixed-precision offers a promising solution, we take this further by leveraging the key observation that the sensitivity of each layer dynamically changes across decoding steps. Building on this insight, we introduce DP-LLM, a novel mechanism that dynamically assigns precision to each layer based on input values. Experimental results across multiple models and benchmarks demonstrate that DP-LLM achieves a superior performance-latency trade-off, outperforming prior approaches.