A Systematic Evaluation of On-Device LLMs: Quantization, Performance, and Resources

TL;DR

This paper systematically evaluates on-device large language models, analyzing quantization effects on performance and resource use, and provides guidelines for optimizing LLM deployment on edge devices.

Contribution

It introduces a comprehensive methodology for evaluating on-device LLMs, highlighting the impact of quantization and resource constraints on performance.

Findings

Heavily quantized large models outperform smaller high-precision models.

Resource utilization scales linearly with effective bits-per-weight.

Throughput constraints shift from communication to computation with model size reduction.

Abstract

Deploying Large Language Models (LLMs) on edge devices enhances privacy but faces performance hurdles due to limited resources. We introduce a systematic methodology to evaluate on-device LLMs, balancing capability, efficiency, and resource constraints. Through an extensive analysis of models (0.5B-14B) and seven post-training quantization (PTQ) methods on commodity hardware, we demonstrate that: 1) Heavily quantized large models consistently outperform smaller, high-precision models, with a performance threshold at ~3.5 effective bits-per-weight (BPW); 2) Resource utilization scales linearly with BPW, though power and memory footprints vary by quantization algorithm; and 3) With a reduction in model size, the primary constraint on throughput transitions from communication overhead to computational latency. We conclude by offering guidelines for optimizing LLMs in resource-constrained edge environments. Our codebase is available at https://anonymous.4open.science/r/LLMOnDevice/.