Optimizing LLMs Using Quantization for Mobile Execution

TL;DR

This paper demonstrates that 4-bit post-training quantization of LLMs significantly reduces model size, enabling efficient deployment on mobile devices while maintaining functional inference capabilities.

Contribution

It introduces a practical workflow for applying 4-bit PTQ to large models like Llama 3.2 3B for mobile deployment using existing tools and formats.

Findings

Achieved 68.66% size reduction with 4-bit quantization.

Successfully ran quantized LLM on Android device.

Validated inference quality of the quantized model.

Abstract

Large Language Models (LLMs) offer powerful capabilities, but their significant size and computational requirements hinder deployment on resource-constrained mobile devices. This paper investigates Post-Training Quantization (PTQ) for compressing LLMs for mobile execution. We apply 4-bit PTQ using the BitsAndBytes library with the Hugging Face Transformers framework to Meta's Llama 3.2 3B model. The quantized model is converted to GGUF format using llama.cpp tools for optimized mobile inference. The PTQ workflow achieves a 68.66% reduction in model size through 4-bit quantization, enabling the Llama 3.2 3B model to run efficiently on an Android device. Qualitative validation shows that the 4-bit quantized model can perform inference tasks successfully. We demonstrate the feasibility of running the quantized GGUF model on an Android device using the Termux environment and the Ollama framework. PTQ, especially at 4-bit precision combined with mobile-optimized formats like GGUF, provides a practical pathway for deploying capable LLMs on mobile devices, balancing model size and performance.