Porting Large Language Models to Mobile Devices for Question Answering

TL;DR

This paper demonstrates how to successfully port large language models to mobile devices, enabling real-time question answering with high accuracy on smartphones using a 6-bit quantized model.

Contribution

The authors show how to adapt state-of-the-art LLMs for mobile devices using llama.cpp and 6-bit quantization, achieving interactive speed and high-quality responses.

Findings

LLM inference runs interactively on Galaxy S21.

High-quality answers across diverse subjects.

Effective use of 6-bit quantization for mobile deployment.

Abstract

Deploying Large Language Models (LLMs) on mobile devices makes all the capabilities of natural language processing available on the device. An important use case of LLMs is question answering, which can provide accurate and contextually relevant answers to a wide array of user queries. We describe how we managed to port state of the art LLMs to mobile devices, enabling them to operate natively on the device. We employ the llama.cpp framework, a flexible and self-contained C++ framework for LLM inference. We selected a 6-bit quantized version of the Orca-Mini-3B model with 3 billion parameters and present the correct prompt format for this model. Experimental results show that LLM inference runs in interactive speed on a Galaxy S21 smartphone and that the model delivers high-quality answers to user queries related to questions from different subjects like politics, geography or history.