Enabling On-Device LLMs Personalization with Smartphone Sensing

TL;DR

This paper introduces a novel framework that personalizes on-device large language models using smartphone sensing, addressing privacy, latency, and cost issues of cloud-based solutions.

Contribution

It is the first framework to enable on-device LLM personalization with multimodal sensor data and customized prompts for privacy-aware, context-aware services.

Findings

Achieves a favorable privacy-performance trade-off.

Demonstrates effective personalized recommendations in a case study.

Reduces latency and energy consumption compared to cloud LLMs.

Abstract

This demo presents a novel end-to-end framework that combines on-device large language models (LLMs) with smartphone sensing technologies to achieve context-aware and personalized services. The framework addresses critical limitations of current personalization solutions via cloud LLMs, such as privacy concerns, latency and cost, and limited personal information. To achieve this, we innovatively proposed deploying LLMs on smartphones with multimodal sensor data through context-aware sensing and customized prompt engineering, ensuring privacy and enhancing personalization performance. A case study involving a university student demonstrated the capability of the framework to provide tailored recommendations. In addition, we show that the framework achieves the best trade-off in privacy, performance, latency, cost, battery and energy consumption between on-device and cloud LLMs. To the best of our knowledge, this is the first framework to provide on-device LLMs personalization with smartphone sensing. Future work will incorporate more diverse sensor data and involve extensive user studies to enhance personalization. Our proposed framework has the potential to substantially improve user experiences across domains including healthcare, productivity, and entertainment.