Accelerating Local LLMs on Resource-Constrained Edge Devices via Distributed Prompt Caching

TL;DR

This paper introduces distributed prompt caching with Bloom filters to improve local LLM inference on resource-limited edge devices, significantly reducing response times.

Contribution

It proposes a novel distributed caching mechanism with partial matching and Bloom-filter-based state catalog to accelerate local LLM inference on edge devices.

Findings

Reduces TTFT by 93.12% on average

Reduces TTLT by 50.07% on average

Demonstrates effectiveness on Raspberry Pi Zero 2W with Gemma-3 270M model

Abstract

Since local LLM inference on resource-constrained edge devices imposes a severe performance bottleneck, this paper proposes distributed prompt caching to enhance inference performance by cooperatively sharing intermediate processing states across multiple low-end edge devices. To fully utilize prompt similarity, our distributed caching mechanism also supports partial matching. As this approach introduces communication overhead associated with state sharing over a wireless network, we introduce a Bloom-filter-based data structure, referred to as a catalog, to determine whether a remote server possesses the desired internal states, thereby suppressing unnecessary communication. Experiments using the Gemma-3 270M model and the MMLU dataset on the Raspberry Pi Zero 2W platform demonstrate that the proposed approach reduces TTFT (Time to First Token) and TTLT (Time to Last Token) by 93.12% and 50.07% on average, respectively.