POaaS: Minimal-Edit Prompt Optimization as a Service to Lift Accuracy and Cut Hallucinations on On-Device sLLMs

TL;DR

POaaS introduces a minimal-edit prompt optimization method that enhances accuracy and reduces hallucinations in on-device small language models by routing queries through specialized lightweight modules, outperforming traditional search-based methods.

Contribution

The paper presents POaaS, a novel minimal-edit prompt optimization layer tailored for on-device sLLMs, which improves task accuracy and factuality without extensive search procedures.

Findings

POaaS improves accuracy and factuality on on-device sLLMs.

POaaS recovers up to +7.4% accuracy under certain perturbations.

Traditional APO methods degrade performance in on-device settings.

Abstract

Small language models (sLLMs) are increasingly deployed on-device, where imperfect user prompts--typos, unclear intent, or missing context--can trigger factual errors and hallucinations. Existing automatic prompt optimization (APO) methods were designed for large cloud LLMs and rely on search that often produces long, structured instructions; when executed under an on-device constraint where the same small model must act as optimizer and solver, these pipelines can waste context and even hurt accuracy. We propose POaaS, a minimal-edit prompt optimization layer that routes each query to lightweight specialists (Cleaner, Paraphraser, Fact-Adder) and merges their outputs under strict drift and length constraints, with a conservative skip policy for well-formed prompts. Under a strict fixed-model setting with Llama-3.2-3B-Instruct and Llama-3.1-8B-Instruct, POaaS improves both task accuracy and factuality while representative APO baselines degrade them, and POaaS recovers up to +7.4% under token deletion and mixup. Overall, per-query conservative optimization is a practical alternative to search-heavy APO for on-device sLLMs.