Efficient Reasoning on the Edge

TL;DR

This paper introduces a lightweight, resource-efficient method for enabling reasoning in small language models suitable for mobile devices, using LoRA adapters, reinforcement learning, and dynamic mechanisms to reduce costs and maintain accuracy.

Contribution

The authors propose a novel approach combining LoRA adapters, reinforcement learning, and dynamic adapter switching to enable efficient reasoning in small LLMs for edge deployment.

Findings

Achieves significant reduction in response length with minimal accuracy loss.

Improves on-device reasoning efficiency using memory and computation optimizations.

Demonstrates practical reasoning capabilities on mobile devices with Qwen2.5-7B.

Abstract

Large language models (LLMs) with chain-of-thought reasoning achieve state-of-the-art performance across complex problem-solving tasks, but their verbose reasoning traces and large context requirements make them impractical for edge deployment. These challenges include high token generation costs, large KV-cache footprints, and inefficiencies when distilling reasoning capabilities into smaller models for mobile devices. Existing approaches often rely on distilling reasoning traces from larger models into smaller models, which are verbose and stylistically redundant, undesirable for on-device inference. In this work, we propose a lightweight approach to enable reasoning in small LLMs using LoRA adapters combined with supervised fine-tuning. We further introduce budget forcing via reinforcement learning on these adapters, significantly reducing response length with minimal accuracy loss. To address memory-bound decoding, we exploit parallel test-time scaling, improving accuracy at minor latency increase. Finally, we present a dynamic adapter-switching mechanism that activates reasoning only when needed and a KV-cache sharing strategy during prompt encoding, reducing time-to-first-token for on-device inference. Experiments on Qwen2.5-7B demonstrate that our method achieves efficient, accurate reasoning under strict resource constraints, making LLM reasoning practical for mobile scenarios. Videos demonstrating our solution running on mobile devices are available on our project page.