EdgeReasoning: Characterizing Reasoning LLM Deployment on Edge GPUs

TL;DR

EdgeReasoning provides a comprehensive analysis of deploying reasoning large language models on edge GPUs, balancing latency, accuracy, and resource constraints to guide optimal deployment strategies.

Contribution

It systematically characterizes latency-accuracy tradeoffs and evaluates techniques for optimizing reasoning LLM deployment on edge GPUs, filling a guidance gap.

Findings

Mapped the Pareto frontier of accuracy and latency configurations.

Evaluated prompt and tuning techniques for token reduction.

Profiled test-time scaling methods for latency optimization.

Abstract

Edge intelligence paradigm is increasingly demanded by the emerging autonomous systems, such as robotics. Beyond ensuring privacy-preserving operation and resilience in connectivity-limited environments, edge deployment offers significant energy and cost advantages over cloud-based solutions. However, deploying large language models (LLMs) for reasoning tasks on edge GPUs faces critical challenges from strict latency constraints and limited computational resources. To navigate these constraints, developers must balance multiple design factors - choosing reasoning versus non-reasoning architectures, selecting appropriate model sizes, allocating token budgets, and applying test-time scaling strategies - to meet target latency and optimize accuracy. Yet guidance on optimal combinations of these variables remains scarce. In this work, we present EdgeReasoning, a comprehensive study characterizing the deployment of reasoning LLMs on edge GPUs. We systematically quantify latency-accuracy tradeoffs across various LLM architectures and model sizes. We systematically evaluate prompt-based and model-tuning-based techniques for reducing reasoning token length while maintaining performance quality. We further profile test-time scaling methods with varying degrees of parallelism to maximize accuracy under strict latency budgets. Through these analyses, EdgeReasoning maps the Pareto frontier of achievable accuracy-latency configurations, offering systematic guidance for optimal edge deployment of reasoning LLMs.