Edge Deployment of Small Language Models, a comprehensive comparison of CPU, GPU and NPU backends

TL;DR

This paper compares CPU, GPU, and NPU hardware for running small language models at the edge, highlighting that NPUs offer the best performance and energy efficiency for resource-constrained environments.

Contribution

It provides a comprehensive evaluation of different hardware backends for SLM inference, demonstrating the superiority of NPUs in performance and energy efficiency.

Findings

NPUs outperform CPUs and GPUs in inference speed and energy efficiency.

Bandwidth normalization is crucial for fair cross-architecture comparison.

NPUs are the most suitable hardware for edge SLM deployment.

Abstract

Edge computing processes data where it is generated, enabling faster decisions, lower bandwidth usage, and improved privacy. However, edge devices typically operate under strict constraints on processing power, memory, and energy consumption, making them unsuitable for large language models (LLMs). Fortunately, Small Language Models (SLMs) offer lightweight alternatives that bring AI inference to resource-constrained environments by significantly reducing computational cost while remaining suitable for specialization and customization. In this scenario, selecting the hardware platform that best balances performance and efficiency for SLM inference is challenging due to strict resource limitations. To address this issue, this study evaluates the inference performance and energy efficiency of commercial CPUs (Intel and ARM), GPUs (NVIDIA), and NPUs (RaiderChip) for running SLMs. GPUs, the usual platform of choice, are compared against commercial NPUs and recent multi-core CPUs. While NPUs leverage custom hardware designs optimized for computation, modern CPUs increasingly incorporate dedicated features targeting language-model workloads. Using a common execution framework and a suite of state-of-the-art SLMs, we analyze both maximum achievable performance and processing and energy efficiency across commercial solutions available for each platform. The results indicate that specialized backends outperform general-purpose CPUs, with NPUs achieving the highest performance by a wide margin. Bandwidth normalization proves essential for fair cross-architecture comparisons. Although low-power ARM processors deliver competitive results when energy usage is considered, metrics that combine performance and power (such as EDP) again highlight NPUs as the dominant architecture. These findings show that designs optimized for both efficiency and performance offer a clear advantage for edge workloads.