NPUEval: Optimizing NPU Kernels with LLMs and Open Source Compilers

TL;DR

NPUEval introduces a benchmark for NPU kernel optimization, evaluating LLM-generated code on real hardware, revealing significant challenges and progress in automating efficient kernel development.

Contribution

The paper presents NPUEval, a new benchmark dataset and evaluation framework for NPU kernel code generation using LLMs, addressing the lack of specialized benchmarks in this domain.

Findings

DeepSeek R1 achieves over 50% vectorization on some kernels

Average vectorization score across dataset is about 10%

Open source tools enable functional correctness and efficiency evaluation

Abstract

Neural processing units (NPUs) are gaining prominence in power-sensitive devices like client devices, with AI PCs being defined by their inclusion of these specialized processors. Running AI workloads efficiently on these devices requires libraries of optimized kernels. Creating efficient kernels demands expertise in domain-specific C++ with vector intrinsics and in-depth knowledge of the target architecture. Unlike GPU programming, which has had years to mature, NPU programming is new, with smaller and more fragmented developer communities across hardware platforms. This fragmentation poses a challenge when utilizing LLMs to assist in writing NPU kernels, as domain-specific optimized code examples are underrepresented in LLM pre-training data. In this paper we introduce NPUEval -- a benchmark for writing and evaluating NPU kernels, consisting of 102 common operators for machine learning workloads. We evaluate LLM generated code on actual hardware based on both functional correctness and vectorization efficiency using open source compiler tools targeting the AMD NPU. We evaluate a range of state-of-the-art LLMs with a mix of proprietary and open-weight models. Latest reasoning models like DeepSeek R1, show promising results achieving out-of-the-box 50%+ vectorization on select kernels. However, the average score across the entire dataset remains roughly 10% even with compiler feedback and vectorized kernel examples -- showing that this is a challenging dataset even for frontier models. The dataset and evaluation code will be released with a permissive open source license, providing an essential benchmark for advancing research in code generation and NPU kernel optimization.