MobileKernelBench: Can LLMs Write Efficient Kernels for Mobile Devices?

TL;DR

This paper evaluates the ability of large language models to generate efficient kernels for mobile devices, introduces a benchmark and framework for testing, and proposes a multi-agent system that significantly improves kernel compilation success and performance.

Contribution

It introduces MobileKernelBench, a comprehensive evaluation framework, and proposes MoKA, a multi-agent system that enhances kernel generation success and efficiency for mobile devices.

Findings

Current LLMs have high failure rates (>54%) in mobile kernel generation.

MoKA improves compilation success to 93.7%.

27.4% of generated kernels show measurable speedups.

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities in code generation, yet their potential for generating kernels specifically for mobile devices remains largely unexplored. In this work, we extend the scope of automated kernel generation to the mobile domain to investigate the central question: Can LLMs write efficient kernels for mobile devices? To enable systematic investigation, we introduce MobileKernelBench, a comprehensive evaluation framework comprising a benchmark prioritizing operator diversity and cross-framework interoperability, coupled with an automated pipeline that bridges the host-device gap for on-device verification. Leveraging this framework, we conduct extensive evaluation on the CPU backend of Mobile Neural Network (MNN), revealing that current LLMs struggle with the engineering complexity and data scarcity inherent to mobile frameworks; standard models and even fine-tuned variants exhibit high compilation failure rates (over 54%) and negligible performance gains due to hallucinations and a lack of domain-specific grounding. To overcome these limitations, we propose the Mobile Kernel Agent (MoKA), a multi-agent system equipped with repository-aware reasoning and a plan-and-execute paradigm. Validated on MobileKernelBench, MoKA achieves state-of-the-art performance, boosting compilation success to 93.7% and enabling 27.4% of generated kernels to deliver measurable speedups over native libraries.