KernelCraft: Benchmarking for Agentic Close-to-Metal Kernel Generation on Emerging Hardware

TL;DR

KernelCraft introduces a benchmark for evaluating LLM agents in generating and optimizing low-level kernels for emerging hardware accelerators, demonstrating rapid, valid, and efficient kernel production across diverse platforms.

Contribution

This work presents the first benchmark to assess LLM agent capabilities in generating and refining kernels for new ISAs using feedback-driven workflows.

Findings

Agents produce valid kernels within few refinement steps

Optimized kernels match or outperform compiler baselines

Effective across multiple emerging hardware platforms

Abstract

New AI accelerators with novel instruction set architectures (ISAs) often require developers to manually craft low-level kernels -- a time-consuming, laborious, and error-prone process that cannot scale across diverse hardware targets. This prevents emerging hardware platforms from reaching the market efficiently. While prior LLM-based code generation has shown promise in mature GPU ecosystems, it remains unclear whether agentic LLM systems can quickly produce valid and efficient kernels for emerging hardware with new ISAs. We present KernelCraft: the first benchmark to evaluate an LLM agent's ability to generate and optimize low-level kernels for customized accelerators via a function-calling, feedback-driven workflow. Within KernelCraft, the agent refines kernels under ISA and hardware constraints using automated feedback derived from compilation checks, simulation, and correctness validation against ground truth. In our experiments, we assess agent performance across three emerging accelerator platforms on more than 20 ML tasks, each with 5 diverse task configurations, with special evaluation of task configuration complexity. Across four leading reasoning models, top agents produce functionally valid kernels for previously unseen ISAs within a few refinement steps, with optimized kernels that match or outperform template-based compiler baselines. With that, we demonstrate the potential for reducing the cost of kernel development for accelerator designers and kernel developers.