QiMeng-NeuComBack: Self-Evolving Translation from IR to Assembly Code

TL;DR

This paper introduces NeuComBack, a benchmark dataset and a self-evolving prompt optimization method that significantly improves the correctness and performance of LLM-generated assembly code from IR, advancing neural compilation.

Contribution

It presents a dedicated IR-to-assembly benchmark, evaluates LLMs' neural compilation capabilities, and proposes a novel self-evolving prompt technique to enhance code correctness and efficiency.

Findings

Functional correctness improved from 44% to 64% on x86_64.

Functional correctness improved from 36% to 58% on aarch64.

Most correctly generated x86_64 programs outperformed clang-O3.

Abstract

Compilers, while essential, are notoriously complex systems that demand prohibitively expensive human expertise to develop and maintain. The recent advancements in Large Language Models (LLMs) offer a compelling new paradigm: Neural Compilation, which could potentially simplify compiler development for new architectures and facilitate the discovery of innovative optimization techniques. However, several critical obstacles impede its practical adoption. Firstly, a significant lack of dedicated benchmarks and robust evaluation methodologies hinders objective assessment and tracking of progress in the field. Secondly, systematically enhancing the reliability and performance of LLM-generated assembly remains a critical challenge. Addressing these challenges, this paper introduces NeuComBack, a novel benchmark dataset specifically designed for IR-to-assembly compilation. Leveraging this dataset, we first define a foundational Neural Compilation workflow and conduct a comprehensive evaluation of the capabilities of recent frontier LLMs on Neural Compilation, establishing new performance baselines. We further propose a self-evolving prompt optimization method that enables LLMs to iteratively evolve their internal prompt strategies by extracting insights from prior self-debugging traces, thereby enhancing their neural compilation capabilities. Experiments demonstrate that our method significantly improves both the functional correctness and the performance of LLM-generated assembly code. Compared to baseline prompts, the functional correctness rates improved from 44% to 64% on x86_64 and from 36% to 58% on aarch64, respectively. More significantly, among the 16 correctly generated x86_64 programs using our method, 14 (87.5%) surpassed clang-O3 performance.