Exploring the Feasibility of End-to-End Large Language Model as a Compiler

TL;DR

This paper investigates the potential of large language models to function as end-to-end compilers, evaluating their current capabilities, limitations, and future improvements for code translation tasks.

Contribution

It introduces the CompilerEval dataset and framework to assess LLMs as compilers, highlighting their basic abilities and proposing methods to enhance their performance.

Findings

LLMs show basic compilation capabilities but have low success rates.

Prompt optimization and model scaling improve code quality.

Future research can enable LLMs to generate high-quality assembly code.

Abstract

In recent years, end-to-end Large Language Model (LLM) technology has shown substantial advantages across various domains. As critical system software and infrastructure, compilers are responsible for transforming source code into target code. While LLMs have been leveraged to assist in compiler development and maintenance, their potential as an end-to-end compiler remains largely unexplored. This paper explores the feasibility of LLM as a Compiler (LaaC) and its future directions. We designed the CompilerEval dataset and framework specifically to evaluate the capabilities of mainstream LLMs in source code comprehension and assembly code generation. In the evaluation, we analyzed various errors, explored multiple methods to improve LLM-generated code, and evaluated cross-platform compilation capabilities. Experimental results demonstrate that LLMs exhibit basic capabilities as compilers but currently achieve low compilation success rates. By optimizing prompts, scaling up the model, and incorporating reasoning methods, the quality of assembly code generated by LLMs can be significantly enhanced. Based on these findings, we maintain an optimistic outlook for LaaC and propose practical architectural designs and future research directions. We believe that with targeted training, knowledge-rich prompts, and specialized infrastructure, LaaC has the potential to generate high-quality assembly code and drive a paradigm shift in the field of compilation.