Meta Large Language Model Compiler: Foundation Models of Compiler Optimization
Chris Cummins, Volker Seeker, Dejan Grubisic, Baptiste Roziere, Jonas, Gehring, Gabriel Synnaeve, Hugh Leather

TL;DR
This paper introduces Meta LLM Compiler, a set of pre-trained language models designed specifically for code and compiler optimization, trained on extensive data, and capable of improving code size and disassembly tasks.
Contribution
The paper presents the first large-scale, open, pre-trained LLMs for compiler optimization, built on Code Llama with extensive training on LLVM-IR and assembly code, and demonstrates their effectiveness.
Findings
Achieved 77% of autotuning search optimization potential.
Disassembled code with 45% accuracy in round-trip conversion.
Released models in two sizes: 7B and 13B parameters.
Abstract
Large Language Models (LLMs) have demonstrated remarkable capabilities across a variety of software engineering and coding tasks. However, their application in the domain of code and compiler optimization remains underexplored. Training LLMs is resource-intensive, requiring substantial GPU hours and extensive data collection, which can be prohibitive. To address this gap, we introduce Meta Large Language Model Compiler (LLM Compiler), a suite of robust, openly available, pre-trained models specifically designed for code optimization tasks. Built on the foundation of Code Llama, LLM Compiler enhances the understanding of compiler intermediate representations (IRs), assembly language, and optimization techniques. The model has been trained on a vast corpus of 546 billion tokens of LLVM-IR and assembly code and has undergone instruction fine-tuning to interpret compiler behavior. LLM…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsDistributed and Parallel Computing Systems · Model-Driven Software Engineering Techniques · Parallel Computing and Optimization Techniques
